Using ZChunk for setup…

Using ZChunk for setup…

[Achim Gratz]

I've been experimenting with ZChunk with the idea of eventually using it
for setup:

https://www.jdieter.net/posts/2018/05/31/what-is-zchunk/
https://github.com/zchunk/zchunk

The chunked ini file is ~10…15% larger than the original (after
compression).  In order to minimize the overhead, I've re-arranged the
package entries to have one chunk for every source package.  The actual
benefit is that the typical download size reduces to less than 5% of the
original.  Two examples of much longer timespans between updates are
provided at the end, which would still download only around a third of
the original:

--8<---------------cut here---------------start------------->8---
# no changes, only header gets downloaded
update from 20220109 to 20220109
Would download 82277 of 4061994 bytes
Matched 4103 of 4103 chunks

update from 20220106 to 20220109
Would download 112022 of 4061994 bytes
Matched 4078 of 4103 chunks

update from 20211221 to 20220106
Would download 172069 of 4061583 bytes
Matched 4024 of 4103 chunks

update from 20211218 to 20211221
Would download 216879 of 4052330 bytes
Matched 4012 of 4098 chunks

update from 20211204 to 20211218
Would download 248101 of 4020714 bytes
Matched 3997 of 4097 chunks

update from 20200102 to 20210703
Would download 1438581 of 3960442 bytes
Matched 2938 of 4087 chunks

update from 20190101 to 20200102
Would download 1139408 of 3723670 bytes
Matched 3142 of 3987 chunks
--8<---------------cut here---------------end--------------->8---

WDYT?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Using ZChunk for setup…

[Jon Turney]

On 09/01/2022 11:05, Achim Gratz wrote:
> 
> I've been experimenting with ZChunk with the idea of eventually using it
> for setup:
> 
> https://www.jdieter.net/posts/2018/05/31/what-is-zchunk/
> https://github.com/zchunk/zchunk
> 
> The chunked ini file is ~10…15% larger than the original (after
> compression).  In order to minimize the overhead, I've re-arranged the
> package entries to have one chunk for every source package.  The actual
> benefit is that the typical download size reduces to less than 5% of the
> original.  Two examples of much longer timespans between updates are
> provided at the end, which would still download only around a third of
> the original:
> 
[...]
> 
> WDYT?

This seems like a nice thing to have.

How would the signature be checked?  Is it for the reconstructed full 
zchunk file?

One slightly related issue which it would be good to address if possible 
when adding this is that rsync is only file-atomic, not repo-atomic, so 
we may get a compressed ini file and signature which don't match as they 
are different moments in time during an update. I think currently no-one 
notices this if it happens, as setup silently falls back to an older 
compression type, but it would be nice to stop generating those eventually.

Re: Using ZChunk for setup…

[Achim Gratz]

Jon Turney writes:
> How would the signature be checked?  Is it for the reconstructed full
> zchunk file?

The signature should be for the full (reconstructed) file I think, the
individual chunks of the changeset are locked in by SHA256 checksums
anyway.

> One slightly related issue which it would be good to address if
> possible when adding this is that rsync is only file-atomic, not
> repo-atomic, so we may get a compressed ini file and signature which
> don't match as they are different moments in time during an update. I
> think currently no-one notices this if it happens, as setup silently
> falls back to an older compression type, but it would be nice to stop
> generating those eventually.

One of the things I am still thinking about is switching from the ini
format to YAML.  In this case the signature could/should be part of the
data structure (JOSE/COSE style), so you'd only ever have to look at a
single file and it wouldn't matter how you got it together.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds