On Mon, 2018-04-23 at 00:27 -0400, Neal Gompa wrote:
On Tue, Apr 17, 2018 at 3:05 PM, Jonathan Dieter
<jdieter(a)gmail.com> wrote:
> I'm assuming that you're referring here to getting zchunk packaged into
> Fedora. I'd really like to finalize the file format (we're close, but
> I still need a good way of storing signatures in it) and the download
> API before releasing it into Fedora proper.
>
I'm looking forward to this!
I've updated the file format to allow for multiple signatures, updated
the zchunk code to recognize the existence of a signature (while still
not checking it), and have released as zchunk-0.3.0 in COPR. I've also
added in 32-bits of flags that we can use to extend the format in a
backwards-compatible way.
The current zchunk format description is at:
https://github.com/jdieter/zchunk/blob/master/zchunk_format.txt
I would recommend using the dicts mentioned above as they give me
over
> 40% space savings for both other.xml.zck and primary.xml.zck. Do
> please let me know if you run into any problems.
>
Are those dictionaries Fedora specific? If so, how can other
distributions generate similar ones? If not, still, how were they
made? :)
They were generated from Fedora metadata, but they should help with any
distribution's repodata. I generated them by splitting a few day's
worth of metadata along package boundaries, stripping out any
checksums, and then running zstd --train * on the directory containing
the split metadata. The script I used is available at
https://www.jdieter.net/downloads/zchunk-dicts/split.py, and I hope to
write up proper instructions at some point.
Jonathan