I've been working on a C implementation of this spec, and came up with
a few other changes. I think it's important to have a checksum of the
index as well as the data as we want to be able to verify that the
index is as expected before trying to parse it.
I've also added in the ability to use a different hash type for the
chunk checksums versus the index checksum and overall checksum. The
idea is that a weaker checksum may be chosen for the chunks to reduce
the size of the index without weakening overall verification.
+-+-+-+-+-+---------------+================+------------------+
| ID | Checksum type | Index checksum | Compression type |
+-+-+-+-+-+---------------+================+------------------+
+-+-+-+-+-+-+-+-+==================+=================+
| Index size | Compressed Index | Compressed Dict |
+-+-+-+-+-+-+-+-+==================+=================+
+===========+===========+
| Chunk | Chunk | ==> More chunks
+===========+===========+
ID
'\0ZCK1', identifies file as zchunk version 1 file
Checksum type
This is an 8-bit unsigned integer containing the type of checksum
used to generate the index checksum and the total data checksum, but
*not* the chunk checksums
Current values:
0 = SHA-1
1 = SHA-256
Index checksum
This is the checksum of everything from this point until the end of
the index. It includes the compression type, the index size, and the
compressed index.
Compression type
This is an 8-bit unsigned integer containing the type of compression
used to compress dict and chunks.
Current values:
0 - Uncompressed
2 - zstd
Index size
This is a 64-bit LE unsigned integer containing the size of compressed
index.
Compressed Index
This is the index, which is described in the next section. The index
is compressed without a custom dictionary.
Compressed Dict (optional)
This is a custom dictionary used when compressing each chunk.
Because each chunk is compressed completely separately from the
others, the custom dictionary gives us much better overall
compression. The custom dictionary is compressed without a custom
dictionary (for obvious reasons).
Chunk
This is a chunk of data, compressed with the custom dictionary
provided above.
The index:
+---------------------+-+-+-+-+-+-+-+-+======================+
| Chunk checksum type | Chunk count | Checksum of all data |
+---------------------+-+-+-+-+-+-+-+-+======================+
+================+-+-+-+-+-+-+-+-+
| Dict checksum | End of dict |
+================+-+-+-+-+-+-+-+-+
+================+-+-+-+-+-+-+-+-+
| Chunk checksum | End of chunk | ==> More
+================+-+-+-+-+-+-+-+-+
Chunk checksum type
This is an 8-bit unsigned integer containing the type of checksum used
to generate the chunk checksums.
Current values:
0 = SHA-1
1 = SHA-256
Chunk count
This is a count of the number of chunks in the zchunk file.
Checksum of all data
This is the checksum of everything after the index, including the
compressed dict and all the compressed chunks. This checksum is
generated using the overall checksum type, *not* the chunk checksum
type.
Dict checksum
This is the checksum of the compressed dict, used to detect whether
two dicts are identical. If there is no dict, the checksum must be
all zeros.
End of dict
This is a 64-bit LE unsigned integer containing the location of the
end of the dict starting from the end of the index. This gives us the
information we need to find and decompress the dict. If there is no
dict, this must be a zero.
Chunk checksum
This is the checksum of the compressed chunk, used to detect whether
any two chunks are identical.
End of chunk
This is the location of the end of the chunk starting from the end of
the index. This gives us the information we need to find and
decompress each chunk.
The index is designed to be able to be extracted from the file on the
server and downloaded separately, to facilitate downloading only the
parts of the file that are needed, but must then be re-embedded when
assembling the file so the user only needs to keep one file.