Neal, thanks for the feedback. After taking your comments into
consideration, here's version 2.
+-+-+-+-+-+------------------+-+-+-+-+-+-+-+-+
| ID | Compression type | Index size |
+-+-+-+-+-+------------------+-+-+-+-+-+-+-+-+
+==================+=================+
| Compressed Index | Compressed Dict |
+==================+=================+
+===========+===========+
| Chunk | Chunk | ==> More chunks
+===========+===========+
ID
'\0ZCK1', identifies file as zchunk version 1 file
Compression type
Type of compression used to compress dict and chunks
Current values:
0 - Uncompressed
2 - zstd
Index size
This is a 64-bit unsigned integer containing the size of compressed
index.
Compressed Index
This is the index, which is described in the next section. The index
is compressed without a custom dictionary.
Compressed Dict (optional)
This is a custom dictionary used when compressing each chunk.
Because each chunk is compressed completely separately from the
others, the custom dictionary gives us much better overall
compression. The custom dictionary is compressed without a custom
dictionary (for obvious reasons).
Chunk
This is a chunk of data, compressed with the custom dictionary
provided above.
The index:
+---------------+======================+
| Checksum type | Checksum of all data |
+---------------+======================+
+================+-+-+-+-+-+-+-+-+
| Dict checksum | End of dict |
+================+-+-+-+-+-+-+-+-+
+================+-+-+-+-+-+-+-+-+
| Chunk checksum | End of chunk | ==> More
+================+-+-+-+-+-+-+-+-+
Checksum type
This is the type of checksum used to generate the checksums in the
index.
Current values:
0 = SHA-256
Checksum of all data
This is the checksum of the compressed dict and all the compressed
chunks, used to verify that the file is actually the same, even in
the unlikely event of a hash collision for one of the chunks
Dict checksum
This is the checksum of the compressed dict, used to detect whether
two dicts are identical. If there is no dict, the checksum must be
all zeros.
End of dict
This is the location of the end of the dict starting from the end of
the index. This gives us the information we need to find and
decompress the dict. If there is no dict, the checksum must be all
zeros.
Chunk checksum
This is the checksum of the compressed chunk, used to detect whether
any two chunks are identical.
End of chunk
This is the location of the end of the chunk starting from the end of
the index. This gives us the information we need to find and
decompress each chunk.
The index is designed to be able to be extracted from the file on the
server and downloaded separately, to facilitate downloading only the
parts of the file that are needed, but must then be re-embedded when
assembling the file so the user only needs to keep one file.