On Sat, Jul 27, 2019 at 10:04:06AM -0400, Stephen John Smoogen wrote:
I did a bunch of copying of packages over to archives at the end of
May. I
do all the copies using cp -l to preserve links. I then ran the
update-archives for full file list but then did not see it did not work.
This we got several requests from mirrors to get rid of f28 out of normal
space. I confirmed that f28 was copied over and then I synced over f29 and
f30 so I would not be behind in a couple of months. All of these were also
done with cp -l also and I checked that the hardlink numbers had increased
on files. I gave the info to Adrian and he let me know that the
update-archives was not updated. I then fixed that and again didnt send any
email about this.
I'm not sure whether q-f-m didn't work quite right, whether there was
a server-side issue (update-archives?), or whether this was just all
due to timing of cp -l vs. update-archives vs. my q-f-m runs, but
somehow I received over 1 TB of downloads that were not hardlinked.
It did not finish (I ran out of disk space) so I cleaned that up
yesterday, moved the files from .~tmp/* to their final locations, ran
a local hardlink.py, and re-ran q-f-m. The hardlink restored the 1+
TB of free space. The files are all staying hardlinked, but due to
q-f-m optimizing the rsync runs, that doesn't provide any assurance
that the master mirror is as hardlinked as it could be.
To focus on the good parts:
My local hardlink.py run saved at least 600 GB additional space
compared to before Friday. I think some of the additional savings
comes from internal hardlinking between e.g. Workstation, Everything,
Spins, noarch across different arch directories, etc. that one might
expect to be already hardlinked, but weren't for some reason.
I'm going to look into running quick-fedora-harlink locally regularly.