Hello,
has anyone successfully managed to get ccache to speed up kernel mockbuilds, be it SRPMS from kernel-ark or from dist-git? I gave it a brief shot but saw no difference, if anything the build was slower. The machine I am building on has 32 GB RAM which is not quite enough to fit everything on a ramdisk, and the drive I am building to is an NVME SSD, "only" PCI Express 3.0. CPU is a Ryzen 5 5600x. A baseonly builds takes about 25 minutes, which is somewhat annoying when one has to bisect something. If ccache can speed this up, how big does it need to be? I understand the default 4 GB is nowhere near enough but what would be needed to actually help? Thanks!
Best regards, Julian
Dne 12. 02. 24 v 19:29 Julian Sikorski napsal(a):
Hello,
has anyone successfully managed to get ccache to speed up kernel mockbuilds, be it SRPMS from kernel-ark or from dist-git? I gave it a brief shot but saw no difference, if anything the build was slower. The machine I am building on has 32 GB RAM which is not quite enough to fit everything on a ramdisk, and the drive I am building to is an NVME SSD, "only" PCI Express 3.0. CPU is a Ryzen 5 5600x. A baseonly builds takes about 25 minutes, which is somewhat annoying when one has to bisect something. If ccache can speed this up, how big does it need to be? I understand the default 4 GB is nowhere near enough but what would be needed to actually help? Thanks!
Just to be sure - did you configured Mock to use it?
https://rpm-software-management.github.io/mock/Plugin-CCache
the build from build container cannot see host ccache, so Mock must specially configure it.
This is by default off, because in multi user environment you can easily poison the cache and affect different build. This should be enabled only on VM where you trust all users that can run Mock. E.g. your personal developer workstation.
Am 12.02.24 um 22:10 schrieb Miroslav Suchý:
Dne 12. 02. 24 v 19:29 Julian Sikorski napsal(a):
Hello,
has anyone successfully managed to get ccache to speed up kernel mockbuilds, be it SRPMS from kernel-ark or from dist-git? I gave it a brief shot but saw no difference, if anything the build was slower. The machine I am building on has 32 GB RAM which is not quite enough to fit everything on a ramdisk, and the drive I am building to is an NVME SSD, "only" PCI Express 3.0. CPU is a Ryzen 5 5600x. A baseonly builds takes about 25 minutes, which is somewhat annoying when one has to bisect something. If ccache can speed this up, how big does it need to be? I understand the default 4 GB is nowhere near enough but what would be needed to actually help? Thanks!
Just to be sure - did you configured Mock to use it?
https://rpm-software-management.github.io/mock/Plugin-CCache
the build from build container cannot see host ccache, so Mock must specially configure it.
This is by default off, because in multi user environment you can easily poison the cache and affect different build. This should be enabled only on VM where you trust all users that can run Mock. E.g. your personal developer workstation.
I am building with: --enable-plugin=ccache --plugin-option=ccache:max_cache_size='40G' Building with cold cache took 29:29, building with hot cache 29:45, so there appears to be no benefit so far. /var/cache/mock/fedora-36-x86_64/ccache is 9,5G in size, suggesting that size is not the problem. I am down to 21 commits to bisect after the current one, so changes to kernel source tree should be minimal at this point. Do I need to install ccache on the host as well?
Best regards, Julian
Dne 12. 02. 24 v 23:01 Julian Sikorski napsal(a):
Do I need to install ccache on the host as well?
No. Mock installs the ccache into buildroot. It is injected as additional Build requires
https://github.com/rpm-software-management/mock/blob/main/mock/py/mockbuild/...
Am 12.02.24 um 23:01 schrieb Julian Sikorski:
Am 12.02.24 um 22:10 schrieb Miroslav Suchý:
Dne 12. 02. 24 v 19:29 Julian Sikorski napsal(a):
Hello,
has anyone successfully managed to get ccache to speed up kernel mockbuilds, be it SRPMS from kernel-ark or from dist-git? I gave it a brief shot but saw no difference, if anything the build was slower. The machine I am building on has 32 GB RAM which is not quite enough to fit everything on a ramdisk, and the drive I am building to is an NVME SSD, "only" PCI Express 3.0. CPU is a Ryzen 5 5600x. A baseonly builds takes about 25 minutes, which is somewhat annoying when one has to bisect something. If ccache can speed this up, how big does it need to be? I understand the default 4 GB is nowhere near enough but what would be needed to actually help? Thanks!
Just to be sure - did you configured Mock to use it?
https://rpm-software-management.github.io/mock/Plugin-CCache
the build from build container cannot see host ccache, so Mock must specially configure it.
This is by default off, because in multi user environment you can easily poison the cache and affect different build. This should be enabled only on VM where you trust all users that can run Mock. E.g. your personal developer workstation.
I am building with: --enable-plugin=ccache --plugin-option=ccache:max_cache_size='40G' Building with cold cache took 29:29, building with hot cache 29:45, so there appears to be no benefit so far. /var/cache/mock/fedora-36-x86_64/ccache is 9,5G in size, suggesting that size is not the problem. I am down to 21 commits to bisect after the current one, so changes to kernel source tree should be minimal at this point. Do I need to install ccache on the host as well?
Best regards, Julian
Installing ccache on the host did not help either. If anything, builds are getting slower, last one took 29:15. My guess would be that this is due to cache getting bigger (at 15G now) but no matches getting found. I am down to two commits to test left after the current one so source changes cannot possibly be the reason. Is the kernel version defined across the source perhaps? I am changing the LOCALVERSION from build to build in order to be able to tell the kernels apart, i.e. I have: - kernel-5.19.0-0.rc0.fdaf9a5840ac.200.bs11.fc36.src.rpm - kernel-5.19.0-0.rc0.fdaf9a5840ac.200.bs12.fc36.src.rpm - etc. Even the commit in the SRPM name does not change anymore at this point. Other than that, the build command is the same every single time:
- git bisect good/bad - git branch bs12 - git checkout os-build-5.19-alt - git merge bs12 - make LOCALVERSION=".bs12" BUILD=200 dist-srpm - mock --chain --localrepo /mnt/openmediavault/kernel/ -r eol/fedora-36-x86_64 --disable-plugin=tmpfs --enable-plugin=yum_cache --enable-plugin=ccache --plugin-option=ccache:max_cache_size='40G' --kernel-5.19.0-0.rc0.fdaf9a5840ac.200.bs12.fc39.src.rpm --with baseonly
Could this be the reason for ccache not working?
Best regards, Julian
Dne 13. 02. 24 v 9:08 Julian Sikorski napsal(a):
Could this be the reason for ccache not working?
I wonder whether it is Mock problem, Ccache issue or problem in packaging? Does the ccache speadup the build when you run it with plain rpmbuild and ccache on host?
On Tue, Feb 13, 2024 at 9:52 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 13. 02. 24 v 9:08 Julian Sikorski napsal(a):
Could this be the reason for ccache not working?
I wonder whether it is Mock problem, Ccache issue or problem in packaging? Does the ccache speadup the build when you run it with plain rpmbuild and ccache on host?
I have lost track of the details (and the version of ccache when the issue was addressed/patched), but ccache at one time included SOURCE_DATE_EPOCH in the default hash, resulting in no cache hits.
Am 13.02.24 um 16:16 schrieb Gary Buhrmaster:
On Tue, Feb 13, 2024 at 9:52 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 13. 02. 24 v 9:08 Julian Sikorski napsal(a):
Could this be the reason for ccache not working?
I wonder whether it is Mock problem, Ccache issue or problem in packaging? Does the ccache speadup the build when you run it with plain rpmbuild and ccache on host?
I have lost track of the details (and the version of ccache when the issue was addressed/patched), but ccache at one time included SOURCE_DATE_EPOCH in the default hash, resulting in no cache hits.
This appears to have been fixed in ccache-4.2. F36, obsolete as it is, shipped with 4.5.1. Having said that, with current kernel there is this line in the log: + perl -p -i -e 's/^CONFIG_BUILD_SALT.*/CONFIG_BUILD_SALT="6.7.4-200.lacie03.fc39.x86_64"/' .config This would change every build.
Best regards, Julian
Am 13.02.24 um 18:52 schrieb Julian Sikorski:
Am 13.02.24 um 16:16 schrieb Gary Buhrmaster:
On Tue, Feb 13, 2024 at 9:52 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 13. 02. 24 v 9:08 Julian Sikorski napsal(a):
Could this be the reason for ccache not working?
I wonder whether it is Mock problem, Ccache issue or problem in packaging? Does the ccache speadup the build when you run it with plain rpmbuild and ccache on host?
I have lost track of the details (and the version of ccache when the issue was addressed/patched), but ccache at one time included SOURCE_DATE_EPOCH in the default hash, resulting in no cache hits.
This appears to have been fixed in ccache-4.2. F36, obsolete as it is, shipped with 4.5.1. Having said that, with current kernel there is this line in the log:
- perl -p -i -e
's/^CONFIG_BUILD_SALT.*/CONFIG_BUILD_SALT="6.7.4-200.lacie03.fc39.x86_64"/' .config This would change every build.
Best regards, Julian
I had to bisect a kernel regression once again so I revisited this. I have modified my mock with PR showing ccache stats [1]. Interestingly enough, rebuilding the exact same srpm does result in a significant speedup and almost 100% cache hits. Build took ~12 minutes (as opposed to ~25 when building without cache) and cache stats looks as follows:
DEBUG util.py:463: Cacheable calls: 21570 / 21954 (98.25%) DEBUG util.py:463: Hits: 21554 / 21570 (99.93%) DEBUG util.py:463: Direct: 21458 / 21554 (99.55%) DEBUG util.py:463: Preprocessed: 96 / 21554 ( 0.45%) DEBUG util.py:463: Misses: 16 / 21570 ( 0.07%) DEBUG util.py:463: Uncacheable calls: 384 / 21954 ( 1.75%)
Next bisect step, couple hundred commits away (from fe46a7dd189e to eaf0e7a3d271 in case this is relevant), is a completely different story. Build took ~45 minutes and there are almost no cache hits:
DEBUG util.py:463: Cacheable calls: 21554 / 21938 (98.25%) DEBUG util.py:463: Hits: 1421 / 21554 ( 6.59%) DEBUG util.py:463: Direct: 1365 / 1421 (96.06%) DEBUG util.py:463: Preprocessed: 56 / 1421 ( 3.94%) DEBUG util.py:463: Misses: 20133 / 21554 (93.41%) DEBUG util.py:463: Uncacheable calls: 384 / 21938 ( 1.75%)
Further down the bisect, despite differences from build to build getting smaller, there is almost no difference. From 902861e34c40 to e5eb28f6d1af, build took around ~45 minutes too, and almost no cache hits were found either:
DEBUG util.py:463: Cacheable calls: 21571 / 21955 (98.25%) DEBUG util.py:463: Hits: 1395 / 21571 ( 6.47%) DEBUG util.py:463: Direct: 1373 / 1395 (98.42%) DEBUG util.py:463: Preprocessed: 22 / 1395 ( 1.58%) DEBUG util.py:463: Misses: 20176 / 21571 (93.53%) DEBUG util.py:463: Uncacheable calls: 384 / 21955 ( 1.75%)
Can the cache files be inspected with something to see what is getting cached?
Best regards, Julian
[1] https://github.com/rpm-software-management/mock/pull/1299
-
Gary Buhrmaster -
Julian Sikorski -
Miroslav Suchý