Hi folks! Today I woke up and found https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me down a bit of an "installer environment size" rabbit hole.
As of today, with that new dep in webkitgtk, Rawhide's network install images are 703M in size. Here's a potted history of network install image sizes:
Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M) Fedora 13: 208M Fedora 17: 162M (last "old UI") Fedora 18: 294M (first "new UI") Fedora 23: 415M Fedora 28: 583M Fedora 33: 686M Fedora 37: 665M Fedora Rawhide: 703M
The installer does not really do much more in Rawhide than it did in FC8. Even after the UI rewrite in F18, we were only at 294M. Now the image is well over 2x as big and does...basically the same.
Why does this matter? Well, the images being large is moderately annoying in itself just in terms of transfer times and so on. But more importantly, AIUI at least, the entire installer environment is loaded into RAM at startup - it kinda has to be, we don't have anywhere else to put it. The bigger it is, the more RAM you need to install Fedora. The size of the installer environment (for which the size of the network install image is more or less a perfect proxy) is one of the two key factors in this, the other being how much RAM DNF uses during package install.
So, I did a bit of poking about into *what* is taking up all that space. There's a variety of answers, but there's two major culprits:
1. firmware 2. yelp (which pulls in webkitgtk and its deps)
I've been using du and baobab (the GNOME visual disk usage analyzer, which is great) to examine the filesystems, but I ran a couple of test builds to confirm these suspects, especially after the impact of compression (it's hard to check the *compressed* size of things in the installer environment directly).
I did a scratch build of lorax which does not pull in firmware packages, and had openQA build a netinst using that lorax. It came out at 489M - 214M smaller than current netinsts, a size we last managed in Fedora 26. I did a scratch build of anaconda with its requirement of yelp dropped (which would break help pages), and built a netinst with that; it came out at 662M - 41M smaller than current images. I haven't run a combined test yet, but it ought to come out around 448M, around the size of Fedora 24.
Even then we'd still be about 50% larger than the Fedora 18 image, for not really any added functionality.
I've moaned about the sheer amount and size of firmware blobs in other forums before, but 214M compressed is *really* obnoxious. We must be able to do something to clean this up (further than it's already cleaned up - this is *after* we dropped low-hanging fruit like enterprise switch 'firmwares' and garbage like that; most of the remaining size seems to be huge amounts of probably-very-similar firmware files for AMD graphics adapters and Intel wireless adapters). I know some folks were trying to work on this (there was talk that we could drop quite a lot of files that would only be loaded by older kernels no longer in Fedora); any news on how far along that effort is?
Other obvious things that take up a lot of space:
1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is 224M uncompressed. A quick test just compressing the file with xz on my system shows it compresses to around 11M, though, so that's probably all it adds up to after compression (the image is an xz-compressed squashfs)
2. /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to 23M. We are, I think, basically stuck with this for mesa-dri-drivers , but does it have to be so *big*?
3. libicudata.so.71.1 - 30.4M, compresses to 7M. This is in the webkitgtk dep chain but seems to still be pulled in without it, not sure what else is requiring it.
4. /usr/share/locale - 112M in total (uncompressed, not sure how much compressed) of translated strings from a ton of packages. No idea how many of these are really *needed* in the installer environment. We can maybe come up with a way to have lorax strip some, if we can come up with a viable way to figure out which. Obviously-fairly-large ones are from gnupg2 and libgweather4. I do recall we have some logic somewhere to decide which languages have a certain level of translation in anaconda; perhaps we could only include the strings for these languages?
* Adam Williamson:
- /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is
224M uncompressed. A quick test just compressing the file with xz on my system shows it compresses to around 11M, though, so that's probably all it adds up to after compression (the image is an xz-compressed squashfs)
Isn't the compression block-based? I think it would be interesting to measure the image size with the file removed.
For the non-live installer, we can *significantly* cut down its size, without degrading localization of the installer itself.
- /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to
23M. We are, I think, basically stuck with this for mesa-dri-drivers , but does it have to be so *big*?
It has all the targets in it. As it's for JIT, we'd only need one target.
Thanks, Florian
On Thu, 2022-12-08 at 07:57 +0100, Florian Weimer wrote:
- Adam Williamson:
- /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is
224M uncompressed. A quick test just compressing the file with xz on my system shows it compresses to around 11M, though, so that's probably all it adds up to after compression (the image is an xz-compressed squashfs)
Isn't the compression block-based? I think it would be interesting to measure the image size with the file removed.
I'll try it tomorrow, it's not too hard.
For the non-live installer, we can *significantly* cut down its size, without degrading localization of the installer itself.
- /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to
23M. We are, I think, basically stuck with this for mesa-dri-drivers , but does it have to be so *big*?
It has all the targets in it. As it's for JIT, we'd only need one target.
That sounds interesting, though of course the details of how to implement it could be a bit tricky, I guess...
Thanks for the ideas!
* Adam Williamson:
On Thu, 2022-12-08 at 07:57 +0100, Florian Weimer wrote:
- Adam Williamson:
- /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is
224M uncompressed. A quick test just compressing the file with xz on my system shows it compresses to around 11M, though, so that's probably all it adds up to after compression (the image is an xz-compressed squashfs)
Isn't the compression block-based? I think it would be interesting to measure the image size with the file removed.
I'll try it tomorrow, it's not too hard.
Have you posted the outcome of the experiment somewhere?
Thanks, Florian
On Wed, Dec 7, 2022, at 11:57 PM, Florian Weimer wrote:
- Adam Williamson:
- /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is
224M uncompressed. A quick test just compressing the file with xz on my system shows it compresses to around 11M, though, so that's probably all it adds up to after compression (the image is an xz-compressed squashfs)
Isn't the compression block-based? I think it would be interesting to measure the image size with the file removed.
Squashfs defaults to a 128 KiB block size. I'm not 100% sure, but I'm pretty sure this means the compression algorithm is getting discrete 128 KiB (max) tranches to compress, meaning it's not able to see the big picture as well as when e.g. a tar.xz is created, and thus can see the redundancies better. So it's possible using xz on a file will give the impression of much better compression than what will happen with squashfs.
There are optimizations we're using with squashfs and xz though too that (a) have a higher computational cost up front in releng, improving the compression ratio (b) makes it harder to just use squashfs on CLI, you need to know the full incantation to do an apples to apples comparison. I think it's is in the log lorax produces.
On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks! Today I woke up and found https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me down a bit of an "installer environment size" rabbit hole.
As of today, with that new dep in webkitgtk, Rawhide's network install images are 703M in size. Here's a potted history of network install image sizes:
Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M) Fedora 13: 208M Fedora 17: 162M (last "old UI") Fedora 18: 294M (first "new UI") Fedora 23: 415M Fedora 28: 583M Fedora 33: 686M Fedora 37: 665M Fedora Rawhide: 703M
The installer does not really do much more in Rawhide than it did in FC8. Even after the UI rewrite in F18, we were only at 294M. Now the image is well over 2x as big and does...basically the same.
Why does this matter? Well, the images being large is moderately annoying in itself just in terms of transfer times and so on. But more importantly, AIUI at least, the entire installer environment is loaded into RAM at startup - it kinda has to be, we don't have anywhere else to put it. The bigger it is, the more RAM you need to install Fedora. The size of the installer environment (for which the size of the network install image is more or less a perfect proxy) is one of the two key factors in this, the other being how much RAM DNF uses during package install.
So, I did a bit of poking about into *what* is taking up all that space. There's a variety of answers, but there's two major culprits:
- firmware
- yelp (which pulls in webkitgtk and its deps)
I've been using du and baobab (the GNOME visual disk usage analyzer, which is great) to examine the filesystems, but I ran a couple of test builds to confirm these suspects, especially after the impact of compression (it's hard to check the *compressed* size of things in the installer environment directly).
I did a scratch build of lorax which does not pull in firmware packages, and had openQA build a netinst using that lorax. It came out at 489M - 214M smaller than current netinsts, a size we last managed in Fedora 26. I did a scratch build of anaconda with its requirement of yelp dropped (which would break help pages), and built a netinst with that; it came out at 662M - 41M smaller than current images. I haven't run a combined test yet, but it ought to come out around 448M, around the size of Fedora 24.
Even then we'd still be about 50% larger than the Fedora 18 image, for not really any added functionality.
I've moaned about the sheer amount and size of firmware blobs in other forums before, but 214M compressed is *really* obnoxious. We must be able to do something to clean this up (further than it's already cleaned up - this is *after* we dropped low-hanging fruit like enterprise switch 'firmwares' and garbage like that; most of the remaining size seems to be huge amounts of probably-very-similar firmware files for AMD graphics adapters and Intel wireless adapters). I know some folks were trying to work on this (there was talk that we could drop quite a lot of files that would only be loaded by older kernels no longer in Fedora); any news on how far along that effort is?
I've done a few passes, dropping a bunch of older firmware upstream that are no longer supported in any stable kernel release, also a bunch of de-dupe and linking of files rather than shipping of multiple copies of the same firmware. It's improved things a bit, unfortunately a lot of the dead firmware was tiny compared to say average modern devices like GPUs or WiFI.
The problem with a lot of the firmware, and with the new nvidia "open driver" which shoves a lot of stuff into firmware in order to have an upstreamable driver apparently the firmwares there are going to be 30+Mb each, is that they're needed to bring up graphics/network etc to even just install so I don't know how we can get around this and still have a device work enough to be able to install the needed firmware across the network.
Ideas on how to solve that problem welcome.
Peter
On Thu, 2022-12-08 at 12:58 +0000, Peter Robinson wrote:
I've done a few passes, dropping a bunch of older firmware upstream that are no longer supported in any stable kernel release, also a bunch of de-dupe and linking of files rather than shipping of multiple copies of the same firmware. It's improved things a bit, unfortunately a lot of the dead firmware was tiny compared to say average modern devices like GPUs or WiFI.
The problem with a lot of the firmware, and with the new nvidia "open driver" which shoves a lot of stuff into firmware in order to have an upstreamable driver apparently the firmwares there are going to be 30+Mb each, is that they're needed to bring up graphics/network etc to even just install so I don't know how we can get around this and still have a device work enough to be able to install the needed firmware across the network.
Ideas on how to solve that problem welcome.
Sorry if this is way off, but - do we need the GPU firmwares to run a graphical install on the fallback path, just using the framebuffer set up by the firmware? How crazy would it be to just do that - ship the installer env with no GPU firmware?
On Thursday, December 8, 2022, Adam Williamson adamwill@fedoraproject.org wrote:
On Thu, 2022-12-08 at 12:58 +0000, Peter Robinson wrote:
I've done a few passes, dropping a bunch of older firmware upstream that are no longer supported in any stable kernel release, also a bunch of de-dupe and linking of files rather than shipping of multiple copies of the same firmware. It's improved things a bit, unfortunately a lot of the dead firmware was tiny compared to say average modern devices like GPUs or WiFI.
The problem with a lot of the firmware, and with the new nvidia "open driver" which shoves a lot of stuff into firmware in order to have an upstreamable driver apparently the firmwares there are going to be 30+Mb each, is that they're needed to bring up graphics/network etc to even just install so I don't know how we can get around this and still have a device work enough to be able to install the needed firmware across the network.
Ideas on how to solve that problem welcome.
Sorry if this is way off, but - do we need the GPU firmwares to run a graphical install on the fallback path, just using the framebuffer set up by the firmware? How crazy would it be to just do that - ship the installer env with no GPU firmware?
That would be very crazy, as you will have a degraded user experience (laggy UI, wrong resolution, ...) to save a couple of megabytes that are a non issue for today's hardware.
Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
desktop mailing list -- desktop@lists.fedoraproject.org To unsubscribe send an email to desktop-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject. org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/desktop@ lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora- infrastructure/new_issue
On Thu, 2022-12-08 at 19:59 +0100, drago01 wrote:
On Thursday, December 8, 2022, Adam Williamson adamwill@fedoraproject.org wrote:
On Thu, 2022-12-08 at 12:58 +0000, Peter Robinson wrote:
I've done a few passes, dropping a bunch of older firmware upstream that are no longer supported in any stable kernel release, also a bunch of de-dupe and linking of files rather than shipping of multiple copies of the same firmware. It's improved things a bit, unfortunately a lot of the dead firmware was tiny compared to say average modern devices like GPUs or WiFI.
The problem with a lot of the firmware, and with the new nvidia "open driver" which shoves a lot of stuff into firmware in order to have an upstreamable driver apparently the firmwares there are going to be 30+Mb each, is that they're needed to bring up graphics/network etc to even just install so I don't know how we can get around this and still have a device work enough to be able to install the needed firmware across the network.
Ideas on how to solve that problem welcome.
Sorry if this is way off, but - do we need the GPU firmwares to run a graphical install on the fallback path, just using the framebuffer set up by the firmware? How crazy would it be to just do that - ship the installer env with no GPU firmware?
That would be very crazy, as you will have a degraded user experience (laggy UI, wrong resolution, ...) to save a couple of megabytes that are a non issue for today's hardware.
I mean, the modern systems that *need* GPU firmware generally seem to do pretty well with using native resolution and don't perform too badly, especially in the simple installer UI. When I test the fallback path on my bare metal test box on UEFI it uses the monitor's native resolution and performs fine (even, honestly, in GNOME), and that motherboard is nearly a decade old even. Don't know if this is the same for everyone, of course.
On Thu, Dec 8, 2022 at 4:56 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Thu, 2022-12-08 at 12:58 +0000, Peter Robinson wrote:
I've done a few passes, dropping a bunch of older firmware upstream that are no longer supported in any stable kernel release, also a bunch of de-dupe and linking of files rather than shipping of multiple copies of the same firmware. It's improved things a bit, unfortunately a lot of the dead firmware was tiny compared to say average modern devices like GPUs or WiFI.
The problem with a lot of the firmware, and with the new nvidia "open driver" which shoves a lot of stuff into firmware in order to have an upstreamable driver apparently the firmwares there are going to be 30+Mb each, is that they're needed to bring up graphics/network etc to even just install so I don't know how we can get around this and still have a device work enough to be able to install the needed firmware across the network.
Ideas on how to solve that problem welcome.
Sorry if this is way off, but - do we need the GPU firmwares to run a graphical install on the fallback path, just using the framebuffer set up by the firmware? How crazy would it be to just do that - ship the installer env with no GPU firmware?
That has crossed my mind, and with simpledrm that may be more straight forward now, but TBH it's not something I am skilled enough to deal with, nor have the resources to test, or actually care enough about, but the big GPU firmwares are now all split out so that should be much more straightforward for someone with the resources to investigate.
On Fri, 2022-12-09 at 11:12 +0000, Peter Robinson wrote:
On Thu, Dec 8, 2022 at 4:56 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Thu, 2022-12-08 at 12:58 +0000, Peter Robinson wrote:
I've done a few passes, dropping a bunch of older firmware upstream that are no longer supported in any stable kernel release, also a bunch of de-dupe and linking of files rather than shipping of multiple copies of the same firmware. It's improved things a bit, unfortunately a lot of the dead firmware was tiny compared to say average modern devices like GPUs or WiFI.
The problem with a lot of the firmware, and with the new nvidia "open driver" which shoves a lot of stuff into firmware in order to have an upstreamable driver apparently the firmwares there are going to be 30+Mb each, is that they're needed to bring up graphics/network etc to even just install so I don't know how we can get around this and still have a device work enough to be able to install the needed firmware across the network.
Ideas on how to solve that problem welcome.
Sorry if this is way off, but - do we need the GPU firmwares to run a graphical install on the fallback path, just using the framebuffer set up by the firmware? How crazy would it be to just do that - ship the installer env with no GPU firmware?
That has crossed my mind, and with simpledrm that may be more straight forward now, but TBH it's not something I am skilled enough to deal with, nor have the resources to test, or actually care enough about, but the big GPU firmwares are now all split out so that should be much more straightforward for someone with the resources to investigate.
Heck, if people want to try it out, we can. I can re-run the openQA test (the old one's assets will have been garbage collected by now) and pull the ISO out and upload it somewhere, and everyone can see how it behaves on their system. maybe I'll do that later...
desktop@lists.fedoraproject.org
-
Adam Williamson -
Chris Murphy -
drago01 -
Florian Weimer -
Peter Robinson