On Thursday, December 8, 2022, Daniel P. Berrangé <berrange(a)redhat.com>
wrote:
> On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote:
> > On Thursday, December 8, 2022, Adam Williamson <
> adamwill(a)fedoraproject.org>
> > wrote:
> >
> > > On Thu, 2022-12-08 at 12:58 +0000, Peter Robinson wrote:
> > > >
> > > > I've done a few passes, dropping a bunch of older firmware upstream
> > > > that are no longer supported in any stable kernel release, also a
> > > > bunch of de-dupe and linking of files rather than shipping of
> multiple
> > > > copies of the same firmware. It's improved things a bit,
> unfortunately
> > > > a lot of the dead firmware was tiny compared to say average modern
> > > > devices like GPUs or WiFI.
> > > >
> > > > The problem with a lot of the firmware, and with the new nvidia "open
> > > > driver" which shoves a lot of stuff into firmware in order to have an
> > > > upstreamable driver apparently the firmwares there are going to be
> > > > 30+Mb each, is that they're needed to bring up graphics/network etc
> to
> > > > even just install so I don't know how we can get around this and
> still
> > > > have a device work enough to be able to install the needed firmware
> > > > across the network.
> > > >
> > > > Ideas on how to solve that problem welcome.
> > >
> > > Sorry if this is way off, but - do we need the GPU firmwares to run a
> > > graphical install on the fallback path, just using the framebuffer set
> > > up by the firmware? How crazy would it be to just do that - ship the
> > > installer env with no GPU firmware?
> >
> >
> > That would be very crazy, as you will have a degraded user experience
> > (laggy UI, wrong resolution, ...) to save a couple of megabytes that are
> a
> > non issue for today's hardware.
>
> Please bear in mind the difference between bare metal and virtual
> machines. The bare metal machine may have 32 GB of RAM, making a
> 800 MB install image a non-issue. For a public cloud virtual machine
> though, this could bump your VM sizing up 1 level from 2 GB quota
> to a 4 GB RAM quota, with correspondingly higher price point. Now
> most people probably don't run the installer in a public cloud,
> preferring pre-built disk images. Even in a local machine though,
> you may be using most of your 32 GB of RAM for other things (well
> firefox/chrome), so allowing extra for the VM is not without
> resource cost. If we could figure out a way to knock a few 100 MB
> off the installer RAM requirements that is valuable.
>
>
The problem I see here is not the presence of the firmware on the image,
but the fact that it seems to be loaded into memory despite not being used.
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/
> dberrange :|
> |: https://libvirt.org -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/
> dberrange :|
> _______________________________________________
> kernel mailing list -- kernel(a)lists.fedoraproject.org
> To unsubscribe send an email to kernel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://docs.fedoraproject.
> org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/kernel@
> lists.fedoraproject.org
> Do not reply to spam, report it: https://pagure.io/fedora-
> infrastructure/new_issue
>
On Thu, 2022-12-08 at 12:50 -0500, David Cantrell wrote:
> On Wed, Dec 07, 2022 at 04:42:05PM -0800, Adam Williamson wrote:
> > Hi folks! Today I woke up and found
> > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me
> > down a bit of an "installer environment size" rabbit hole.
> >
> > As of today, with that new dep in webkitgtk, Rawhide's network install
> > images are 703M in size. Here's a potted history of network install
> > image sizes:
> >
> > Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M)
> > Fedora 13: 208M
> > Fedora 17: 162M (last "old UI")
> > Fedora 18: 294M (first "new UI")
> > Fedora 23: 415M
> > Fedora 28: 583M
> > Fedora 33: 686M
> > Fedora 37: 665M
> > Fedora Rawhide: 703M
> >
> > The installer does not really do much more in Rawhide than it did in
> > FC8. Even after the UI rewrite in F18, we were only at 294M. Now the
> > image is well over 2x as big and does...basically the same.
>
> I take issue with this. It is not accurate to say that the installer now does
> not do much more than it did for Fedora Core 8. There is more to the
> installer than the UI.
>
> Broadly speaking, a lot of the growth came from converging the runtime
> environment for the installer with the installed system. In Fedora Core 8 and
> previous releases, the "installer environment" was a unique and stripped down
> install. This was frustrating because it was effectively maintaining a small
> mini distro for the purposes of running the distro installer.
I meant it doesn't do much more in terms of what it achieves for the
user. But we can also just take F18 as the base point, if you like.
We're still over 2x as big as that was.
> I think some curation on firmware could happen. I think probinson@ mentions
> it later in this thread. If there were a way to identify the firmware
> necessary for the installer environment that could probably simplify things.
We already do a lot of "identifying the firmware necessary for the
installer environment", see my earlier mail about all the stuff lorax
does here. We've done the easy part, unfortunately. The stuff that's
left is stuff that is, in some sense, needed - graphics card and
wireless adapter firmwares, mainly.
>
> > Other obvious things that take up a lot of space:
> >
> > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is
> > 224M uncompressed. A quick test just compressing the file with xz on my
> > system shows it compresses to around 11M, though, so that's probably
> > all it adds up to after compression (the image is an xz-compressed
> > squashfs)
>
> Can this be installed compressed? I'm not sure it can.
The "compressed" size is the effective size we're concerned about,
because the installer filesystem image is an xz-compressed squashfs. So
11M is already the "effective weight" of this file, I think - if I ran
an image build with it removed, it'd probably be ~11M smaller.
>
> > 2. /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to
> > 23M. We are, I think, basically stuck with this for mesa-dri-drivers ,
> > but does it have to be so *big*?
>
> If mesa-dri-drivers is not required for installation, it could be removed from
> the installer environment.
It is required - we don't get any graphics without it.
> > 4. /usr/share/locale - 112M in total (uncompressed, not sure how much
> > compressed) of translated strings from a ton of packages. No idea how
> > many of these are really *needed* in the installer environment. We can
> > maybe come up with a way to have lorax strip some, if we can come up
> > with a viable way to figure out which. Obviously-fairly-large ones are
> > from gnupg2 and libgweather4. I do recall we have some logic somewhere
> > to decide which languages have a certain level of translation in
> > anaconda; perhaps we could only include the strings for these
> > languages?
>
> On that note, /usr/share/doc, /usr/share/man, and /usr/share/info could be
> removed from the installer image if they are present. That likely won't free
> a whole lot of space, but it's not nothing.
All of those are already stripped:
https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/run…
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
On Thu, 2022-12-08 at 08:26 -0500, Stephen Smoogen wrote:
> >
> The only ideas I have seen which 'work'* is to ship a minimal set of
> drivers for some 'chosen' hardware and then you have a bloated kitchen-sink
> iso which has all the drivers in it. The chosen hardware could be a
> 'defined' virtual environment which needs a minimal set of firmware,
> languages, etc. Everything else can be done in the install for getting
> languages, extra firmware, etc. The kitchen-sink.iso is going to be one
> which we know is going to be large.
>
> Now I have doubled the QA, releng, and product work.. I would say we would
> focus most of the work on the mini-installer, but we all know that all the
> work will be in the kitchen-sink one.
Well, if the two images are just "with firmware" and "without firmware"
I suppose in a way the testing load shouldn't be too awful, because we
can be pretty confident of the possible behaviours - nothing except the
kernel should do anything with kernel firmware files, after all, so
you're kinda limited to "it works fine", "some hardware doesn't work
because the firmware isn't there", and the occasional "kernel blew up
because firmware was missing which it really shouldn't do", which is
bad but at least easy to spot and workaround ("you gotta boot with the
firmware, sorry"). We could probably rely mostly on automated testing
to confirm that both images at least work properly in expected cases.
Also it's general not too onerous for us to test "some random alternate
path through the installer" because we have to run several install
tests on bare metal anyway so we can just kinda add it to the 'matrix'
there - "OK, for the firmware RAID install test I'll try booting the
no-firmware route", that knocks out two tests in one. (Of course you're
assuming that firmware RAID handling isn't somehow broken when booting
with the firmware, but that seems sufficiently unlikely not to worry
about :>)
If we want to get fancy, I suppose we could ship a single ISO, but with
two filesystem images - one the main installer environment, one
containing /lib/firmware - and just have a boot arg that (tells dracut
to) mounts the firmware one, and a boot menu option to *not* do it (not
pass the boot arg), which saves memory as long as the system doesn't
need the firmwares. But then of course someone will say "hey, why don't
we build a second ISO without the firmware image included at all, so we
have a small ISO for people who know they don't need it?" and you're
back at option 1. We also I guess have to think about how things work
for things like PXE installs, and maybe update the documentation
there...
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net