Hello guys I hope you're all doing fine, with joy in your hearts. After a adventurous while, I have updated and QAed a new qemu srpm. I worked mostly on x86 and sparc QA, since they are the ones in which we have to build additional firmware. Speaking of firware, for the very same reason as etherboot[1], I'm proposing some new packages. They are: Bochs bios: http://glommer.fedorapeople.org/bochs-bios-2.3.8-0.1.git36989b0d2.fc11.sr... http://koji.fedoraproject.org/koji/taskinfo?taskID=1121809 OpenBIOS: http://glommer.fedorapeople.org/openbios-1.0-0.1.svn450.fc11.src.rpm http://koji.fedoraproject.org/koji/taskinfo?taskID=1119906 VGABIOS: http://glommer.fedorapeople.org/vgabios-0.6-0.1beta.fc11.src.rpm http://koji.fedoraproject.org/koji/taskinfo?taskID=1120633

[1] Most firmware is compiled for one specific architecture, but should run on qemu whichever your host architecture is. We don't have a cross compiler infra in place, so we have to compile it natively. The easiest would be to pick binaries directly from upstream, which usually provides it. But we have policies that say we have to build everything we ship. So we have a crazy bootstrap process in place, in which we first build natively, and then feed a tar.gz for the other architectures. It means every package should be built at least twice in the process of getting something out of the door.

On Fri, Feb 13, 2009 at 09:50:06AM +0000, Mark McLoughlin wrote: > Hi Glauber, > > I'm glad you left the bottle of wine until last - this work requires a > strong stomach! :-) > > On Thu, 2009-02-12 at 13:29 -0200, Glauber Costa wrote: <snip> > > [1] Most firmware is compiled for one specific architecture, but > > should run on qemu whichever your host architecture is. We don't > > have a cross compiler infra in place, so we have to compile it natively. > > The easiest would be to pick binaries directly from upstream, which > > usually provides it. But we have policies that say we have to build > > everything we ship. So we have a crazy bootstrap process in place, > > in which we first build natively, and then feed a tar.gz for the > > other architectures. It means every package should be built at least > > twice in the process of getting something out of the door. > > Okay, let me see if I understand this: > > 1) You need to fix or update something in the package > > 2) You make the change and build it in koji > > 3) The build finishes, you download the binary RPM, unpack with > rpm2cpio and: > > $> cd usr/share > $> tar -cvjf etherboot-binaries-$(nvr).tar.bz etherboot/ > > 4) Update the spec file with the new tarball name, upload the tarball > to the sources repo, check in and build in Koji > > Two suggestions: > > 1) The second build shouldn't rebuild the binaries on any arch - that > way each arch has identical binaries Currently (on etherboot, and I think that's the case on the bios packages made by Glauber), we use prebuilt binaries only on the arches we can't build the binaries. This lets us use exactly the same package for step 2 and 4. But maybe we can change this: - First build: all prebuilt binaries tarballs removed from the spec, only native binaries built (using something like %define only_native 1) - Second build: added the prebuilt binaries from the first build, no binaries actually compiled, but uses only the prebuilt binaries We have a problem with any of the above approaches: we don't want to make the first build a scratch build (we want to keep the build information stored somewhere on Koji), but we don't want this build to be available for the users, as the result is an incomplete package. I am thinking about asking a Koji tag to be created just to store those "intermediate build step" RPMs. What do you think? > > 2) We add a "make update-binaries" the makefile - it would do: > > $> koji download-build $(make verrel) > $> rpm2cpio ... | cpio > $> tar -cvjf ... > $> grep -v etherboot-binaries sources > sources.tmp > $> mv sources.tmp sources > $> sed -i -e ... etherboot.spec > $> make upload FILES=... +1. This should be automated as much as possible.

> We have a problem with any of the above approaches: we don't want to make > the first build a scratch build (we want to keep the build information > stored somewhere on Koji), but we don't want this build to be available > for the users, as the result is an incomplete package. I am thinking > about asking a Koji tag to be created just to store those "intermediate > build step" RPMs. What do you think? > > > > > > 2) We add a "make update-binaries" the makefile - it would do: > > > > $> koji download-build $(make verrel) > > $> rpm2cpio ... | cpio > > $> tar -cvjf ... > > $> grep -v etherboot-binaries sources > sources.tmp > > $> mv sources.tmp sources > > $> sed -i -e ... etherboot.spec > > $> make upload FILES=... > > +1. This should be automated as much as possible. I've talked to some people in #fedora-devel yesterday, for _hours_. It turns out that they're planning on deploying some features in koji by Feb 28th, that _might_ help us a lot. Really, a lot. The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us. I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time.

On Fri, Feb 13, 2009 at 01:09:53PM +0000, Daniel P. Berrange wrote: > On Fri, Feb 13, 2009 at 10:59:04AM -0200, Glauber Costa wrote: > > > We have a problem with any of the above approaches: we don't want to make > > > the first build a scratch build (we want to keep the build information > > > stored somewhere on Koji), but we don't want this build to be available > > > for the users, as the result is an incomplete package. I am thinking > > > about asking a Koji tag to be created just to store those "intermediate > > > build step" RPMs. What do you think? > > > > > > > > > > > > > > 2) We add a "make update-binaries" the makefile - it would do: > > > > > > > > $> koji download-build $(make verrel) > > > > $> rpm2cpio ... | cpio > > > > $> tar -cvjf ... > > > > $> grep -v etherboot-binaries sources > sources.tmp > > > > $> mv sources.tmp sources > > > > $> sed -i -e ... etherboot.spec > > > > $> make upload FILES=... > > > > > > +1. This should be automated as much as possible. > > I've talked to some people in #fedora-devel yesterday, for _hours_. > > > > It turns out that they're planning on deploying some features in koji > > by Feb 28th, that _might_ help us a lot. Really, a lot. > > > > The idea is that you can have your package built on an architecture > > exclusively with BuildArch directive. But your package can then have > > a subpackage with BuildArch: noarch. That way, your package will span > > all repositories. This seems like the perfect solution to us. I've > > helped them to proceed with some tests, and it seem to work. So what > > I'm plannin on doing, is post the rpms for review today, since it is > > unlikely that they will receive more updates in this mean time. > > The main limitation with that approach is the architecture coverage, > and the way it relates to secondary architectures. This will only > let us easily do x86 & ppc. If we want sparc, arm or ia64 BIOS, the > builds would happen in the 2nd arch build ssytem, and the result > would not be available in our main Fedora YUM repos for primary > archs. If we go with your current approach, we can do the build in > the 2nd-ary arch, and then pull the result back into the packages > on the primary arches I don't know if I understood correctly. This means that the new feature could solve our problems, but it won't solve all of them because some arches where we need some binaries to be built are second-class citzens on the build system?

The idea is that you can have your package built on an architecture exclusively with BuildArch directive. But your package can then have a subpackage with BuildArch: noarch. That way, your package will span all repositories. This seems like the perfect solution to us.

I've helped them to proceed with some tests, and it seem to work. So what I'm plannin on doing, is post the rpms for review today, since it is unlikely that they will receive more updates in this mean time. Getting them review and approved is the really important bits. We can set on the ideal solution a bit later.

- First build: all prebuilt binaries tarballs removed from the spec, only native binaries built (using something like %define only_native 1) - Second build: added the prebuilt binaries from the first build, no binaries actually compiled, but uses only the prebuilt binaries

We have a problem with any of the above approaches: we don't want to make the first build a scratch build (we want to keep the build information stored somewhere on Koji),

but we don't want this build to be available for the users, as the result is an incomplete package.

I am thinking about asking a Koji tag to be created just to store those "intermediate build step" RPMs. What do you think?