On 06/29/2016 12:03 PM, Daniel P. Berrange wrote: > Debian handles this by having several packages [1] > > - qemu-user - the dynamic linked qemu user binaries > - qemu-binfmt - binfmt rules registering the dynamic linked binaries > - qemu-user-static - the static linked qemu user binaries *and* binfmt > rules to register them. The static binaries all > have -static suffix on their name Debian builds static libraries and puts them into their -dev packages. Fedora does not do this systematically. Are all the static libraries you need actually available in Fedora?

An alternative would be to bind-mount the real file system into the chroot and run the native dynamic linker with a --library-path command line argument that specifies the library directories in the bind mount. (It's reasonable to specify --inhibit-cache as well.) This would need some adjustments in the binfmt wrapper.

On Wed, 2016-06-29 at 11:34 +0100, Daniel P. Berrange wrote: > On Wed, Jun 29, 2016 at 12:15:02PM +0200, Florian Weimer wrote: > > On 06/29/2016 12:03 PM, Daniel P. Berrange wrote: > > > Debian handles this by having several packages [1] > > > > > > - qemu-user - the dynamic linked qemu user binaries > > > - qemu-binfmt - binfmt rules registering the dynamic linked binaries > > > - qemu-user-static - the static linked qemu user binaries *and* binfmt > > > rules to register them. The static binaries all > > > have -static suffix on their name > > > > Debian builds static libraries and puts them into their -dev packages. > > Fedora does not do this systematically. Are all the static libraries you > > need actually available in Fedora? > > Yes, everything we need exists - as mentioned its only glibc-static, > glib2-static, zlib-static and pcre-static that we need for this. > > > An alternative would be to bind-mount the real file system into the chroot > > and run the native dynamic linker with a --library-path command line > > argument that specifies the library directories in the bind mount. (It's > > reasonable to specify --inhibit-cache as well.) This would need some > > adjustments in the binfmt wrapper. > > The binfmt registrations are global to the OS, so the same binfmt command > needs to work whether inside or outside the chroot. So a scheme which > requires us to pass special args to make the linker looks elsewhere when > inside the chroot is not so easy. We'd have to create a wrapper program > around the real qemu user binary that decides which args to pass to the > linker, and that wrapper would then have to be statically linked Just an idea: Would it make sense to build the qemu-user binary so that it looks in a different /lib directory (using rpath) like /lib/qemu-user-systemlib, and then symlink into that directory the libraries it needs from the host ? That way if you use chroot and bind mount in that directory in the right place the qemu-user binary uses different libraries from the emulated binaries yet you do not have to resort to static linking.

> > symlink: /usr/lib/qemu-user-root -> / > > dir: /usr/lib/qemu-user-host-lib > > symlink: /usr/lib/qemu-user-host-lib/libc.so -> /usr/lib/qemu-user-root/lib/libc.so > > > > And build qemu-$ARCH with rpath to /usr/lib/qemu-user-root/lib. > > > > Then in the chroot you would need to bind mount the host root into > > /usr/lib/qemu-user-root, to override the symlink that would otherwise > > point back to / Why do you need 2 symlinks ? I was thinking you'd have symlink: /usr/lib/qemu-user-lib-x86_64/ -> /usr/lib In the chroot you just bind mount the host's /usr/lib as /usr/lib/qemu-user-lib-x86_64 This assuming x86-64, you can bind mount in the appropriate arch-target dir on other arches.

On 06/29/2016 12:34 PM, Daniel P. Berrange wrote: > On Wed, Jun 29, 2016 at 12:15:02PM +0200, Florian Weimer wrote: > > On 06/29/2016 12:03 PM, Daniel P. Berrange wrote: > > > Debian handles this by having several packages [1] > > > > > > - qemu-user - the dynamic linked qemu user binaries > > > - qemu-binfmt - binfmt rules registering the dynamic linked binaries > > > - qemu-user-static - the static linked qemu user binaries *and* binfmt > > > rules to register them. The static binaries all > > > have -static suffix on their name > > > > Debian builds static libraries and puts them into their -dev packages. > > Fedora does not do this systematically. Are all the static libraries you > > need actually available in Fedora? > > Yes, everything we need exists - as mentioned its only glibc-static, > glib2-static, zlib-static and pcre-static that we need for this. In this case, linking statically is indeed an option. We currently have little tooling for static libraries. Strictly speaking, all statically linked libraries have to be rebuilt even for minor glibc updates because we do not provide ABI compatibility for static linking (there are no compatibility symbols, static binaries always get the most recent implementation).

On Wed, Jun 29, 2016 at 3:03 AM, Daniel P. Berrange <berrange(a)redhat.com&gt; wrote: > For those who aren't familiar, QEMU actually provides two completely > different sets of emulators > > - system emulators - they emulate a full virtual machine and thus run > a full guest OS. > - user emulators - they emulate the Linux userspace ABI letting you > run non-native arch executables directly. > > The user emulators are what I'm concerned with in this mail, so ignore > the system emulators. > > Currently all the user emulators are provided in the "qemu-user" RPM > which also includes files in /usr/lib/binfmt.d to register each emulator > binary as a binary format handler for its respective architecture. > > This is ok if you have a non-native arch binary that's statically linked > and you just want to run it from context of your main OS root filesystem. > Running dynamic linked binaries won't fly because if say running an arm > binary on x86_64 host, it'll look for /lib/libc.so and find the i386 one, > instead of the arm one. You can't set LD_LIBRARY_PATH to override this > as the env var will apply to both qemu-arm (an x86_64 binary) and the > binary it is trying to run (an arm binary). > > More typical though is that you have a directory containing an fullish > install tree of a non-native architecture and you just want to chroot > into that. When doing such a chroot, the qemu-$ARCH emulator must be > present inside the chroot too. ie the x86_64 build of /usr/bin/qemu-arm > must be present inside at /my/chroot/for/fedora-arm/usr/bin/qemu-arm. > So again you have the potential problem of clashing libc.so in /usr/lib > It is a shame Fedora doesn't have full multi-arch support, instead of > merely multi-lib to avoid these clashing lib dirs across architecture > RPMs. > > The recommended way to deal with this for the qemu user emulator binaries > to be statically linked, so when copied inside the non-native arch chroot, > they never need to resolve any native arch libraries. Fedora's qemu user > binaries are all dynamic linked right now. > > Debian handles this by having several packages [1] > > - qemu-user - the dynamic linked qemu user binaries > - qemu-binfmt - binfmt rules registering the dynamic linked binaries > - qemu-user-static - the static linked qemu user binaries *and* binfmt > rules to register them. The static binaries all > have -static suffix on their name > > NB, this means qemu-binfmt and qemu-user-static are mutually exclusive > since they both provide the same binfmt files. You can however have both > qemu-user and qemu-user-static installed as their binary names won't > clash, and in this case the static ones will be registered as binfmts > > This nice thing about this multiple package approach is that when you > copied the x86_64 build of the "qemu-arm-static" binary into your arm > chroot, you still then have the possibility of installing the arm build > of the "qemu-arm" binary inside that chroot without filename clash. > > An alternative simpler approach would be to just have one package, > qemu-user, which contains the static binaries and never ship any > dynamic linked qemu user binaries. This is slightly more restrictive > though, as explained in the previous paragraph, so I'd like to avoid > doing that. > > > I'd like to make using non-native arch chroots simple with Fedora without > people needing to manually build their own static QEMU binaries, or download > static binaries provided by another distro[2]. So I'm suggesting to make a > change to Fedora qemu packages to essentially copy the way Debian has done > things. Specifically I will > > - Pull the binfmt registration files out of qemu-user and into a > new qemu-binfmt package which depends on qemu-user. > > - Add static builds of qemu user emulators to a new qemu-user-static > package, along with binfmt registration files > > The static build of QEMU user emulators is moderately light on > dependancies, only requiring glib2-static, pcre-static, zlib-static > and glibc-static packages. > > The change to introduce a qemu-binfmt package has small upgrade > implications since anyone with qemu-user installed today, will loose > the binary format rules unless they manually install qemu-binfmt. I > think the number of people affected is probably quite small, and some > of them may well wish to use qemu-user-static instead anyway. > > Obviously this would only be done in rawhide, not any existing stable > releases of Fedora. > > Nothing will change about the rest of QEMU packaging - ie all system > emulators will continue to use dynamic linking > > Regards, > Daniel > Amazingly, I've been pre-empted here. I was going to ask about this for Fedora, since we've got interest in this for Mageia as part of potentially enabling cross-arch building with mock in an easy fashion. Mageia tries to keep its QEMU package in sync with Fedora, so I'd love to see this happen so that it becomes easier to maintain the qemu-user-static stuff. Right now, we only have a couple of the arches built, but one of our guys interested in this might be willing to help out on this side of the fence with it (Per Øyvind Karlsen). I've CC'd him to this email.

On 06/29/2016 12:03 PM, Daniel P. Berrange wrote: > The change to introduce a qemu-binfmt package has small upgrade > implications since anyone with qemu-user installed today, will loose > the binary format rules unless they manually install qemu-binfmt. Not if you add "Obsoletes: qemu-user < V-R" (where V-R is the first version-release that contains this change) to both qemu-binfmt and qemu-user.

"Daniel P. Berrange" <berrange(a)redhat.com&gt; writes: > More typical though is that you have a directory containing an fullish > install tree of a non-native architecture and you just want to chroot > into that. When doing such a chroot, the qemu-$ARCH emulator must be > present inside the chroot too. ie the x86_64 build of /usr/bin/qemu-arm > must be present inside at /my/chroot/for/fedora-arm/usr/bin/qemu-arm. Hi, Dan, Is this work from James Bottomley at all relevant? http://comments.gmane.org/gmane.linux.file-systems/105033 http://blog.hansenpartnership.com/constructing-architecture-emulation-con...