I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdls
It's focusing on the properties that are different in the two schemes as such it is not a stand-alone specification.
The purpose of this write-up is to clarify the issues we're facing and in light of this to stop further adoption of the current kernel module scheme especially in GFS related parts in FC.
Please take some time to read it through, if I forgot some arguments I will add them later to it. I would very much like to discuss it on Thursday and vote on whether the kmod scheme should be blocked in favour of adopting a kmdl scheme.
Beyond fedora-packaging's scope, e.g. hammering together a sane guide for kernel modules, I'm willing to work with build infrastructure to get the kmdl scheme firmly in place if this is approved and ratified. I've done that in the past with ATrpms' buildsystem, so I hope it will not be a too difficult task.
Furthermore I will work with upstream depsolver authors to get the (trivial) kmdl support in all depsolvers. For yum and apt this can be considered as already done, for smart I requested convenient hooks for plugging in kmdl support which upstream already accepted. I will also try to promote this scheme to the kmp/kerneldriver.org folks.
I think the kmdl scheme has a chance of becoming a wider-spread standard that all rpm distros will benefit from.
Thanks.
Axel Thimm schrieb:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdls
Just FYI, I won't oppose this in general -- if people think this is the way forward (*1) and if we get a buy in from Core developers, too, I'm fine with working further on it.
CU thl
(*1) -- But I doubt that it is because some important pieces of this scheme are similar to the one we started with when we developed the current kmod-standard
On Tue, Aug 08, 2006 at 12:04:51PM +0200, Thorsten Leemhuis wrote:
Axel Thimm schrieb:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdlsJust FYI, I won't oppose this in general -- if people think this is the way forward (*1) and if we get a buy in from Core developers, too, I'm fine with working further on it.
Thanks.
CU thl
(*1) -- But I doubt that it is because some important pieces of this scheme are similar to the one we started with when we developed the current kmod-standard
I hope that the wiki page is outlining the issues good enough for non-kernel-module experts, too, so that pointing Core folks disliking this proposal to the wiki may help.
But first the fedora-packaging group needs to give it's blessing before the real nasty guys show up ;)
On Tue, Aug 08, 2006 at 12:04:51PM +0200, Thorsten Leemhuis wrote:
Axel Thimm schrieb:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdlsJust FYI, I won't oppose this in general -- if people think this is the way forward (*1) and if we get a buy in from Core developers, too, I'm fine with working further on it.
What exactly would it mean if you had opposed in general? I see you oppose this on every corner of the proposal, defending the kmod scheme more and more the more flaws are brought up. You need to be able to let leave for the sake of the project.
On Tue, 2006-08-08 at 11:26 +0200, Axel Thimm wrote:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdls
Axel,
Thank you for taking the time to do this. I honestly think it will be helpful for the Packaging Committee to have this information in front of them.
Rather than trying to replace kmod with kmdl, I'd rather look at the key changes that we should consider making.
The biggest one, IMHO, is overloading name with the kernel version. I've been one of the staunchest opponents of doing this, because I think its ugly, a hack, and causes problems.
With all that said: I now think it is necessary for kernel module packages. I did a lot of thinking and reading over the last several days, and overloading the name works. We know it works, whether done with rpm by hand or via depsolvers (yum).
As Axel points out: It makes kernel module packages completely independent across kernels, and within a kernel, the kernel modules are normally updated. I think these things are key. Users expect things to just work without having to worry about doing things differently or special. The reason that third party repositories such as ATrpms have been so successful is because things just work.
So, what problems does it cause to overload the name?
1. cvs: No changes necessary. CVS keys off SRPM name, which remains foo-kmod.
2. buildsystem: The buildsystem needs to treat kernel-module packages differently, but we've got the buildsystem code authors on board to help fit the buildsystem to our standards (within reason). Either way, the buildsystem has to detect kernel modules and build them specially, so this is just a different color of paint. Plus, Axel has volunteered to help with this.
3. bugzilla: Bugzilla pulls from owners.list, which bases off SRPM/CVS, so we're fine here.
4. rpm queries: rpm -q kmod-foo doesn't return anything? Say what? Ehh. If you're a power user enough to be querying with rpm on the commandline, you're geek enough to rpm -qa |grep kmod-foo and find it.
I now believe that the benefits of overloading the name with kver outweigh any pain it causes, and I propose that we amend the existing kernel module standard to include the version of the kernel in the name field.
Here is an updated version of the kmod proposal with kver in the name: http://fedoraproject.org/wiki/PackagingDrafts/KernelModulesWithKverInName
~spot
On Tue, 2006-08-08 at 14:04 -0500, Tom 'spot' Callaway wrote:
On Tue, 2006-08-08 at 11:26 +0200, Axel Thimm wrote:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdlsAxel,
Thank you for taking the time to do this. I honestly think it will be helpful for the Packaging Committee to have this information in front of them.
Rather than trying to replace kmod with kmdl, I'd rather look at the key changes that we should consider making.
The biggest one, IMHO, is overloading name with the kernel version. I've been one of the staunchest opponents of doing this, because I think its ugly, a hack, and causes problems.
With all that said: I now think it is necessary for kernel module packages. I did a lot of thinking and reading over the last several days, and overloading the name works. We know it works, whether done with rpm by hand or via depsolvers (yum).
As Axel points out: It makes kernel module packages completely independent across kernels, and within a kernel, the kernel modules are normally updated. I think these things are key. Users expect things to just work without having to worry about doing things differently or special. The reason that third party repositories such as ATrpms have been so successful is because things just work.
So, what problems does it cause to overload the name?
- cvs: No changes necessary. CVS keys off SRPM name, which remains
foo-kmod.
- buildsystem: The buildsystem needs to treat kernel-module packages
differently, but we've got the buildsystem code authors on board to help fit the buildsystem to our standards (within reason). Either way, the buildsystem has to detect kernel modules and build them specially, so this is just a different color of paint. Plus, Axel has volunteered to help with this.
- bugzilla: Bugzilla pulls from owners.list, which bases off SRPM/CVS,
so we're fine here.
- rpm queries: rpm -q kmod-foo doesn't return anything? Say what? Ehh.
If you're a power user enough to be querying with rpm on the commandline, you're geek enough to rpm -qa |grep kmod-foo and find it.
I now believe that the benefits of overloading the name with kver outweigh any pain it causes, and I propose that we amend the existing kernel module standard to include the version of the kernel in the name field.
Here is an updated version of the kmod proposal with kver in the name: http://fedoraproject.org/wiki/PackagingDrafts/KernelModulesWithKverInName
not to make work but it wouldn't be hard.
would it be worth making a little script to help users manage kernel modules like they would with rpm?
kernel-module-package -q kmod-foo
would look through the package lists/provides lists and find out all the installed packages that way?
-sv
On Tue, 2006-08-08 at 15:07 -0400, seth vidal wrote:
not to make work but it wouldn't be hard.
would it be worth making a little script to help users manage kernel modules like they would with rpm?
kernel-module-package -q kmod-foo
would look through the package lists/provides lists and find out all the installed packages that way?
That would be trivial. It could either share code with repoquery or simply call out to it.
~spot
On Tue, 2006-08-08 at 14:15 -0500, Tom 'spot' Callaway wrote:
On Tue, 2006-08-08 at 15:07 -0400, seth vidal wrote:
not to make work but it wouldn't be hard.
would it be worth making a little script to help users manage kernel modules like they would with rpm?
kernel-module-package -q kmod-foo
would look through the package lists/provides lists and find out all the installed packages that way?
That would be trivial. It could either share code with repoquery or simply call out to it.
exactly.
-sv
On Tue, 2006-08-08 at 14:04 -0500, Tom 'spot' Callaway wrote:
On Tue, 2006-08-08 at 11:26 +0200, Axel Thimm wrote:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdls
One thing completely missing from that is debuginfo packages.
The NEVR of the debuginfo package is derived from the name of the *source* package, which means overlaps/unshippability of debuginfo between builds unless 1) the source package's NEVR changes on every rebuild, *including* just rebuilding it for a newer kernel using different rpmbuild flags or whatever, or 2) the module packages implement their own debuginfo package creation. https://bugzilla.redhat.com/113276
The reason that third party repositories such as ATrpms have been so successful is because things just work.
Things certainly haven't "just work"ed without special user education and making POLA violations a standard practice. Dunno about the current state of affairs with using that scheme, but I've seen the bug reports elsewhere in the past. Before depsolvers are adapted to do "the right thing", their users need to remember to manually pull in module package updates when a new kernel is installed. Granted, with the suggested scheme, they *can* do that without interference from other module packages in more situations than with the current one, ditto with the rpm CLI.
I now believe that the benefits of overloading the name with kver outweigh any pain it causes
I have no doubts that it can be made to work (and there is still some work to do, eg. debuginfo, depsolvers), but I'm still not convinced that it's worth the trouble. But that's moot if consensus says otherwise and there's competent manpower available to do the work.
On Wed, Aug 09, 2006 at 12:29:08AM +0300, Ville Skyttä wrote:
On Tue, 2006-08-08 at 14:04 -0500, Tom 'spot' Callaway wrote:
On Tue, 2006-08-08 at 11:26 +0200, Axel Thimm wrote:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdlsOne thing completely missing from that is debuginfo packages.
The NEVR of the debuginfo package is derived from the name of the *source* package, which means overlaps/unshippability of debuginfo between builds unless 1) the source package's NEVR changes on every rebuild, *including* just rebuilding it for a newer kernel using different rpmbuild flags or whatever, or 2) the module packages implement their own debuginfo package creation. https://bugzilla.redhat.com/113276
The reason that third party repositories such as ATrpms have been so successful is because things just work.
Things certainly haven't "just work"ed without special user education and making POLA violations a standard practice. Dunno about the current state of affairs with using that scheme, but I've seen the bug reports elsewhere in the past. Before depsolvers are adapted to do "the right thing", their users need to remember to manually pull in module package updates when a new kernel is installed. Granted, with the suggested scheme, they *can* do that without interference from other module packages in more situations than with the current one, ditto with the rpm CLI.
I now believe that the benefits of overloading the name with kver outweigh any pain it causes
I have no doubts that it can be made to work (and there is still some work to do, eg. debuginfo, depsolvers), but I'm still not convinced that it's worth the trouble. But that's moot if consensus says otherwise and there's competent manpower available to do the work.
I've spent quite a bit of time working on a Yum plugin to add the proper functionality to Yum. In fact I spent part of today adding some features that Thorsten had requested. My goal was to have decent support in FC6 for kernel modules. The freeze is about a month away.
I will veto Axel's current plugin as it uses regular expressions to parse the name of the package. I've seen some fun kernel versions and parsing that out of the name is just asking for corner cases. This information needs to be pulled from what the package provides or requires.
Most of my hesitation regarding the uname-r in name scheme is because this was fully discussed before and we decided on something different. FESCo ratified it and we were going to be able to support in in FC6 and RHEL 5.
There is another important trade off here. Right now up2date/yum/whatever is able to suck down upgraded kernel modules just like upgrades to a new kernel. This works without modifications to the depresolvers because the package name is always the same. With the uname-r in name scheme we are now dependent on some other additional magic to pull down the new kernel modules for the new kernel everyone got 3 days ago. My shop depends on OpenAFS. AFS issues are a complete show stopper. I manage well over 1,300 servers/workstations. Now instead of writing code to make my job easier I have to write code to keep my work load at its current level.
Jack
On Tue, Aug 08, 2006 at 06:29:41PM -0400, Jack Neely wrote:
I will veto Axel's current plugin as it uses regular expressions to
Let the plugin wars begin ;)
Most of my hesitation regarding the uname-r in name scheme is because this was fully discussed before and we decided on something different. FESCo ratified it and we were going to be able to support in in FC6 and RHEL 5.
Decisions can be wrong especially when it affects complex matters that affects only few packages like less than 0.5% of all packages. The important thing is that any decision needs to be reevaluated from time to time and possibly changed. Otherwise you kill development.
There is another important trade off here. Right now up2date/yum/whatever is able to suck down upgraded kernel modules just like upgrades to a new kernel. This works without modifications to the depresolvers because the package name is always the same.
No, you forget that this scheme either nukes or conflicts/coinstalls the new modules. Imagine running up2date/yum w/o any kmod-plugin support on RHEL5 and nuking your GFS (or AFS) modules away. You (and your clients) will not be amused.
On Wed, Aug 09, 2006 at 01:42:10AM +0200, Axel Thimm wrote:
On Tue, Aug 08, 2006 at 06:29:41PM -0400, Jack Neely wrote:
I will veto Axel's current plugin as it uses regular expressions to
Let the plugin wars begin ;)
Most of my hesitation regarding the uname-r in name scheme is because this was fully discussed before and we decided on something different. FESCo ratified it and we were going to be able to support in in FC6 and RHEL 5.
Decisions can be wrong especially when it affects complex matters that affects only few packages like less than 0.5% of all packages. The important thing is that any decision needs to be reevaluated from time to time and possibly changed. Otherwise you kill development.
There is another important trade off here. Right now up2date/yum/whatever is able to suck down upgraded kernel modules just like upgrades to a new kernel. This works without modifications to the depresolvers because the package name is always the same.
No, you forget that this scheme either nukes or conflicts/coinstalls the new modules. Imagine running up2date/yum w/o any kmod-plugin support on RHEL5 and nuking your GFS (or AFS) modules away. You (and your clients) will not be amused. -- Axel.Thimm at ATrpms.net
Okay...walk me through this then:
Assuming no yum plugins or other mess.
A new kernel is available that corrects some random remote DoS. How do I get all 1300 machines to pull down the new AFS modules?
More complex: How do I get 1300 machines to download both the new kernel and AFS modules together?
Assume that we cannot allow a machine to reboot between getting the new kernel and getting the new AFS modules. (This equals broken for me.)
Jack
On Tue, Aug 08, 2006 at 08:06:49PM -0400, Jack Neely wrote:
On Wed, Aug 09, 2006 at 01:42:10AM +0200, Axel Thimm wrote:
On Tue, Aug 08, 2006 at 06:29:41PM -0400, Jack Neely wrote:
I will veto Axel's current plugin as it uses regular expressions to
Let the plugin wars begin ;)
Most of my hesitation regarding the uname-r in name scheme is because this was fully discussed before and we decided on something different. FESCo ratified it and we were going to be able to support in in FC6 and RHEL 5.
Decisions can be wrong especially when it affects complex matters that affects only few packages like less than 0.5% of all packages. The important thing is that any decision needs to be reevaluated from time to time and possibly changed. Otherwise you kill development.
There is another important trade off here. Right now up2date/yum/whatever is able to suck down upgraded kernel modules just like upgrades to a new kernel. This works without modifications to the depresolvers because the package name is always the same.
No, you forget that this scheme either nukes or conflicts/coinstalls the new modules. Imagine running up2date/yum w/o any kmod-plugin support on RHEL5 and nuking your GFS (or AFS) modules away. You (and your clients) will not be amused.
Okay...walk me through this then:
Assuming no yum plugins or other mess.
A new kernel is available that corrects some random remote DoS. How do I get all 1300 machines to pull down the new AFS modules?
It's in the wiki, but here it comes again:
o current kernel module scheme w/o any special depsolver handling: - broken on rpm level, inherits on all depsolvers - Modules of the current kernel get nuked whether you reboot into the new kernel or not - Module upgrades within the same kernel get blocked due to file conflicts - OR module upgrades within the same kernel get coinstalled and module-init-tools can flip a coin to find out what to choose + but the new kernel gets its kernel modules (and only the new kernel ...)
o proposed kernel module scheme + old and current kernels remain untouched by new kernels and kernel modules for new kernels + Proper upgrades of kernel modules within a kernel - new kernels don't get associated kmdls coinstalled
Now compare and judge for yourself :=)
Next thing to consider is whether and how to fix the remaining issues:
o current kernel module scheme special handling: - no special handling possible with rpm cli will remain forever broken - needs to both mark kernel module packages as coinstallable (which they are not) and to perform non-rpm comformant depsolving - Is required for all depsolvers and is a MUST otherwise depsolvers reduce your system to ashes
o proposed kernel module scheme special handling: + nothing needed for rpm cli + Obviously less than half the code needed than for the current scheme, rather trivial coding, easy maintenance + Is NICE TO HAVE, absence does not break running system
Compare again. :=)
Okay...walk me through this then:
Assuming no yum plugins or other mess.
A new kernel is available that corrects some random remote DoS. How do I get all 1300 machines to pull down the new AFS modules?
It's in the wiki, but here it comes again:
o current kernel module scheme w/o any special depsolver handling:
- broken on rpm level, inherits on all depsolvers
- Modules of the current kernel get nuked whether you reboot into the new kernel or not
Wrong. Both up2date and yum have always marked packages that provide 'kernel-modules' as install only for several years now. Modules don't get "nuked" unless you rpm -U.
- Module upgrades within the same kernel get blocked due to file conflicts
- OR module upgrades within the same kernel get coinstalled and module-init-tools can flip a coin to find out what to choose
Upgrades within the same kernel don't work. This is one point, not multiple.
- but the new kernel gets its kernel modules (and only the new kernel ...)
This point has been used in practice by several large universities. I've been doing this for about 6 years. While not perfect its been proven to be acceptable and allow machines to remain fulled patched.
NC State University. Duke. I believe Matt at Boston U. has used this approch in the past as well.
o proposed kernel module scheme
- old and current kernels remain untouched by new kernels and kernel modules for new kernels
Contrast: The current tools allow this as well. I use up2date. It does not nuke my old kernel modules.
- Proper upgrades of kernel modules within a kernel
- new kernels don't get associated kmdls coinstalled
This is a big problem. Your scheme discurages people from keeping their systems updated. You'd rather have all of my 1300 machines run a swiss cheese kernel from 2003 for RHEL 3.
Now compare and judge for yourself :=)
I did. Its hard enough to get people to keep their kernels upgraded. Your scheme makes it next to impossible without manual touching of all machines. This is not acceptable.
Next thing to consider is whether and how to fix the remaining issues:
o current kernel module scheme special handling:
- no special handling possible with rpm cli will remain forever broken
And it sucks. I agree here. However, ask who did not make compromises when we developed the current scheme.
- needs to both mark kernel module packages as coinstallable (which they are not) and to perform non-rpm comformant depsolving
The co-installable mark has been in production and tested for 3 years now I beleive.
I use the caveat of not issuing an upgrade within a kernel version. Modules act identically to kernel packages themselves.
- Is required for all depsolvers and is a MUST otherwise depsolvers reduce your system to ashes
I've been doing this with up2date for quite a while and manage to maintain mine. Seth? Matt?
o proposed kernel module scheme special handling:
- nothing needed for rpm cli
- Obviously less than half the code needed than for the current scheme, rather trivial coding, easy maintenance
How much code do we need to support pulling in modules for new kernels?
- Is NICE TO HAVE, absence does not break running system
No. Yum/Up2date modifications would be required. Not optional. We must not discurage people from applying security errata.
Compare again. :=)
Jack
-- Axel.Thimm at ATrpms.net
-- Fedora-packaging mailing list Fedora-packaging@redhat.com https://www.redhat.com/mailman/listinfo/fedora-packaging
NC State University. Duke. I believe Matt at Boston U. has used this approch in the past as well.
Agreed. We've been using kernels and kernel-modules in this way for a number of years, now.
example: rpm -q --provides openafs-kernel kernel-module-openafs-2.6.9-22.0.2.EL kernel-modules openafs-kernel = 0:1.3.82-7.duke.1.centos4
-sv
On Wed, 9 Aug 2006, seth vidal wrote:
NC State University. Duke. I believe Matt at Boston U. has used this approch in the past as well.
Agreed. We've been using kernels and kernel-modules in this way for a number of years, now.
example: rpm -q --provides openafs-kernel kernel-module-openafs-2.6.9-22.0.2.EL kernel-modules openafs-kernel = 0:1.3.82-7.duke.1.centos4
Just for the measure, we've been using kernel-module-foo-<uname -r> scheme (basically the old livna.org scheme which is similar to Axels kmdl scheme) very successfully at work for a few years. Been working quite nicely, first with apt and now yum, both with an additional plugin to handle these.
Now, I don't really want to take any sides in this, I just want the dang thing to be decided one way or the other so we can move on to refining our plugins, as plugins are required in both schemes to be correctly handled in all cases. From experience I know the unamer-in-name works quite nicely but has it quirks from user POV [1], OTOH the kmod scheme (currently used by livna) also seems to work just fine, at least it hasn't bitten me yet although I haven't used any plugins to handle it.
[1] When you can't rely on plugins, eg with plain rpm, one needs to remember to append -`uname -r` to the package name. Our users haven't complained about that, mostly I suppose because they never really see it thanks to plugins.
- Panu -
On Thu, 2006-08-10 at 08:08 +0300, Panu Matilainen wrote:
Now, I don't really want to take any sides in this, I just want the dang thing to be decided one way or the other so we can move on to refining our plugins, as plugins are required in both schemes to be correctly handled in all cases. From experience I know the unamer-in-name works quite nicely but has it quirks from user POV [1], OTOH the kmod scheme (currently used by livna) also seems to work just fine, at least it hasn't bitten me yet although I haven't used any plugins to handle it.
+1
I was not taking a side in it. I was just explaining about what has worked about the kmod scheme in the past.
I'd just like one or the other, ultimately, please, for the love of all that's good and holy. :)
-sv
On Thu, 10 Aug 2006, seth vidal wrote:
On Thu, 2006-08-10 at 08:08 +0300, Panu Matilainen wrote:
Now, I don't really want to take any sides in this, I just want the dang thing to be decided one way or the other so we can move on to refining our plugins, as plugins are required in both schemes to be correctly handled in all cases. From experience I know the unamer-in-name works quite nicely but has it quirks from user POV [1], OTOH the kmod scheme (currently used by livna) also seems to work just fine, at least it hasn't bitten me yet although I haven't used any plugins to handle it.
+1
I was not taking a side in it. I was just explaining about what has worked about the kmod scheme in the past.
Yup, that's the way at least I read it.
I'd just like one or the other, ultimately, please, for the love of all that's good and holy. :)
Amen! :)
- Panu -
Panu Matilainen schrieb:
On Thu, 10 Aug 2006, seth vidal wrote:
On Thu, 2006-08-10 at 08:08 +0300, Panu Matilainen wrote:
Now, I don't really want to take any sides in this, I just want the dang thing to be decided one way or the other so we can move on to refining our plugins, as plugins are required in both schemes to be correctly handled in all cases. From experience I know the unamer-in-name works quite nicely but has it quirks from user POV [1], OTOH the kmod scheme (currently used by livna) also seems to work just fine, at least it hasn't bitten me yet although I haven't used any plugins to handle it.
with my livna hat on: the Extras kmod standard we used in FC5 works a lot better out of the box (e.g. without plugins) than the old scheme where we had in FC[1-4] which had the "uname -r" in the name.
I'd just like one or the other, ultimately, please, for the love of all that's good and holy. :)
Amen! :)
+1
If we really want to fix the problem that was brought up in this discussion (the "a update might remove old kmods for older kernels") then we IMHO can do that with an adjusted variant of the current kmod-standard together with the kabi stuff and/or with a proper plugin. I currently prefer the solution with the kabi stuff because that solution will have other benefits as well. I can try to outline my thoughts/ideas later (maybe today, but probably on the weekend) in more detail.
BTW, do we really want to use the "uname -r" again in times where RHEL will have a kabi which means that the "uname -r" is mostly meaningless? I'd really prefer one solution for both FC and RHEL.
CU thl
On Thu, Aug 10, 2006 at 10:47:12AM +0200, Thorsten Leemhuis wrote:
together with the kabi stuff and/or with a proper plugin.
BTW, do we really want to use the "uname -r" again in times where RHEL will have a kabi which means that the "uname -r" is mostly meaningless? I'd really prefer one solution for both FC and RHEL.
Just to make sure everybody understands about kabi myths and doesn't have expectations that are known to not be fulfilled:
o There is never ever going to be an ordered api id like kabi = 1.0 o There is no intention of upstream having a stable api which is a prerequisite for a stable abi, in fact it is considered a feature (less `locate stable_api_nonsense.txt`) o The discussion about recycling old kernel modules using kabi tools means hashing subset information like kabihash(subsystem_abc_def) = 0ed3425d. o The full kabi will be a large set of unorderable hases as above o You don't know from a packging POV which hashes are relevant. And even if you knew you would have to concatenate the relevant parts into the kernel module package's filename o If you consider hashing the hashes into one master (sub-)"kabi" hash item you will - still need the single hashes in the package relations for the kabi recycling mechanism to work at all - end with an unorderable hash
At the very best you would replace uname-r with an unorderable hash which no user can associate to his kernel. E.g. with either scheme can you humanly identify that
foo-kmdl-0ed3425d-1.2.3-4.i686.rpm (or kmod-foo-1.2.3-4_0ed3425d)
can satisfy kabi relationships of
kernel-2.6.18-23.0.1.EL5.i686.rpm and kernel-2.6.18-23.0.2.EL5.i686.rpm
No, you can't. In a pure RHEL5 world with a really stable kabi (note on passing that this is *not* the case with RHEL4 and for example the wireless extensions update breaking both api and abi) you would end up with dropping the evr of the kernel, e.g. uname -r or kabi, to make any positive packaging use of the kabi scheme.
In the world of Fedora Core you will never have the same total kabi hash as the previous kernel, so in this case the discussion is moot anyway.
So then you may wonder, were's the kabi stuff going to help at all and why do people invest time to develop it further?
It will only help in RHEL5 and similar "enterprise" environments if the kernel packages really keep the set of kabi hashes invariant over the years (and the kabi tools will assist kernel engineering in detecting and avoiding such changes). The benefits lie with the IHV/ISVs, who will be able to invest once in a package (either open or closed source) and never have to rebuild it again for RHEL5 if engineering does its work well.
It's of no use to systems that don't firmly commit to a stable kabi/kapi like the upstream kernel itself and Fedora, though. And it will also encourage use of closed source drivers (see stable_api_nonsense.txt for why this is so).
On Wed, Aug 09, 2006 at 09:38:54AM -0400, Jack Neely wrote:
Okay...walk me through this then:
Assuming no yum plugins or other mess.
A new kernel is available that corrects some random remote DoS. How do I get all 1300 machines to pull down the new AFS modules?
It's in the wiki, but here it comes again:
o current kernel module scheme w/o any special depsolver handling:
- broken on rpm level, inherits on all depsolvers
- Modules of the current kernel get nuked whether you reboot into the new kernel or not
Wrong. Both up2date and yum have always marked packages that provide 'kernel-modules' as install only for several years now. Modules don't get "nuked" unless you rpm -U.
Wrong x 3:
o not always, neither yum, not up2date initially had any "kernel-module(s)" support o first implementation had a typo mismatch, kernel-modules vs kernel-module. In fact effectively its a very young approach, I think this was fixed less than a year ago o you asked for "no yum plugins or other mess". Requiring all kernel modules to be initially marked as "always coinstallable" has been proven to be a bug and therefore a "mess".
- Module upgrades within the same kernel get blocked due to file conflicts
- OR module upgrades within the same kernel get coinstalled and module-init-tools can flip a coin to find out what to choose
Upgrades within the same kernel don't work. This is one point, not multiple.
There is a big capitalized "OR" in the list. Which means that if the current /lib/module/<uname-r>/... file paths are kept you are lucky that it's only file conflicts (and "just" an aborted yum update). But if the non-overlapping kabi/whatever file path scheme is adopted you don't have file conflicts, but worse: the modules will get installed and depmod will play oracle on what module to chose.
So yes, it's one point with multiple failure paths all of which are ugly.
- but the new kernel gets its kernel modules (and only the new kernel ...)
This point has been used in practice by several large universities. I've been doing this for about 6 years. While not perfect its been proven to be acceptable and allow machines to remain fulled patched.
6 years? So you've been using yum's secret unannounced and NSA sponsored version back then, huh? ;)
NC State University. Duke. I believe Matt at Boston U. has used this approch in the past as well.
And I know large universities that extensively make use of proprietary operating systems, so what exactly does that say? Mass does not imply infallibility.
- Proper upgrades of kernel modules within a kernel
- new kernels don't get associated kmdls coinstalled
This is a big problem. Your scheme discurages people from keeping their systems updated. You'd rather have all of my 1300 machines run a swiss cheese kernel from 2003 for RHEL 3. [...] Its hard enough to get people to keep their kernels upgraded. Your scheme makes it next to impossible without manual touching of all machines. This is not acceptable.
You asked about "Assuming no yum plugins or other mess", see above. I never suggested using this scheme w/o any support for depsolvers, and I did submit a working yum plugin for this scheme, didn't I?
Don't twist my words, please.
The co-installable mark has been in production and tested for 3 years now I beleive.
Well, you need to decide, is it 6, 3 or perhaps less? ...
No. Yum/Up2date modifications would be required. Not optional. We must not discurage people from applying security errata.
Certainly not, then don't ask about academic exersizes on what the behaviour is w/o any special depsolver handling.
To remain on the subject since there are other depsolvers out there that know nothing yet about any kernel module scheme: The old scheme will nuke your running kernel modules, there is no way out, and currently for smart to not do so requires a special config entry *per module*!!!
Getting back to the real facts: We're interested in
a) supporting rpm cli b) supporting yum c) supporting up2date
In the old case a) is forever tainted, b) needs a yet uncomplete plugin (you never replied on the review on the requirements on recursive package removal your plugin would have to do) and c) is unclear to me.
In the new case a) is already done with, b) needs a plugin that is already finished and c) is unclear, too ;)
[Regarding c) I hope that up2date uses yum more and more, so perhaps at RHEL5 time yum plugins will fix this, too].
Nice to have are
a) support for apt b) support for smart
As said in the old scheme, where you need *both* to mark all packages as coinstallable *and* to undo case-wise this assumption, b) is a big looser when it comes to marking packages as coinstallable. And support for a) for kmdls is trivial and is even shipped with the source tarball (only need to replace "kernel-module-*" matching with "*-kmld-" matching) while for kmods you have the same issues as under yum w/o any plugin.
On Wed, 2006-08-09 at 17:13 +0200, Axel Thimm wrote:
On Wed, Aug 09, 2006 at 09:38:54AM -0400, Jack Neely wrote:
Okay...walk me through this then:
Assuming no yum plugins or other mess.
A new kernel is available that corrects some random remote DoS. How do I get all 1300 machines to pull down the new AFS modules?
It's in the wiki, but here it comes again:
o current kernel module scheme w/o any special depsolver handling:
- broken on rpm level, inherits on all depsolvers
- Modules of the current kernel get nuked whether you reboot into the new kernel or not
Wrong. Both up2date and yum have always marked packages that provide 'kernel-modules' as install only for several years now. Modules don't get "nuked" unless you rpm -U.
Wrong x 3:
o not always, neither yum, not up2date initially had any "kernel-module(s)" support o first implementation had a typo mismatch, kernel-modules vs kernel-module. In fact effectively its a very young approach, I think this was fixed less than a year ago
2003-11-21 01:24 skvidal
* nevral.py: make packages providing 'kernel-modules' installonly.
that was yum 2.0.X
- but the new kernel gets its kernel modules (and only the new kernel ...)
This point has been used in practice by several large universities. I've been doing this for about 6 years. While not perfect its been proven to be acceptable and allow machines to remain fulled patched.
6 years? So you've been using yum's secret unannounced and NSA sponsored version back then, huh? ;)
we used the idea in yup prior to yum. That was about 2000->2001, iirc so yes, about 6 years.
NC State University. Duke. I believe Matt at Boston U. has used
this
approch in the past as well.
And I know large universities that extensively make use of proprietary operating systems, so what exactly does that say? Mass does not imply infallibility.
I don't think he was alleging that. I think he was saying there are some big users with large installations who have used it and it works.
that's all.
-sv
On Wed, Aug 09, 2006 at 09:38:54AM -0400, Jack Neely wrote:
NC State University. Duke. I believe Matt at Boston U. has used this approch in the past as well.
My current approach is to build a subpackage containing kernel modules for the latest three kernels all bundled together. This is horrible but works fine in our environment. (And, works perfectly fine with buildrequires and makes repeatable builds without passing in special parameters on the build command line.)
(If we wanted to have both i586 and i686 kernels, there would be a problem, but we simply don't support i586.)
I was working on updating to the Fedora standard, but it's a lot of work and the incentive isn't high. :)
I am inclined to believe the only real solution here is to get the in-kernel AFS client up to snuff and abandon the OpenAFS kernel module. I don't know when this will happen, but I think it's easier than solving this problem. :)
On Wed, Aug 09, 2006 at 11:27:11AM -0400, Matthew Miller wrote:
On Wed, Aug 09, 2006 at 09:38:54AM -0400, Jack Neely wrote:
NC State University. Duke. I believe Matt at Boston U. has used this approch in the past as well.
My current approach is to build a subpackage containing kernel modules for the latest three kernels all bundled together. This is horrible but works fine in our environment. (And, works perfectly fine with buildrequires and makes repeatable builds without passing in special parameters on the build command line.)
doesn't this imply that the kernels are also effectively bundled?
(If we wanted to have both i586 and i686 kernels, there would be a problem, but we simply don't support i586.)
:/
I was working on updating to the Fedora standard, but it's a lot of work and the incentive isn't high. :)
I am inclined to believe the only real solution here is to get the in-kernel AFS client up to snuff and abandon the OpenAFS kernel module. I don't know when this will happen, but I think it's easier than solving this problem. :)
It looks like everyone is only (really) interested in openafs kernel modules. How about
http://atrpms.net/name/openafs/
These are openafs kmdls for FC3-FC5 and RHEL4 (FC6 is being worked on). Less than RH9 was abandoned and RHEL3 could be resurrected (there were some issues with x86_64) if there were anyone interested (but I'm running out of licenses and need them for RHEL5).
If you have spare boxes and time and like to test you're very welcome!
On Wed, Aug 09, 2006 at 05:45:42PM +0200, Axel Thimm wrote:
My current approach is to build a subpackage containing kernel modules for the latest three kernels all bundled together. This is horrible but works fine in our environment. (And, works perfectly fine with buildrequires and makes repeatable builds without passing in special parameters on the build command line.)
doesn't this imply that the kernels are also effectively bundled?
No. There's no dependency tracking going on. It only works because everyone gets updates.
On Wed, Aug 09, 2006 at 12:29:08AM +0300, Ville Skyttä wrote:
On Tue, 2006-08-08 at 14:04 -0500, Tom 'spot' Callaway wrote:
On Tue, 2006-08-08 at 11:26 +0200, Axel Thimm wrote:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdlsOne thing completely missing from that is debuginfo packages.
Debuginfo packages are handled in the kmdl scheme, I just didn't outline the details in the wiki to keep the document from growing larger than a packaging comittee member is expected to invest in reading.
On Wed, 2006-08-09 at 01:21 +0200, Axel Thimm wrote:
On Wed, Aug 09, 2006 at 12:29:08AM +0300, Ville Skyttä wrote:
On Tue, 2006-08-08 at 14:04 -0500, Tom 'spot' Callaway wrote:
On Tue, 2006-08-08 at 11:26 +0200, Axel Thimm wrote:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdlsOne thing completely missing from that is debuginfo packages.
Debuginfo packages are handled in the kmdl scheme, I just didn't outline the details in the wiki to keep the document from growing larger than a packaging comittee member is expected to invest in reading.
Without that information, folks who really want to review the suggestion now need to abort the review due to incompleteness, invest much more time than reading it would take in (re?)inventing how it could be done, or take things for granted.
As discussed previously in private mail, if you don't have time to document something, just upload a few example packages built for let's say two successive FC kernels (complete with SRPMS, macro definitions, binaries and debuginfos) somewhere and post a link to that somewhere in public.
On Wed, Aug 09, 2006 at 07:20:56PM +0300, Ville Skyttä wrote:
On Wed, 2006-08-09 at 01:21 +0200, Axel Thimm wrote:
On Wed, Aug 09, 2006 at 12:29:08AM +0300, Ville Skyttä wrote:
On Tue, 2006-08-08 at 14:04 -0500, Tom 'spot' Callaway wrote:
On Tue, 2006-08-08 at 11:26 +0200, Axel Thimm wrote:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdlsOne thing completely missing from that is debuginfo packages.
Debuginfo packages are handled in the kmdl scheme, I just didn't outline the details in the wiki to keep the document from growing larger than a packaging comittee member is expected to invest in reading.
Without that information, folks who really want to review the suggestion now need to abort the review due to incompleteness, invest much more time than reading it would take in (re?)inventing how it could be done, or take things for granted.
Until now I'm fighting the uname-r-in-name and one-specfile battles, I really don't want to distract people from these issues until they are done with.
To answer your question on debuginfos: You simply change the name of the debuginfo package to match that of the kmdl, e.g. one debuginfo package per kmdl package.
As discussed previously in private mail, if you don't have time to document something, just upload a few example packages built for let's say two successive FC kernels (complete with SRPMS, macro definitions, binaries and debuginfos) somewhere and post a link to that somewhere in public.
You mean like http://ATrpms.net/? The kmdl scheme as proposed is in production there for quite a while. Due to size constraints I don't keep too many kmdls for different kernels around. The currently supported kernels are:
fc5/2.6.17-1.2157_1.rhfc5.cubbi_suspend2 \ fc5/2.6.17-1.2157_0.99.rhfc5.cubbi_suspend2_8k \ fc4/2.6.17-1.2142_1.rhfc4.cubbi_swsusp2 \ fc4/2.6.17-1.2142_0.99.rhfc4.cubbi_swsusp2_8k \ el4/2.6.9-34.0.2.EL \ el3/2.4.21-47.EL \ fc6/2.6.17-1.2530.fc6 \ fc5/2.6.17-1.2157_FC5 \ fc4/2.6.17-1.2142_FC4 \ fc3/2.6.12-2.3.legacy_FC3 \ fc2/2.6.10-2.3.legacy_FC2 \ fc1/2.4.22-1.2199.8.legacy.nptl \ rh9/2.4.20-46.9.legacy \ rh7.3/2.4.20-46.7.legacy \
So if you want to compare kernel modules upgradablity you can either pick a module across two releases, or perhaps assume that the swsusp2 kernels are upgrades of the others (technically from rpm's POV they are, that's why they are not shipped in ATrpms "stable", but "testing").
On Wed, 2006-08-09 at 18:34 +0200, Axel Thimm wrote:
Until now I'm fighting the uname-r-in-name and one-specfile battles, I really don't want to distract people from these issues until they are done with.
debuginfos are intimately part of that "battle". It's not a distraction, it's required information.
To answer your question on debuginfos: You simply change the name of the debuginfo package
In the specfile? How? Examples? Surely you're not suggesting to just rename the actual debuginfo rpm file on disk after it has been built and be happy?
As discussed previously in private mail, if you don't have time to document something, just upload a few example packages built for let's say two successive FC kernels (complete with SRPMS, macro definitions, binaries and debuginfos) somewhere and post a link to that somewhere in public.
You mean like http://ATrpms.net/?
I have no idea where to find the macro definitions and debuginfo packages for your suggested scheme there.
Ville Skyttä wrote:
On Wed, 2006-08-09 at 18:34 +0200, Axel Thimm wrote:
Until now I'm fighting the uname-r-in-name and one-specfile battles, I really don't want to distract people from these issues until they are done with.
debuginfos are intimately part of that "battle". It's not a distraction, it's required information.
To answer your question on debuginfos: You simply change the name of the debuginfo package
In the specfile? How? Examples? Surely you're not suggesting to just rename the actual debuginfo rpm file on disk after it has been built and be happy?
I think he is, and stop calling him Surely. (:
-- Rex
On Wed, Aug 09, 2006 at 01:25:08PM -0500, Rex Dieter wrote:
Ville Skyttä wrote:
On Wed, 2006-08-09 at 18:34 +0200, Axel Thimm wrote:
Until now I'm fighting the uname-r-in-name and one-specfile battles, I really don't want to distract people from these issues until they are done with.
debuginfos are intimately part of that "battle". It's not a distraction, it's required information.
OK, if you feel that way I'm attaching the requested details below. I hope that this thread doesn't stall into some implementation detail issues.
It's rather trivial and really doesn't deserve the attention it receives, but at least I hope that it makes clear that all implementation details have been carefully thought of already or matured in the last three years.
So, do you buy it now? :)
To answer your question on debuginfos: You simply change the name of the debuginfo package
In the specfile? How? Examples?
No, remember that the kmdl scheme is a clean and KISS interface/ implementation design. Furthermore debuginfos are hidden as an implementation detail in conventional packages, too. So starting to add such information to the specfile or the kmdl interface part would be wrong or a hack. It belongs to the implementation part, aka macro definitions.
Surely you're not suggesting to just rename the actual debuginfo rpm file on disk after it has been built and be happy?
I think he is, and stop calling him Surely. (:
Surely is my middle name, but I prefer Axel ;)
In fact renaming after the fact would be dirty and would leave traces in the resulting debuginfo package (the internal rpm name would still be the one before renaming). I prefer modding the macros. It's a simple change, but since I don't want to touch the upstream macros (well, I do want to, but rather in the source itself ;), I override the macro with a local copy. No, let's not get distracted about what I have in mind about future redhat-rpm-config contents ...
In order to avoid any further inquiries on this rather trivial issue I even diffed the original macro to the one I use to reveal all the magic at once (note again: the macro is overridden, not replaced in the original definition, this is a convenience diff).
%kmdl_userland is always 1 unless the package to be built is a kernel specific kernel module (the default for %kmdl_userland is an implementation detail, for FE's buildsystem it would probably be the other way around). So project foo ends up with debuginfos naturally named as
foo-debuginfo-1.2.3 foo-kmdl-2.6.20-4.5.6-debuginfo-1.2.3 foo-kmdl-2.6.20-4.5.6smp-debuginfo-1.2.3 foo-kmdl-2.6.20-4.5.6PAE-debuginfo-1.2.3 [...]
| +## Note^2: A modified copy to reflect different debuginfo packages for kmdls | +%debuginfoname %(test "%{kmdl_userland}" = 1 && echo debuginfo || echo "-n %{kmdl_name}-debuginfo") | | # Template for debug information sub-package. | # NOTE: This is a copy from rpm to get the ifnarch noarch fix, it can be removed later | %debug_package \ | %ifnarch noarch\ | %global __debug_package 1\ | -%package debuginfo \ | +%package %{debuginfoname} \ | Summary: Debug information for package %{name}\ | Group: Development/Debug\ | -%description debuginfo\ | +%description %{debuginfoname}\ | This package provides debug information for package %{name}.\ | Debug information is useful when developing applications that use this\ | package or when debugging this package.\ | -%files debuginfo -f debugfiles.list\ | +%files %{debuginfoname} -f debugfiles.list\ | %defattr(-,root,root)\ | %endif\ | %{nil}
On Tue, Aug 08, 2006 at 02:04:24PM -0500, Tom 'spot' Callaway wrote:
On Tue, 2006-08-08 at 11:26 +0200, Axel Thimm wrote:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"):
http://fedoraproject.org/wiki/AxelThimm/kmdls
Thank you for taking the time to do this. I honestly think it will be helpful for the Packaging Committee to have this information in front of them.
Rather than trying to replace kmod with kmdl, I'd rather look at the key changes that we should consider making.
The biggest one, IMHO, is overloading name with the kernel version. I've been one of the staunchest opponents of doing this, because I think its ugly, a hack, and causes problems.
With all that said: I now think it is necessary for kernel module packages. I did a lot of thinking and reading over the last several days, and overloading the name works. We know it works, whether done with rpm by hand or via depsolvers (yum).
I'm glad I'm getting things rolling finally :)
But please consider the following: Changing some key elements like uname-r-in-name and one-specfile you inevitably end very near to kmdl scheme. If these idioms are decided upon to be used in the future it would be a pity to create yet-another-standard if the one existing (kmdl) really covers everything we wish.
It's also a scheme in use by ATrpms since quite some time and a vehicle for me to push some kernel modules to FE (provided the legal/quality issues are covered).
That having said if something needs improvement in the kmdl scheme I'm open to suggestions. But it would really be a pity to come so close and still have a forked scheme.
Tom 'spot' Callaway schrieb:
On Tue, 2006-08-08 at 11:26 +0200, Axel Thimm wrote:
I've created a wiki page outlining the kmdl design as well as showing the flaws of the current kernel module scheme ("kmod"): http://fedoraproject.org/wiki/AxelThimm/kmdls
[...]
So, what problems does it cause to overload the name?
- cvs: No changes necessary. CVS keys off SRPM name, which remains
foo-kmod.
- buildsystem: The buildsystem needs to treat kernel-module packages
differently, but we've got the buildsystem code authors on board to help fit the buildsystem to our standards (within reason). Either way, the buildsystem has to detect kernel modules and build them specially, so this is just a different color of paint. Plus, Axel has volunteered to help with this.
- bugzilla: Bugzilla pulls from owners.list, which bases off SRPM/CVS,
so we're fine here.
- rpm queries: rpm -q kmod-foo doesn't return anything? Say what? Ehh.
If you're a power user enough to be querying with rpm on the commandline, you're geek enough to rpm -qa |grep kmod-foo and find it.
The most important thing didn't come up in this discussion yet (or I overlooked it):
5. None of the depsolvers will install new kernel-modules for newly installed kernels by default. All need a special plugin that handles that.
That's no problem with the current kmod standard.
CU thl
On Wed, Aug 09, 2006 at 06:06:04AM +0200, Thorsten Leemhuis wrote:
The most important thing didn't come up in this discussion yet (or I overlooked it):
- None of the depsolvers will install new kernel-modules for newly
installed kernels by default. All need a special plugin that handles that.
which was already submitted
That's no problem with the current kmod standard.
but this needs a plugin/patch even more to avoid nuking of kernel modules of the running kernel or depsolvers bailing out due to file conflicts or partial kernel module overwrites or non-wanted coinstalltion per kernel.
See the wiki and my reply to Jack.
packaging@lists.fedoraproject.org
-
Axel Thimm -
Jack Neely -
Matthew Miller -
Panu Matilainen -
Rex Dieter -
seth vidal -
Thorsten Leemhuis -
Tom Callaway -
Ville Skyttä