Dependencies on Bodhi Updates

As requested during the FESCo meeting, I am going to try to summarize some of the issues inherent in the way that Bodhi updates currently work. First, I'll try to explain the goals and constraints: 1) The stable 'fedora-updates' yum repository should NEVER exist in a state where any package has dependency issues. In other words, it should never be possible for an update to be pushed to stable that breaks cleanly updating any other package. 2) Updates must be possible and (ideally) timely. This is probably self-evident. 3) Packages pushed to the stable 'fedora-updates' yum repository should (ideally) not introduce regressions in packages that depend on them. 4) New features in "superpackages" such as Firefox, GNOME or FreeIPA that have many and varied dependencies may require new features in packages they depend on in order to enhance or fix the superpackage. In the trivial example, a package (let's say libtalloc) needs to make an update to fix a bug. This package requires nothing new from its dependencies and is a self-contained fix. For this example, it is simple to just build libtalloc in koji and then create a Bodhi update and pass it through "updates-testing", get karma and *poof* off to "fedora-updates". Now let's extend the example. Suppose that we have another package libtevent that has libtalloc as a dependency. Libtevent's maintainer wants to add a new feature to libtevent, but the patch from upstream depends on the bug in libtalloc having been fixed in order for the new feature to work properly. In this situation, the maintainer of libtevent would build libtevent with an explicit Requires: libtalloc >= <version> in the specfile (possibly pulling libtalloc into the BuildRoot overrides if necessary) and then test it locally to see that it works. So now we have our first updates dependency issue. If we submit libtevent as its own update, it is possible that it will achieve its karma requirement before libtalloc does. It would then be pushed to the "fedora-updates" repository and then introduce a dependency issue in the stable repository (because users trying to update libtevent would be unable to update libtalloc without enabling the updates-testing repository). The current recommended approach is to bundle the two updates into a single one carrying multiple packages. The first problem with this is that you must have commit privilege on all packages that you are bundling into an update. If you do not, then you need to track down a provenpackager to do it for you. Now let's make the problem even more fun. Consider that the update to libtevent might be coming in because it is necessary for a new feature in libldb, which is in turn providing new functionality necessary for SSSD. So now we have four packages all sitting in the same update. The problem with this is that the tendency will be to only test the most user-visible package(s) in the set. In this particular case, that might be SSSD. So people would likely test SSSD and, if nothing went wrong, consider the entire update stable. But wait! SSSD isn't the only package that depends on libldb, libtevent and libtalloc. So too does the samba package. Suppose that the bugfix in libtalloc, after resolving the original issue, results in exposing another more serious bug in samba? Now we need to pull a samba update into this same update series. A contrived example, you say? That would never happen, bugfixes aren't likely to do that. Well, for one example: https://admin.fedoraproject.org/updates/FEDORA-2011-11845 In this particular example, we knew up-front that it was going to necessitate a rebuild of several dependent packages and we coordinated a single release to address them. So in this case, the proper approach was to bundle them together in a single update. This worked because we specifically knew that the libtevent change was going to break other packages. But what about when we don't know that? Let's take another example: https://admin.fedoraproject.org/updates/FEDORA-2011-17399 In this case, there was a security bug reported against Firefox. Such things are serious, and acted on quickly. However, the bug was actually fixed in the nss package, and Firefox, Xulrunner and friends were rebuilt against that nss package. The problem was this: the fix made to the nss package introduced regressions in every other package that depended on it. However, because the default install of Firefox contained no issues, it rapidly received the necessary karma points and the whole update was pushed to stable. It then broke nearly every application in Fedora that relied on cryptography. The problem here was sociological, not technological. The only package that received testing was Firefox. It's hard to say without evidence whether the problem would have been averted by having nss go through its own update, but I strongly suspect that what we would have seen was greater testing on actual nss features for that specific update. Of course, we now have the same potential for an issue that I described above: If we had separate updates for nss and for Firefox, chances would be highly-likely that Firefox would be pushed to stable via karma points rapidly, whereas nss (which requires much more careful testing) might be left behind in updates-testing. So I really see two options for improving these situations: 1) https://fedorahosted.org/bodhi/ticket/663 I opened this ticket two months ago (to silence). The idea would be to add the ability for bodhi updates to mark other updates as a dependency, so that in the example above, Firefox could have been marked as ready for stable, but not pushed until the nss update was also marked as ready for stable. This to me seems like the best long-term solution. I'd also like to mention that Ubuntu's Launchpad system has this capability. 2) We could continue on the "single update for multiple packages" approach, but revamp the karma system so that each SRPM gets its own karma, rather than the update as a whole. Then, the whole update would not be pushed via autokarma until all of the dependent packages had sufficient karma (or the owner of the update could push them after the stable wait period, of course). Discuss.