Does Core have an accurate count of packages which are built against libraries from firefox?
Since the last firefox update I have been trying to trackdown ALL the packages which are affected by the change in location of libraries that come in the versioned firefox. The problem here is, is that the autogenerated requires on libraries doesn't catch the fact that the library has moved, so you won't actually get any depchain errors from rpm. This makes its pretty tough to track all the affected packages down as an end-user without installing the world.
Does anyone anywhere have a complete list of affected packages, or is clever enough to generate such a list programmatically that doesn't involved installing all of fedora space? In Core I already know about gnome-python2-extras because it affected applications that I use.
But I just ran into the fact that esc was also affect by the firefox update by accident. /usr/lib/esc-1.0.0/xulrunner/xulrunner-bin links against libmozjs.so libxpcom.so which can't be found by the linker since these are both packages in the versioned firefox directory tree. My god damn home workstation died yesterday(merry christmas to me!) so if someone else can bugzilla this before I get a chance...go right ahead.
The main point of this post however is that I don't think we are doing a good job of coordinating maintainers so they can respond as quickly as possible when a firefox update is pushed so they can rebuild their packages. We've had a lovely little chat in #fedora-extras with the firefox maintainer, and he's absolutely right there needs to be a toolized way for each maintainer with a package who builds against firefox to set a sort of watch in the build or update system against firefox looking for new builds.
So what exactly can we do, in the form of policy and in the form toolized automation which can help prevent or mitigate an unreasonable amount of lag between when a firefox package is built and when maintainers finally notice that other packages need to be rebuilt? Are there additional review steps we can do to check for this condition at the time of package review? The Core packages have to be reviewed before the merger right? I'd like to see if we can bury this issue as part of those reviews. Should everyone building against firefox libraries be using versioned firefox requires? Can we add an rpmlint check for this?
-jef"word to the wise, do not eat expresso chocolate mousse cake at 9 PM if you want to go to sleep before 2 AM"spaleta
On 12/26/06, Jeff Spaleta jspaleta@gmail.com wrote:
We've had a lovely little chat in #fedora-extras with the firefox maintainer, and he's absolutely right there needs to be a toolized way for each maintainer with a package who builds against firefox to set a sort of watch in the build or update system against firefox looking for new builds.
Actually, I think such a tool would be useful also for others: for example, in order to know when a new kernel is released and I need to rebuild the kmod I am maintaining I can: 1. watch each and every Build report (rawhide), test and/or pkg-announce lists (FC5 and FC6) 2. wait for the Broken deps script to wake me.
I am trying to use the RSS infofeeds but right now the solution is still suboptimal.
"Gianluca Sforna" giallu@gmail.com writes:
On 12/26/06, Jeff Spaleta jspaleta@gmail.com wrote:
We've had a lovely little chat in #fedora-extras with the firefox maintainer, and he's absolutely right there needs to be a toolized way for each maintainer with a package who builds against firefox to set a sort of watch in the build or update system against firefox looking for new builds.
Actually, I think such a tool would be useful also for others:
+1 --- although I suspect Jeff meant that already, ie, he was just using firefox as an example of a package with many dependent packages. I'd like to see an automatic notification mechanism that "package FOO that your package BAR Requires: or BuildRequires: was just updated".
To make life interesting: sometimes a package rebuild is for trivial reasons. If there were such a notification mechanism, then ideally the owner of the package-having-dependencies should be able to mark a particular rebuild as "all hands on deck!" or "take a look but probably no problemo" or "fear not, nothing interesting here".
regards, tom lane
On 12/26/06, Tom Lane tgl@redhat.com wrote:
+1 --- although I suspect Jeff meant that already, ie, he was just using firefox as an example of a package with many dependent packages.
Its more than that, the firefox situation is extraordinary because of the way the dependencies are handled. Something needs to be done, which goes above and beyond the available dependency chain checking script mechanisms that we currently have to catch what is happening with broken packages with firefox library deps. If whatever that new mechanism is, is a generally welcomed tool which all maintainers can benefit from to help manage dependency coordination issues, great.
But I want to be clear, this is not a navel gazing exercise, where I want to dream up the ideal notification tools meant to solve all maintainer coordination problems and cure cancer. I am looking to solve a very specific problem. We need a way to prevent undue delay in correcting packages which are broken by firefox rebuilds (or other packages) because of changes in the versioned directory tree location which are currently undiscoverable from an rpm based dependency checking perspective. We have a situation right now where linked in libraries are going missing on installed system and our available rpm based depchecking scripts are not catching it. I've caught two CORE packages now which are broken on my system in this way, as an end-user. We have to do better, we need to find a way to detect this broken dependancy chain situation asap in the published tree so the community of maintainers can react without waiting for bugreports from endusers which could come days or weeks after the breakage actually occured. I'd be happy if there was a scriptable way to detect this and add it to the summary of broken dep reports that go out to the lists.
-jef
On Tue, 2006-12-26 at 20:30 -0900, Jeff Spaleta wrote:
On 12/26/06, Tom Lane tgl@redhat.com wrote:
+1 --- although I suspect Jeff meant that already, ie, he was just using firefox as an example of a package with many dependent packages.
Its more than that, the firefox situation is extraordinary because of the way the dependencies are handled. Something needs to be done, which goes above and beyond the available dependency chain checking script mechanisms that we currently have to catch what is happening with broken packages with firefox library deps. If whatever that new mechanism is, is a generally welcomed tool which all maintainers can benefit from to help manage dependency coordination issues, great.
But I want to be clear, this is not a navel gazing exercise, where I want to dream up the ideal notification tools meant to solve all maintainer coordination problems and cure cancer. I am looking to solve a very specific problem. We need a way to prevent undue delay in correcting packages which are broken by firefox rebuilds (or other packages) because of changes in the versioned directory tree location which are currently undiscoverable from an rpm based dependency checking perspective. We have a situation right now where linked in libraries are going missing on installed system and our available rpm based depchecking scripts are not catching it. I've caught two CORE packages now which are broken on my system in this way, as an end-user. We have to do better, we need to find a way to detect this broken dependancy chain situation asap in the published tree so the community of maintainers can react without waiting for bugreports from endusers which could come days or weeks after the breakage actually occured. I'd be happy if there was a scriptable way to detect this and add it to the summary of broken dep reports that go out to the lists.
Isn't this a function of the package database some of the infrastructure folks are working on? Or, alternatively, couldn't we have it tracked from the srpm builddep relationships?
here's what it sounds like to me:
1. package X gets rebuilt/updated 2. package Y has a dep/build dep relationship with package X 3. package owner of package Y gets a notice that package Y has been rebuilt/updated.
none of the above is really all that hard, you'd just have to iterate the packages looking for ones that have any dep relationship with any of the packages that have been updated/rebuilt.
It'd probably be best if it could be a script that we ran automatically and one that a package maintainer could run to report on any of his/her packages.
-sv
seth vidal skvidal@linux.duke.edu writes:
On Tue, 2006-12-26 at 20:30 -0900, Jeff Spaleta wrote:
On 12/26/06, Tom Lane tgl@redhat.com wrote:
+1 --- although I suspect Jeff meant that already, ie, he was just using firefox as an example of a package with many dependent packages.
Its more than that, the firefox situation is extraordinary because of the way the dependencies are handled. Something needs to be done,
Isn't this a function of the package database some of the infrastructure folks are working on? Or, alternatively, couldn't we have it tracked from the srpm builddep relationships?
well, Jeff makes it perfectly clear that he thinks firefox has a special problem. I'm still in favor of some kind of generic solution, because I know I've broken a few downstream packages in my time :-( ... but perhaps firefox needs some extra mechanism.
regards, tom lane
On Wed, 2006-12-27 at 00:52 -0500, seth vidal wrote:
On Tue, 2006-12-26 at 20:30 -0900, Jeff Spaleta wrote:
Its more than that, the firefox situation is extraordinary because of the way the dependencies are handled. Something needs to be done, which goes above and beyond the available dependency chain checking script mechanisms that we currently have to catch what is happening with broken packages with firefox library deps. If whatever that new mechanism is, is a generally welcomed tool which all maintainers can benefit from to help manage dependency coordination issues, great.
But I want to be clear, this is not a navel gazing exercise, where I want to dream up the ideal notification tools meant to solve all maintainer coordination problems and cure cancer. I am looking to solve a very specific problem. We need a way to prevent undue delay in correcting packages which are broken by firefox rebuilds (or other packages) because of changes in the versioned directory tree location which are currently undiscoverable from an rpm based dependency checking perspective. We have a situation right now where linked in libraries are going missing on installed system and our available rpm based depchecking scripts are not catching it. I've caught two CORE packages now which are broken on my system in this way, as an end-user. We have to do better, we need to find a way to detect this broken dependancy chain situation asap in the published tree so the community of maintainers can react without waiting for bugreports from endusers which could come days or weeks after the breakage actually occured. I'd be happy if there was a scriptable way to detect this and add it to the summary of broken dep reports that go out to the lists.
Isn't this a function of the package database some of the infrastructure folks are working on? Or, alternatively, couldn't we have it tracked from the srpm builddep relationships?
Not for the first iteration[1]. I think we should be able to pull the initial data from the repodata. After that we'd want our ties into the buildsystem to alert us to each rebuild. The notification can trigger a dep checking run against that package and notification to all the dependent package maintainers.
here's what it sounds like to me:
- package X gets rebuilt/updated
- package Y has a dep/build dep relationship with package X
- package owner of package Y gets a notice that package Y has been
rebuilt/updated.
none of the above is really all that hard, you'd just have to iterate the packages looking for ones that have any dep relationship with any of the packages that have been updated/rebuilt.
Sounds right.
[1] Note -- I don't know how many revisions we'll get to before FC7. The essential functions that I want to definitely see are:
* Replacement for owners.list * Able to configure ACLs in the VCS * Able to branch packages in the VCS * Web front end for both read-only, anonymous access and read-write, package owner access.
-Toshio
On Tue, 2006-12-26 at 20:30 -0900, Jeff Spaleta wrote:
But I want to be clear, this is not a navel gazing exercise, where I want to dream up the ideal notification tools meant to solve all maintainer coordination problems and cure cancer. I am looking to solve a very specific problem. We need a way to prevent undue delay in correcting packages which are broken by firefox rebuilds (or other packages) because of changes in the versioned directory tree location which are currently undiscoverable from an rpm based dependency checking perspective. We have a situation right now where linked in libraries are going missing on installed system and our available rpm based depchecking scripts are not catching it.
Seems to me FireFox's packaging is fundamentally broken if RPM dep tracking can't detect this already. Some judicious use of virtual provides and dep-filtering maybe? Since the packages are apparently dependent on a versioned directory, maybe they all need to just Require: firefox = 1.5.foo.bar ...
On 12/27/06, Callum Lerwick seg@haxxed.com wrote:
firefox = 1.5.foo.bar ...
I'm thinking this is the best way to prevent this situation from going unnoticed by the currently available rpm dep checking. It's a policy fix. Make an explicit check during the review process to ensure that a versioned firefox requires is used if and only if libraries from inside the versioned firefox directory tree are pulled in as requires. We may want to think about this as part of Core package review for the upcoming merger.
Is anyone got a few spare cycles to script a Core+Extras wide check looking for all packages which end up requiring shared libraries required by firefox? My frelling home workstation died, so I don't have my normal development environment at my fingertips. I'm thinking we can brute-force this with some repetitive calls of repoquery --whatrequires using the output rpm -q --provides firefox |grep .so
Doing some simple exploratory surgery with rpm --provides and --whatrequires I noticed that esc doesn't actually require libxpcom.so. I took a closer look at the esc in Core 6 and the esc situation is even more confusing than I originally thought.
esc-1.0.0-16.fc6 actually includes its own copy of /usr/lib/esc-1.0.0/xulrunner/libxpcom.so but ldd /usr/lib/esc-1.0.0/xulrunner/xulrunner-bin returns: libmozjs.so => not found libxpcom.so => not found libxul.so => not found
I'd be interested to know if esc in the development tree as a similiar problem.
All three of these libraries are provided by firefox AND by esc itself. WTF. esc's specfile must completely short-circuit rpm's normal auto-generation of library dependancies, and according to the changelog entries it does short circuit the process. Whereever esc's xulrunner-bin is looking for libxpcom.so its not from its own xulrunner library directory nor from any firefox versioned directory of any firefox rpm released for fc6 afaict. I'm gonna have to try to file this in bugzilla tomorrow. Something tells me I'm going to need a couple of ultra-strong eye-poking sticks. I might even have to buy the white-hot heated upgrade this time.
-jef"Is it okay if I make MST3K-like commentary on some of the more interesting Core package reviews when they start happening. The esc review is already on my short-list. I want to understand wtf is so wrong with esc's build process that automatic library dependency calculations had to be explictly turned off. There is absolutely no way to catch that xulrunner-bin is missing library references at the packaging level as it stands. I had to blunder into this looking for other problems on an installed system. I'm coming to this review with a large tub of popcorn and bat wielding sock puppets."spaleta
Hi all!
BTW, I added "the firefox updates in stable often breaks quite some packages (galeon, liferia, ...); do we need some kind of task force that steps up to rebuild the stuff? Same for kmods!" to the FESCo schedule some days ago already ;-)
Gianluca Sforna schrieb:
On 12/26/06, Jeff Spaleta jspaleta@gmail.com wrote:
We've had a lovely little chat in #fedora-extras with the firefox maintainer, and he's absolutely right there needs to be a toolized way for each maintainer with a package who builds against firefox to set a sort of watch in the build or update system against firefox looking for new builds.
Well, maybe that could work. But I think for the merged Core and Extras world the maintainer of a update for now should be responsible of poking the downstream packagers and/or even rebuild the packages and push them at the same time.
Sure, a technical solution would be best, but I want a solution now until we have a technical one in place.
I'd suggest: For a stable distro (like FC6 in this case) we normally should not update stuff that breaks other stuff in the repo in any case. If we have to (firefox) then it's up to the maintainer of that package and his co-maintainers to make sure the updates works smoothly for most of the users. That means: make sure other stuff in the repos (galeon, liferia,...) depending on that package still works. That would normal mean: just rebuild it and poke the signers afterwards to push the rebuild stuff. That should be able to script and should not take more then 5 minutes of work in total.
Sure, that would mean a bit more work for the firefox maintainer in this case (sorry callion, but that's due to firefox fault as it does not provide a stable ABI/API), but it's the best for all afaics as firefox updates are often security relevant and that's the quickest and smoothest way to get it out for everyone. And btw, sure, if a downstream package (galeon, liferia,...) breaks with the new firefox, then their maintainers should fix it and it should not delay the push of the new firefox.
That will get a bit easier in the merged Core and Extras world. For now (FC5 and FC6) I'd say: Maintain a list somewhere in the wiki with the names of packages that need a rebuild when a new firefox is published. If a new firefox was pushed by Core let someone (callion, me, f13, the signers, FESCo members, ...) increase EVR on these packages and queue a rebuild.
How does that sound?
Actually, I think such a tool would be useful also for others: for example, in order to know when a new kernel is released and I need to rebuild the kmod I am maintaining I can:
The plan for kmods always was to let the buildsys handle that automatically. Just no one got around to implement it. Any volunteers?
CU thl
Jeff Spaleta wrote:
But I just ran into the fact that esc was also affect by the firefox update by accident. /usr/lib/esc-1.0.0/xulrunner/xulrunner-bin links against libmozjs.so libxpcom.so which can't be found by the linker since these are both packages in the versioned firefox directory tree. My god damn home workstation died yesterday(merry christmas to me!) so if someone else can bugzilla this before I get a chance...go right ahead.
This is a serious bug. What the hell is ESC doing packaging an entire copy of every gecko library and shipping with an entire gecko source tree if it's going to rely on the system installed copies? This aside from the fact that ESC using gecko at all is a calamity of an engineering decision, IMO.
-
Callum Lerwick -
Christopher Aillon -
Gianluca Sforna -
Jef Spaleta -
seth vidal -
Thorsten Leemhuis -
Tom Lane -
Toshio Kuratomi