Hi, Imagine situation that somebody is working on KDE rebase and me on libgit2 rebase. Both involve rebuilding/updating some package, let's say kf5-ktexteditor. We both work in different side tags, in KDE rebase kf5-ktexteditor gets updated to a new version. In libgit2 rebase, old version gets rebuilt.

Do you think that scales?

On Sat, Jun 8, 2019 at 12:54 PM Nicolas Mailhot via devel <devel(a)lists.fedoraproject.org&gt; wrote: > > Le samedi 08 juin 2019 à 11:23 +0200, Igor Gnatenko a écrit : > > Hi, > > > > Imagine situation that somebody is working on KDE rebase and me on > > libgit2 rebase. Both involve rebuilding/updating some package, let's > > say kf5-ktexteditor. > > > > We both work in different side tags, in KDE rebase kf5-ktexteditor > > gets updated to a new version. In libgit2 rebase, old version gets > > rebuilt. > > […] > > > Do you think that scales? > > But, what is different here from the Fedora circles / Fedora modules / > etc endeavours? Isn’t the root problem synchronizing common code paths, > because free software means pervasive code reuse, apps ends up being > deployed together, and un-sharing generates collisions / API > incompatibilities / behaviour incompatibilities / config file > incompatibilities / un-adressed security issues? > > What makes it possible in modules but not in side tags? I never said that it is better / possible in modules. I just wanted to point out that expecting that people will do double, triple, … work in side tags is not something what I would like to see (see thread about libgit 0.28.x update). Also rawhide gating just makes this problem even worse.

On Sat, Jun 8, 2019 at 2:10 PM Igor Gnatenko <ignatenkobrain(a)fedoraproject.org&gt; wrote: > > On Sat, Jun 8, 2019 at 12:54 PM Nicolas Mailhot via devel > <devel(a)lists.fedoraproject.org&gt; wrote: > > > > Le samedi 08 juin 2019 à 11:23 +0200, Igor Gnatenko a écrit : > > > Hi, > > > > > > Imagine situation that somebody is working on KDE rebase and me on > > > libgit2 rebase. Both involve rebuilding/updating some package, let's > > > say kf5-ktexteditor. > > > > > > We both work in different side tags, in KDE rebase kf5-ktexteditor > > > gets updated to a new version. In libgit2 rebase, old version gets > > > rebuilt. > > > > […] > > > > > Do you think that scales? > > > > But, what is different here from the Fedora circles / Fedora modules / > > etc endeavours? Isn’t the root problem synchronizing common code paths, > > because free software means pervasive code reuse, apps ends up being > > deployed together, and un-sharing generates collisions / API > > incompatibilities / behaviour incompatibilities / config file > > incompatibilities / un-adressed security issues? > > > > What makes it possible in modules but not in side tags? > > I never said that it is better / possible in modules. I just wanted to > point out that expecting that people will do double, triple, … work in > side tags is not something what I would like to see (see thread about > libgit 0.28.x update). Also rawhide gating just makes this problem > even worse. Would it be possible to block builds of packages for certain target tags? Like a "mutex for packages"? I'm thinking about specifying a list of packages at side-tag creation time, and then builds of these packages targeting any other tag could be blocked by koji, until the side "blocking" tag is merged back.

From: "Kevin Kofler" <kevin.kofler(a)chello.at&gt; To: devel(a)lists.fedoraproject.org Sent: Sunday, June 9, 2019 12:20:34 AM Subject: Re: RFC: Multiple parallel side tags Nicolas Mailhot via devel wrote: > But, what is different here from the Fedora circles / Fedora modules / > etc endeavours? Isn’t the root problem synchronizing common code paths, > because free software means pervasive code reuse, apps ends up being > deployed together, and un-sharing generates collisions / API > incompatibilities / behaviour incompatibilities / config file > incompatibilities / un-adressed security issues? > > What makes it possible in modules but not in side tags? Modules are just as broken. We really need to go back to just doing all the work directly in the one and only Rawhide, no modules, side tags, or other such "features" that are just conflicts waiting to happen. Yes, this means Rawhide will go back to being broken for people trying to actually use it while a transition is ongoing, but the point of Rawhide is to do development in it, NOT to be used. It should be fully expected that Rawhide does NOT work as an actual distro, ever. Its one and only purpose should be to coordinate the package development so it all happens in one place.

Martin Kolman wrote: > But even for package development and integration you need something that > works at least a bit. If you won't get a compose for a few weeks due to > the constant breakage you won't get much work done on landing your latest > library rebase or integrating a new package. In the past, the compose would not fail just because a release-blocking deliverable failed to compose, the package compose would just ship without the deliverables that failed to compose. That's what needs to happen again.

There is no technical reason why an error composing a live ISO prevents the changed RPMs from going out. Rawhide is not a release, there should be time until the actual release to fix the release-blocking deliverable.

And ideally, we would keep the last successful compose of each deliverable on the mirrors, no matter whether it matches the current package compose or not. (We did not do that in the past, the ISO was just missing when the latest version failed to compose, which is suboptimal.) We need to stop having unreasonable expectations about Rawhide's quality, and ISOs always composing successfully is one such unreasonable expectation.

Kevin Fenzi wrote: > I disagree. I think we need gating to block as much stuff that breaks > things from landing as we can and then we should find that keeping > composes going is much easier on all of us. Then things can be fixed > when gating catches them and it's on the person who broke things. And that is going to make development completely cringe to a halt. It is the nature of a distribution branch under development that things will sometimes be completely broken for a couple weeks. There needs to be a place to do development that can cause such temporary breakage.

The side tag approach already does not scale, as evidenced by this thread.

On Mon, Jun 17, 2019 at 8:53 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote: > > On 6/17/19 4:47 PM, Kevin Kofler wrote: > > Kevin Fenzi wrote: > >> I disagree. I think we need gating to block as much stuff that breaks > >> things from landing as we can and then we should find that keeping > >> composes going is much easier on all of us. Then things can be fixed > >> when gating catches them and it's on the person who broke things. > > > > And that is going to make development completely cringe to a halt. It is the > > nature of a distribution branch under development that things will sometimes > > be completely broken for a couple weeks. There needs to be a place to do > > development that can cause such temporary breakage. > > I again completely disagree. There is no reason for weeks of breakage. > Most of the issues that break composes are unannounced abi bumps where > just rebuilding dependent packages fixes it. Or broken deps (likewise). > Or mistakes made in kickstarts/comps. Or something that doesn't even > run. What good does having everyone broken for weeks do? > And that comes down to people shouldn't need to have to think about these things when working in Rawhide. While I don't disagree that Rawhide should be usable, I fundamentally disagree with making it harder for people to put things in Rawhide. We should be developing our tooling to make it _easier_ for stuff to go into Rawhide, and have Rawhide fix itself when the issues are relatively trivial to fix (such as reverse dependency rebuilding).

> > The side tag approach already does not scale, as evidenced by this thread. > > It does. You just need to communicate with others working in the same > area, IMHO. I don't think we need some technical thing for something > that happens rarely and can be solved by more communication. > Side tags will happen a lot more often because the tooling is pushing us to do it that way. Don't discount the potential for future insanity. I'm still not sure side-tags are enough. Could we have a concept of a "scratch side tag"? Something like a scratch build, but contains a collection of builds and creates an overlay repo that can be used to run checks on for auto-merging? If they're good, then it would get auto-built properly into the main rawhide tag (or even stable tag!).

Hi, On Tue, Jun 18, 2019 at 3:05 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote: > > On Mon, Jun 17, 2019 at 8:53 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote: > > > > On 6/17/19 4:47 PM, Kevin Kofler wrote: > > > Kevin Fenzi wrote: > > >> I disagree. I think we need gating to block as much stuff that breaks > > >> things from landing as we can and then we should find that keeping > > >> composes going is much easier on all of us. Then things can be fixed > > >> when gating catches them and it's on the person who broke things. > > > > > > And that is going to make development completely cringe to a halt. It is the > > > nature of a distribution branch under development that things will sometimes > > > be completely broken for a couple weeks. There needs to be a place to do > > > development that can cause such temporary breakage. > > > > I again completely disagree. There is no reason for weeks of breakage. > > Most of the issues that break composes are unannounced abi bumps where > > just rebuilding dependent packages fixes it. Or broken deps (likewise). > > Or mistakes made in kickstarts/comps. Or something that doesn't even > > run. What good does having everyone broken for weeks do? > > > > And that comes down to people shouldn't need to have to think about > these things when working in Rawhide. While I don't disagree that > Rawhide should be usable, I fundamentally disagree with making it > harder for people to put things in Rawhide. We should be developing > our tooling to make it _easier_ for stuff to go into Rawhide, and have > Rawhide fix itself when the issues are relatively trivial to fix (such > as reverse dependency rebuilding). I think this is exactly what gating is supposed to do. Let's compare it in this way: currently to add a feature which may break Rawhide and some unknown dependencies of the component you need to write a HEADS-UP e-mail, or, better, submit a Change request with the analysis of the change. People who read this e-mail would need to make a guess on whether or not his change affects them, then they have to fetch it and test it somehow, then they have to provide the feedback back to you. You need to wait for feedback, then you get the reports, in best cases - bugs, which you need to debug, requesting more info. Then you implement the change, hit the unexpected bug, which was unnoticed, and block others from building their packages and implementing their changes for unknown amount of time. With gating you can submit a code change, the tooling will take care of building it, building its dependencies, informing you of possible breakages, giving you the list of actual issues with all the debug logs. And then based on this data you can proceed or stop and rework the change a bit more in collaboration with exactly those people affected. I also think that we need a second point of view here: you were talking about not driving away the developer, who makes the change. But there are also other people who we shouldn't drive away. For example developers who depend on the change. Or QA who need to react on such changes. These people are have their part in the process. But, to be honest, I think there is a bit of overreacting on the entire Gating topic. It doesn't do a hard block. It is included in the design that gating can be bypassed. But it supposed to provide better analysis of the change. Bypassing of the gate can happen, The key here is that it won't be a surprise, rather informed decision.

> > > The side tag approach already does not scale, as evidenced by this thread. > > > > It does. You just need to communicate with others working in the same > > area, IMHO. I don't think we need some technical thing for something > > that happens rarely and can be solved by more communication. > > > > Side tags will happen a lot more often because the tooling is pushing > us to do it that way. Don't discount the potential for future > insanity. I'm still not sure side-tags are enough. Could we have a > concept of a "scratch side tag"? Something like a scratch build, but > contains a collection of builds and creates an overlay repo that can > be used to run checks on for auto-merging? If they're good, then it > would get auto-built properly into the main rawhide tag (or even > stable tag!). Afaik, this is exactly the concept of a dynamic sidetag as Fedora Infra is currently implementing. Sidetag in koji as a sort of pull-request: you create sidetag, get repos and composes built out of it, run tests, get results. Then "merge" this sidetag into the main repo. And while clashing sidetags is a problem, it is the same problem as it happens with overlapping pull-requests when several people are working on the source code. There are several ways to address it: 1) create smaller pull-requests and merge them more often This is one of the core points of CI workflows and it also seems to be most often underestimated. There is an overhead in creating smaller changes, which we are trying to address with tooling. But the overhead is _linear_. While the complexity of one merge grows exponentially with the size of the branch you are merging. The smaller the changes - the easier the rebase. 2) reorganize the codebase When working with source code, it usually means splitting files. If there are many changes targeting the same file again and again, causing conflicts, it might mean this file has too much logic bundled in it and needs a split up. It might also be merging files. For example if there are many changes targeting two files and it causes problems when these two files get out of sync. When applied to RPM packages, it may be also a good question: if there is circular dependency between packages, so that these are strictly tied to each other by version and can only be processed together - why these are separate packages and not subpackages? 3) Talk about changes in advance This of course is the default, and it doesn't go anywhere with gating or without it. Again I believe gating can make it easier. For example, we can add a note on conflicting "pull-requests" and put a rebase requirement as a gating check.

There are several ways how one can implement it. I would personally prefer pull requests to dist-git over scratch builds. I think it is important to keep some visibility in the system, so that not just the owner of a change, but also random lurkers can see what is being prepared.

(Also solves the issue )

Dne 18. 06. 19 v 16:54 Aleksandra Fedorova napsal(a): > On Tue, Jun 18, 2019 at 1:31 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote: >> On Tue, Jun 18, 2019 at 5:21 AM Aleksandra Fedorova <alpha(a)bookwar.info&gt; wrote: >>> Hi, >>> On Tue, Jun 18, 2019 at 3:05 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote: >>>> On Mon, Jun 17, 2019 at 8:53 PM Kevin Fenzi <kevin(a)scrye.com&gt; wrote: >>>>> On 6/17/19 4:47 PM, Kevin Kofler wrote: >>>>>> Kevin Fenzi wrote: >>>>>>> I disagree. I think we need gating to block as much stuff that breaks >>>>>>> things from landing as we can and then we should find that keeping >>>>>>> composes going is much easier on all of us. Then things can be fixed >>>>>>> when gating catches them and it's on the person who broke things. >>>>>> And that is going to make development completely cringe to a halt. It is the >>>>>> nature of a distribution branch under development that things will sometimes >>>>>> be completely broken for a couple weeks. There needs to be a place to do >>>>>> development that can cause such temporary breakage. >>>>> I again completely disagree. There is no reason for weeks of breakage. >>>>> Most of the issues that break composes are unannounced abi bumps where >>>>> just rebuilding dependent packages fixes it. Or broken deps (likewise). >>>>> Or mistakes made in kickstarts/comps. Or something that doesn't even >>>>> run. What good does having everyone broken for weeks do? >>>>> >>>> And that comes down to people shouldn't need to have to think about >>>> these things when working in Rawhide. While I don't disagree that >>>> Rawhide should be usable, I fundamentally disagree with making it >>>> harder for people to put things in Rawhide. We should be developing >>>> our tooling to make it _easier_ for stuff to go into Rawhide, and have >>>> Rawhide fix itself when the issues are relatively trivial to fix (such >>>> as reverse dependency rebuilding). >>> I think this is exactly what gating is supposed to do. >>> >>> Let's compare it in this way: >>> >>> currently to add a feature which may break Rawhide and some unknown >>> dependencies of the component you need to write a HEADS-UP e-mail, or, >>> better, submit a Change request with the analysis of the change. >>> People who read this e-mail would need to make a guess on whether or >>> not his change affects them, then they have to fetch it and test it >>> somehow, then they have to provide the feedback back to you. You need >>> to wait for feedback, then you get the reports, in best cases - bugs, >>> which you need to debug, requesting more info. Then you implement the >>> change, hit the unexpected bug, which was unnoticed, and block others >>> from building their packages and implementing their changes for >>> unknown amount of time. >>> >>> With gating you can submit a code change, the tooling will take care >>> of building it, building its dependencies, informing you of possible >>> breakages, giving you the list of actual issues with all the debug >>> logs. And then based on this data you can proceed or stop and rework >>> the change a bit more in collaboration with exactly those people >>> affected. >>> >>> I also think that we need a second point of view here: you were >>> talking about not driving away the developer, who makes the change. >>> But there are also other people who we shouldn't drive away. For >>> example developers who depend on the change. Or QA who need to react >>> on such changes. These people are have their part in the process. >>> >>> >>> But, to be honest, I think there is a bit of overreacting on the >>> entire Gating topic. >>> >>> It doesn't do a hard block. It is included in the design that gating >>> can be bypassed. But it supposed to provide better analysis of the >>> change. Bypassing of the gate can happen, The key here is that it >>> won't be a surprise, rather informed decision. >>> >> I think this is the first time I've heard a coherent description of >> the intent of this stuff. If it really is intended to work this way, >> it'd be very helpful. >> >>>>>> The side tag approach already does not scale, as evidenced by this thread. >>>>> It does. You just need to communicate with others working in the same >>>>> area, IMHO. I don't think we need some technical thing for something >>>>> that happens rarely and can be solved by more communication. >>>>> >>>> Side tags will happen a lot more often because the tooling is pushing >>>> us to do it that way. Don't discount the potential for future >>>> insanity. I'm still not sure side-tags are enough. Could we have a >>>> concept of a "scratch side tag"? Something like a scratch build, but >>>> contains a collection of builds and creates an overlay repo that can >>>> be used to run checks on for auto-merging? If they're good, then it >>>> would get auto-built properly into the main rawhide tag (or even >>>> stable tag!). >>> Afaik, this is exactly the concept of a dynamic sidetag as Fedora >>> Infra is currently implementing. Sidetag in koji as a sort of >>> pull-request: you create sidetag, get repos and composes built out of >>> it, run tests, get results. Then "merge" this sidetag into the main >>> repo. >>> >>> And while clashing sidetags is a problem, it is the same problem as it >>> happens with overlapping pull-requests when several people are working >>> on the source code. >>> >>> There are several ways to address it: >>> >>> 1) create smaller pull-requests and merge them more often >>> >>> This is one of the core points of CI workflows and it also seems to be >>> most often underestimated. There is an overhead in creating smaller >>> changes, which we are trying to address with tooling. But the overhead >>> is _linear_. While the complexity of one merge grows exponentially >>> with the size of the branch you are merging. >>> >>> The smaller the changes - the easier the rebase. >>> >>> 2) reorganize the codebase >>> >>> When working with source code, it usually means splitting files. If >>> there are many changes targeting the same file again and again, >>> causing conflicts, it might mean this file has too much logic bundled >>> in it and needs a split up. >>> >>> It might also be merging files. For example if there are many changes >>> targeting two files and it causes problems when these two files get >>> out of sync. >>> >>> When applied to RPM packages, it may be also a good question: if there >>> is circular dependency between packages, so that these are strictly >>> tied to each other by version and can only be processed together - why >>> these are separate packages and not subpackages? >>> >>> 3) Talk about changes in advance >>> >>> This of course is the default, and it doesn't go anywhere with gating >>> or without it. Again I believe gating can make it easier. For example, >>> we can add a note on conflicting "pull-requests" and put a rebase >>> requirement as a gating check. >>> >> I think you kind of missed what I was asking: I want side tags that >> are composed entirely of scratch builds. The result *must not* be used >> to merge into the distribution. It *must* be thrown away and when the >> pull request (or whatever) is merged, it should trigger the build >> again, which would be a "real" side tag that would get auto-merged on >> success. > You are right, I missed that part. And I agree this is needed. But I > am not sure if there is already a proposal for that. > > There are several ways how one can implement it. I would personally > prefer pull requests to dist-git over scratch builds. I think it is > important to keep some visibility in the system, so that not just the > owner of a change, but also random lurkers can see what is being > prepared. (Also solves the ) > > And if we look into PR-based workflows, then Zuul team has a proof of > concept pipeline which builds packages on PR in Pagure. > Which is not your ordinary Jenkins PR-triggered pipeline. > > The killer-feature of Zuul Zuul team? Zuul feature? I am probably missing some context ... It does not look like you are talking about https://github.com/Netflix/zuul which is the first reference to Zuul returned by Google.

On 6/17/19 4:47 PM, Kevin Kofler wrote: > Kevin Fenzi wrote: >> I disagree. I think we need gating to block as much stuff that breaks >> things from landing as we can and then we should find that keeping >> composes going is much easier on all of us. Then things can be fixed >> when gating catches them and it's on the person who broke things. > And that is going to make development completely cringe to a halt. It is the > nature of a distribution branch under development that things will sometimes > be completely broken for a couple weeks. There needs to be a place to do > development that can cause such temporary breakage. I again completely disagree. There is no reason for weeks of breakage.

Most of the issues that break composes are unannounced abi bumps where just rebuilding dependent packages fixes it. Or broken deps (likewise). Or mistakes made in kickstarts/comps. Or something that doesn't even run. What good does having everyone broken for weeks do? > The side tag approach already does not scale, as evidenced by this thread. It does. You just need to communicate with others working in the same area, IMHO. I don't think we need some technical thing for something that happens rarely and can be solved by more communication. kevin _______________________________________________ devel mailing list -- devel(a)lists.fedoraproject.org To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Kevin Fenzi wrote: > I again completely disagree. There is no reason for weeks of breakage. > Most of the issues that break composes are unannounced abi bumps where > just rebuilding dependent packages fixes it. Or broken deps (likewise). > Or mistakes made in kickstarts/comps. Or something that doesn't even > run. What good does having everyone broken for weeks do? There are several reasons why weeks of breakage are entirely reasonable for a distribution under development: 1. A bump of a library to a new major version. It can take a few weeks for dependent packages to be ported to the new version. But it is often worth waiting rather than introducing a compatibility library with all the mess that comes with it. (Of course, that does not hold for a library like Qt where getting everything ported to the new version is not realistic even in a couple weeks. Those are the cases where a compatibility library is needed basically forever. But there are libraries where it makes sense to just port everything.) The workflow would be to just bump the library to the new version in Rawhide immediately and then let dependent packages get fixed over time as their upstreams, Fedora packagers, or other distros' packagers port them. 2. Vacation. The maintainer of the dependent package may be on vacation (or even just busy with paid work, in case of volunteers) and therefore unable to fix their package immediately. So you may have to wait a few weeks for the broken dependency to get fixed if a simple rebuild is not enough.

This has all worked just fine in the past, before it was decided to make basically every single broken dependency fail the entire Rawhide compose. Soname bumps did not even have to be announced, they would get announced by the broken dependency report within less than 24 hours anyway, and then eventually fixed (without somebody unrealistically expecting maintainers to fix the broken dependency in less than 24 hours to make the next compose succeed, which is just plain impossible for the average volunteer packager, especially if source code changes are needed).

Hi, Imagine situation that somebody is working on KDE rebase and me on libgit2 rebase. Both involve rebuilding/updating some package, let's say kf5-ktexteditor. We both work in different side tags, in KDE rebase kf5-ktexteditor gets updated to a new version. In libgit2 rebase, old version gets rebuilt. Once any of us finished with side tag, we merge it. Let's say that was KDE rebase. That means, new kf5-ktexteditor is merged into the rawhide which is built against old libgit2. Then I finish with libgit2 things and we merge it into the rawhide.. That means it just downgrades kf5-ktexteditor version because koji looks at the build time and not the NVR. And even if it did, that would mean I have to manually go and rebuild all packages which are handled in multiple side tags. Do you think that scales?

_______________________________________________ devel mailing list -- devel(a)lists.fedoraproject.org To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

On Sat, Jun 08, 2019 at 10:29:29AM -0400, Stephen John Smoogen wrote: > On Sat, Jun 8, 2019 at 6:12 AM Igor Gnatenko < > ignatenkobrain(a)fedoraproject.org&gt; wrote: > >> Hi, >> >> Imagine situation that somebody is working on KDE rebase and me on >> libgit2 rebase. Both involve rebuilding/updating some package, let's >> say kf5-ktexteditor. >> >> We both work in different side tags, in KDE rebase kf5-ktexteditor >> gets updated to a new version. In libgit2 rebase, old version gets >> rebuilt. >> >> Once any of us finished with side tag, we merge it. Let's say that was >> KDE rebase. That means, new kf5-ktexteditor is merged into the rawhide >> which is built against old libgit2. Then I finish with libgit2 things >> and we merge it into the rawhide.. That means it just downgrades >> kf5-ktexteditor version because koji looks at the build time and not >> the NVR. Is that true? If yes, that seems to be the wrong thing to do. Instead, koji should say "not merging a-nn.rpm, because a-nn.rpm is already in the tag", or "... a-nn+1.rpm is already in the tag". Ideally, this would cause the whole merge to be rejected, and you can bump and rebuild the offending packages, and try to merge again later.

> And even if it did, that would mean I have to manually go and > rebuild all packages which are handled in multiple side tags. Yes. But I think that'd be OK. This isn't *that* much work, and at least there'd be a clear result and a precise list of packages that need to be rebuilt. That's much better than a partial merge that breaks the next compose. >> Do you think that scales? > > > So I believe at one point we only allowed one side tag at a time, but > people complained that most of the time we were wasting packagers time > waiting for someone else to get stuff done. There was an idea about trying > to solve conflicts but that was seen as too much time to solve and too much > bureaucracy to live with. Instead we assumed that these would happen and > that packagers would work things out between themselves. > > So no it doesn’t scale. The various solutions to make it scale are not > written and will not be written in any near time as they will need some > sort of project, some amount of resources added and a backlog of more > critical work done first. Until then packagers need to realize that there > are going to be build conflicts every now and then and factor in that they > will have to work out between themselves how to solve them. Yeah, I don't think any technical solution is worth the trouble here. But we can coordinate, e.g. by announcing on fedora-devel.