+1 Trying to continually level set to the HEAD of Fedora has introduced patch sets which only continue to diverge over time. Upstream(s) have expressed little/no interest in accepting some of these patches, and I can hardly blame them.

From: "Robert Rati" <rrati(a)redhat.com&gt; To: devel(a)lists.fedoraproject.org Sent: Tuesday, July 23, 2013 7:01:33 PM Subject: Re: RFC: Proposal for a more agile "Fedora.next" (draft of my Flock talk) I think there is another data point to include in the discussion. As Pete mentioned, a lot of the effort to get Hadoop into Fedora has revolved around updating the dependencies. Most of these changes are being sent upstream in the hopes of moving them forward on their dependencies. However, there are cases where we have had to patch the source and the result was not thing we could push upstream. The tomcat and jsp compiler changes was one such patch, but the bigger one in my mind is jetty. We patched upward to jetty8 when we were testing against F18, and that patch has a chance of being included upstream. F19 moved to jetty9 which drastically changed some of the interfaces and dropped support for java6 (the version upstream Hadoop uses). I was able to patch Hadoop to make it work with jetty9, but I would not claim it to be an optimal solution nor is there much hope upstream would accept a patch to jetty9 anytime soon because of the java version incompatibility. While packagers can update dependencies, they may not be the best people to decide on the correct technical direction. Using jetty9 as an example, the interface changed substantially and in such a way that it might make more sense to re-architect some areas of Hadoop. The team packaging Hadoop is certainly not the right people to take on that task. With the java incompatibility issues with jetty9, upstream is unlikely to take that on any time soon. Having packagers continually patch to newer versions of dependencies not only potentially moves them further away from upstream, but also could decrease the likelihood of patches being accepted upstream because the solution is not the direction the project wants to head in.

Rob On 07/22/2013 05:54 PM, Peter MacKinnon wrote: > Shout out to my fellow Flocker, Matt... > > -------- Original Message -------- >> Subject: RFC: Proposal for a more agile "Fedora.next" (draft of my >> Flock talk) >> Date: Mon, 22 Jul 2013 09:38:54 -0400 >> From: Matthew Miller <mattdm(a)fedoraproject.org&gt; >> Reply-To: Development discussions related to Fedora >> <devel(a)lists.fedoraproject.org&gt; >> To: Fedora Development List <devel(a)lists.fedoraproject.org&gt; >> >> <snip> >> >> Obviously, no-bundled-libs is a crucial part of the packaging guidelines >> today. As a sysadmin, I know why it's important. This is not just a noble >> goal, but also something that pragmatically makes systems better. But, >> it's >> also keeping us from having software that people really use in Fedora. >> Chef >> and Hadoop are two big examples. This hurts us more than it helps the >> world. >> So, in some areas, we need a different approach. >> > > The Big Data SIG is trying to adapt Hadoop 2.x into Fedora for F20 > <https://fedoraproject.org/wiki/Changes/Hadoop>;, and I'll be sharing our > insights on this at Flock <http://sched.co/185PNI9> in a couple of > weeks. In Matt's conceptual architecture I suppose Hadoop Common would > live in the Ring 2-to-3 orbit somewhere. It is a core in it's own right > (it provides a distributed, replicated file system) in that there is an > every growing software ecosystem that has emerged around it, and the SIG > would like Fedora to be the OS of choice for that ecosystem. Stable > enough for deployment but a feature-rich, current and productive > environment for the developers in that evolving ecosystem. The Hadoop > runtime is an orchestration of JVM-based daemons which can be viewed as > system-level services, thus an obvious candidate for well-defined > integration with Fedora via packaging: correct permissions, systemd > scripts, logs, etc. > > However, the root of that core is a set of older and deprecated Java > dependencies (e.g., Jetty 6, Tomcat 5.5) which are expressed via the > Apache Maven build tool. The "quick and dirty" label used by another > poster of a VERY popular build tool like Maven does it a disservice. The > fact is that it is exceedingly popular in the Java development community > and has been for some time. Anyway, the challenge for this project is > the reconciliation of it's stable dependencies versus the ever-changing > bleeding edge that is typically found in the latest Fedora release. A > lot of our efforts so far have been the various API and build > specification changes necessary to try to make Hadoop fit into Fedora. > > So far, so good...sort of. We can make the basic use case and tests work > with the modified dependencies but in doing so we risk giving up parity > with the Apache baseline (including the JRE) and potentially lose out to > other so-called "dirty RPMs". Ideally, we wouldn't be forced into some > of these adaptations and compromises if there were Fedora packaging > alternatives that would give us (a SIG ring?) more control over the > bundles needed by Hadoop as opposed to the ones mandated by the latest > Fedora release. Make no mistake: patches are fed from the SIG to the > Hadoop community to try to bump the versions there. But the upstream > project can't and won't chase an ever-vanishing point in the distance. > They view their lower dependencies much like a stable OS such as RHEL > and change should be deliberated there. > > I feel like Matt has at least kick-started the discussion around how > Fedora could evolve to support orthogonal dependency models that more > readily adapt to external projects like Hadoop. Not that our SIG has any > profound answers. :-) > > Thus, we are very interested in _any_ packaging architecture proposals > that could help relieve our initiative's pain points, and look forward > to further constructive discussion of the same. > > My $0.02, > \Pete > > > -- > Peter MacKinnon > MRG Grid/Big Data > Red Hat Inc. > Raleigh, NC > > > -- devel mailing list devel(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel

On Mon, Jul 22, 2013 at 05:54:27PM -0400, Peter MacKinnon wrote: > So far, so good...sort of. We can make the basic use case and tests work with > the modified dependencies but in doing so we risk giving up parity with the > Apache baseline (including the JRE) and potentially lose out to other so-called > "dirty RPMs". Ideally, we wouldn't be forced into some of these adaptations and > compromises if there were Fedora packaging alternatives that would give us (a > SIG ring?) more control over the bundles needed by Hadoop as opposed to the > ones mandated by the latest Fedora release. Make no mistake: patches are fed > from the SIG to the Hadoop community to try to bump the versions there. But the > upstream project can't and won't chase an ever-vanishing point in the distance. > They view their lower dependencies much like a stable OS such as RHEL and > change should be deliberated there. > So -- would you be willing to go on a tangent with me for a short bit? Looking at your issue, I have a few comments, some of which might even be helpful :-) I'm bringing this up because the way you describe Hadoop makes it sound like people are going to want to build on top of it. having it in a Ring 2/3 non-rpm universe makes it more tempting for people to treat Hadoop as an alternative choice which might mean that we get multiple copies of hadoop there as people build on different hadoop foundations. Figuring out whether it's possible to build this in Fedora Commons might be better. Observations-only: * Bravo for your patching effort! My experience in open source is that projects which relentlessly stick to old versions eventually fork and the fork which is supporting the newer dep stack. Unfortunately, this switch of project direction is painful and can take a long while to come about. So even though your experience with patching up to more recent versions will be valuable when/if this actually happens, I understand that you will likely want to work on some other strategy first. Ways to improve within the Fedora Core/Fedora Commons model: * Although bundled libraries are not allowed in Fedora Core and Commons; both compat libraries and forks are allowed. There are downsides to each: * Compat libraries may not have support from upstream. Once upstream moves onto libfoo-2.0, they may no longer be interested in releasing bugfixes or even security fixes for libfoo-1.x. This means that you, as the owner of the compat package would be responsible for those. On the other hand, you'd be responsible for those even if they were bundled into the package you're really interested in. It's just that you would mostly pretend that the Hadoop upstream was on top of those problems instead of having to deal with them yourself. You could still rely heavily on the Hadoop upstream by syncing their fixes to the compat library in their bundled copy to your copy. * Forking increases the number of copies of virtually the same code, at least, until the forks diverge significantly. Forking also means that someone needs to setup upstream infrastructure: an upstream SCM, a mailing list somewhere, an issue tracker (a lot of times, the ml and issue tracker start off as a piece of an existing project; only the SCM differs). What does forking bring you, though? The potential benefit is that you get a better upstream experience. It's a place where people who need the same API version of libfoo can congregate and supply fixes that can be used in common with every other project that uses that code. * The upsides to both of these over bundled libraries are that you have better tracking of your dependency chain (Security issue discovered in libfoo that affects everything from 1.0.5 through 2.8. When you have an external package, it's easier to see whether the code is affected and how many packages are affected than if it's bundled. This can also be overcome using Virtual Provides to "tag" a package that's bundling a specific version.) and the possibility for more people to collaborate on this (We've often seen that multiple upstream packages are bundling upstream versions of a libfoo-1.x API. When we pull these into a compat package, all the bugfix work can be pushed to the central package which cuts down on work for everyone.) * Compat packages only help us if there's a way to parallel install library packages (and it sounds like for hadoop you might need parallel versions of the JRE as well). Forking usually implies a new library name so that usually doesn't need parallel library support. So the first implementation question might be: Is parallel insallation and usage supported in your language? if it isn't out of the box, is there a way we could create it (for instance, via a wrapper script that set CLASSPATH appropriately)? For the JRE, is there a way to parallel iinstall it? Is there also a way that the application being run can choose which JRE it needs to be run under? -Toshio

On Tue, Jul 23, 2013 at 03:34:15PM -0400, Peter MacKinnon wrote: > > Well, compat libraries are certainly an option but I view that as a tactical > solution to an institutional, um, challenge. > And I believe that is what Matt is driving at: sustainable solutions that > satisfy the user/admin need for stability and > "cleanliness" while also providing an OS that developers from all manner of > technology and language profiles would > gravitate toward. > So actually... I think that I would describe the two strategies the other way. Bundling libraries into stacks is an expedient solution, not a sustainable one. As time goes on, you get more and more programs bundling more and more.

Which means that there's more maintainance burden and less sharing of that burden between interested stakeholders.

I'm not opposed to trying to find a way to host the expedient solution but in the end, the sustainable solution is to make changes to what is shipped.

At some point, the software that gets bundled is only being maintained by a handful of pople in the bundling project. At that point, it gets real lonely when a security issue or two pop up.

-Toshio

Le Mer 24 juillet 2013 16:17, Peter MacKinnon a écrit : > A generalization which I would disagree with in the Java space. I think > many projects > eventually reach "steady-state" where they have acquired the set of dep > bundles > they need to satisfy their runtime and test requirements. For example, > Hadoop > would not bundle multiple versions of Jetty. However, it's bundled deps > may differ > slightly (perhaps even by just API-compatible versions) say from Tomcat's > (another Hadoop dependency). And that's where you see the proliferation > of bundled > jar libraries as you walk down (up?) the dep tree. Having been involved in java packaging in the past, I can only agree with the generalization. Most java projects will affirm their bundling is minimal, but java projects are deeply interconnected and unwinding recursively the dependencies of the bundled deps (which can also do their own 'sparse' bundling) always gave frightening results in my time. Java projects only consider the tip of the iceberg. They quickly lose count of the layers of obsolete components their bundled deps drag. And because they lose count, they are not aware of the security problems that caused each of those to become obsolete upstream in the first place.

I wouldn't object to bundling if the bundlers did due security diligence on the bits they bundled. But they don't. That's the *only* reason they feel bundling is cheaper. They skim on the security maintenance costs.