I've been working on updating the hadoop package to the latest 2.4.0 release and at this point I've resolved all the issues but I'm now blocked by the log4j2 update. log4j2 breaks the hadoop build pretty severely, and it doesn't seem the log4j2 team has spent much time thinking about how to provide backwards compatibility to existing log4j1.2 users. From my investigation of log4j2: log4j.properties file is no longer read at all configuration file is now in XML or JSON configuration file name is log4j2.[xml|json|jsn] V2 isn't backwards compatible with V1. There's a shim for v1 api but it will only work for a limited set of cases, and for some cases it does work for it turns some operations into noop calls. This is a pretty major change and even the compatibility layer, if it will work for a project, does not seem to guarantee like functionality and minimally will require a re-do of all log4j configuration files a project ships. I'm not sure many sizable upstream projects would undertake/accept such a drastic change very quickly. The list of projects currently blocked by this update are: hadoop hbase oozie hive apache-log4j-extras amplab-tachyon I would be surprised if there aren't a lot more. I understand Fedora is always pushing for the latest versions, but for some fundamental packages can there be compatibility packages introduced at the same time as an incompatible update? Package maintainers of dependent packages will still need to touch their packages and determine if the new version will work for them. Providing a compat package will also allow packages to update to their newer versions while not held up on trying to integrate changes from a compatibility breaking dependency update. Rob --

From: "Robert Rati" <rrati(a)redhat.com&gt; To: "Fedora Big Data SIG" <bigdata(a)lists.fedoraproject.org&gt;, "Development discussions related to Fedora" <devel(a)lists.fedoraproject.org&gt;, "Fedora Java Development List" <java-devel(a)lists.fedoraproject.org&gt; Sent: Wednesday, May 21, 2014 7:03:42 PM Subject: Hadoop + log4j2 = fullstop I've been working on updating the hadoop package to the latest 2.4.0 release and at this point I've resolved all the issues but I'm now blocked by the log4j2 update. log4j2 breaks the hadoop build pretty severely, and it doesn't seem the log4j2 team has spent much time thinking about how to provide backwards compatibility to existing log4j1.2 users. From my investigation of log4j2: log4j.properties file is no longer read at all configuration file is now in XML or JSON configuration file name is log4j2.[xml|json|jsn] V2 isn't backwards compatible with V1. There's a shim for v1 api but it will only work for a limited set of cases, and for some cases it does work for it turns some operations into noop calls. This is a pretty major change and even the compatibility layer, if it will work for a project, does not seem to guarantee like functionality and minimally will require a re-do of all log4j configuration files a project ships. I'm not sure many sizable upstream projects would undertake/accept such a drastic change very quickly. The list of projects currently blocked by this update are: hadoop hbase oozie hive apache-log4j-extras amplab-tachyon I would be surprised if there aren't a lot more. I understand Fedora is always pushing for the latest versions, but for some fundamental packages can there be compatibility packages introduced at the same time as an incompatible update?

Package maintainers of dependent packages will still need to touch their packages and determine if the new version will work for them. Providing a compat package will also allow packages to update to their newer versions while not held up on trying to integrate changes from a compatibility breaking dependency update. Rob -- devel mailing list devel(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

From: "Robert Rati" <rrati(a)redhat.com&gt; To: "Aleksandar Kurtakov" <akurtako(a)redhat.com&gt;, "Development discussions related to Fedora" <devel(a)lists.fedoraproject.org&gt; Cc: "Fedora Big Data SIG" <bigdata(a)lists.fedoraproject.org&gt;, "Fedora Java Development List" <java-devel(a)lists.fedoraproject.org&gt; Sent: Wednesday, May 21, 2014 9:06:07 PM Subject: Re: [fedora-java] Hadoop + log4j2 = fullstop So, who needed log4j2? It is massively incompatible with log4j1.2 and isn't a simple port job. I would argue if log4j2 was actually needed, it should have been introduced as a separate log4j2 package and allow projects to port to it as they have time/need. This update log4j to an incompatible version with no compat package provided at the same time is not the way to handle such an upgrade. Giving advanced notice that the world will come crumbling down and you'll have to deal with it is not enough.

Rob On 05/21/2014 02:01 PM, Aleksandar Kurtakov wrote: > > > Alexander Kurtakov > Red Hat Eclipse team > > ----- Original Message ----- >> From: "Robert Rati" <rrati(a)redhat.com&gt; >> To: "Fedora Big Data SIG" <bigdata(a)lists.fedoraproject.org&gt;, "Development >> discussions related to Fedora" >> <devel(a)lists.fedoraproject.org&gt;, "Fedora Java Development List" >> <java-devel(a)lists.fedoraproject.org&gt; >> Sent: Wednesday, May 21, 2014 7:03:42 PM >> Subject: Hadoop + log4j2 = fullstop >> >> I've been working on updating the hadoop package to the latest 2.4.0 >> release and at this point I've resolved all the issues but I'm now >> blocked by the log4j2 update. log4j2 breaks the hadoop build pretty >> severely, and it doesn't seem the log4j2 team has spent much time >> thinking about how to provide backwards compatibility to existing >> log4j1.2 users. From my investigation of log4j2: >> >> log4j.properties file is no longer read at all >> configuration file is now in XML or JSON >> configuration file name is log4j2.[xml|json|jsn] >> V2 isn't backwards compatible with V1. There's a shim for v1 api but it >> will only work for a limited set of cases, and for some cases it does >> work for it turns some operations into noop calls. >> >> This is a pretty major change and even the compatibility layer, if it >> will work for a project, does not seem to guarantee like functionality >> and minimally will require a re-do of all log4j configuration files a >> project ships. I'm not sure many sizable upstream projects would >> undertake/accept such a drastic change very quickly. >> >> The list of projects currently blocked by this update are: >> hadoop >> hbase >> oozie >> hive >> apache-log4j-extras >> amplab-tachyon >> >> I would be surprised if there aren't a lot more. I understand Fedora is >> always pushing for the latest versions, but for some fundamental >> packages can there be compatibility packages introduced at the same time >> as an incompatible update? > > Nothing stops compatibility packages from appearing. But it's the people > that need it that have to drive it. Everyone is time constrained so > whoever needs something must do it. > It's really is as simple as that. > > Alexander Kurtakov > Red Hat Eclipse team > >> Package maintainers of dependent packages >> will still need to touch their packages and determine if the new version >> will work for them. Providing a compat package will also allow packages >> to update to their newer versions while not held up on trying to >> integrate changes from a compatibility breaking dependency update. >> >> Rob >> -- >> devel mailing list >> devel(a)lists.fedoraproject.org >> https://admin.fedoraproject.org/mailman/listinfo/devel >> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct > -- > java-devel mailing list > java-devel(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/java-devel >

From: "David M. Lloyd" <david.lloyd(a)redhat.com&gt; To: java-devel(a)lists.fedoraproject.org Sent: Wednesday, May 21, 2014 9:29:56 PM Subject: Re: [fedora-java] Hadoop + log4j2 = fullstop On 05/21/2014 01:16 PM, Aleksandar Kurtakov wrote: > ----- Original Message ----- >> From: "Robert Rati" <rrati(a)redhat.com&gt; >> To: "Aleksandar Kurtakov" <akurtako(a)redhat.com&gt;, "Development discussions >> related to Fedora" >> <devel(a)lists.fedoraproject.org&gt; >> Cc: "Fedora Big Data SIG" <bigdata(a)lists.fedoraproject.org&gt;, "Fedora Java >> Development List" >> <java-devel(a)lists.fedoraproject.org&gt; >> Sent: Wednesday, May 21, 2014 9:06:07 PM >> Subject: Re: [fedora-java] Hadoop + log4j2 = fullstop >> >> So, who needed log4j2? It is massively incompatible with log4j1.2 and >> isn't a simple port job. I would argue if log4j2 was actually needed, >> it should have been introduced as a separate log4j2 package and allow >> projects to port to it as they have time/need. This update log4j to an >> incompatible version with no compat package provided at the same time is >> not the way to handle such an upgrade. Giving advanced notice that the >> world will come crumbling down and you'll have to deal with it is not >> enough. > > Now this it the wrong question. Questioning the right of someone doing the > update and maintaining something for others to use is unfair at least. The > one that *does* decides. Others are free to *do* themself. > I can write an essay why keeping old software around is bad and how often > it hurts people that care for the whole distro. It's pretty easy for > others caring for just their own thing to dismiss this work and any > explanation would simply be not understood until they try to look at the > bigger picture. I really recommend you joining in the big effort to keep > the Java ecosystem in good shape on fedora to feel the pain and understand > why so many don't want to deal with such things. Before you wax poetic, it's worth mentioning that log4j2 is not in any way considered to be an appropriate successor to log4j1 in the greater community, despite the similarity in name (which, as I understand, is largely a political artifact rather than technical). It could be argued that logback is closer to being a successor, but even this project is not compatible with the original log4j API or configuration mechanism, despite being written by the same original author. As long as applications bundle log4j (as a backend), and as long as shipped packages use log4j (as a frontend), the log4j1 package should continue to be maintained and updated.

It makes the most sense to picture log4j1 and log4j2 as completely separate and independent projects with little in common other than an accident of naming. > P.S. 1. Just trying to get explanation for something or make the build > system work with latest build systems around once new major release is > available upstream will show you what I speak for. > P.S. 2. Two of my packages are failing because of this. I still haven't > investigated it but if moving to log4j2 is not easy it's my problem not > log4j maintainer one (who by the way is awesome guy and provides more > support than people can ask for, making the question even more unfair). > > > Alexander Kurtakov > Red Hat Eclipse team > >> >> Rob >> >> On 05/21/2014 02:01 PM, Aleksandar Kurtakov wrote: >>> >>> >>> Alexander Kurtakov >>> Red Hat Eclipse team >>> >>> ----- Original Message ----- >>>> From: "Robert Rati" <rrati(a)redhat.com&gt; >>>> To: "Fedora Big Data SIG" <bigdata(a)lists.fedoraproject.org&gt;, >>>> "Development >>>> discussions related to Fedora" >>>> <devel(a)lists.fedoraproject.org&gt;, "Fedora Java Development List" >>>> <java-devel(a)lists.fedoraproject.org&gt; >>>> Sent: Wednesday, May 21, 2014 7:03:42 PM >>>> Subject: Hadoop + log4j2 = fullstop >>>> >>>> I've been working on updating the hadoop package to the latest 2.4.0 >>>> release and at this point I've resolved all the issues but I'm now >>>> blocked by the log4j2 update. log4j2 breaks the hadoop build pretty >>>> severely, and it doesn't seem the log4j2 team has spent much time >>>> thinking about how to provide backwards compatibility to existing >>>> log4j1.2 users. From my investigation of log4j2: >>>> >>>> log4j.properties file is no longer read at all >>>> configuration file is now in XML or JSON >>>> configuration file name is log4j2.[xml|json|jsn] >>>> V2 isn't backwards compatible with V1. There's a shim for v1 api but it >>>> will only work for a limited set of cases, and for some cases it does >>>> work for it turns some operations into noop calls. >>>> >>>> This is a pretty major change and even the compatibility layer, if it >>>> will work for a project, does not seem to guarantee like functionality >>>> and minimally will require a re-do of all log4j configuration files a >>>> project ships. I'm not sure many sizable upstream projects would >>>> undertake/accept such a drastic change very quickly. >>>> >>>> The list of projects currently blocked by this update are: >>>> hadoop >>>> hbase >>>> oozie >>>> hive >>>> apache-log4j-extras >>>> amplab-tachyon >>>> >>>> I would be surprised if there aren't a lot more. I understand Fedora is >>>> always pushing for the latest versions, but for some fundamental >>>> packages can there be compatibility packages introduced at the same time >>>> as an incompatible update? >>> >>> Nothing stops compatibility packages from appearing. But it's the people >>> that need it that have to drive it. Everyone is time constrained so >>> whoever needs something must do it. >>> It's really is as simple as that. >>> >>> Alexander Kurtakov >>> Red Hat Eclipse team >>> >>>> Package maintainers of dependent packages >>>> will still need to touch their packages and determine if the new version >>>> will work for them. Providing a compat package will also allow packages >>>> to update to their newer versions while not held up on trying to >>>> integrate changes from a compatibility breaking dependency update. >>>> >>>> Rob >>>> -- >>>> devel mailing list >>>> devel(a)lists.fedoraproject.org >>>> https://admin.fedoraproject.org/mailman/listinfo/devel >>>> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct >>> -- >>> java-devel mailing list >>> java-devel(a)lists.fedoraproject.org >>> https://admin.fedoraproject.org/mailman/listinfo/java-devel >>> >> > -- > java-devel mailing list > java-devel(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/java-devel > -- - DML -- java-devel mailing list java-devel(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/java-devel

On 05/21/2014 08:06 PM, Robert Rati wrote: > So, who needed log4j2? It is massively incompatible with log4j1.2 and > isn't a simple port job. I would argue if log4j2 was actually needed, > it should have been introduced as a separate log4j2 package and allow > projects to port to it as they have time/need. This update log4j to an > incompatible version with no compat package provided at the same time is > not the way to handle such an upgrade. Giving advanced notice that the > world will come crumbling down and you'll have to deal with it is not > enough. 1. This change was announced [1] in advance and I didn't hear any concerns from anyone until the change was implemented. hadoop and all other packages were mentioned in the announcement as possibly being affected by the update. 2. The original announcement [1] and follow-up messages contain some of the reasons for updating to 2.0. But there are other reasons as well -- log4j 2.0 was blocking mybatis package update [3].

3. As I already said on IRC multiple times, I am going to introduce a compat package log4j12 before 6 Jun 2014. If that's too late then feel free to package it yourself. 4. I shared my general opinions about Java compat packages and update procedures on java-devel [2] and I didn't hear any concerns about them. Recent log4j update follows the procedure and best practices I presented there, like keeping the name log4j instead using log4j2. [1] https://lists.fedoraproject.org/pipermail/devel/2014-May/198934.html [2] https://lists.fedoraproject.org/pipermail/java-devel/2013-October/005000.... [3] https://bugzilla.redhat.com/show_bug.cgi?id=1091042