Hi, Stephen. On Wednesday, 05 August 2020 at 21:36, Stephen Gallagher wrote: [...] [1] Transitions to/from modular are already forbidden in Fedora releases, so saying "in the same release" makes no sense. I'd say "in rawhide or during Fedora upgrades". I guess "transition to another default module" is what is meant above. This is redundant, already mentioned above[1]. Regards, Dominik -- Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org There should be a science of discontent. People need hard times and oppression to develop psychic muscles. -- from "Collected Sayings of Muad'Dib" by the Princess Irulan

As general feedback, the footnotes make it hard to read the rendered version of the document, forcing me to scroll up and down. More comments below. On Wed, Aug 5, 2020, at 3:36 PM, Stephen Gallagher wrote: Can these be directed to a disabled-by-default build-dep repo of some kind for those trying to do local builds? Do these "non-shipped" packages shadow non-modular versions of the same packages? I'd propose an alternative to Default Streams. Any package part of a default stream should instead be auto-built in a given release where the stream is marked as default. Pushes to the dist-git branch for that release would be blocked by all but the auto build bot, and all changes should be made to the stream branch. The stream marked "default" for the particular module would be enabled as a buildroot override, including build only packages, for such automated non-modular rebuilds of streams marked "default". This would obviate the need of stream transition, except in cases if inter-module deps. Even given the status quo, I'd argue that streams enabled by nature of being Default rather than being explicitly enabled, should not shadow non-modular packages. As-is today, third party repos are marking themselves as module_hotfixes to skip the shadowing issues. Say EPEL, don't mention version. "default stream of another module" V/r, James Cassell

On Wed, Aug 05, 2020 at 10:50:59PM -0400, Stephen Gallagher wrote: These build-only components are handy for running tests after the build. E.g. a non-build component has upstream tests. They are run at build-time, and thus the module contains build-only components for performing the test. The tests are also packaged into a subpackage and the subpackage is filtered from the module. ("buildonly: true" is only a different form of "filter" modulemd section.) Later a CI may want to install the subpackage and reproduce the tests. -- Petr

On Thu, Aug 6, 2020, at 10:45 AM, Miro Hrončok wrote: The concerns sound valid to me. Despite the concerns, it sounds like the decision has been made to have Default Streams in RHEL Next, so we should develop an appropriate policy for them in ELN. Let's not impede RHEL development unnecessarily, but let's work to make RHEL better. Part of that might be convincing the powers-that-be to give up default streams, but the topic that started this thread is "what is the policy, assuming they've already been allowed." I'd propose that any default module stream in ELN should have its packages built only from the "master" branch in dist-git, and should not have any separate branch. This is the item that I did not see in the proposed policy, and a necessary addition, in my opinion. Perhaps with an exception process for software that requires older bulid deps, but even in that case, they should be added as compat packages if at all possible. Similarly, I'd argue that ELN shouldn't have default content that isn't also available by default in Fedora. this would likely be solved by my suggestion above for default modules to use only the master branch. I appreciate your efforts here! Thanks for raising the concerns. They echo my own. V/r, James Cassell

On Thu, Aug 6, 2020 at 10:46 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote: Those comments were from Daniel when talking about Fedora as it exists today. Not about ELN. Yes, he discourages their use in Fedora. However, ELN has a mission to be a bridge to future RHEL and that distribution has committed to default streams as a business necessity for multiple reasons (among them support lifecycle which is much harder to address via non-modular content). It was written that way *at your request*. You specifically asked for a proposal for "Fedora" that we could initially apply only to "Fedora ELN". It seems disingenuous to now use that as an argument against it. Josh listed some of the key reasons behind default streams: that enterprise customers don't like to learn new commands. So default streams allowed us to package content with shorter-than-RHEL-lifetime and still `yum install foo` would install something the customer could use. I agree with this statement, hence why the policy expressly requires "Changes to default streams *MUST* be approved via a Change Proposal". This means that the maintainer must get FESCo approval to become a default stream. Decisions will not be made behind closed doors, but Red Hat *will* have a strong opinion on them. And if ELN diverges too far from Red Hat's needs, the experiment of ELN will have failed. Red Hat is committed to making its reasons for such needs as transparent as possible, so claims of "decisions behind closed doors" are rather alarmist. That said, I cannot deny that there is a crucial balance to be struck here and that, I think, is one of the key pieces that default streams can help with. As I have said elsewhere in this email (I'm jumping around while writing, so I think this part is below): ELN will *always* be built from sources that are available for Rawhide to deliver. In effect, it will be equivalent to installing Rawhide with that module stream enabled in the kickstart. In cases where Red Hat strongly believes that a particular alternative stream in Rawhide is more in line with the needs of RHEL, a Change Proposal will be filed with a request to have that stream become the default in ELN. Because software never changes or improves over time. Right, I forgot we gave up on Python after we discovered that its unicode handling wasn't great... Honestly, I don't understand your argument here. The *implementation* had issues. Those issues have been identified and plans exist to resolve them. We agreed not to put that in Fedora until we could prove it was fixed. I'm... not really sure what you're comparing here, but I'll try to make it clearer. * Anything that exists as non-modular content in ELN will have been built from the Rawhide branch of the same non-modular RPM. * The set of non-modular packages built in ELN will be a subset of the non-modular packages in Rawhide * Module streams built for Fedora Rawhide *may* also be built for Fedora ELN. * Module streams built for Fedora ELN *may* be set as the default stream in ELN. If so, those RPMs coming from the default stream *must not* overlap the non-modular content. Does that make more sense? See the previous statement. The module streams for ELN will be built from the exact same YAML as the equivalent Rawhide module stream. Thus what is being delivered is still "Rawhide content", it just happens to differ about which is the *default content*. I feel that this is highly unlikely since ELN won't be a deliverable in the traditional sense of the term. If they were to do this, their package would stagnate in Fedora. If that happens, that's what the non-responsive maintainer process is for. I guess we haven't been clear enough about the artifacts that ELN produces. They are intended as a compose for testing/CI purposes *only*. We will not be mirroring them or making them publicly visible on getfedora.org or anything of the sort. It's a bootstrapping effort towards RHEL X+1. I intend to make it VERY clear that working in ELN but not maintaining that content in the rest of Fedora is unacceptable. That's part of the reason behind the package comaintainership requirement in this policy. I understand your stance on the matter at large. I hope that my responses above have provided you with at least some measure of improved clarity.

On Thu, Aug 20, 2020 at 02:17:09PM -0400, Stephen Gallagher wrote: Modularity that exists in Fedora today is also the Modularity as it exists in ELN. We use the same DNF and associated tools everywhere, and I hope nobody is proposing to diverge on that. The arguments that were made in those comments apply to ELN too equally well. I don't think that "RHEL ... has committed to default streams as a business necessity" is really true. The long reply that Terry Bowling sent in parallel makes it clear that RH wants a certain *user experience* (existing tools and commands continue to work, more software is available). Technical details like whether there is a default stream or not are not mentioned. If the default stream is replaced by non-modular rpms, users are not likely to notice or care. First, I don't remember Miro making such a request. I looked at the meeting [1] notes where we agreed to take the Change-proposal route: 14:19:36 <sgallagh> mhroncok: Once it's fleshed out, I can submit it as a Self-Contained Change, if you like 14:19:45 <sgallagh> For ELN 14:19:53 <Son_Goku> sgallagh, I think that's acceptable 14:20:03 <nirik> +1 14:20:04 <mhroncok> sgallagh: I wasn't going to "require" this, but I would certainly appreciate that, thanks 14:20:04 <ignatenkobrain> sgallagh: yeah, that probably would be the best... so it follows our standard processes 14:20:04 <Son_Goku> we can also approve it essentially out of band since ELN rolls 14:20:09 <mhroncok> sgallagh++ 14:20:09 <zodbot> mhroncok: Karma for sgallagh changed to 7 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 14:20:36 <ignatenkobrain> #action sgallagh to submit Self-Contained Change Proposal once ready [1] https://meetbot-raw.fedoraproject.org/teams/fesco/fesco.2020-06-24-14.04.... Second, for the sake of discussion, even if Miro did request this, with all the respect I have for Miro, his opinion is just that ... one opinion. Right now the text *is* misleading, and we shouldn't accept that just because somebody (maybe) in the past said something. I guess that "shorter-than-RHEL-lifetime" is the big differentiator, i.e. normal rpms cannot be yanked from the distribution, but a module can be. But technically there isn't much difference, and it's only policy that sets those two cases apart. So instead of using default modules, why not adjust the policy and use non-modular rpms with plain Obsoletes? (In fact, this simpler approach could be argued to be better, since the technology to put Obsoletes in rpms is well established and understood and works nicely, while stream Obsoletes are only being conceived.) [...] Does this last point mean that modular content in Rawhide can contain packages which are available only as modular content? The important part is whether those issues have been resolved. As we all know, *plans* in software engineering are worth even less than in other human endeavors. You brought up Unicode in Python, i.e. the transition from 2 to 3. It is a good example: we waited 10 years to make python3 the default while *the implementation* was being improved, even though PEP3000 back in 2006 had all the great ideas and plans. > > OTOH, why don't just let the ELN SIG do this if it doesn't affect "Fedora > > proper"? Because I think it does. Since the beginning of the ELN project, it has > > been said that it ships Fedora rawhide content, built differently. Yet if we > > ship the default modular streams content in Fedora ELN and the nonmodular > > defaults in Fedora Rawhide, suddenly we ship different things, unless we have > > technical means to ensure the content is synchronized. > > > > I'm... not really sure what you're comparing here, but I'll try to > make it clearer. > > > Does that make more sense? > > > When the ELN proposal was discussed, several package maintainers (both from > > Fedora and RHEL) proposed to have an eln branch. It received a big NO because > > the content would diverge and keeping it in sync would be (close to) impossible. > > > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o... > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o... > > ..and many times more... > > > > Fedora ELN should not be a fork. Yet (some of the) default content of Fedora ELN > > would be built from modules, which packages built from different branches. > > Has the objective been changed? Is having separate branches for Fedora and > > Fedora ELN no longer a big NO? What changed? > > See the previous statement. The module streams for ELN will be built > from the exact same YAML as the equivalent Rawhide module stream. Thus > what is being delivered is still "Rawhide content", it just happens to > differ about which is the *default content*. > > > I am worried about the divergence between Fedora and Fedora ELN. I am worried > > about certain maintainers only maintaining their modules without upgrading > > packages in rawhide > > I feel that this is highly unlikely since ELN won't be a deliverable > in the traditional sense of the term. If they were to do this, their > package would stagnate in Fedora. If that happens, that's what the > non-responsive maintainer process is for. > > > -- and a need of a large volunteer-driven effort to backport > > improvements and fixes from modules to nonmodular packages: From Fedora ELN to > > Fedora. This seems wrong to me, as I've always considered Fedora to be the > > upstream of RHEL (and by extension of Fedora ELN) rather than downstream. > > I guess we haven't been clear enough about the artifacts that ELN > produces. They are intended as a compose for testing/CI purposes > *only*. We will not be mirroring them or making them publicly visible > on getfedora.org or anything of the sort. It's a bootstrapping effort > towards RHEL X+1. > > I intend to make it VERY clear that working in ELN but not maintaining > that content in the rest of Fedora is unacceptable. That's part of the > reason behind the package comaintainership requirement in this policy. I accept the fact that this will probably be approved, because there is such a strong desire to do so from a small but important group of contributors. But neither the motivation why this is needed is clear, nor is the wording of the proposal very appealing. Zbyszek > > When I expressed this concern in the "RHEL 9 and modularity" thread, the > > discussion went nowhere: > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o... > > (and replies) > > > > > > I believe that allowing default modular streams in any Fedora project is a bad > > thing to do before the technology sees large design-level changes. I believe > > that RHEL completely ignoring Fedora's feedback is a very sad thing to happen. I > > believe that having default modular streams in ELN will negatively affect > > general Fedora packagers. But I realize that this is juts one person's opinion. > > At FESCo, I try to represent the opinion of Fedora package maintainers, not just > > my own. Hence I am sharing my concerns here to hear feedback from others. > > I understand your stance on the matter at large. I hope that my > responses above have provided you with at least some measure of > improved clarity.

On Fri, Aug 21, 2020 at 07:55:57PM +0200, Miro Hrončok wrote: Modules are not removed from RHEL repositories either. The difference is that in RHEL they stop being supported. That's something what Fedora has not yet experienced. Bedides EPEL. I think that packages in EPEL are sometimes rebased to incompatible versions and what happen with the unneded dependencies of the old versions is not clear to me. I think they are marked as a dead package in dist-git and blocked from Koji. But do they disappear from the repository? If the stable repository is based on Koji blocked status, then then should the removed. -- Petr

On Mon, Aug 24, 2020 at 08:45:49AM -0400, Stephen Gallagher wrote: Well, that confirms what I wrote in the part that was snipped: I'll ask again: why not? Zbyszek

On Mon, Aug 24, 2020 at 09:02:55AM -0400, Stephen Gallagher wrote: We use fedora-obsolete-packages to remove packages from end-user systems. Users can opt-out of installation of fedora-obsolete-packages and retain packages that would be obsoleted [*]. The same mechanism could be used RHEL, for example by having 'rhel-is-up-to-date.rpm' installed by default, with newer versions doing the obsoletes for packages that have been dropped. Users *may* stop the upgrade of rhel-is-up-to-date, but then they know their system is using outdated packages. DNF will even nicely tell them which ones. This is: a) simple, b) well-understood, c) already implemented. Zbyszek [*] Right now fedora-obsolete-packages has Provides:libsolv-self-destruct-pkg(), so this muddies the situation a bit. Let's assume that the packages in RHEL would not have that set.

On 20. 08. 20 20:17, Stephen Gallagher wrote: I would appreciate if you could stop saying what I have done or requested when in fact this was not the case. If it was, please provide quotes. Never have I requested a proposal for "Fedora" that we could initially apply only to "Fedora ELN". I requested that if you want default modular streams in ELN, that you propose it via regular channels and discuss this with the wider community. You have volunteered to do a Fedora change proposal. You did (except the proposal said nothing about default modular streams in ELN). The conversation was logged and is already posted elsewhere in this thread. I am genuinely insulted by your false claim. It is hard enough to argue about the technical side of things, but impossible when you paint me as the unreasonable troll that just changes the narrative every time you make a step forward. I understand that your position is complicated and I block you. But please use fair discussion over calumny. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

On 21. 08. 20 15:29, Stephen Gallagher wrote: I want a policy for Fedora (and I see ELN as part of Fedora) that says: "Default streams are not allowed." We already have that policy. In the IRC meeting line you quoted I expressed my feeling that the policy also applies (and should continue to apply) to ELN, because ELN **is** Fedora. This has been my view on this topic since the original FESCo ticket. I don't understand how should I be more ingenuous. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

On 21. 08. 20 16:38, Stephen Gallagher wrote: Thank you. I would not expect anything different. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

On Wed, 5 Aug 2020 15:36:53 -0400 Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote: I find that all the discussion of non-ELN Fedora and EPEL distract from the subject of ELN. The proposal would be clearer if it restricted itself to ELN and left non-ELN fedora for a separate proposal. I also have several questions that the proposal does not address. I believe that modularity has fundamental deep and hard design problems. My evidence for this is the several years of FESCO tickets. Many of these tickets were not resolved or resolved only with hacks, eventually compelling FESCO to restrict the scope of modularity. I am skeptical that these issues can be solved at the policy level. Please go back to the drawing board and don't push modularity at us until they're fixed. I think that build-only components are antithetical to Fedora's "Friends" foundation. It interferes with my ability to consume, rebuild, and modify the affected packages. I would like to see build-only content forbidden except for well-defined situations. This is especially important for default modules. Proposal: Build-only content is forbidden in default modules and is permitted in other modules only with FESCO exception if no other solution is possible. This is supposed to be a policy for ELN, yet this is the last time ELN is mentioned. I am unsure how the following is supposed to apply to ELN. And what about ELN? What standards apply to judging the change proposal? And what about ELN? Will this also need a change proposal with FESCO approval? Approved by whom? Also, what is the relation between ELN and rawhide? Are ELN default modules built from the same sources as the rawhide non-modular content? What if patches are needed or useful? Do ELN default modules have identical non-default rawhide modules? What happens when the rawhide package is rebuilt or upgraded? What if the upgrade is not compatible with a ELN default module? Are rawhide maintainers supposed to care about the corresponding ELN default modules? Are rawhide maintainers supposed to care about dependent ELN default modules? Are rawhide maintainers allowed to "clean up" their spec files to remove ELN "clutter"? What infrastructure tooling exists or is needed to implement this proposal? Can it be implemented in a timely manner. I recall several requests for tracing modular dependencies, which could not be answered easily. Has that been fixed? What is the current state of modularity with respect to .so name bumps, mass rebuilds, and orphaned and retired packages? How are users supposed to install ELN and test it and it modules? I understand ELN is not producing installers. This is not part of the policy, but it is still relevant, because the ELN as a test bed will be much more meaningful if ELN can be consumed at least as easily as rawhide. How are users and 3rd party packagers supposed to rebuild modular content? Where are the docs? As a user I consider this a hard requirement, and ask that FESCO do the same. How are users and 3rd party packagers supposed to build RPMS that depend on modular content? Where are the docs? As a user I consider this a hard requirement, and ask that FESCO do the same. Thanks, Jim