Modularity is Dead, Long Live Modularity!

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Modularity is Dead, Long Live Modularity! ========================================= See this post in glorious technicolor at:https://communityblog.fedoraproject.org/modularity-dead-long-live-modu... Summary - ------- Fedora’s Modularity initiative aims to make it easy for packagers to create alternative versions of software and for users to consume those streams simply. We’ve been working on this for several years, resulting in the “Boltron” prototype this summer and the recent Fedora Modular Server beta. Feedback shows that these test releases didn’t meet the goal, and we’re incorporating that in a modified design which we think _will_. We plan to demo the new approach by DevConf.cz and FOSDEM. Retrospective - ------------- As you have probably read, the members of the Modularity Working Group and the Server Working Group decided that the current implementation of Modularity would not meet Fedora’s standard of excellence in time for a release in Fedora 27 — even with the extra time that the Fedora Council granted to the project. In particular, feedback received from the Fedora 27 Modular Server Beta release alerted us to several serious problems in the design. The most common feedback from users was: “How do I install package `foo` in Modular Server?”. In this case, `foo` ranged across a wide variety of software, including everything from the screen package to complex third-party applications. In that version of Modularity, a system was either all Modular or none of it was. To make software available in the fully-modular system, its packages needed to be part of a module — and unfortunately, we didn’t succeed in making many of those. The lack of available modules was a symptom of a larger problem with packager involvement. Some of this problem was due to disagreements with the implementation or design, and some of it was because creating modules turned out to be more time-consuming and had a steeper learning curve than we wanted. Additionally, the whole process was not obvious or well-socialized. The team produced a lot of documentation, but packagers needed to understand a whole lot of it before even getting started. As a result, all of the modules slated for delivery in F27 Modular Server were being developed by the Modularity team, a group that had limited expertise around many of the packages we desired to ship. There were other problems, of course. We didn’t really have a strong plan for how to handle upgrades from traditional deployments. We lacked a story for how third-party software should be loaded onto the system. And, we never got away from needing a “bootstrap” module (as explained below). So, we decided not to ship Modular Server in Fedora 27, and we activated our fallback plan of releasing Fedora 27 Server Edition in its classic incarnation at https://getfedora.org/server/. Moving Forward - -------------- With all of this in mind, we took Modularity back to the drawing board (or perhaps “chopping board”). We analyzed the various functionality that the design offered us and made some trade-offs. First and foremost, we gave up on the idea of a strictly-maintained, stable buildroot. Traditional Fedora builds are performed by building an RPM in a buildroot containing the latest packages available in the stable updates repository for a release (plus the overrides repo when needed to allow building against pre-release packages). With modularity, we hoped that we could define a small and specific buildroot which would be stable and maintained for the life of a release. We would then build every module against it. It turned out when we tried to implement this that it was basically impossible. It required us to find a point at which the entire buildroot could be used to build _itself_, a feat which has not been accomplished in Fedora for many years (if ever). Fedora packages are really only guaranteed to build successfully against a buildroot for which they have _already_ been built. If it builds against any other buildroot, this is a happy accident. There are many reasons for this; something in the dependency chain for one of the build dependencies may have broken, a bug may have been introduced in the compiler or linker resulting in an inability to continue, new default compiler flags may have been added that result in warnings becoming errors and so on. This was the easiest requirement to drop as it had never worked. We used a kludgy “bootstrap” module to work around this which provided a set of packages copied from non-Modular Fedora to be able to build the platform. In its place, **we will now build modules against the standard Fedora buildroot**. Since our buildroot now has access to everything that has been built for Fedora, we reconsidered another of our original goals that of providing a module to be the base platform upon which all of the other modules would depend. Originally, this was intended to define the provided API of the Base and indicate which portions of it should be considered stable. However, Fedora already has a stable updates policy for how things must act within a release. This policy has only a few outliers (Kernel, KDE, etc.), but for the most part, once we release the Beta, the platform API is stable for the life of the Fedora release. By relaxing this requirement, we realized that we could eliminate (or at least postpone) the requirement for having a defined platform module. What we decided instead is to treat Fedora’s “Everything” repository (essentially, the complete set of software available within a Fedora release) as the “platform module”, though the tooling will not report this content as a module. In practical terms, this means that **creators of modules will no longer need to go through the very painful process of tracking down which modules provide a dependency that they need**. Instead, they will be able to depend on the system version available in the Everything repo. If that version does not meet their needs, they will have several options described later. This provides us with several advantages, including **a straightforward upgrade path from a traditional deployment**, because we will retain the traditional repositories as well as a set of default modules. This means it will be possible to upgrade from a current Fedora 27 system to a modular Fedora 28 system without any special steps. In fact, this approach will also mean that modularity need not be limited to the Server Edition. Another practical advantage to this change is that **module-creation will become _significantly_ simpler**. Instead of a complex collection of a package and all of its dependencies, modules will now only need to describe the parts that differ from the base repository. For example, Fedora 28 will ship with the Node.js 8.x LTS release in the standard repository, and a module could be built to provide the 9.x experimental release as an option. We could also easily provide the older 6.x LTS release to support older applications. In these cases, we can ship very simple module definitions which just lists the dist-git branches matching the desired upstream releases. In fact, this will now be so simple that **we plan to provide tools to automate the creation of these modules**. Since most modules will now only require a single source package be in the components list, we plan to enable support for automatically building a single branch in dist-git for all active Fedora (and EPEL) releases. Even for more complex multi-package modules, the automatically-created module definitions provide an easy and obvious starting point. What Will It Look Like? - ----------------------- From an end-user’s perspective, **Fedora will ship with two sets of repositories.** One will be the traditional Fedora repositories (fedora, updates, and updates-testing) and the other will be a new set of repositories providing alternative and supplementary modules. We haven’t decided on a final name for these yet, so we will use the placeholder terms modular, modular-updates, and modular-updates-testing. With this design, **anyon who does not wish to access the additional versions of software provided by modules can disable the modular and modular-updates repositories and their system will function exactly as it does today**. Packages built with Fedora’s traditional process will be installed and managed from the regular fedora repository, as will default versions of packages which use the new process behind the scenes. **For anyone who wants access to additional versions of packages, these new module repositories will make them available**. Users will be able to interact with these new repositories by taking advantage of some new syntax in DNF, the same as was used in the Modular Server Beta. If a user wants to install a particular module stream, they can do `dnf install module[:stream[/profile]]` (e.g. `dnf install @nodejs:6`, which will install the default profile containing the nodejs and npm packages). If the user just wants to install whatever version is the default for this system, a `dnf install package` will continue to work as it always has. Conclusion - ---------- This refined plan offers an understandable, approachable, and deliverable future for Modularity. Packagers who don’t want to produce modules will be able to continue packaging exactly as they always have with **no modification** to their workflows. Those who want to provide alternative versions of software in a single release or to easily provide the same version across multiple releases will have new tools to simplify this. As the number of available modules grows, users of Fedora will have a much easier access to the exact version of software they want to accomplish their tasks. People doing rapid-prototyping can more easily access newer versions of packages and at the same time people running older applications can continue to access the older streams that they need.