Good news, module maintainers. I'm relieved to announce an availability of the new module packaging format, modulemd-packager, version 3. Skip to "WHAT IS CHANGING" section to see the main message. HISTORY ======= Up to now, you wrote a YAML document in modulemd-v2 format <https://raw.githubusercontent.com/fedora-modularity/libmodulemd/main/yaml...;: document: modulemd version: 2 data: name: perl-DBI stream: '1.643' license: module: - MIT dependencies: - buildrequires: platform: [] perl: [] requires: platform: [] perl: [] buildopts: rpms: macros: | %this_is_my_module 1 components: rpms: perl-DBI: rationale: API ref: f34 You built it and an output were multiple module builds, one for each combination of a Fedora release and a perl stream. E.g. this matrix: buildrequires: platform: [f35, f34] perl: [5.30, 5.32] requires: platform: [f35, f34] perl: [5.30, 5.32] was expaned into 4 unique builds by MBS, Module Build Service. Observe the last, "random" value, context: perl-DBI:1.643:3420210211145152:7f057cb7 perl-DBI:1.643:3420210211145152:a450835c perl-DBI:1.643:3520210211145152:2c956793 perl-DBI:1.643:3520210211145152:e12b6f3a These output modules used the same modulemd-v2 format. It was deemed that having the same input and output format is a benefit. WHY === But it turned out that DNF had difficulties to upgrade from an older version of a module to a newer one. The problem was that DNF could not identify which build upgrades which one. E.g. having a Fedora 35 system with these builds in a repository: 1 perl-DBI:1.643:3520210211145152:2c956793 * this one is installed 2 perl-DBI:1.643:3520210211145152:e12b6f3a 3 perl-DBI:1.643:3520210212105947:ab395794 4 perl-DBI:1.643:3520210212105947:c39b3af3 DNF did not know whether to upgrade to the 3rd build or the 4th one. Both of them have the highest version. But which one to use? Context value is no clue here because it differs from the older builds. This happens when a module is changing its build-requires. (Actually first we observed this on RHEL where the platform changes with each minor RHEL release.) Run-time dependencies are also of no help because they could also change. (We also observed this on RHEL. Original design used run-time dependencies for the selection.) Therefore DNF maintainers declared that this problem is undecidable and requested a change in Modularity. Modularity team together with DNF maintainers identified that a culprit is the random nature of the context and decided to sacrifice a stream expansion for making the context static and predictable. It was designed that the context value will be fully in hands of the module maintainers who will be able freely add, remove, and maintain a module upgrade path by the means of the context: Context Context Context A B ↓ ↓ Version 3520210211145152 3520210211145152 requires: perl:5.30 requires: perl:5.32 C (perl:5.34 added) ↓ ↓ ↓ Version 3520210212105947 3520210212105947 3520210212105947 (a new dep. requires: perl:5.30 requires: perl:5.32 requires: perl:5.34 added) requires: nginx:1.20 requires: nginx:1.20 requires: nginx:1.20 (perl:5.30 ended) ↓ ↓ Version 3520210930210943 3520210930210943 (nginx not requires: perl:5.32 requires: perl:5.34 req. anymore) ↓ ↓ WHAT IS CHANGING ================ Since there was no place for the contexts in the old module format, a new, input-only format "modulemd-packager", version 3, was designed and implemented in MBS and other tools. At the opportunity of the format change, some other pet peeves of the old format were solved. Changes in the format include: New document type and version: document: modulemd-packager version: 3 A "module" middle-field removed from the "license" section: license: - MIT A "dependencies" section replaced with a "configurations" section: configurations: - context: A platform: f34 buildrequires: perl: ['5.30'] requires: perl: ['5.30'] - context: B platform: f34 buildrequires: perl: ['5.32'] requires: perl: ['5.32'] - context: C platform: f35 buildrequires: perl: ['5.30'] requires: perl: ['5.30'] - context: D platform: f35 buildrequires: perl: ['5.32'] requires: perl: ['5.32'] The context value is a string and its alphabet and length is limited. All the contexts must be unique among a module stream. The plaform field is a scalar now. However, the stream values of the "buildrequires" and "requires" dependencies are a list. Although the specification requires a single value. A "buildopts" section was moved from /data to /data/configurations/*/context section. Therefore it's specific to a context and should be repeated if needed any time: configurations: - context: A platform: f34 buildrequires: perl: ['5.30'] requires: perl: ['5.30'] buildopts: rpms: macros: | %this_is_my_module A - context: B platform: f34 buildrequires: perl: ['5.32'] requires: perl: ['5.32'] buildopts: rpms: macros: | %this_is_my_module B And that's all. The complete specification of modulemd-packager-v3 format is at <https://raw.githubusercontent.com/fedora-modularity/libmodulemd/main/yaml...;. WHEN ==== Modularity team started to work on the new format a year a ago, the implementation was deployed to Fedora infrastructure few months ago and I've been waiting with the announcement until fixing the most serious bugs. That, I hope, happened last week. In other words, you can start using the new format right now. DO I HAVE TO MIGRATE MY MODULES? ================================ No, you don't have to. But Modularity people would like to see moving all Fedora content there. The old modulemd-v2 format is still supported by MBS. DNF handles outputs of both of them (the output format is distinguished with "static_context: true" field), but it behaves differently. With the old format, DNF can produce weird warnings or errors. Especially when your module is required by another module. Will the old format be supporterd forever? I don't know. Does have the new format some drawbacks? Yes, it has: It's longer than the previous one. You need to update it for each new Fedora release. You cannot run-require an unspecified (= any) module stream (perl:[]). (Though a DNF maintainer says it should not be a problem.) HOW TO MIGRATE ============== Read the "WHAT IS CHANGING" section of this e-mail message. Follow the specification if in doubts. When migrating your modules, you will have to come up with the context value. To preserve a compatibility with the old builds and to preserve the upgrade path, I strongly recommend reusing the old context values. Use Koji <https://admin.fedoraproject.org/pkgdb/package/MODULE/>;, or MBS <https://mbs.fedoraproject.org/module-build-service/2/module-builds/?name=...;, or "dnf module info MODULE:STREAM" command to locate the lastest version of the module, and pick a context matching the desired dependencies as depicted here: # dnf -q module info perl-DBI:1.643 Name : perl-DBI Stream : 1.643 Version : 3520210722203952 Context : e12b6f3a ------+ […] | Requires : perl:[5.32] ----+ | platform:[f35] --+ | | | | | The new YAML file: | | | | | | data: | | | configurations: | | | - context: 'e12b6f3a' <----|-|-+ platform: f35 <----+ | buildrequires: | perl: ['5.32'] | requires: | perl: ['5.32'] <------+ buildopts: You can validate your YAML files with "modulemd-validator --type modulemd-packager-v3 FILE" command. For a real example look at any perl* module. I've already migrated them. For instance this commit <https://src.fedoraproject.org/modules/perl-DBI/c/2200d21f0fab5295226d915c...;. FEEDBACK ======== General help on modularity <devel(a)lists.fedoraproject.org&gt;. Modularity design issues <https://pagure.io/modularity/issues>;. Fedora MBS instance issues <https://pagure.io/fedora-infrastructure/issues>;. MBS implementation bugs <https://pagure.io/fm-orchestrator/issues>;. DISCLAIMER ========== The examples in this document are simplified and sometimes redacted to better illustrate the discussed topic. I joined Modularity team way after designing this change. Therefore I cannot guarantee that all data provided here are accurate. I'd like to thank all the people who participated in this change and who helped me to understand it. Especially Martin Curlej, Jaroslav Mracek, Daniel Mach, and Mike McLean. -- Petr