Hello Fedora people! I would like to offer a solution for dynamic changelog and release generation and potentially more. Intro ------ The solution employs a new type of macros, which can be placed into an rpm spec file and afterward parsed by preproc utility (https://pagure.io/rpkg-util/blob/master/f/preproc/preproc) which executes the macros (by using bash interpreter) and replaces the occurrences of the macros in the original file with standard output of those macros. The main use of those macros is intended to be reading git metadata and using it to automatically generate rpm fields like Name, Release, or %changelog but the macros are, in the end, a turing-complete tool so they can be potentially used to do more than that (if desired). The tags for those macros are {{{ and }}} and a header of an example spec file could then look like this: Name: {{{ git_dir_name }}} Version: 1.18 Summary: Remove old packages from rpm-md repository Release: {{{ git_dir_release_branched }}} The `git_dir_name` and `git_dir_release_branched` keywords are bash functions that already implement some logic for automatic generation of package name (from git repository name) and release (from commit and tag history), respectively. There is also `git_dir_changelog` macro to generate a changelog from content of annotated tags. This proposal, however, is not about usage of those particular macro functions (this can be discussed later as well as the usage of annotated tags in particular) but only about usage of the new macro class {{{ ... }}} recognized by the preproc utility. The reason why we need this new macro class and we couldn't rely just on shell expansion already offered by rpm is that %(…) (the shell expansion rpm tag) gets carried over verbatim into the spec file which is stored inside a built source rpm. If the code in such tag relied on git metadata it would work only when the srpm is originally built from dist-git sources but later when the source rpm gets rebuilt (e.g. internally in mock), the code would fail because at that point, the git metadata is already missing (they are not included in srpms). Therefore we need a new type of macros in a spec file that, when evaluated, are replaced with their stdout before the spec file is used to build an srpm. Implementation --------------------- I have some packages in reviews and ready for a review to provide the building stones. These are: https://bugzilla.redhat.com/show_bug.cgi?id=1809644 (preproc - the preprocessor) https://bugzilla.redhat.com/show_bug.cgi?id=1811432 (rpkg-macros - basic set of macros that can be used by packagers) https://bugzilla.redhat.com/show_bug.cgi?id=1811432 (preproc-rpmspec - simple tool that makes it easier to invoke preproc on a spec file with the rpkg-macros included) These packages are functional from fedora-rawhide down to epel-6. Only preproc is essential for this proposal but I would like to, at the same time, also provide some basic starting set of macros that packagers can easily try and comment on. Either we can stick with those macros, tweak them, or provide an entirely different set of macros that uses a different logic. Next, I was looking into koji code and found out that no changes should be actually needed there to support this. The changes, in my opinion, should be placed into mock here, just before the start of srpm build: https://github.com/rpm-software-management/mock/blob/5c08f5b74/mock/py/mo... together with adding three new configuration options: config_opts['rpkg_preprocessing'] = True # by default False config_opts['rpkg_preprocessing_requires'] = ["preproc-rpmspec"] config_opts['rpkg_preprocessing_cmd'] = '/usr/bin/preproc-rpmspec %(source_spec)s --output %(target_spec)s' The first one `rpkg_preprocessing` tells us if the feature should be enabled or not (by default, it will be probably False in mock). The second one tells us what package should be installed into the chroot before the start of srpm build to provide the preprocessing functionality. And the third one is a concrete invocation of a preprocessing command. The logic in mock (at the corresponding place) should go as follows: if rpkg_preprocessing is enabled; then if rpkg.conf file is present in git toplevel directory; then if the rpkg.conf file contains `preprocess_spec = True` directive under [rpkg] section; then install `rpkg_preprocessing_requires` into the chroot run the preprocessing command as specified by the config And these changes in mock (apart from the above packages being brought into Fedora and EPEL) are the only changes that need to be done for the solution to already work in Fedora (and EPEL). I scanned master branches of all packages in Fedora dist-git and none of them contains rpkg.conf file in the git top-level directory so there should be no conflict there (i can do the checks on other branches too but at this point, I am already quite certain that 'rpkg.conf' is a non-conflicting filename). I have run all spec files from master through preproc and all of them came out unchanged except two: python-dns-lexicon.spec - uses {{{ }}} in comments python-suds.spec - uses {{{ }}} in changelog at one place All the other packages are just left untouched after the preproc run (by the way, that wouldn't be the case for {{, }} tags if used - there are many more conflicts there). Those two packages would need to escape the {{{ }}} tags to keep using them if they wanted to also start using the preprocessing functionality by placing rpkg.conf file into dist-git repo with the following content: [rpkg] preprocess_spec = True I already have this working on my machine so now it is a matter of opening a PR and polishing the code. Basically, I tested it on a dist-git repo (on prunerepo package, specifically) where I added rpkg.conf with the above content and used some preproc macros in the spec file and I was able to successfully build an srpm with a preprocessed spec file by invoking: mock --buildsrpm --spec prunerepo.spec --sources . This is pretty much what koji does when building from SCM (it just adds more switches to the above command). Implementation continued ----------------------------------- The next thing that needs to be adjusted is the mass-rebuild script (https://pagure.io/releng/blob/master/f/scripts/mass-rebuild.py) and rpmdev-bumpspec (https://github.com/rsrchboy/rpmdevtools/blob/master/rpmdev-bumpspec), however also here the changes should be quite minimal. My plan is to tweak rpmdev-bumpspec to return a dedicated exit code when it encounters: Release: {{{ ... }}} and not to do any changes on release or changelog then with the assumption that the macros will do it themselves. The mass rebuild script when being given this exit code will not error out but instead continue by adding a new commit (https://pagure.io/releng/blob/682804c8/f/scripts/mass-rebuild.py#_162) and building as usual. This is not a 100% percent solution because one could e.g. do this in a spec file: {{{{ echo "Release: $(git_dir_release_branched)" }}} in which case regular expression checking e.g. for "Release: {{{" wouldn't work. It might be therefore better to check for the rpkg.conf file and check if it contains `preprocess_spec = True` in the [rpkg] section. In either case, we can make this work by communicating with packagers the following: "You are free to use preproc macros but if a new commit lands, the Release field needs to change based on that". That's also the behavior of `git_dir_release` and `git_dir_release_branched` macros included in rpkg-macros library. Implementation continued continued ------------------------------------------------ If the mock changes arrive to koji builders, you will be already able to play around with the macros and I recommend using the latest version of rpkg-utility (https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/) that can interpret the macros client-side and tell you exactly what the koji builders produce before you push. Anyway, I would like to get proper support for this in fedpkg. I think we could immediately start supporting this on the level of the rpkg library (https://pagure.io/rpkg) so that the other dist-git clients get this functionality as well (in opt-in manner). This is not a difficult task - i have already done the support in rpkg-util so I (or maybe someone else) will just open a pull request which should port the code from rpkg-util to the rpkg library. Implementation finished? --------------------------------- Is there anything else I should think about or some other scripts in infra that should be adjusted? About the approach -------------------------- - the preproc macros (together with rpkg-macros) are already supported in Copr when you use SCM method with rpkg srpm build method - the approach is generic in the sense that it makes it very easy to start generating other parts of spec files from git (or possibly other) content: e.g. VCS field, Source field by packing the whole dist-git repo content into a tarball, and %prep field - it depends on what community and packaging committee decide is acceptable - the approach gives packagers the same freedom they have with rpm macros. While there can be some preconfigured macros, everybody can roll out their own variants or entirely new macros - the approach is suitable to implement "Git is the single source of truth" paradigm at least with respect to the basic package information like Release, Version or Changelog. - the approach needs very little changes in our tooling - it produces the same results locally which will be produced by build system for a given commit - guys from "Ideas and proposal for removing changelog and release fields from spec file" are currently trying something else which involves build system and, from my POV, magic spec file manipulations but I still hope we can find a common language at some point and finally start cooperating Thanks clime