Proposal to enable spec file preprocessing step before srpm build
by clime
Hello Fedora people!
I would like to offer a solution for dynamic changelog and release
generation and potentially more.
Intro
------
The solution employs a new type of macros, which can be placed into an
rpm spec file and afterward parsed by preproc utility
(https://pagure.io/rpkg-util/blob/master/f/preproc/preproc) which
executes the macros (by using bash interpreter) and replaces the
occurrences of the macros in the original file with standard output of
those macros.
The main use of those macros is intended to be reading git metadata
and using it to automatically generate rpm fields like Name, Release,
or %changelog but the macros are, in the end, a turing-complete tool
so they can be potentially used to do more than that (if desired).
The tags for those macros are {{{ and }}} and a header of an example
spec file could then look like this:
Name: {{{ git_dir_name }}}
Version: 1.18
Summary: Remove old packages from rpm-md repository
Release: {{{ git_dir_release_branched }}}
The `git_dir_name` and `git_dir_release_branched` keywords are bash
functions that already implement some logic for automatic generation
of package name (from git repository name) and release (from commit
and tag history), respectively. There is also `git_dir_changelog`
macro to generate a changelog from content of annotated tags.
This proposal, however, is not about usage of those particular macro
functions (this can be discussed later as well as the usage of
annotated tags in particular) but only about usage of the new macro
class {{{ ... }}} recognized by the preproc utility.
The reason why we need this new macro class and we couldn't rely just
on shell expansion already offered by rpm is that %(…) (the shell
expansion rpm tag) gets carried over verbatim into the spec file which
is stored inside a built source rpm. If the code in such tag relied on
git metadata it would work only when the srpm is originally built from
dist-git sources but later when the source rpm gets rebuilt (e.g.
internally in mock), the code would fail because at that point, the
git metadata is already missing (they are not included in srpms).
Therefore we need a new type of macros in a spec file that, when
evaluated, are replaced with their stdout before the spec file is used
to build an srpm.
Implementation
---------------------
I have some packages in reviews and ready for a review to provide the
building stones. These are:
https://bugzilla.redhat.com/show_bug.cgi?id=1809644 (preproc - the preprocessor)
https://bugzilla.redhat.com/show_bug.cgi?id=1811432 (rpkg-macros -
basic set of macros that can be used by packagers)
https://bugzilla.redhat.com/show_bug.cgi?id=1811432 (preproc-rpmspec
- simple tool that makes it easier to invoke preproc on a spec file
with the rpkg-macros included)
These packages are functional from fedora-rawhide down to epel-6.
Only preproc is essential for this proposal but I would like to, at
the same time, also provide some basic starting set of macros that
packagers can easily try and comment on. Either we can stick with
those macros, tweak them, or provide an entirely different set of
macros that uses a different logic.
Next, I was looking into koji code and found out that no changes
should be actually needed there to support this. The changes, in my
opinion, should be placed into mock here, just before the start of
srpm build: https://github.com/rpm-software-management/mock/blob/5c08f5b74/mock/py/mo...
together with adding three new configuration options:
config_opts['rpkg_preprocessing'] = True # by default False
config_opts['rpkg_preprocessing_requires'] = ["preproc-rpmspec"]
config_opts['rpkg_preprocessing_cmd'] = '/usr/bin/preproc-rpmspec
%(source_spec)s --output %(target_spec)s'
The first one `rpkg_preprocessing` tells us if the feature should be
enabled or not (by default, it will be probably False in mock). The
second one tells us what package should be installed into the chroot
before the start of srpm build to provide the preprocessing
functionality. And the third one is a concrete invocation of a
preprocessing command.
The logic in mock (at the corresponding place) should go as follows:
if rpkg_preprocessing is enabled; then
if rpkg.conf file is present in git toplevel directory; then
if the rpkg.conf file contains `preprocess_spec = True` directive
under [rpkg] section; then
install `rpkg_preprocessing_requires` into the chroot
run the preprocessing command as specified by the config
And these changes in mock (apart from the above packages being brought
into Fedora and EPEL) are the only changes that need to be done for
the solution to already work in Fedora (and EPEL).
I scanned master branches of all packages in Fedora dist-git and none
of them contains rpkg.conf file in the git top-level directory so
there should be no conflict there (i can do the checks on other
branches too but at this point, I am already quite certain that
'rpkg.conf' is a non-conflicting filename).
I have run all spec files from master through preproc and all of them
came out unchanged except two:
python-dns-lexicon.spec - uses {{{ }}} in comments
python-suds.spec - uses {{{ }}} in changelog at one place
All the other packages are just left untouched after the preproc run
(by the way, that wouldn't be the case for {{, }} tags if used - there
are many more conflicts there).
Those two packages would need to escape the {{{ }}} tags to keep using
them if they wanted to also start using the preprocessing
functionality by placing rpkg.conf file into dist-git repo with the
following content:
[rpkg]
preprocess_spec = True
I already have this working on my machine so now it is a matter of
opening a PR and polishing the code. Basically, I tested it on a
dist-git repo (on prunerepo package, specifically) where I added
rpkg.conf with the above content and used some preproc macros in the
spec file and I was able to successfully build an srpm with a
preprocessed spec file by invoking:
mock --buildsrpm --spec prunerepo.spec --sources .
This is pretty much what koji does when building from SCM (it just
adds more switches to the above command).
Implementation continued
-----------------------------------
The next thing that needs to be adjusted is the mass-rebuild script
(https://pagure.io/releng/blob/master/f/scripts/mass-rebuild.py) and
rpmdev-bumpspec
(https://github.com/rsrchboy/rpmdevtools/blob/master/rpmdev-bumpspec),
however also here the changes should be quite minimal.
My plan is to tweak rpmdev-bumpspec to return a dedicated exit code
when it encounters:
Release: {{{ ... }}}
and not to do any changes on release or changelog then with the
assumption that the macros will do it themselves. The mass rebuild
script when being given this exit code will not error out but instead
continue by adding a new commit
(https://pagure.io/releng/blob/682804c8/f/scripts/mass-rebuild.py#_162)
and building as usual.
This is not a 100% percent solution because one could e.g. do this in
a spec file:
{{{{ echo "Release: $(git_dir_release_branched)" }}} in which case
regular expression checking e.g. for "Release: {{{" wouldn't work. It
might be therefore better to check for the rpkg.conf file and check if
it contains `preprocess_spec = True` in the [rpkg] section.
In either case, we can make this work by communicating with packagers
the following: "You are free to use preproc macros but if a new commit
lands, the Release field needs to change based on that". That's also
the behavior of `git_dir_release` and `git_dir_release_branched`
macros included in rpkg-macros library.
Implementation continued continued
------------------------------------------------
If the mock changes arrive to koji builders, you will be already able
to play around with the macros and I recommend using the latest
version of rpkg-utility
(https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/) that can
interpret the macros client-side and tell you exactly what the koji
builders produce before you push.
Anyway, I would like to get proper support for this in fedpkg. I think
we could immediately start supporting this on the level of the rpkg
library (https://pagure.io/rpkg) so that the other dist-git clients
get this functionality as well (in opt-in manner).
This is not a difficult task - i have already done the support in
rpkg-util so I (or maybe someone else) will just open a pull request
which should port the code from rpkg-util to the rpkg library.
Implementation finished?
---------------------------------
Is there anything else I should think about or some other scripts in
infra that should be adjusted?
About the approach
--------------------------
- the preproc macros (together with rpkg-macros) are already supported
in Copr when you use SCM method with rpkg srpm build method
- the approach is generic in the sense that it makes it very easy to
start generating other parts of spec files from git (or possibly
other) content: e.g. VCS field, Source field by packing the whole
dist-git repo content into a tarball, and %prep field - it depends on
what community and packaging committee decide is acceptable
- the approach gives packagers the same freedom they have with rpm
macros. While there can be some preconfigured macros, everybody can
roll out their own variants or entirely new macros
- the approach is suitable to implement "Git is the single source of
truth" paradigm at least with respect to the basic package information
like Release, Version or Changelog.
- the approach needs very little changes in our tooling
- it produces the same results locally which will be produced by build
system for a given commit
- guys from "Ideas and proposal for removing changelog and release
fields from spec file" are currently trying something else which
involves build system and, from my POV, magic spec file manipulations
but I still hope we can find a common language at some point and
finally start cooperating
Thanks
clime
3 years, 11 months
HEADS UP: repoquery --whatrequires yields incomplete results when run
from Fedora 32+
by Miro Hrončok
I have upgraded to Fedora 32 today and after a while I have noticed that
`repoquery --whatrequires` yields incomplete results.
From Fedora 31:
$ dnf --refresh repoquery --repo=rawhide{,-source} --whatrequires python3-flaky
pipenv-0:2018.11.26-13.fc32.src
python-ipykernel-0:5.1.4-1.fc33.src
From Fedora 32/33:
$ dnf --refresh repoquery --repo=rawhide{,-source} --whatrequires python3-flaky
python-ipykernel-0:5.1.4-1.fc33.src
I have reported this here: https://bugzilla.redhat.com/show_bug.cgi?id=1812596
Be extra careful when taking decisions based on `repoquery --whatrequires` (such
as: I can safely retire this or upgrade that, nothing else requires it).
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 years, 11 months
Fedora 32 Self-Contained Change proposal: Build Python with
-fno-semantic-interposition for better performance
by Ben Cotton
https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup
Simplified version of another change proposal|This change was
originally proposed for [[Releases/32|Fedora 32]] as
[[Changes/PythonStaticSpeedup]], however based on
[https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
community feedback], it has been significantly reduced.
== Summary ==
We add the <code>-fno-semantic-interposition</code> compiler/linker
flag when building Python interpreters, as it provides significant
performance improvement, up to 27% depending on the workload. Users
will no longer be able to use LD_PRELOAD to override a symbol from
libpython, which we consider a good trade off for the speedup.
== Owner ==
* Name: [[User:Cstratak| Charalampos Stratakis]], [[User:Vstinner|
Victor Stinner]], [[User:Churchyard| Miro Hrončok]]
* Email: python-maint(a)redhat.com
* Shout-out: [[User:Jankratochvil|Jan Kratochvíl]] for first
suggesting this instead of the original proposal, followed by
[[User:Kkofler|Kevin Kofler]]. [[User:Fweimer|Florian Weimer]] for
providing answers to our questions. David Gray for originally
suggesting to link Python statically to gain performance.
== Detailed Description ==
When we build the Python interpreter with the
<code>-fno-semantic-interposition</code> compiler/linker flag, we can
achieve a performance gain of 5% to 27% depending on the workload.
Link time optimizations and profile guided optimizations also have a
greater impact when python3 is built this way.
As a negative side effect, it disables the LD_PRELOAD feature: it's no
longer possible to override symbols in libpython with LD_PRELOAD.
Interposition is enabled by default in compilers like GCC: function
calls to a library goes through a "Procedure Linkage Table" (PLT).
This indirection is required to allow a library loaded by LD_PRELOAD
environment variable to override a function. The indirection puts more
pressure on the CPU level 1 cache (instruction cache). In term of
performance, the main drawback is that function calls from a library
to the same library cannot be inlined, to respect the interposition
semantics. Inlining is usually a big win in term of performance.
Disabling interposition for libpython removes the overhead on function
calls by avoiding the PLT indirection, and allows to inline more
function calls. We're describing function calls from libpython to
libpython, something which is very common in Python: almost all
function calls are calls from libpython to libpython.
If Fedora users need to use LD_PRELOAD to override symbols in
libpython, the recommend way is to build a custom Python without
<code>-fno-semantic-interposition</code>.
It is still possible to use LD_PRELOAD to override symbols in other
libraries (for example in glibc).
=== Affected Pythons ===
Primarily, we will change the interpreter in the {{package|python3}}
package, that is Python 3.8 in Fedora 32 and any later version of
Python in future Fedora releases.
Impact on other Python packages (and generally software using Python)
is not anticipated (other than the possible speedup).
We will also change the
[https://developer.fedoraproject.org/tech/languages/python/multiple-python...
alternate Python interpreters] where possible and useful, primarily
the upstream supported versions of CPython, such as
{{package|python39}} (if already packaged), {{package|python37}} and
{{package|python36}}.
=== Affected Fedora releases ===
This is a Fedora 32 change and it will be implemented in Rawhide
(Fedora 32) only. Any future versions of Fedora will inherit the
change until it is reverted for some reason.
If it turns out that there are absolutely no issues, we might consider
backporting the speedup to already released Fedora versions (for
example Fedora 31). Such action would be separately coordinated with
[https://docs.fedoraproject.org/en-US/fesco/ FESCo].
== Benefit to Fedora ==
Python's performance will increase significantly depending on the
workload. Since many core components of the OS also depend on Python
this could lead to an increase in their performance as well, however
individual benchmarks will need to be conducted to verify the
performance gain for those components.
[https://pyperformance.readthedocs.io/ pyperformance] results,
ignoring differences smaller than 5%:
(See change proposal)
== Scope ==
* Proposal owners:
** Review and merge the
[https://src.fedoraproject.org/rpms/python3/pull-request/151 pull
request with the implementation].
** Monitor Koschei for significant problems.
** Backport the change to alternate Python versions.
* Other developers are encouraged to check if their package works as expected
* Release engineering: N/A (not needed for this Change) -- this change
does not require a mass rebuild nor any other special releng work
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
Python package maintainers should verify that their packages work as
expected and the only impact the end users should see is a performance
increase for workloads relying on Python.
== How To Test ==
Test that everything Python related in Fedora works as usual.
=== Was the flag applied test ===
You can test whether the <code>-fno-semantic-interposition</code> flag
was applied for your Python build:
<pre>
>>> import sysconfig
>>> '-fno-semantic-interposition' in (sysconfig.get_config_var('PY_CFLAGS') + sysconfig.get_config_var('PY_CFLAGS_NODIST'))
True
>>> '-fno-semantic-interposition' in (sysconfig.get_config_var('PY_LDFLAGS') + sysconfig.get_config_var('PY_LDFLAGS_NODIST'))
True
</pre>
Before the change, you would see <code>False</code>, <code>False</code>.
=== Performance test ===
The performance speedup can be measured using the official Python
benchmark suite [https://pyperformance.readthedocs.io/ pyperformance]:
see [https://pyperformance.readthedocs.io/usage.html#run-benchmarks
Run benchmarks].
== User Experience ==
Python based workloads should see a performance gain of up to 27%.
== Dependencies ==
This change is not dependent on anything else.
== Contingency Plan ==
* Contingency mechanism: If issues appear that cannot be fixed in a
timely manner the change can be easily reverted and will be considered
again for the next fedora release.
* Contingency deadline: Before the beta freeze of Fedora 32 (2020-02-25)
* Blocks release? Yes
* Blocks product? None
== Documentation ==
This change proposal has all the documentation.
See the [[Changes/PythonStaticSpeedup|previous change proposal]] and
the [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
thread about it on the devel mailing list] for more relevant
information about what we are not doing
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
3 years, 11 months
Announcing new anitya integration and de-orphaning process
by Pierre-Yves Chibon
Good Morning Everyone,
Tomorrow we are planning on deploying a new version of pagure and
pagure-dist-git on production.
These changes come with two changes to the packager workflow:
* Anitya integration in dist-git
Something we lost when loosing pkgdb was the easy integration to anitya
(https://release-monitoring.org). With the coming changes we are getting them
back.
On the left hand-side column, there will be a drop-down button allowing to
change the settings for anitya for the project.
Existing status will be migrated from the fedora-scm-requests repo on pagure to
use this drop-down.
Using the fedora-scm-requests repo for the anitya integration will no longer be
supported.
* Change in the de-orphaning process
Currently if a package is orphaned, one has to open a ticket against the releng
project to adopt it. With these changes, anyone will be able to adopt orphaned
projects (not retired on master) directly from dist-git's UI.
If the project is retired or has been orphaned for too long, a ticket on the
releng project will still be required though.
Both of these changes can already be reviewed in staging at:
https://src.stg.fedoraproject.org
Looking forward for your feedback!
Pierre
_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedora...
4 years
python-msgpack
by Fabian Affolter
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi all,
A month ago was msgpack 1.0.0 released. I'm planning to update the
package in rawhide in the near future.
This should start a conversation about a coordinated approach to keep
the update smooth for all packages which depends on python-msgpack.
Especially borgbackup as there is a restriction in place that will
break installations.
Kind regards,
Fabian
-----BEGIN PGP SIGNATURE-----
iF0EARECAB0WIQQvbJMP08R+OGr6TrTiPNLdNqQ5fwUCXnnltQAKCRDiPNLdNqQ5
f7t6AJ9j7JdEqUqp3YXfPB4PGX04HunqqACgjhzp4EmVy644q+nzcSTP1J4W+mY=
=D1+a
-----END PGP SIGNATURE-----
4 years
Fedora 33 System-Wide Change proposal: Introduce module Obsoletes and EOL
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Module_Obsoletes_and_EOL
== Summary ==
Fix Fedora upgradability issues when upgrading systems with module
streams enabled.
== Owner ==
* Name: [[User:dmach| Daniel Mach]]
* Email: dmach(a)redhat.com
== Detailed Description ==
DNF currently doesn't have sufficient information to perform system
upgrades of systems with enabled module streams correctly.
To solve upgradability problems, we will add additional modulemd metadata.
This includes information about a stream obsoleting another one or a
stream being EOLed.
Use Cases:
* A module (all its streams) is removed from a distribution.
* A stream has reached its EOL and must no longer be available.
** The stream packages have no replacements and must be removed from
the system (an extension of fedora-obsolete-packages?)
** The stream packages get replaced with non-modular packages
* A stream has reached its EOL and is Obsoleted by another stream.
* A stream has reached its EOL and a user wants to continue using it
regardless of EOL/Obsoletes.
* A stream has reached its EOL and a user wants to install a new host
with the EOLed stream enabled.
There is going to be a new DNF config option:
* (name: TBD) - automatically follow stream EOL/Obsoletes; only print
a warning if turned off
Proposed new modulemd document:
<pre>
document: TBD
version: 1
data:
# A string representing UTC date in ISO 8601 format: YYYY-MM-DD[T ]HH:MMZ
# When merging, entries with a newer 'modified' value will override
any earlier values.
# MANDATORY
modified: 2020-03-19T23:26Z
# A boolean option to cancel/reset all previously specified obsoletes
# Example: repo 'fedora' has Obsoletes:nodejs:12; we want to bring
nodejs:12 back in 'updates'
# If used, following options will be ignored: eol_date, obsoleted_by
# OPTIONAL
<TBD>: <bool>
# A string representing a Name of a module that is EOLed
# MANDATORY
module: nodejs
# A string representing a Stream of a module that is EOLed
# MANDATORY
stream: 11
# A string representing a Context of a module that is EOLed
# If not specified, all contexts get EOLed.
# NOTE: consider specifying a list of contexts
# OPTIONAL
context: aabbccddee
# A string representing UTC date in ISO 8601 format: YYYY-MM-DD[T ]HH:MMZ
# It is strongly recommended to keep HH:MM to 00:00.
# If not specified, the module is EOLed immediately.
OPTIONAL
eol_date: 2020-03-19T00:00Z
# A string describing the change, reason, etc.
# MANDATORY
message: "Module stream nodejs:11 is no longer supported. It is
recommended to switch to nodejs:12"
# If a stream is not EOLed but Obsoleted, provide details about the
obsoleting stream:
# OPTIONAL
obsoleted_by:
module: nodejs
stream: 12
</pre>
== Benefit to Fedora ==
Seamless system upgrades of systems with module streams enabled.
== Scope ==
* Proposal owners:
** Introduce modularity features to:
** Obsolete a module stream
** Stream EOL
** Design a new modulemd documents for Obsoletes and EOL
** Get the new documents supported by libmodulemd
** Implement the new functionality in the DNF stack (libdnf, dnf,
microdnf) and PackageKit
* Other developers:
** Follow updated packaging policy. See the "Policies and guidelines" section.
* Release engineering: [https://pagure.io/releng/issue/9359] (a check
of an impact with Release Engineering is needed) <!-- REQUIRED FOR
SYSTEM WIDE CHANGES -->
Maintain and distribute new module metadata.
* Policies and guidelines:
** Packaging policy and modularity documents require a change:
*** When a module stream gets removed from a Fedora release, the
maintainer of the module stream must provide a modulemd document with
Obsoletes or EOL data.
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
This feature solves upgradability problems of systems with module
streams enabled.
== How To Test ==
Stream Obsoletes:
* Enable a stream that is not available in the next version of Fedora
and is obsoleted by another stream in the metadata
* Run system upgrade
* Verify that the obsoleting stream is enabled instead of the obsoleted one
Stream EOL:
* Enable a stream that is not available in the next version of Fedora
and is obsoleted in its module metadata
* Run system upgrade
* Verify that the EOLed stream is no longer enabled and does not show
up in `dnf module list`
Fedora upgradability:
* Install Fedora 31 or 32
* For each $stream in available streams
** dnf module reset '*'
** dnf module enable $stream
** dnf distro-sync --releasever=33 --assumeno
** # the command must not throw any errors related to modularity,
especially related to the enabled stream
== User Experience ==
Fedora users experience upgradability problems when upgrading to Fedora 32
when they have any module streams enabled.
If a stream no longer exists on the version of Fedora they are upgrading to,
DNF used to error out on resolving modular dependencies which was a
clear release blocker.
To workaround this case, all module streams are reset during the system upgrade.
By doing this, modularity users lose information about enabled streams
and they need to re-enable them by hand.
This Change aims at removing the upgradability problems and allowing
Fedora modularity users
to upgrade their systems while keeping their streams enabled, reset or
switched (obsoleted)
according to newly provided modular metadata.
== Dependencies ==
* libmodulemd
* libdnf
* dnf
* microdnf
* PackageKit
== Contingency Plan ==
* Contingency mechanism: Remove the new metadata from Fedora composes
and repositories. Fedora will stay at the current behavior.
* Contingency deadline: Beta Freeze
* Blocks release? Yes
== Documentation ==
TBD
== Release Notes ==
Modularity supports Obsoleting streams and stream EOL (end-of-life).
Enabled streams are no longer reset during the system upgrade.
When a stream reaches its EOL, it's reset.
When a stream is Obsoleted, it's switched to the obsoleting stream.
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
4 years
