Good Morning Everyone,
A little while ago[1], we integrated anitya in dist-git itself, allowing to
stop using fedora-scm-request's[2] git repository to store this information.
However, this git repository is still being used to store bugzilla overrides
(i.e.: default assignee on bugzilla ticket when they differ from the point of
contact (main admin) of the package in dist-git).
Together with Karsten Hopp we worked on integrating this functionality on
pagure-dist-git[3], thus allowing to get rid entirely of the git repository at
fedora-scm-request[2].
This work has been deployed in staging today. We would very much appreciate if
you could take a few minute of your time and see if it works to your
liking: https://src.stg.fedoraproject.org/
The overrides information from production has been migrated yesterday to the
staging dist-git, so what you see in the UI reflects the current state of the
overrides in production as of yesterday.
Here is an example with an override:
https://src.stg.fedoraproject.org/rpms/0ad
One note: in the rpms namespace, the UI will always show you the default
assignee for Fedora and Fedora EPEL, regardless of whether the package is in
EPEL.
This is a shortcoming we are aware of and will be looking at fixing in the near
future but potentially after it has reached production.
Thank you for your understanding and help testing this,
Pierre
[1] https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapr…
[2] https://pagure.io/releng/fedora-scm-requests
[3] https://pagure.io/pagure-dist-git/
_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapr…
Hello,
I've orphaned python-pep8. pep8 was renamed to pycodestyle in 2016; it
received its last release in 2017. It should be removed from Fedora in a
future release.
I unfortunately don't have time to proceed with the full retirement
process myself. If somebody would like to pick it up:
https://fedoraproject.org/wiki/Orphaned_package_that_need_new_maintainers#C…
$ dnf repoquery --whatrequires python2-pep8
python2-autopep8-0:1.2.4-9.fc29.noarch
python2-pytest-pep8-0:1.0.6-15.fc29.noarch
python2-spyder-0:3.3.1-3.fc29.noarch
$ dnf repoquery --whatrequires python3-pep8
python3-autopep8-0:1.2.4-9.fc29.noarch
python3-hacking-0:1.1.0-3.fc29.noarch
python3-pytest-pep8-0:1.0.6-15.fc29.noarch
python3-spyder-0:3.3.1-3.fc29.noarch
See also https://bugzilla.redhat.com/show_bug.cgi?id=1667200's dependent
bugs.
(Please CC me on replies that need my attention.)
--
iliana weller <ilianaw(a)buttslol.net>
Hello Fedora people!
I would like to offer a solution for dynamic changelog and release
generation and potentially more.
Intro
------
The solution employs a new type of macros, which can be placed into an
rpm spec file and afterward parsed by preproc utility
(https://pagure.io/rpkg-util/blob/master/f/preproc/preproc) which
executes the macros (by using bash interpreter) and replaces the
occurrences of the macros in the original file with standard output of
those macros.
The main use of those macros is intended to be reading git metadata
and using it to automatically generate rpm fields like Name, Release,
or %changelog but the macros are, in the end, a turing-complete tool
so they can be potentially used to do more than that (if desired).
The tags for those macros are {{{ and }}} and a header of an example
spec file could then look like this:
Name: {{{ git_dir_name }}}
Version: 1.18
Summary: Remove old packages from rpm-md repository
Release: {{{ git_dir_release_branched }}}
The `git_dir_name` and `git_dir_release_branched` keywords are bash
functions that already implement some logic for automatic generation
of package name (from git repository name) and release (from commit
and tag history), respectively. There is also `git_dir_changelog`
macro to generate a changelog from content of annotated tags.
This proposal, however, is not about usage of those particular macro
functions (this can be discussed later as well as the usage of
annotated tags in particular) but only about usage of the new macro
class {{{ ... }}} recognized by the preproc utility.
The reason why we need this new macro class and we couldn't rely just
on shell expansion already offered by rpm is that %(…) (the shell
expansion rpm tag) gets carried over verbatim into the spec file which
is stored inside a built source rpm. If the code in such tag relied on
git metadata it would work only when the srpm is originally built from
dist-git sources but later when the source rpm gets rebuilt (e.g.
internally in mock), the code would fail because at that point, the
git metadata is already missing (they are not included in srpms).
Therefore we need a new type of macros in a spec file that, when
evaluated, are replaced with their stdout before the spec file is used
to build an srpm.
Implementation
---------------------
I have some packages in reviews and ready for a review to provide the
building stones. These are:
https://bugzilla.redhat.com/show_bug.cgi?id=1809644 (preproc - the preprocessor)
https://bugzilla.redhat.com/show_bug.cgi?id=1811432 (rpkg-macros -
basic set of macros that can be used by packagers)
https://bugzilla.redhat.com/show_bug.cgi?id=1811432 (preproc-rpmspec
- simple tool that makes it easier to invoke preproc on a spec file
with the rpkg-macros included)
These packages are functional from fedora-rawhide down to epel-6.
Only preproc is essential for this proposal but I would like to, at
the same time, also provide some basic starting set of macros that
packagers can easily try and comment on. Either we can stick with
those macros, tweak them, or provide an entirely different set of
macros that uses a different logic.
Next, I was looking into koji code and found out that no changes
should be actually needed there to support this. The changes, in my
opinion, should be placed into mock here, just before the start of
srpm build: https://github.com/rpm-software-management/mock/blob/5c08f5b74/mock/py/mock…
together with adding three new configuration options:
config_opts['rpkg_preprocessing'] = True # by default False
config_opts['rpkg_preprocessing_requires'] = ["preproc-rpmspec"]
config_opts['rpkg_preprocessing_cmd'] = '/usr/bin/preproc-rpmspec
%(source_spec)s --output %(target_spec)s'
The first one `rpkg_preprocessing` tells us if the feature should be
enabled or not (by default, it will be probably False in mock). The
second one tells us what package should be installed into the chroot
before the start of srpm build to provide the preprocessing
functionality. And the third one is a concrete invocation of a
preprocessing command.
The logic in mock (at the corresponding place) should go as follows:
if rpkg_preprocessing is enabled; then
if rpkg.conf file is present in git toplevel directory; then
if the rpkg.conf file contains `preprocess_spec = True` directive
under [rpkg] section; then
install `rpkg_preprocessing_requires` into the chroot
run the preprocessing command as specified by the config
And these changes in mock (apart from the above packages being brought
into Fedora and EPEL) are the only changes that need to be done for
the solution to already work in Fedora (and EPEL).
I scanned master branches of all packages in Fedora dist-git and none
of them contains rpkg.conf file in the git top-level directory so
there should be no conflict there (i can do the checks on other
branches too but at this point, I am already quite certain that
'rpkg.conf' is a non-conflicting filename).
I have run all spec files from master through preproc and all of them
came out unchanged except two:
python-dns-lexicon.spec - uses {{{ }}} in comments
python-suds.spec - uses {{{ }}} in changelog at one place
All the other packages are just left untouched after the preproc run
(by the way, that wouldn't be the case for {{, }} tags if used - there
are many more conflicts there).
Those two packages would need to escape the {{{ }}} tags to keep using
them if they wanted to also start using the preprocessing
functionality by placing rpkg.conf file into dist-git repo with the
following content:
[rpkg]
preprocess_spec = True
I already have this working on my machine so now it is a matter of
opening a PR and polishing the code. Basically, I tested it on a
dist-git repo (on prunerepo package, specifically) where I added
rpkg.conf with the above content and used some preproc macros in the
spec file and I was able to successfully build an srpm with a
preprocessed spec file by invoking:
mock --buildsrpm --spec prunerepo.spec --sources .
This is pretty much what koji does when building from SCM (it just
adds more switches to the above command).
Implementation continued
-----------------------------------
The next thing that needs to be adjusted is the mass-rebuild script
(https://pagure.io/releng/blob/master/f/scripts/mass-rebuild.py) and
rpmdev-bumpspec
(https://github.com/rsrchboy/rpmdevtools/blob/master/rpmdev-bumpspec)
however also here the changes should be quite minimal.
My plan is to tweak rpmdev-bumpspec to return a dedicated exit code
when it encounters:
Release: {{{ ... }}}
and not to do any changes on release or changelog then with the
assumption that the macros will do it themselves. The mass rebuild
script when being given this exit code will not error out but instead
continue by adding a new commit
(https://pagure.io/releng/blob/682804c8/f/scripts/mass-rebuild.py#_162)
and building as usual.
This is not a 100% percent solution because one could e.g. do this in
a spec file:
{{{{ echo "Release: $(git_dir_release_branched)" }}} in which case
regular expression checking e.g. for "Release: {{{" wouldn't work. It
might be therefore better to check for the rpkg.conf file and check if
it contains `preprocess_spec = True` in the [rpkg] section.
In either case, we can make this work by communicating with packagers
the following: "You are free to use preproc macros but if a new commit
lands, the Release field needs to change based on that". That's also
the behavior of `git_dir_release` and `git_dir_release_branched`
macros included in rpkg-macros library.
Implementation continued continued
------------------------------------------------
If the mock changes arrive to koji builders, you will be already able
to play around with the macros and I recommend using the latest
version of rpkg-utility
(https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/) that can
interpret the macros client-side and tell you exactly what the koji
builders produce before you push.
Anyway, I would like to get proper support for this in fedpkg. I think
we could immediately start supporting this on the level of the rpkg
library (https://pagure.io/rpkg) so that the other dist-git clients
get this functionality as well (in opt-in manner).
This is not a difficult task - i have already done the support in
rpkg-util so I (or maybe someone else) will just open a pull request
which should port the code from rpkg-util to the rpkg library.
Implementation finished?
---------------------------------
Is there anything else I should think about or some other scripts in
infra that should be adjusted?
About the approach
--------------------------
- the preproc macros (together with rpkg-macros) are already supported
in Copr when you use SCM method with rpkg srpm build method
- the approach is generic in the sense that it makes it very easy to
start generating other parts of spec files from git (or possibly
other) content: e.g. VCS field, Source field by packing the whole
dist-git repo content into a tarball, and %prep field - it depends on
what community and packaging committee decide is acceptable
- the approach gives packagers the same freedom they have with rpm
macros. While there can be some preconfigured macros, everybody can
roll out their own variants or entirely new macros
- the approach is suitable to implement "Git is the single source of
truth" paradigm at least with respect to the basic package information
like Release, Version or Changelog.
- the approach needs very little changes in our tooling
- it produces the same results locally which will be produced by build
system for a given commit
- guys from "Ideas and proposal for removing changelog and release
fields from spec file" are currently trying something else which
involves build system and, from my POV, magic spec file manipulations
but I still hope we can find a common language at some point and
finally start cooperating
Thanks
clime
Hello,
Just a general request: could more community folks please spend a few
minutes a day to check AskFedora[1]. With the beta release, we're
already seeing some users asking questions, which often turn out to be
bugs. So, if the package maintainers in the community can please keep an
eye on these, that'll be helpful.
https://ask.fedoraproject.org
--
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London
I have upgraded to Fedora 32 today and after a while I have noticed that
`repoquery --whatrequires` yields incomplete results.
From Fedora 31:
$ dnf --refresh repoquery --repo=rawhide{,-source} --whatrequires python3-flaky
pipenv-0:2018.11.26-13.fc32.src
python-ipykernel-0:5.1.4-1.fc33.src
From Fedora 32/33:
$ dnf --refresh repoquery --repo=rawhide{,-source} --whatrequires python3-flaky
python-ipykernel-0:5.1.4-1.fc33.src
I have reported this here: https://bugzilla.redhat.com/show_bug.cgi?id=1812596
Be extra careful when taking decisions based on `repoquery --whatrequires` (such
as: I can safely retire this or upgrade that, nothing else requires it).
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup
Simplified version of another change proposal|This change was
originally proposed for [[Releases/32|Fedora 32]] as
[[Changes/PythonStaticSpeedup]], however based on
[https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org…
community feedback], it has been significantly reduced.
== Summary ==
We add the <code>-fno-semantic-interposition</code> compiler/linker
flag when building Python interpreters, as it provides significant
performance improvement, up to 27% depending on the workload. Users
will no longer be able to use LD_PRELOAD to override a symbol from
libpython, which we consider a good trade off for the speedup.
== Owner ==
* Name: [[User:Cstratak| Charalampos Stratakis]], [[User:Vstinner|
Victor Stinner]], [[User:Churchyard| Miro Hrončok]]
* Email: python-maint(a)redhat.com
* Shout-out: [[User:Jankratochvil|Jan Kratochvíl]] for first
suggesting this instead of the original proposal, followed by
[[User:Kkofler|Kevin Kofler]]. [[User:Fweimer|Florian Weimer]] for
providing answers to our questions. David Gray for originally
suggesting to link Python statically to gain performance.
== Detailed Description ==
When we build the Python interpreter with the
<code>-fno-semantic-interposition</code> compiler/linker flag, we can
achieve a performance gain of 5% to 27% depending on the workload.
Link time optimizations and profile guided optimizations also have a
greater impact when python3 is built this way.
As a negative side effect, it disables the LD_PRELOAD feature: it's no
longer possible to override symbols in libpython with LD_PRELOAD.
Interposition is enabled by default in compilers like GCC: function
calls to a library goes through a "Procedure Linkage Table" (PLT).
This indirection is required to allow a library loaded by LD_PRELOAD
environment variable to override a function. The indirection puts more
pressure on the CPU level 1 cache (instruction cache). In term of
performance, the main drawback is that function calls from a library
to the same library cannot be inlined, to respect the interposition
semantics. Inlining is usually a big win in term of performance.
Disabling interposition for libpython removes the overhead on function
calls by avoiding the PLT indirection, and allows to inline more
function calls. We're describing function calls from libpython to
libpython, something which is very common in Python: almost all
function calls are calls from libpython to libpython.
If Fedora users need to use LD_PRELOAD to override symbols in
libpython, the recommend way is to build a custom Python without
<code>-fno-semantic-interposition</code>.
It is still possible to use LD_PRELOAD to override symbols in other
libraries (for example in glibc).
=== Affected Pythons ===
Primarily, we will change the interpreter in the {{package|python3}}
package, that is Python 3.8 in Fedora 32 and any later version of
Python in future Fedora releases.
Impact on other Python packages (and generally software using Python)
is not anticipated (other than the possible speedup).
We will also change the
[https://developer.fedoraproject.org/tech/languages/python/multiple-pythons.…
alternate Python interpreters] where possible and useful, primarily
the upstream supported versions of CPython, such as
{{package|python39}} (if already packaged), {{package|python37}} and
{{package|python36}}.
=== Affected Fedora releases ===
This is a Fedora 32 change and it will be implemented in Rawhide
(Fedora 32) only. Any future versions of Fedora will inherit the
change until it is reverted for some reason.
If it turns out that there are absolutely no issues, we might consider
backporting the speedup to already released Fedora versions (for
example Fedora 31). Such action would be separately coordinated with
[https://docs.fedoraproject.org/en-US/fesco/ FESCo].
== Benefit to Fedora ==
Python's performance will increase significantly depending on the
workload. Since many core components of the OS also depend on Python
this could lead to an increase in their performance as well, however
individual benchmarks will need to be conducted to verify the
performance gain for those components.
[https://pyperformance.readthedocs.io/ pyperformance] results,
ignoring differences smaller than 5%:
(See change proposal)
== Scope ==
* Proposal owners:
** Review and merge the
[https://src.fedoraproject.org/rpms/python3/pull-request/151 pull
request with the implementation].
** Monitor Koschei for significant problems.
** Backport the change to alternate Python versions.
* Other developers are encouraged to check if their package works as expected
* Release engineering: N/A (not needed for this Change) -- this change
does not require a mass rebuild nor any other special releng work
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
Python package maintainers should verify that their packages work as
expected and the only impact the end users should see is a performance
increase for workloads relying on Python.
== How To Test ==
Test that everything Python related in Fedora works as usual.
=== Was the flag applied test ===
You can test whether the <code>-fno-semantic-interposition</code> flag
was applied for your Python build:
<pre>
>>> import sysconfig
>>> '-fno-semantic-interposition' in (sysconfig.get_config_var('PY_CFLAGS') + sysconfig.get_config_var('PY_CFLAGS_NODIST'))
True
>>> '-fno-semantic-interposition' in (sysconfig.get_config_var('PY_LDFLAGS') + sysconfig.get_config_var('PY_LDFLAGS_NODIST'))
True
</pre>
Before the change, you would see <code>False</code>, <code>False</code>.
=== Performance test ===
The performance speedup can be measured using the official Python
benchmark suite [https://pyperformance.readthedocs.io/ pyperformance]:
see [https://pyperformance.readthedocs.io/usage.html#run-benchmarks
Run benchmarks].
== User Experience ==
Python based workloads should see a performance gain of up to 27%.
== Dependencies ==
This change is not dependent on anything else.
== Contingency Plan ==
* Contingency mechanism: If issues appear that cannot be fixed in a
timely manner the change can be easily reverted and will be considered
again for the next fedora release.
* Contingency deadline: Before the beta freeze of Fedora 32 (2020-02-25)
* Blocks release? Yes
* Blocks product? None
== Documentation ==
This change proposal has all the documentation.
See the [[Changes/PythonStaticSpeedup|previous change proposal]] and
the [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org…
thread about it on the devel mailing list] for more relevant
information about what we are not doing
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
Good Morning Everyone,
Tomorrow we are planning on deploying a new version of pagure and
pagure-dist-git on production.
These changes come with two changes to the packager workflow:
* Anitya integration in dist-git
Something we lost when loosing pkgdb was the easy integration to anitya
(https://release-monitoring.org) With the coming changes we are getting them
back.
On the left hand-side column, there will be a drop-down button allowing to
change the settings for anitya for the project.
Existing status will be migrated from the fedora-scm-requests repo on pagure to
use this drop-down.
Using the fedora-scm-requests repo for the anitya integration will no longer be
supported.
* Change in the de-orphaning process
Currently if a package is orphaned, one has to open a ticket against the releng
project to adopt it. With these changes, anyone will be able to adopt orphaned
projects (not retired on master) directly from dist-git's UI.
If the project is retired or has been orphaned for too long, a ticket on the
releng project will still be required though.
Both of these changes can already be reviewed in staging at:
https://src.stg.fedoraproject.org
Looking forward for your feedback!
Pierre
_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapr…
There will be an outage starting at 2020-02-23 06:00 UTC, which will last
approximately 7 hours (but can take more if something goes wrong).
To convert UTC to your local time, take a look at
https://fedoraproject.org/wiki/UTCHowto
or run:
date -d '2020-02-23 06:00 UTC'
Reason for outage:
Moving Copr servers from OpenStack to Amazon AWS because OpenStack is
going to be shut down soon.
Affected Services:
copr-frontend - https://copr.fedorainfracloud.org
copr-backend - https://copr-be.cloud.fedoraproject.org/
copr-distgit
copr-keygen
Backend repositories should remain available during the outage.
Ticket Link:
https://pagure.io/fedora-infrastructure/issue/8668
Contact Information:
Please join #fedora-admin in irc.freenode.net or add comments to the
ticket for this outage above.
Jakub