F37 proposal: Python: Add -P to default shebangs (System-Wide Change proposal)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/PythonSafePath
== Summary ==
The [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P `-P`
flag] will be added to the Python shebang macros
(`%{py3_shbang_opts}`, `%{py3_shebang_flags}`, ...). Packages that
adhere to those macros will change their Python shebangs from `#!
/usr/bin/python3 -s` to `#! /usr/bin/python3 -sP` and as a result,
will no longer have the directory of the script (such as `/usr/bin`)
in `sys.path`. An opt-out mechanism exists.
== Owner ==
* Name: [[User:Churchyard|Miro Hrončok]], [[User:Vstinner|Victor Stinner]]
* Email: python-maint(a)redhat.com
== Detailed Description ==
All Python 3 shebang RPM macros will be changed to contain one more
flag: `-P`. Previously, they contained `-s`, now they will contain
`-sP`.
From the [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
documentation for the `-P` option]:
:Don’t prepend a potentially unsafe path to `sys.path`:
:
:* `python -m module` command line: Don’t prepend the current working directory.
:* `python script.py` command line: Don’t prepend the script’s
directory. If it’s a symbolic link, resolve symbolic links.
:* `python -c code` and `python` (REPL) command lines: Don’t prepend
an empty string, which means the current working directory.
In shebangs, only the middle option (''don’t prepend the script’s
directory'') is relevant.
Consider the following executbale script installed as
`/usr/bin/let-there-be-fun`:
#! /usr/bin/python3 -s
import abc
...
When the script is directly executed (e.g. by running
`let-there-be-fun` from the console), the script's directory
(`/usr/bin`) is prepended to `sys.path`. Python tries to locate an
importable `abc` module in `/usr/bin` first. This can cause real
issues: [https://bugzilla.redhat.com/2057340 python3-notebook:
ImportError: bad magic number in six] and
[https://github.com/benjaminp/six/issues/359 bad magic number in six].
When the shebang includes `-P`:
#! /usr/bin/python3 -sP
import abc
...
...the script's directory (`/usr/bin`) is '''not''' prepended to
`sys.path`. The change owners consider this approach safer for the
majority of Fedora's RPM packages.
By default, '''all standardly RPM-packaged Python packages with
scripts in `/usr/bin` will gain the `-P` flag in their shebang''',
assuming the software is packaged in a way that respects the Python
shebang RPM macros (see below for opt-out and explicit opt-in
mechanisms). Due to the variety of ways such scripts can be
created/packaged, there will likely be packages that will not be
affected by the change automatically. (In other words, the change is
applied on RPM macro level, no added mechanics to force the flag, such
as BRP scripts, are planned as part of this change.)
=== List of RPM macros that will gain `-P` ===
* `%{py3_shbang_opts}`
* `%{py3_shbang_opts_nodash}`
* `%{py3_shebang_flags}`
* `%{py_shbang_opts}`
* `%{py_shbang_opts_nodash}`
* `%{py_shebang_flags}`
=== Opting out ===
If the new behavior is not desirable to your package amend the macros
(e.g. with `sed`) to remove the `P` flag.
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
current Python packaging guidelines], e.g. `%pyproject_wheel` and
`%pyproject_install`, use:
# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own
rationale here)
%global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//)
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python packaging guidelines], e.g. `%py3_build` and
`%py3_install`, use:
# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own
rationale here)
%global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//)
(The only difference is the name of the macro.)
=== Opting in ===
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
current Python packaging guidelines], e.g. `%pyproject_wheel` and
`%pyproject_install`, the standard set of Python shebang flags is
applied to all files with Python shebangs installed in `/usr/bin/`.
If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python packaging guidelines], e.g. `%py3_build` and
`%py3_install`, the standard set of Python shebang flags might be
applied to some files and not applied to others depending on the exact
structure of the packaged software.
If you wish to explicitly apply the standard set of Python shebang
flags on a certain file that is not handled automatically, use
[https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#py3_she...
the `%py3_shebang_fix` macro].
=== What if the packager changes `%__python3` to an older version of Python ===
The `-P` flag was introduced in Python 3.11. When `%__python3` is
redefined to an older Python version, e.g. `/usr/bin/python3.10`,
including the `-P` flag in shebangs would break the scripts. Hence,
the flag will be included conditionally, presumably somehow like this:
<nowiki>%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P'
if hasattr(sys.flags, 'safe_path') else '')")</nowiki>
=== What if the admin/user changes `/usr/bin/python3` to an older
version of Python ===
The `-P` flag was introduced in Python 3.11. When an admin/user
changes `/usr/bin/python3` to point to an older version of Python,
e.g. `/usr/bin/python3.10`, including the `-P` flag in shebangs would
break the scripts.
However, changing `/usr/bin/python3` to a different Python would
''brick'' a Fedora system even now. So we don't consider that an
issue. See an example that changes `/usr/bin/python3` to Python 3.9
(don't try this at home):
[root@086a2804411a /]# head -n1 /usr/bin/dnf
#!/usr/bin/python3
[root@086a2804411a /]# dnf --version
4.12.0
...
[root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3
[root@086a2804411a /]# dnf --version
Traceback (most recent call last):
File "/usr/bin/dnf", line 61, in <module>
from dnf.cli import main
ModuleNotFoundError: No module named 'dnf'
=== Risks ===
As with any other change, there is a risk that this will break things.
The change owners plan to test the change extensively via Copr before
they deploy the change in Rawhide. If things go badly, they are
prepared to delay or cancel the change.
The [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python RPM macros] set the shebang flags by a weird hack. As
a result, it is not well-defined what scripts will be affected by this
change. The change owners are aware that:
# not all Python scripts in `/usr/bin` will have the `-P` flag automatically
# some scripts '''not''' in `/usr/bin` might gain the `-P` flag as well
The first point is an acceptable gradual deployment of the default
flag. The second point is not very dangerous because we don't except
users to directly execute Python scripts via shebangs when such
scripts are not in `$PATH`. If a problematic package is found, it can
opt-out easily. If this causes too much friction, we will only change
the flag used in the `%pyproject_install` macro, leaving packages with
the "legacy" macros intact.
== Feedback ==
* Relevant upstream discussion: [https://discuss.python.org/t/13896
Should console_scripts entry points exclude the scripts directory from
sys.path?]
* Adding `-P` to Python:
[https://github.com/python/cpython/issues/57684 GitHub issue],
[https://mail.python.org/archives/list/python-dev@python.org/thread/IU5Q2A...
email thread]
* [https://github.com/rpm-software-management/dnf/pull/1815#pullrequestrevie...
dnf maintainers response for doing this explicitly in dnf first]
Generally, people seem to think that this is a good thing. They are
afraid to change the default behavior of Python but welcome using this
flag for system-installed scripts.
== Benefit to Fedora ==
Python programs in `/usr/bin` will be less fragile to other random
files being present in `/usr/bin`.
Real hard-to-debug issues like [https://bugzilla.redhat.com/2057340
python3-notebook: ImportError: bad magic number in six] and
[https://github.com/benjaminp/six/issues/359 bad magic number in six]
will not happen.
== Scope ==
* Proposal owners:
** Test everything in Copr
** If everything works, change the flags either before the Python 3.11
rebuild or before the Fedora 37 Mass Rebuild
** Provide guidance to packagers, fix bugs if needed
* Other developers:
** Observe their packages, find and report bugs, opt-out if needed
** Volunteerily opt-in by calling the `%py3_shebang_fix` macro and/or
by converting their packages to `%pyproject_install`
* Release engineering: [https://pagure.io/releng/issue/10784 #10784]
* Policies and guidelines: The new flag needs to be documented in the
Python packaging guidelines (old and new)
* Trademark approval: not needed for this Change
* Alignment with Objectives: no
== Upgrade/compatibility impact ==
No impact is anticipated.
== How To Test ==
* Low level: Examine the value of the changed RPM macros, it should
contain the `-P` flag
* Middle level: Examine the shebang lines of RPM-installed Python
scripts in `/usr/bin`, it should contain the `-P` flag
* High level: Tests that RPM-installed Python scripts still behave as
expected but don't try to import stuff from `/usr/bin`
== User Experience ==
Users of RPM-installed scripts should get a safer experience by default.
Users of Python should not observe a difference, the behavior is not a
new default: the `-P` flag needs to be explicitly used.
== Dependencies ==
We need [[Changes/Python3.11]] first.
== Contingency Plan ==
* Contingency mechanism: defer to F38; only add `-P` to shebangs in
`%pyproject_install` to keep backward compatibility of the old macros;
revert and rebuild
* Contingency deadline: 1 week before the beta freeze
* Blocks release? No
== Documentation ==
* This page
* TBD Updated Python guidelines
* https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
* https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights
(Security improvements)
== Release Notes ==
TBD
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1 year, 9 months
F37 Change: Drop i686 builds of jdk8,11,17 and latest (18) rpms from
f37 onwards (System-Wide Change)
by Ben Cotton
https://fedoraproject.org/wiki/Changes/Drop_i686_JDKs
== Summary ==
java-1.8.0-openjdk, java-11-openjdk, java-17-openjdk and
java-latest-openjdk packages will no longer build i686 subpackages
== Owner ==
* Name: [[User:jvanek| Jiri Vanek]]
* Email: <jvanek(a)redhat.com>
* Product: java and java stack
* Responsible WG: java-sig (java and java-maint)(which no longer exists)
=== Expected schedule ===
* during march, drop i686 builds from all jdks in fedora rawhide
== Detailed Description ==
Fedora currently ships:
* java-1.8.0-openjdk (LTS)
* java-11-openjdk (LTS)
* java-17-openjdk (LTS)
* java-latest-openjdk (STS, jdk18).
All those builds on all architectures except jdk8, where arm32 with
jit is built by different package.
Unluckily, the i686 bit builds of jdk are rotten in upstream. The
recent breakage of i686 JIT just before branching nearly killed jdk17
as system jdk feature.
The rotting have main visibility with newer GCCs. If GCC bump, and it
does, it always triggers new issues in i686 JIT, and there is less and
less people to somehow workaround them. Unluckily, there is probably
no longer anyone willing to really fix them
== Benefit to Fedora ==
The i686 builds are rotten in usptream, and to patch them localy had
become pain. We may be introducing very bugy i686 jdk. Better then to
do so, we would rather not ship that at all.
This will untie hands of both JDK and GCC developers, who will no
longer need to dive into nasty legacy code.
== Scope ==
==== Change owners ====
* we will simiply stop building i686 pkg in rawhide
==== Other developers ====
* may notice the multilib i686 java missing.
* it is up to them to drop i686 builds or to povide workaround (if possible)
==== Other ====
* Release engineering: https://pagure.io/releng/issue/10686
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
* The upgrade on multilib systems will lead to autoremoval of i686 javastack
* which should be minimum - 99% of javastack is noarch
== How To Test ==
install i686 java will result to not packages found
== User Experience ==
User experience on multilib systems will be bad. Bad reasonable.
== Dependencies ==
There are is unknown number of multilib java consumers. I expect some
of them may rise voice, but that will have to handled one by one.
== Contingency Plan ==
* Contingency mechanism: return i686 packages
* Contingency date: (not provided)
== Documentation ==
Will be neded...
== Release Notes ==
None yet...
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
1 year, 9 months
[HEADS UP] -Wl,--as-needed is added in rawhide
by Igor Gnatenko
It's in redhat-rpm-config-118-1.fc30.
If it causes any problems for you - let me know. In the meantime, you can
use `%undefine _ld_as_needed` to disable it.
Thanks for attention!
--
-Igor Gnatenko
1 year, 9 months
`ansible` Package License Change
by Maxwell G
The license of `ansible` 2.9.x has been corrected from `GPLv3+` to `GPLv3+ and BSD and Python and MIT and ASL 2.0`. The previous `License:` tag did not properly account for the multi-licensing.
Please note that this only applies to EPEL and Fedora 34 and 35. The license of `ansible` 5.x (the collections bundle), which is available on Fedora 36 and above, remains the same.
--
Thanks,
Maxwell G (@gotmax23)
Pronouns: He/Him/Him
1 year, 10 months
Heads up: openjpeg2-2.5.0 and gdal-3.5.0 coming to rawhide
by Sandro Mani
Hi
I'll build openjpeg2-2.5.0 and gdal-3.5.0 for rawhide in the side tag
f37-build-side-53899. I've switched gdal to the new cmake based build,
and at the same time merged the mingw package with the native one. I'll
rebuild affected packages as listed below.
Thanks
Sandro
---
lizams
bes
blender
cloudcompare
compat-ffmpeg4
darktable
dcm2niix
eccodes
efl
engauge-digitizer
ffmpeg
freeimage
gazebo
gdal
gdcm
geeqie
ghostscript
gimp
GMT
gpac
grass
gstreamer1-plugins-bad-free
ImageMagick
liblas
librasterlite2
mapnik
mapserver
merkaartor
mingw-freeimage
mingw-gstreamer1-plugins-bad-free
mingw-opencv
mingw-poppler
mtpaint
mupdf
ncl
opencv
OpenImageIO
OpenSceneGraph
openslide
osgearth
paraview
PDAL
poppler
postgis
python-fiona
python-pillow
python-PyMuPDF
python-rasterio
qgis
qmapshack
qsstv
R-rgdal
saga
vfrnav
vips
vtk
webkit2gtk3
zathura-pdf-mupdf
1 year, 10 months
PSA: openQA failures in tests using Firefox
by Adam Williamson
Hi, folks. You may be noticing more failures than usual in openQA tests
that use Firefox. This is caused by a rather vexing combination of
Firefox wanting to show "helpful" popups on startup:
https://bugzilla.mozilla.org/show_bug.cgi?id=1703903
and what I think is some kind of odd regression in Firefox 100 which
makes typing into some text entry boxes in the Cockpit and FreeIPA web
UIs not work reliably (this affects realmd_join_cockpit and
upgrade_realmd_client).
I *think* I've figured out a way to deal with the popups for now (touch
wood, testing it right now). I'm still kinda stuck on the typing
problem. It's very late here now and I'll look into it more tomorrow. I
do apologize if an update is blocked on such a failure, I've been
restarting failed tests constantly all day but they keep hitting these
failure modes. I'll do my best to have it sorted by tomorrow one way or
another, and if I can't, I'll adjust the greenwave policy to
temporarily disable gating on the problematic tests.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
1 year, 10 months
News from printing world aka PWG May 2022 meetup
by Zdenek Dohnal
Hi all,
it is another year and we have an aggregated report of what changed in
the printing world since the last year.
The whole notes are as the attachment, but highlights are:
- myself (Zdenek Dohnal) being release manager for CUPS 2.4.x series
- Till Kamppeter wrote printer applications which covers all printer
drivers in Debian distribution - we don't have any additional printer
driver package in Fedora, so all our driver packages are covered as well
- printer applications (the solution for driver-only printers how to
work with IPP-only CUPS) are available as SNAPs in Fedora (feel free to
try them and leave feedback at the respective OpenPrinting project
https://github.com/orgs/OpenPrinting/repositories ), packaging them as
RPMs is blocked due dependency on cups-filters 2.0, which is not
released yet (though IIRC someone from Fedora community - maybe Brandon
Nielsen - has them in copr)
- in case of proprietary drivers which aren't packaged in OS there is a
legacy printer application inside of pappl-retrofit project, which, once
you install the printer in this printer application with the proprietary
driver you need, gives the printer IPP Everywhere interface, which makes
it visible for CUPS
- flatpak isn't designed for system services such as CUPS and printer
applications, so Till will investigate shipping containerized printer
applications via OCI containers on dockerhub
Enjoy!
--
Zdenek Dohnal
Software Engineer
Red Hat, BRQ-TPBC
1 year, 10 months
