python-devel

python-devel@lists.fedoraproject.org

934 discussions

Loudly deprecating setup.py-based macros %py3_build %py3_install
by Miro Hrončok 30 Sep '24

30 Sep '24

Hello Pythonistas, setuptools has deprecated executing setup.py (e.g. setup.py build/install) long time ago. It is only a matter of time when it won't be possible to use %py3_build, %py3_install, and %py3_build_wheel. Recently, the removal of setup.py test broke almost 200 packages. In Fedora, the "old" macros (%py3_build, %py3_install) are: 1. deprecated: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_deprecat… 2. yet allowed: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/ Considering the fact that 1.6k Python packages use the macros (while 2k use the %pyproject_ ones), we are in for a lot of trouble when setuptools removes the support for setup.py build/install. To prevent such disaster I'd like to: - emit loud deprecation warnings from the macros, possibly with 10 sec sleep - officially disallow https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/ A proof of concept for the deprecation is https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/180 Later, I'd like to experiment with https://github.com/packit/specfile to write a minimal automatic convertor from the old macros to the new. It won't be perfect and likely won't be able to ditch any manually listed BuildRequires at first, but at least it might allow us to mass update spec files while keeping them buildable. Let me know what you think about this. -- Miro Hrončok -- Phone: +420777974800 Fedora Matrix: mhroncok

3 5

%pyproject_buildrequires -r/-x: Attempt to read runtime dependencies from pyproject.toml?
by Karolina Surma 24 Sep '24

24 Sep '24

Hello, recently, we were suggested an improvement for %pyproject_buildrequires -r/-x. We could read the project's runtime dependencies (if they're not marked as `dynamic`) from pyproject.toml [project] table directly, without calling prepare_metadata_for_build_wheel or building the wheel to read the dependency metadata from it. Reading the metadata via prepare_metadata_for_build_wheel is already quite fast, however implementing that hook is optional for the build backends. When the hook is not available, we need to build the wheel, which can be potentially very slow and resource-hungry (imagine building the entire scipy project). Metadata in pyproject.toml is standardized (PEP 621), however there is no way of telling ahead of time if the build backend already supports that standard. See the details and discussion in bugzilla: https://bugzilla.redhat.com/2261939 Maxwell has raised some valid concerns there: - Other tools (build frontends, e.g. pip/build) call prepare_metadata_for_build_wheel, our macros would diverge in functionality compared to the rest of the landscape. - pyproject.toml's [project] may not be the source of metadata that the build backend uses. A project could switch to a build backend without PEP 621 support (e.g. poetry-core) and forget to remove the [project] table. - There can be potential differences between BuildRequires and Requires generators when one generates dependencies based on the pyproject.toml [project] table while the other on the packaged dist-info metadata. - Using macros to build on older systems: e.g. RHEL 9's old setuptools version silently ignores the pyproject.toml [project] table - %pyproject_buildrequires could still pull the dependency information from it, but the resulting package would be broken because it did not include those in the packaged dist-info metadata. One way to mitigate would be to make the proposed behavior opt-in only, with the possibility to either build wheel with -w option (already existing) or e.g. -p (now-proposed: reading from pyproject.toml) in case backend doesn't have prepare_metadata_for_build_wheel. However, this adds a layer of complexity for packagers and macros maintainers. The questions we'd love your input for: - Should %pyproject_buildrequires try to read dependencies from pyproject.toml first and fall back to calling hooks only if that fails? - Should %pyproject_buildrequires call the hook and try to fall back to reading dependencies from pyproject.toml when the hook is not availbale? - Should this behavior exist but not be the default (explicit flag would be required to opt-in)? - Can you think of a better alternative than the ones described here? Cheers, Karolina

4 4

Django packaging options for alternate stacks - advice needed
by Michel Lind 05 Sep '24

05 Sep '24

Hi all, Since we need the Django stack in EPEL for running the mailing list infra, a few months ago the EPEL Steering Committee discussed a plan to package Django LTS versions in EPEL, with RPM-based Provides/Conflicts to allow swapping between different Django stacks https://pagure.io/epel/issue/271 (Documentation still forthcoming, and it's probably a good idea to formalize the Provides/Conflicts convention first) Carl has since pointed out that there are situations where python-django (the old package, which we now intend to track the latest Django release) and python-djangoX.Y (the LTS package, the first and currently only available is python-django4.2) happen to be the same version, and that it could be confusing* * I personally think it's fine, as the two are interchangeable anyway - * and if a dependency is not compatible with python-django, then users * can swap to the LTS package and keep everything * ... but this can be better documented if we continue down this route, * e.g. with a README.Fedora pointing to the website docs The alternative Carl suggested is to package each python-djangoX.Y we need separately - and that way avoid ever having two package having the same version, similar to how the multiple Python versions are packaged. We thought the SIG might want to chime in on this (might as well sort it out before I start writing documentation anyway) - any pros/cons we have not thought up, and any recommendation about which is better overall? Thanks, -- _o) Michel Lind _( ) identities: https://keyoxide.org/5dce2e7e9c3b1cffd335c1d78b229d2f7ccc04f2

1 1

Proposal: Let's update virtualenv in Rawhide (drops Python 3.6 support)
by Miro Hrončok 26 Aug '24

26 Aug '24

Hello Pythonistas. The virtualenv version in Fedora is currently stuck at 20.21.1 (April 2023). This is because version 20.22+ dropped support for Python 2.7 and 3.6 virtual environments. Now when Python 2.7 was retired via https://fedoraproject.org/wiki/Changes/RetirePython2.7 the only difference is Python 3.6. If we upgrade virtualenv, developers targeting Python 3.6 (e.g. RHEL 8 platform Python) can use `python3.6 -m venv` to create Python 3.6 virtual environments. Developers using tox can follow https://tox.wiki/en/4.18.0/faq.html#testing-end-of-life-python-versions I propose we bite the bullet and update virtualenv to the latest version in Fedora 42+. -- Miro Hrončok -- Phone: +420777974800 Fedora Matrix: mhroncok

3 3

setuptools 72 removes setup.py test, your package(s) will break
by Miro Hrončok 16 Aug '24

16 Aug '24

Hello. We will eventually update python-setuptools in Fedora to version 72 and that version removes support for the `setup.py test` command. We will probably use the change process to propose the upgrade, so don't expect it to land soon. However, if your package still uses the `setup.py test` command (deprecated for 5 years), please update to %pytest, %tox, %{python3} -m unittest, etc. You can use this opportunity to update your package to the current Python packaging guidelines, see e.g. https://src.fedoraproject.org/rpms/python-cachetools/pull-request/4 Affected 199 packages by maintainers based on a simple specfile grep for ^[^-#].+\bsetup\.py\s+test\b abbra python-whoosh abompard python-flufl-i18n python-jinja2_pluralize python-pyramid_sawing python-pysocks python-translationstring python-zope-deprecation python-zope-testing adamwill python-mwclient aekoroglu python-django-prometheus python-repoze-lru python-serpy python-xunitparser ajax binwalk alexl pycairo setools amigadave isrcsubmit python3-mallard-ducktype amoralej python-rsd-lib python-yaql ankursinha python-klusta python-num2words python-pyfim apevec python-appdirs python-distroinfo python-oslo-context python-testresources python-testscenarios python-wsgi_intercept asn cepces athoscr python-booleanoperations python-epub python-fontMath python-glyphsLib python-mutatormath python-pyclipper atim kitty aviso python-enlighten besser82 python-jcconv caolanm pycairo setools carlwgeorge python-testresources chandankumar python-oslo-context python-tosca-parser python-wsgi_intercept chedi python-jinja2-time python-poyo python-pytest-flakes python-tree-format chkr calibre churchyard python-appdirs python-cachetools python-genty python-launchpadlib python-lazr-uri python-scripttest python-simplegeneric python-wadllib cicku python-apsw ckalina ddiskit cqi python-django-contrib-comments dcallagh python-phonenumbers python-www-authenticate dcavalca tpm2-pkcs11 dchen cepces defolos osc dfateyev python-apsw dherrera python-pytest-runner dmach osc python-productmd dminer python-zstd dshea python-jinja2_pluralize dwalsh setools echevemaster python-pydispatcher transifex-client eeickmeyer python-jack-client esyr ddiskit fab ioc-writer past-time python-cx-oracle python-dfdatetime python-geojson python-gmqtt python-imagehash python-ipgetter2 python-jsonpath-ng python-jsonpath-rw python-shodan python-slacker python-textparser python-zc-customdoctests fale python-decorator fedepell python-jinja2-time filiperosset hddfancontrol fjanus PyGreSQL frostyx python-flask-wtf-decorators fschwarz python-digitalocean python-genshi girst mopidy mopidy-mpd gsauthof python-aioresponses gui1ty python-moksha-hub hguemar python-appdirs hhorak PyGreSQL hiwkby python-oslo-metrics hobbes1069 python-collada hvad python-events ignatenkobrain python-pytest-flakes python-simplegeneric jamatos python-markups python-smartypants jcapitao python-cliff python-distroinfo python-timeout-decorator jcaratzas python-repoze-lru python-simplegeneric python-singledispatch python-sphinxcontrib-trio python-zc-lockfile jcline python-ipdb jeckersb python-cachetools jjelen tpm2-pkcs11 jmontleon python-dictdiffer jonathanspw kitty python-freeipa python-pystray python-pytest-runner python-zstd jonny python-libusb1 jorti python-pysrt jpena python-murano-pkg-check python-tosca-parser jruzicka python-tosca-parser jstanek udiskie jujens python-CommonMark python-hgapi jussilehtola fprettify kalev pycairo kevin calibre python-chai python-decorator python-fedmsg-meta-fedora-infrastructure python-html5-parser python-pyramid-tm python-pysocks python-pytest-fixture-config python-svg-path python-tgscheduler kni python-ouimeaux python-pysignals ktdreyer python-testscenarios laxathom python-gammu lbalhar python-boolean.py python-feedgen python-hypothesis-fspaths lbazan python-neurosynth transifex-client lberk python-rmtest lbrabec python-xunitparser lcons python-auth-credential python-dirq python-messaging python-simplevisor limb pyicu python-gattlib python-scour python-svg python-tubes python-typing-inspect lnie libvirt-test-API lsedlar python-productmd lyessaadi python-yapsy maha python-sphinxcontrib-autoprogram melmorabity grammalecte mfabian nototools mohanboddu python-openidc-client mrunge python-case python-pytimeparse python-roman python-singledispatch python-testresources python-testscenarios msuchy osc nathans python-rmtest nb python-digitalocean ngompa osc python-case python-dkimpy python-freeipa python-pystray python-tubes python-zstd omos python-r128gain ondrejj python-chameleon python-nine python-xlrd opohorel python-launchpadlib python-lazr-uri python-wadllib orion python-iptools python-xlib python-zope-testing orphan python-gerritlib transtats-cli osloup PyGreSQL pbrobinson adapt gtts gtts-token pyjokes python-can python-casttube tpm2-pkcs11 pcmoore setools pghmcfc python-crypto pingou python-binaryornot python-case python-chai python-flask-xml-rpc python-vobject python-xlrd pkubat PyGreSQL plautrba setools pnemade ansible-inventory-grapher python-bracex python-cpuinfo poros ddiskit praiskup PyGreSQL puiterwijk python-cccolutils pwhalen pyjokes pyowm pwu nototools qulogic python-json-minify python-xmp-toolkit radez python-zc-lockfile python-zope-testing rajeeshknambiar python-ofxparse raphgro python-pdfkit rathann lazygal rcritten cepces python-pyasn1 rdieter python-mwclient rebus python-colorclass python-oletools python-pcodedmp python-roman rhughes pycairo setools robert python-oletools python-pcodedmp rstrode pycairo setools sagitter petsc salimma kitty python-lupa python-pytest-runner python-volatile python-zstd sergiomb python-gammu python-xlib sgallagh python-whoosh sharkcz python-sql python-vobject shoracek tpm2-pkcs11 slinabery python-jsonpath-rw solopasha kitty spichugi python-pyasn1 suanand transtats-cli suraia python-precis_i18n suve python-ssdeep swt2c binwalk tdecacqu python-statsd terjeros hgsvn python-cwcwidth python-httpsig-cffi python-pyte python-tidy tflink python-xunitparser thebeanogamer classification-banner thm python-sphinxcontrib-trio python-textile python3-lxc thofmann btrfs-sxbackup tomspur python-simplegeneric trawets fros trb143 python-pysword troycurtisjr python-ipdb ttorling python-pysocks vascom python-pymediainfo vmojzis sepolicy_analysis setools vtrefny python-pytimeparse ykarel python-os-service-types zaneb python-autopage zawertun kitty zbyszek calibre zsun ddiskit -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 1

Re: setuptools 72 removes setup.py test, your package(s) will break
by Miro Hrončok 15 Aug '24

15 Aug '24

On 14. 08. 24 18:21, Paul Howarth wrote: >> You can use this opportunity to update your package to the current >> Python packaging guidelines, see e.g. >> https://src.fedoraproject.org/rpms/python-cachetools/pull-request/4 >> >> Affected 199 packages by maintainers based on a simple specfile grep >> for ^[^-#].+\bsetup\.py\s+test\b > There are probably quite a few packages not caught by this, for example > python-pycryptodomex, which has: > > %check > PYTHONPATH=$RPM_BUILD_ROOT%{python3_sitearch}/ %{__python3} %{py_setup} test $ rg 'setup\} test\b' python-journal-brief.spec 64:%{__python3} %{py_setup} test python-pycryptodomex.spec 107:PYTHONPATH=$RPM_BUILD_ROOT%{python3_sitearch}/ %{__python3} %{py_setup} test > There might also be packages using tox with a tox.ini like this one: > https://github.com/pycrypto/pycrypto/blob/master/tox.ini Yeah, invocations hidden in tox.ini, Makefile, etc. are harder to grep for. I might end up building everything in copr and grepping the logs for the error messages. -- Miro Hrončok -- Phone: +420777974800 Fedora Matrix: mhroncok

3 2

The -O3 flag leaked into sysconfig CFLAGS, should we keep it?
by Miro Hrončok 14 Aug '24

14 Aug '24

Hello Pythonistas. For years, the CFLAGS embedded in Python sysconfig contained -O2 in Fedora. This was how Python was built and by default, all flags used to build Python were embedded. Later, the flag was removed in Fedora 39 via this change: https://fedoraproject.org/wiki/Changes/Python_Extension_Flags_Reduction We wanted to remove as many flags as possible, with this motivation: """ Python developers will get more upstream-like experience when building Python extension modules and also closer to building vanilla C programs. """ Note that removing -O2 specifically was not the primary motivation, but the removal was intentional at that time. -------- Now we build Python 3.13 with -O3 via this change: https://fedoraproject.org/wiki/Changes/Python_built_with_gcc_O3 The change proposal said: """ Other Python extension modules will remain bulidng as before, e.g. in RPM packages, they will still be built with -O2... """ However, I made a mistake and the -O3 flag leaked into sysconfig CFLAGS. The good news is this does not seem to affect RPM packages, they are still being built with -O2, like this: <flags embedded in sysconfig> <$CFLAGS from automatic %set_build_flags> E.g. ... -fcf-protection -fexceptions -O3 -O2 -flto=auto ... The latter flag wins. ---- OTOH users building their own extension modules will get -O3. This is not what was intended. However, "Python developers will get more upstream-like experience" is more true now, because upstream Python builds use -O3: $ podman run --rm -ti python:3.12 /usr/bin/bash # python >>> import sysconfig >>> sysconfig.get_config_var('CFLAGS') '-fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall' So the question is: Do we keep -O3 for user-built extension modules for speed and upstream-like experience? (I would update the -O3 change proposal.) Or do we loose the flag, as currently documented? Alternatively, do something else entirely (e.g. embed -O2, or other flag...)? -- Miro Hrončok -- Phone: +420777974800 Fedora Matrix: mhroncok

2 6

The RPM pyproject Declarative Buildsystem is available in Rawhide
by Miro Hrončok 13 Aug '24

13 Aug '24

Hello Pythonistas, pyproject-rpm-macros 1.14.0 landed in Fedora Rawhide and ELN. It defines a provisional Declarative Buildsystem for RPM 4.20+. It allows you to write Python specfiles like this: --------------------------------- Name: python-pello Version: 1.0.4 Release: %autorelease Summary: Example Python library License: MIT-0 URL: https://github.com/fedora-python/Pello Source: %{url}/archive/v%{version}/Pello-%{version}.tar.gz BuildSystem: pyproject BuildOption(install): -l pello BuildArch: noarch %description ... %package -n python3-pello Summary: %{summary} %description -n python3-pello ... %files -n python3-pello -f %{pyproject_files} %doc README.md %{_bindir}/pello_greeting --------------------------------- It will fill-in %prep, %generate_buildrequires, %build, %install, and %check sections for you. Yet you can configure/extend/override the sections if needed. Read about all the possibilities at https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/READ… in the *Provisional: Declarative Buildsystem (RPM 4.20+)* section. This is still *provisional* and the API might change in the future. Beware that RPM in Fedora 39/40 and EPELs does not understand this and RPM < 4.20 will not parse such spec files, so this is inconvenient to use unless you run Rawhide -- fedpkg prep, rpmbuild -bs, spectool -g, rpmdev-bumpspec won't work properly... If you don't use Rawhide, you can build a SRPM like this: mock -r fedora-rawhide-x86_64 --enablerepo=local --resultdir=./ --buildsrpm --spec python-pello.spec --source Pello-1.0.4.tar.gz Or use fedpkg with --srpm-mock. However, even some infra tools that don't run on Rawhide will likely fail with a package like this (e.g. Copr or Zuul). Use this at your own risk. Koji builds SRPMs for Rawhide on Rawhide, so it works there. Please experiment with this and report back how does it work for you. Happy packaging. -- Miro Hrončok -- Phone: +420777974800 Fedora Matrix: mhroncok

1 1

Default BuildOption(install) for the RPM pyproject declarative buildsystem
by Miro Hrončok 13 Aug '24

13 Aug '24

Hello. I am working on the RPM pyproject declarative buildsystem. t;dr it turns this: BuildSystem: pyproject into this: %prep %autosetup -p1 -C %generate_buildrequires %pyproject_buildrequires %build %pyproject_wheel %install %pyproject_install %pyproject_save_files ... %check %pyproject_check_import It allows to override options to sections with BuildOption. For detailed documentation, see https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/455 (it has README changes in it). --------------------------------- What should be the default for BuildOption(install)? That is, what should be passed to %pyproject_save_files if the packager does not override it? 1. The safe+hard default: nothing ================================= The %pyproject_save_files macro currently has no default. It requires at least one argument. This is to avoid accidentally packaging something we don't want. Explicit is better than implicit. For this reason, users of the pyproject declarative buildsystem would be required to always set BuildOption(install) explicitly. Why I don't like this option: The declarative buildsystem was invented to make spec files simpler. Making BuildOption(install) mandatory defeats the purpose. 2. The easy+dangerous default: '*' ================================== When we pass '*' to %pyproject_save_files by default, we ensure all packages work without BuildOption(install). This is the easy way. Everything is automagic, no boilerplate necessary. Passing '*' to %pyproject_save_files is forbidden by the Fedora packaging guidelines because it is dangerous. If upstream removes a Python module or (perhaps accidentally) adds a new one that conflicts with other packages, the Fedora packager should notice. Why I don't like this option: The default is forbidden in Fedora and hence it is only useful elsewhere (e.g. Copr). Defaults should encourage the desired pattern, defaults should not be forbidden. Packagers will use the default and the rule will be ignored as not practical. 3. Guess the module name: python-foo -> foo =========================================== When the package is named somehow, we like the Python modules to be named identical-ish. We can create a transformation of %{name} and make that the default. Something like this: 1. Strip python- prefix if found. 2. Strip everything after a dot (including the dot) if found. 3. Convert dashes, dots etc. to underscores. Some nice examples: python-requests -> requests python-zope.interface -> zope python-flit-core -> flit_core tomcli -> tomcli Examples that would need manual BuildOption(install): python-zope-interface -> zope_interface (we need zope) python-djangorestframework -> djangorestframework (we need rest_framework) pytest -> pytest (we need pytest _pytest py) This would produce a reasonable default that is easy and safe for sane packages but it is not dangerous for the weird ones. Option 3a) is to do this in the pyproject declarative buildsytsem macros implementation. Explicit callers of %pyproject_save_files would not be able to use this, which only has downsides. Option 3b) is to make this the behavior of %pyproject_save_files when it has no arguments. If additional module globs are necessary, all module globs need to be passed explicitly. Option 3c) is to make this the behavior of `%pyproject_save_files -n` and make -n the default BuildOption(install). Users of %pyproject_save_files could combine -n with explicit module glob list (e.g. `%pyproject_save_files -n _pytest py` for pytest). Why I don't like this: Its' heuristics. It smells like guesswork. I prefer things explicit. 4. Default to %{pypi_name} ========================== Similarly to option 3, we prefer the PyPI name and the module name to match. Hence, we could default to %{pypi_name} or %{srcname} like %{pypi_source} does. There would still need to be some transformations applied, because %{pypi_name} can contain dashes and dots, etc... Why I don't like this: %{pypi_name} is not a cardinal part of a package. It's not part of the Python packaging guidelines and does not have a clearly defined semantics (is it the project name or the name part of the source tarball?). It encourages packagers to define the macro and use it everywhere in the spec, making it harder to read. Moreover, we need to apply text transformations to it anyway, hence we might as well do them on %{name} directly (which does not have clear semantics either, but is always there). --------------------------------- If I had to pick, I'd go with 3c. IMHO it is the most sane option. What are your thoughts? Do you see other options? -- Miro Hrončok -- Phone: +420777974800 Fedora Matrix: mhroncok

4 7

New to the list, keen to contribute!
by Igor Jagec 12 Jul '24

12 Jul '24

Hey all, I'm Igor, a long-time Linux and middleware admin (Red Hat/Fedora user since 2001!). I'm currently diving into Python and figuring it all out – always been a big fan of learning by doing, so I thought this would be a great place to get stuck in. Hoping to find my place within the Fedora community and contribute properly. Any pointers on where I could start would be much appreciated. Cheers, Igor

2 1

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel