Re: [Fedora-packaging] Re: RPM 4.19: Dynamic subpackages with Python extras
by Igor Raits
Hey,
On Mon, May 15, 2023, 22:04 Dan Čermák <dan.cermak(a)cgc-instruments.de>
wrote:
> Hi Igor,
>
> On May 15, 2023 6:24:07 PM UTC, Igor Raits <igor.raits(a)gmail.com> wrote:
> >On Thu, Mar 30, 2023 at 11:56 PM Miro Hrončok <mhroncok(a)redhat.com>
> wrote:
> >>
> >> Hello Python packagers.
> >>
> >> RPM 4.19 introduces this feature:
> >>
> >> https://rpm-software-management.github.io/rpm/manual/dynamic_specs.html
> >>
> >> I decided to write this email to gather my thoughts. I believe that
> with this,
> >> we can turn manual Python extras subpackages like this:
> >
> >Is there a reason not to write a brp script so that users won't have
> >to do anything manually at all? Basically scan the sitelibdir for the
> >dist-info and create necessary parts from there?
>
> Don't the brp-scripts run after the build? I though the dynamic subpackage
> creation has to occur beforehand.
>
I think brp script runs exactly before subpackages are created.
> >>
> >> %package -n python3-...
> >> Summary: %{summary}
> >>
> >> %description -n python3-... %_description
> >>
> >> %pyproject_extras_subpkg -n python3-xxx extra1 extra2
> >>
> >> (See
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Extras
> >> for what that means.)
> >>
> >> Into something like this:
> >>
> >> %package -n python3-...
> >> Summary: %{summary}
> >>
> >> %description -n python3-... %_description
> >> ...
> >>
> >> %install
> >> %pyproject_install
> >> ...
> >> %pyproject_generate_extras_subpkgs -n python3-xxx
> >>
> >>
> >> The %pyproject_generate_extras_subpkgs macro would parse the installed
> >> .dist-info directory to find out what extras are available and generate
> >> subpackages for all of them.
> >>
> >> (Obviously, the macro name is open up for discussion.)
> >>
> >> ----------------
> >>
> >> An API would be required to exclude extras:
> >>
> >> - that are not useful for other packages
> >> (for example build/development requirements, commonly named dev, doc
> or test)
> >> - that have requirements that are not packaged in Fedora
> >>
> >> For example (mimicking the API of %pyproject_check_import):
> >>
> >> %pyproject_generate_extras_subpkgs -n python3-xxx -e test -e
> 'nonfree*'
> >>
> >> ----------------
> >>
> >> However, extras are also currently manually passed to
> %pyproject_buildrequires:
> >>
> >> %generate_buildrequires
> >> %pyproject_buildrequires -x extra1 -x extra2 -x test
> >>
> >> It should already be possible to implement automatic extras discovery in
> >> %pyproject_buildrequires with older RPM versions and allow it to be
> used this way:
> >>
> >> %generate_buildrequires
> >> %pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*'
> >>
> >> RPM macros can only accept short flags, so <FLAG_TO_ENABLE_ALL_EXTRAS>
> can
> >> either be -x '*' (if we start treating -x values as globs, which is
> backwards
> >> compatible and probably generally useful), or a single-letter switch
> such as -a
> >> (but honestly we are running out of meaningful letters).
> >>
> >> (When -X is used, <FLAG_TO_ENABLE_ALL_EXTRAS> can probably be implied.
> However,
> >> an explicit form needs to exist for packages that don't need to exclude
> any
> >> extras at all.)
> >>
> >>
> >> Eventually, I'd like to make <FLAG_TO_ENABLE_ALL_EXTRAS> the default,
> once RPM
> >> 4.19 is omnipresent.
> >>
> >> ----------------
> >>
> >> Combined, this would mean that the packager needs to:
> >>
> >> 1. specify extras that are not supposed to be used as BRs
> >> 2. specify extras that are not supposed to be packaged
> >>
> >> In the ideal word (2) is a superset of (1).
> >>
> >> Should %pyproject_generate_extras_subpkgs somehow inherit the -Xes from
> >> %pyproject_buildrequires?
> >>
> >> When a package has extra1, extra2, nonfree1, nonfree2 and test extras,
> one
> >> could do:
> >>
> >> %generate_buildrequires
> >> %pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*'
> >>
> >> ...
> >>
> >> %pyproject_install
> >> ...
> >> %pyproject_generate_extras_subpkgs -X test
> >>
> >> That would mean:
> >>
> >> - extra1 is BRed and packaged
> >> - extra2 is BRed and packaged
> >> - test is BRed but not packaged
> >> - nonfree1 is neither
> >> - nonfree2 is neither
> >>
> >> ----------------
> >>
> >> Alternatively the information could be supplied by %globals:
> >>
> >> %global _python_ignored_extras nonfree*
> >> %global _python_unpackaged_extras test
> >>
> >> However, I somehow dislike this approach.
> >>
> >> ----------------
> >>
> >> I'd appreciate your thoughts on the matter.
> >>
> >> --
> >> Miro Hrončok
> >> --
> >> Phone: +420777974800
> >> IRC: mhroncok
> >> _______________________________________________
> >> packaging mailing list -- packaging(a)lists.fedoraproject.org
> >> To unsubscribe send an email to packaging-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproje...
> >> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
> >
> >
> >
>
4 days, 20 hours
Bringing PyTorch to Fedora
by Kaitlyn Abdo
Hello everyone,
I hope this email finds everyone well. I’m here to introduce myself but also ask for anyone who would be interested in helping me out on a fun side project that me and my colleague, Teng, have planned. I’m Kaitlyn Abdo and I’m an Associate Technical Marketing Manager at Red Hat, specifically for the AI/ML products and offerings. Teng is a Principal Technical Product Manager at Red Hat as well, but I’ll let Teng formally introduce himself :).
To give the TLDR, we want to package PyTorch for Fedora. We are looking for individuals in the community who are interested in making this happen with us. We will be hosting most of the discussion moving forward on the AI/ML discussion page (https://discussion.fedoraproject.org/tag/ai-ml-sig) on Fedora Discussion, but we will still send out major communication through the mailing lists.
We are aware of several existing SIGs (ML SIG-https://fedoraproject.org/wiki/SIGs/ML, Heterogeneous Computing Special Interest SIG-https://fedoraproject.org/wiki/SIGs/HC, Big Data SIG-https://fedoraproject.org/wiki/SIGs/bigdata) and would like to help reignite activity in a coordinated way. I talked to the leadership of the ML SIG about revamping and possibly merging other less active SIGs to create a broader AI/ML SIG. The ML SIG leadership agreed that this could be beneficial for communication and project purposes, so I’m open to input from both leadership and individuals in other SIGs.
Please reach out to either me at kabdo(a)redhat.com or Teng at tema(a)redhat.com if you are interested. We are in the very early stages of this and want to build a team of anyone who would be interested. Thank you for your time and we look forward to hearing from you!
Best,
Kaitlyn Abdo
1 week, 2 days
RPM 4.19: Dynamic subpackages with Python extras
by Miro Hrončok
Hello Python packagers.
RPM 4.19 introduces this feature:
https://rpm-software-management.github.io/rpm/manual/dynamic_specs.html
I decided to write this email to gather my thoughts. I believe that with this,
we can turn manual Python extras subpackages like this:
%package -n python3-...
Summary: %{summary}
%description -n python3-... %_description
%pyproject_extras_subpkg -n python3-xxx extra1 extra2
(See https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Extras
for what that means.)
Into something like this:
%package -n python3-...
Summary: %{summary}
%description -n python3-... %_description
...
%install
%pyproject_install
...
%pyproject_generate_extras_subpkgs -n python3-xxx
The %pyproject_generate_extras_subpkgs macro would parse the installed
.dist-info directory to find out what extras are available and generate
subpackages for all of them.
(Obviously, the macro name is open up for discussion.)
----------------
An API would be required to exclude extras:
- that are not useful for other packages
(for example build/development requirements, commonly named dev, doc or test)
- that have requirements that are not packaged in Fedora
For example (mimicking the API of %pyproject_check_import):
%pyproject_generate_extras_subpkgs -n python3-xxx -e test -e 'nonfree*'
----------------
However, extras are also currently manually passed to %pyproject_buildrequires:
%generate_buildrequires
%pyproject_buildrequires -x extra1 -x extra2 -x test
It should already be possible to implement automatic extras discovery in
%pyproject_buildrequires with older RPM versions and allow it to be used this way:
%generate_buildrequires
%pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*'
RPM macros can only accept short flags, so <FLAG_TO_ENABLE_ALL_EXTRAS> can
either be -x '*' (if we start treating -x values as globs, which is backwards
compatible and probably generally useful), or a single-letter switch such as -a
(but honestly we are running out of meaningful letters).
(When -X is used, <FLAG_TO_ENABLE_ALL_EXTRAS> can probably be implied. However,
an explicit form needs to exist for packages that don't need to exclude any
extras at all.)
Eventually, I'd like to make <FLAG_TO_ENABLE_ALL_EXTRAS> the default, once RPM
4.19 is omnipresent.
----------------
Combined, this would mean that the packager needs to:
1. specify extras that are not supposed to be used as BRs
2. specify extras that are not supposed to be packaged
In the ideal word (2) is a superset of (1).
Should %pyproject_generate_extras_subpkgs somehow inherit the -Xes from
%pyproject_buildrequires?
When a package has extra1, extra2, nonfree1, nonfree2 and test extras, one
could do:
%generate_buildrequires
%pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*'
...
%pyproject_install
...
%pyproject_generate_extras_subpkgs -X test
That would mean:
- extra1 is BRed and packaged
- extra2 is BRed and packaged
- test is BRed but not packaged
- nonfree1 is neither
- nonfree2 is neither
----------------
Alternatively the information could be supplied by %globals:
%global _python_ignored_extras nonfree*
%global _python_unpackaged_extras test
However, I somehow dislike this approach.
----------------
I'd appreciate your thoughts on the matter.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 week, 4 days
python3.11-rpm to EPEL 9
by Miro Hrončok
Hello,
I'm working on adding python3.11-rpm to EPEL 9 and EPEL 9 Next.
See https://src.fedoraproject.org/rpms/python3-rpm/pull-request/3 and 4.
I decided to reuse the python3-rpm component (currently epel7 only). Let me
know if I should not.
If there is a significant demand, I can try add this (and python39-rpm) to EPEL
8 as well.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 week, 4 days
pyproject-rpm-macros 1.8.0: self-referential extras,
%{pyproject_build_lib} deprecated
by Miro Hrončok
Hello Pythonistas,
I've built pyproject-rpm-macros-1.8.0-1.fc39 in Rawhide and updates are
available for older Fedoras. Later on I plan to sync this to c9s as well.
https://bodhi.fedoraproject.org/updates/pyproject-rpm-macros
The release brings a bugfix/enhancement (depending on how you look at it) and a
deprecation.
## %pyproject_buildrequires now supports self-referential extras requirements
That is, if you package e.g. attrs and use:
%pyproject_buildrequires -x tests
It correctly handles the requirement on attrs[tests-no-zope] as defined here:
https://github.com/python-attrs/attrs/blob/23.1.0/pyproject.toml#L45
When a self-referential dependency on extras is found, the required extra is
added to the list of desired extras, and all the previously discarded "alien"
requirements are reevaluated using the new extras list.
Example use: https://src.fedoraproject.org/rpms/python-fiona/pull-request/2
## The provisional %{pyproject_build_lib} macro was deprecated
There is no scheduled removal, but it may break entirely in the future.
See
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
for motivation.
Packagers are encouraged to use
$PWD/build/lib.%{python3_platform}-cpython-%{python3_version_nodots} instead if
they need to (works on all current Fedoras but not on RHEL 9's Python 3.9).
The following Fedora Rawhide packages use it:
asv
buku
ocrmypdf
pytest
python-Automat
python-constantly
python-cppy
python-cypari2
python-cysignals
python-elasticsearch
python-graphviz
python-hyperframe
python-hyperlink
python-h2
python-libdiscid
python-mistune
python-mplcursors
python-nb2plots
python-networkx
python-nipy
python-numcodecs
python-opentracing
python-pbr
python-persistent
python-pikepdf
python-pplpy
python-primecountpy
python-priority
python-pybtex
python-pyedflib
python-pytest-regressions
python-service-identity
python-simplejson
python-stestr
python-transforms3d
python-twisted
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 week, 4 days
Package naming for vkbasalt-cli
by Sandro
Hi,
I need to package vkbasalt-cli [1] as a dependency for Bottles. Bottles
will use the Python library, but vkbasalt-cli also provides a command
for independent/direct usage.
My plan is to name the Fedora package 'vkbasalt-cli' in line with
upstream and add a virtual provide for 'python3-vkbasalt-cli'.
Could someone confirm that this is the right approach for vkbasalt-cli?
Thanks in advance,
--
Sandro
FAS: gui1ty
IRC: Penguinpee
Elsewhere: [Pp]enguinpee
1 week, 4 days
How to drop 32bit support from the scientific Python stack
by Miro Hrončok
Hello folks,
Couple months ago, we discussed with @psimovec if it's possible to ExcludeArch
i686 from scipy.
This Python discussion brought the topic back:
https://discuss.python.org/t/dropping-32-bit-packages/5476/9
I've tried to see how many packages would be affected and the short answer is:
all of them, unless we break the dependency chains somewhere.
There are many deep transitive build dependency chains on scipy, but the
perhaps most interesting one is:
(everything) <- rpm <- rust-rpm-sequoia <- rust-packaging <- pytest <-
python-hypothesis <- python-pandas <- scipy
Another interesting fact is that the majority of the dependency chains I found
traverses through a path of noarch Python packages (such as pytest).
If scipy drops i686 we would need to conditionalize the test-related
BuildRequires of such noarch packages -- unfortunately that would mean that
based on randomness (noarch packages are built on random archotcture), not all
tests would always run and we could potentially ship packages that are broken /
untested properly.
And if we don't conditionalize the dependencies, the packages would randomly
fail to build.
An alternative is to make all the related packages archful with noarch
python3-xxx subpackages -- which would be quite tedious and wasteful resource-wise.
The best way forward for this use case (and many others that will show up
sooner or later) would be to stop building noarch packages on i686. That way,
only archful packages that (Build)Require scipy would need to be changed.
Would that be possible?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 weeks, 2 days
