Hello Python packagers.
After touching the %python_provide topic in:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
I have realized several things I don't like about %python_provide:
1) It must be used conditionally (it is not defined in python-srpm-macros). That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed).
2) You cannot use it with arbitrary versions. Suppose you package python3-foo 1.0 but you want to provide python3-bar 2.0 for some reason -- this is not very common, but it happens. %python_provide only takes the "name" as an argument, always using %{?epoch} %{version} and %{release}.
3) You need to both add a virtual provide + use the macro. Suppose you want to provide python3-pkg_resources from python3-setuptools. Currently, you need to add:
Provides: python3-pkg_resources = %{version}-%{release} %{?python_provide:%python_provide python3-pkg_resources}
4) When used with (sub)package name, it generates a duplicate dependency on Fedora 33+ (and an rpmlint error).
5) It produces Obsoletes, but that might no longer be necessary nor desired.
6) Broken expectations about %_isa. It used to add %_isa provides based on wrong data, it no longer does that (except on old releases and EPELs), can be used manually with name%{?_isa}, but not on the old releases.
7) Undocuemnted error handling (e.g. the macro expands to nothing when used with pypy-foo, but errors when used with foo).
Hence, I was thinking (for the sake of backwards compatibility) to provide a new mechanism to do this and preserve the old macro as is, deprecating it in Fedora 36-ish, actually maybe removing it once RHEL 9 goes EOL (or never, which is basically the same from today's perspective).
The new macro should solve the problems from above, my current (quick, untested) ideas are:
1) Define the macro in python-srpm-macros. No need to use it conditionally. We can backport it to EPEL 8 and define a "stub" macro in EPEL 7 and 6. (An if we start using the macro only after Fedora 30 goes EOL, we can make the macro behave consistently across all Fedora versions.)
2) Accept version identifier as an optional argument, use %{?epoch} %{version} and %{release} as default. (See for example %{pypi_source} on how this can be done.)
3) Make the macro also produce the provide for the given name and document that. E.g. when you call it with python3-pkg_resources, it also provides python3-pkg_resources (not only python-pkg_resources etc.).
4) Make it so that for given arguments, the macro will only expand to something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
5) No obsoletes with the new macro. Packagers use manual obsoletes when desired.
6) Document clearly that there is no %{?_isa} (and there is no "backwards compatibility" load to carry). When absolutely desired, packagers can call the macro with %{name}%{?_isa}.
7) Support arbitrary names. Only provide the given name and nothing else if not "recognized".
As a bonus, I think the current if-elif-elif-elif-elif code can be replaced with lua patterns (imagine regex).
As always, this leaves us with the name problem, but I'd very much like to use %python_provides (note the s). The only problem I see is that it is likely to be mistaken for the old one, but IMHO it shouldn't really hurt that much.
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
Another example:
%package -n python3-pillow %python_provides python3-PIL 1.1.6-100
Resulting provides:
python3-pillow = 7.1.1-1.fc33 python-pillow = 7.1.1-1.fc33 python39-pillow = 7.1.1-1.fc33 python3-PIL = 1.1.6-100 python-PIL = 1.1.6-100 python39-PIL = 1.1.6-100
Dummy when used with package name:
%package -n python3-pip %python_provides python3-pip
Provides:
python3-pip = 20.0.2-1.fc33 python-pip = 20.0.2-1.fc33 python39-pip = 20.0.2-1.fc33 (all of them only once)
Any name is OK:
%python_provides wroom 666
Provides:
wroom = 666
What do you think?
Dne 19. 04. 20 v 16:55 Miro Hrončok napsal(a):
Hello Python packagers.
After touching the %python_provide topic in:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
I have realized several things I don't like about %python_provide:
- It must be used conditionally (it is not defined in
python-srpm-macros). That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed).
This is an advantage. People does not need to have installed python-srpm-macros just to build SRPM, when they are using Mock or Koji to build the package. Please keep it this way.
Vít
On 20. 04. 20 9:35, Vít Ondruch wrote:
Dne 19. 04. 20 v 16:55 Miro Hrončok napsal(a):
Hello Python packagers.
After touching the %python_provide topic in:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
I have realized several things I don't like about %python_provide:
- It must be used conditionally (it is not defined in
python-srpm-macros). That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed).
This is an advantage. People does not need to have installed python-srpm-macros just to build SRPM, when they are using Mock or Koji to build the package. Please keep it this way.
What you say is not true.
redhat-rpm-config requires python-srpm-macros.
People already do need to have redhat-rpm-config (and hence python-srpm-macros) installed when they are using Mock or Koji to build the package. They even need it when they use `rpmbuild -bs`.
I'm keeping it that way.
python-srpm-macros contain macros that are essential to make building SRPM work. There is no external dependency brought in by python-srpm-macros, just the macros. Currently, it has:
%__python{,2,3} %python{,2,3} %python3_pkgversion %__python_other %py3_other_{build,install} (that does not need to be in srpm macros) %py_dist_name %py{2,3}_dist %pypi_source
Do you suggest that the need for "python-srpm-macros is always guaranteed to be there" is artificial? The following (quite extensively used) constructs would not be possible:
Source0: %{pypi_source} BuildRequires: %{python3} // or the more widespread older %{__python3} BuildRequires: %{py3_dist sphinx} BuildRequires: python%{python3_pkgversion}-pytest
Whether or not that is "an advantage" is very much out of scope of what I have proposed. You might argue that by adding one more macro to python-srpm-macros and by using it unconditionally, I am cementing the status quo. However I consider the status quo cemented beyond point of return. Hundreds of Perl, Go, Haskell, Rust etc. packages (there are 13 *-srpm-macros packages required by redhat-rpm-config) won't build SRPMs correctly if this guarantee is gone. All the packages would need to be adapted to different constructs, often throwing away "magic" and going back to "boilerplate" (see e.g. %gometa).
I won't fight you discussing whether this is good or bad, but your feedback is not relevant in this thread.
Dne 20. 04. 20 v 12:02 Miro Hrončok napsal(a):
On 20. 04. 20 9:35, Vít Ondruch wrote:
Dne 19. 04. 20 v 16:55 Miro Hrončok napsal(a):
Hello Python packagers.
After touching the %python_provide topic in:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
I have realized several things I don't like about %python_provide:
- It must be used conditionally (it is not defined in
python-srpm-macros). That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed).
This is an advantage. People does not need to have installed python-srpm-macros just to build SRPM, when they are using Mock or Koji to build the package. Please keep it this way.
What you say is not true.
redhat-rpm-config requires python-srpm-macros.
I don't know if it does or doesn't, but I am quite sure that this is not correct. Dependencies like this should be removed and minimized and not used as justification. `rpmbuild -bs` should work without as much macros as possible.
Currently, I have on my computer these `srpm-macros` packages (and also some others with different names):
```
$ rpm -qa *srpm* fonts-srpm-macros-2.0.5-1.fc33.noarch nim-srpm-macros-3-2.fc32.noarch openblas-srpm-macros-2-7.fc32.noarch gnat-srpm-macros-4-11.fc32.noarch rust-srpm-macros-13-2.fc33.noarch fpc-srpm-macros-1.3-1.fc32.noarch python-srpm-macros-3.8-2.fc33.noarch ghc-srpm-macros-1.5.0-2.fc32.noarch go-srpm-macros-3.0.8-5.fc32.noarch efi-srpm-macros-4-4.fc32.noarch ocaml-srpm-macros-6-2.fc32.noarch perl-srpm-macros-1-34.fc32.noarch qt5-srpm-macros-5.13.2-2.fc32.noarch
```
I don't remember I would ever need any of them.
Vít
People already do need to have redhat-rpm-config (and hence python-srpm-macros) installed when they are using Mock or Koji to build the package. They even need it when they use `rpmbuild -bs`.
I'm keeping it that way.
python-srpm-macros contain macros that are essential to make building SRPM work. There is no external dependency brought in by python-srpm-macros, just the macros. Currently, it has:
%__python{,2,3} %python{,2,3} %python3_pkgversion %__python_other %py3_other_{build,install} (that does not need to be in srpm macros) %py_dist_name %py{2,3}_dist %pypi_source
Do you suggest that the need for "python-srpm-macros is always guaranteed to be there" is artificial? The following (quite extensively used) constructs would not be possible:
Source0: %{pypi_source} BuildRequires: %{python3} // or the more widespread older %{__python3} BuildRequires: %{py3_dist sphinx} BuildRequires: python%{python3_pkgversion}-pytest
Whether or not that is "an advantage" is very much out of scope of what I have proposed. You might argue that by adding one more macro to python-srpm-macros and by using it unconditionally, I am cementing the status quo. However I consider the status quo cemented beyond point of return. Hundreds of Perl, Go, Haskell, Rust etc. packages (there are 13 *-srpm-macros packages required by redhat-rpm-config) won't build SRPMs correctly if this guarantee is gone. All the packages would need to be adapted to different constructs, often throwing away "magic" and going back to "boilerplate" (see e.g. %gometa).
I won't fight you discussing whether this is good or bad, but your feedback is not relevant in this thread.
On 20. 04. 20 12:27, Vít Ondruch wrote:
- It must be used conditionally (it is not defined in
python-srpm-macros). That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed).
This is an advantage. People does not need to have installed python-srpm-macros just to build SRPM, when they are using Mock or Koji to build the package. Please keep it this way.
What you say is not true.
redhat-rpm-config requires python-srpm-macros.
I don't know if it does or doesn't, but I am quite sure that this is not correct. Dependencies like this should be removed and minimized and not used as justification. `rpmbuild -bs` should work without as much macros as possible.
You provide opinions. You don't provide justifications for them. Please, base the discussion on facts, not feelings. What's wrong with the dependencies and why is it wrong now and wasn't wrong until now?
I don't remember I would ever need any of them.
If you don't create srpms from affected packages, you don't need them. Users who do (such as our releng or our buildsystem or me) do need them.
Should we do this differently? I don't know, maybe you are right. But this is how we do this currently and this is what I'll use. Once somebody designs a better way of doing this, I'll adapt. Until then, I refuse the "don't use this mechanism because it's not perfect" argument.
Le lundi 20 avril 2020 à 12:27 +0200, Vít Ondruch a écrit :
$ rpm -qa *srpm* fonts-srpm-macros-2.0.5-1.fc33.noarch nim-srpm-macros-3-2.fc32.noarch openblas-srpm-macros-2-7.fc32.noarch gnat-srpm-macros-4-11.fc32.noarch rust-srpm-macros-13-2.fc33.noarch fpc-srpm-macros-1.3-1.fc32.noarch python-srpm-macros-3.8-2.fc33.noarch ghc-srpm-macros-1.5.0-2.fc32.noarch go-srpm-macros-3.0.8-5.fc32.noarch efi-srpm-macros-4-4.fc32.noarch ocaml-srpm-macros-6-2.fc32.noarch perl-srpm-macros-1-34.fc32.noarch qt5-srpm-macros-5.13.2-2.fc32.noarch
I don't remember I would ever need any of them.
Those are domain-specific macros. You use them for specific domain packages. Some of those are hard requirements for a 4-digit number of packages.
You don’t use them with C/C++ software because autoconf/automake/make/C/C++ stuff is hidden in redhat-rpm-config & rpm for historical reasons (and quite non-regular by modern domain standards, making automation hard).
On 4/19/20 4:55 PM, Miro Hrončok wrote:
Hello Python packagers.
After touching the %python_provide topic in:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
I have realized several things I don't like about %python_provide:
- It must be used conditionally (it is not defined in
python-srpm-macros). That means you always wrap it in %{?python_provide:...} in order to have a "valid" specfile even when the macro is not yet defined (e.g. during SRPM creation in Koji or on a packager's machine without python-rpm-macros installed).
- You cannot use it with arbitrary versions. Suppose you package
python3-foo 1.0 but you want to provide python3-bar 2.0 for some reason -- this is not very common, but it happens. %python_provide only takes the "name" as an argument, always using %{?epoch} %{version} and %{release}.
- You need to both add a virtual provide + use the macro. Suppose you
want to provide python3-pkg_resources from python3-setuptools. Currently, you need to add:
Provides: python3-pkg_resources = %{version}-%{release} %{?python_provide:%python_provide python3-pkg_resources}
- When used with (sub)package name, it generates a duplicate
dependency on Fedora 33+ (and an rpmlint error).
- It produces Obsoletes, but that might no longer be necessary nor
desired.
- Broken expectations about %_isa. It used to add %_isa provides
based on wrong data, it no longer does that (except on old releases and EPELs), can be used manually with name%{?_isa}, but not on the old releases.
- Undocuemnted error handling (e.g. the macro expands to nothing when
used with pypy-foo, but errors when used with foo).
Hence, I was thinking (for the sake of backwards compatibility) to provide a new mechanism to do this and preserve the old macro as is, deprecating it in Fedora 36-ish, actually maybe removing it once RHEL 9 goes EOL (or never, which is basically the same from today's perspective).
The new macro should solve the problems from above, my current (quick, untested) ideas are:
- Define the macro in python-srpm-macros. No need to use it
conditionally. We can backport it to EPEL 8 and define a "stub" macro in EPEL 7 and 6. (An if we start using the macro only after Fedora 30 goes EOL, we can make the macro behave consistently across all Fedora versions.)
- Accept version identifier as an optional argument, use %{?epoch}
%{version} and %{release} as default. (See for example %{pypi_source} on how this can be done.)
- Make the macro also produce the provide for the given name and
document that. E.g. when you call it with python3-pkg_resources, it also provides python3-pkg_resources (not only python-pkg_resources etc.).
- Make it so that for given arguments, the macro will only expand to
something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
- No obsoletes with the new macro. Packagers use manual obsoletes
when desired.
- Document clearly that there is no %{?_isa} (and there is no
"backwards compatibility" load to carry). When absolutely desired, packagers can call the macro with %{name}%{?_isa}.
- Support arbitrary names. Only provide the given name and nothing
else if not "recognized".
Given the limitations, I agree with this choice.
As a bonus, I think the current if-elif-elif-elif-elif code can be replaced with lua patterns (imagine regex).
As always, this leaves us with the name problem, but I'd very much like to use %python_provides (note the s). The only problem I see is that it is likely to be mistaken for the old one, but IMHO it shouldn't really hurt that much.
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
What provides the names `python-setuptools` and `python39-pkg_resources`? From the code [0] it looks like the current rawhide implementation of %py_provides doesn't. On the other hand, it might be a good idea to also provide the names for the given %subpackage name (with an option to disable this). In that case, the macros should also without any parameters at all.
[0] https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.py...
Thanks for a nifty new macro! Tomas
Another example:
%package -n python3-pillow %python_provides python3-PIL 1.1.6-100
Resulting provides:
python3-pillow = 7.1.1-1.fc33 python-pillow = 7.1.1-1.fc33 python39-pillow = 7.1.1-1.fc33 python3-PIL = 1.1.6-100 python-PIL = 1.1.6-100 python39-PIL = 1.1.6-100
Dummy when used with package name:
%package -n python3-pip %python_provides python3-pip
Provides:
python3-pip = 20.0.2-1.fc33 python-pip = 20.0.2-1.fc33 python39-pip = 20.0.2-1.fc33 (all of them only once)
Any name is OK:
%python_provides wroom 666
Provides:
wroom = 666
What do you think?
On 25. 05. 20 18:33, Tomas Orsava wrote:
On 4/19/20 4:55 PM, Miro Hrončok wrote:
- Make it so that for given arguments, the macro will only expand to
something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
Not if one if manual and one from a generator. See:
https://github.com/rpm-software-management/rpm/issues/1166
- Support arbitrary names. Only provide the given name and nothing else if
not "recognized".
Given the limitations, I agree with this choice.
What limitations?
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
What provides the names `python-setuptools`
The generator.
and `python39-pkg_resources`?
%py_provides python3-pkg_resources
(Note that it is python3.9-pkg_resources now.)
From the code [0] it looks like the current rawhide implementation of %py_provides doesn't. On the other hand, it might be a good idea to also provide the names for the given %subpackage name (with an option to disable this). In that case, the macros should also without any parameters at all.
I don't get this part, sorry.
[0] https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.py...
On 5/25/20 7:42 PM, Miro Hrončok wrote:
On 25. 05. 20 18:33, Tomas Orsava wrote:
On 4/19/20 4:55 PM, Miro Hrončok wrote:
- Make it so that for given arguments, the macro will only expand
to something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
Not if one if manual and one from a generator. See:
Interesting. One thing not mentioned is, does this cause any issues besides the visual issue of the provide being listed twice?
- Support arbitrary names. Only provide the given name and nothing
else if not "recognized".
Given the limitations, I agree with this choice.
What limitations?
I was referring to this bit from the other thread:
we cannot error out with arbitrary package names, so we currently
need to pre-filter the arguments before using them
Did I misunderstand it?
Usage example:
%package -n python3-setuptools %python_provides python3-pkg_resources
Resulting provides:
python3-setuptools = 46.6.6-6.fc33 python-setuptools = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33 python3-pkg_resources = 46.6.6-6.fc33 python-pkg_resources = 46.6.6-6.fc33 python39-pkg_resources = 46.6.6-6.fc33
What provides the names `python-setuptools`
The generator.
Oh right, I thought you were only talking about the provides from the macro. Makes sense.
and `python39-pkg_resources`?
%py_provides python3-pkg_resources
(Note that it is python3.9-pkg_resources now.)
From the code [0] it looks like the current rawhide implementation of %py_provides doesn't. On the other hand, it might be a good idea to also provide the names for the given %subpackage name (with an option to disable this). In that case, the macros should also without any parameters at all.
I don't get this part, sorry.
Irrelevant due to my misunderstanding above.
Tomas
[0] https://src.fedoraproject.org/rpms/python-rpm-macros/blob/master/f/macros.py...
On 26. 05. 20 11:29, Tomas Orsava wrote:
On 5/25/20 7:42 PM, Miro Hrončok wrote:
On 25. 05. 20 18:33, Tomas Orsava wrote:
On 4/19/20 4:55 PM, Miro Hrončok wrote:
- Make it so that for given arguments, the macro will only expand to
something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
Not if one if manual and one from a generator. See:
Interesting. One thing not mentioned is, does this cause any issues besides the visual issue of the provide being listed twice?
Technical issues? No. Createrepo merges those, so in the repo, no difference.
But rpmlint is very loud about those. It tells you: This provide you put in is not needed (which is great except it only applies to rawhide and we don't want packagers start removing it and merging the removal to older branches).
Side note: Once Fedora 32 goes EOL, we can change the behavior, so rpmlint starts bugging packagers about this again.
- Support arbitrary names. Only provide the given name and nothing else if
not "recognized".
Given the limitations, I agree with this choice.
What limitations?
I was referring to this bit from the other thread:
we cannot error out with arbitrary package names, so we currently need to
pre-filter the arguments before using them
Did I misunderstand it?
Not really. I did. Nothing to discuss here, these aren't the droids you're looking for... move along.
On 5/26/20 11:52 AM, Miro Hrončok wrote:
On 26. 05. 20 11:29, Tomas Orsava wrote:
On 5/25/20 7:42 PM, Miro Hrončok wrote:
On 25. 05. 20 18:33, Tomas Orsava wrote:
On 4/19/20 4:55 PM, Miro Hrončok wrote:
- Make it so that for given arguments, the macro will only expand
to something once per build. Hence when you use it with package name, the automatic provides won't re-add the same provide again. This also means you cannot have 2 different (sub)packages provide the same name-version-release, but that shall be very very very uncommon need and can always be workarounded somehow if needed.
I thought multiple identical provides were just unified and didn't pose a problem. So what is the reason to focus on this once-per-build expansion?
Not if one if manual and one from a generator. See:
Interesting. One thing not mentioned is, does this cause any issues besides the visual issue of the provide being listed twice?
Technical issues? No. Createrepo merges those, so in the repo, no difference.
But rpmlint is very loud about those. It tells you: This provide you put in is not needed (which is great except it only applies to rawhide and we don't want packagers start removing it and merging the removal to older branches).
Side note: Once Fedora 32 goes EOL, we can change the behavior, so rpmlint starts bugging packagers about this again.
Ah, makes sense.
Overall, good work!
Tomas
- Support arbitrary names. Only provide the given name and
nothing else if not "recognized".
Given the limitations, I agree with this choice.
What limitations?
I was referring to this bit from the other thread:
> we cannot error out with arbitrary package names, so we currently need to pre-filter the arguments before using them
Did I misunderstand it?
Not really. I did. Nothing to discuss here, these aren't the droids you're looking for... move along.
packaging@lists.fedoraproject.org
-
Miro Hrončok -
Nicolas Mailhot -
Tomas Orsava -
Vít Ondruch