python-devel June 2017

python-devel@lists.fedoraproject.org

9 participants
6 discussions

Re: Proposed Mass Bug Filing: Renaming "python-" binary packages to "python2-"
by Jason L Tibbitts III 29 Jun '17

29 Jun '17

First off, I really want to say that we should resist the idea that Red Hat product decisions should hold back Fedora's progress. This python-*/python2-*/python3-* mess has been with us for far too long, and I applaud the Python SIG for putting in the effort to try and get this regularized. Python packaging is still terrible, but regularizing this one thing makes it a little bit less terrible. >>>>> "AW" == Adam Williamson <adamwill(a)fedoraproject.org> writes: AW> I'm not sure this list is terribly useful, because of the AW> above. There are thousands of packages that do this, because the AW> 'python2-' provide is not available on some older Fedora release, or AW> on EPEL (and the package is maintained for EPEL as well as AW> Fedora). We can fix older releases and we can fix EPEL. We could potentially fix RHEL by means of metapackages that live in EPEL, if we were serious about it. Just add a "python2-foo" package that has nothing but "Requires: python-foo". We've talked about this for ages but until now there was never any driving need, and while I'm sure it would work fine I don't know exactly what would happen if the RHEL package suddenly acquires its own Provides: python2-foo, or in any other weird dependency scenarios that someone might invent. If nobody can see a problem with that and we agree it would help, I will volunteer create and maintain this metapackage thing. AW> Sprinkling "if (some release number condition) then AW> Requires: python2-foo else Requires: python-foo" all over your spec AW> is a giant PITA and I for one am not very interested in doing it. Well, here's the big thing: Read https://pagure.io/packaging-committee/issue/686#comment-443129. In fact, I'll quote it here. The indented text is my question, the unindented text is ishcherb's reply: " First off, do we really intend for the unversioned "python-foo" to give you the python3 version of that package? We do, and that is what %python_provide macro was intended for. When needed, it will be enough to only modify the macro to make Python 3 subpackages provide python- prefixed names. " So, basically, you simply cannot have a spec which depends on python-foo and assumes it's going to get the python2 version. Obviously this hasn't been done yet because it would break the world, but the point behind posting this big list of packages is to fix them all. And even if Fedora waits two years to do this, old EPEL releases will still be around. So if you really want that magical non-ifdef spec compatibility between Fedora and EPEL, then either RHEL is going to have to add the python2- provides soon or EPEL is going to have to introduce the metapackages I mentioned above. And if you want compatibility between Fedora and RHEL without EPEL present, then %if/%endif is your only real option. The only other way I can see around this is to have some ugly macro which hardcodes knowledge of packages which provide python-* but not python2-* and generates your Requires: or BuildRequires: based on that. I could write one, but I don't particularly want to. If people talk it out and it turns out that's the only solution which people can accept, then I will volunteer to write and maintain a macro like this. Finally, Red Hat itself could clear up this mess a bit by officially saying that it intends to add the python2-* provides on some schedule. Then Fedora could coordinate with that, and we'd all live in a land of ponies and rainbows. What a concept. But I know Red Hat never comments on what they're doing in upcoming products and never commits to dates for upcoming products, so.... - J<

1 0

Re: Proposed Mass Bug Filing: Renaming "python-" binary packages to "python2-"
by Petr Viktorin 29 Jun '17

29 Jun '17

On 06/29/2017 04:20 AM, Adam Williamson wrote: > On Thu, 2017-06-29 at 12:11 +1000, Nick Coghlan wrote: >> On 29 June 2017 at 11:39, Adam Williamson <adamwill(a)fedoraproject.org> wrote: >>> On Wed, 2017-06-28 at 16:21 +0200, Iryna Shcherbina wrote: >>>> 2) Using `python-` instead of `python2-` in the dependencies for the >>>> Python 2 binary RPM [2]. >>> >>> I'm not sure this list is terribly useful, because of the above. There >>> are thousands of packages that do this, because the 'python2-' provide >>> is not available on some older Fedora release, or on EPEL (and the >>> package is maintained for EPEL as well as Fedora). Sprinkling "if (some >>> release number condition) then Requires: python2-foo else Requires: >>> python-foo" all over your spec is a giant PITA and I for one am not >>> very interested in doing it. >>> >>> IMHO, if there is going to be some kind of requirement that all Python >>> requires be explicitly versioned, there needs to be a co-ordinated >>> effort to make sure the versioned Provides are available across at >>> least EL6, EL7, and all supported Fedora releases *first*. >> >> This was the key concern I raised in response to the initial email, >> and our conclusion at the time was: >> >> 1. This case does need to be addressed >> 2. Adding an opaque dependency on buildroot configuration settings >> wouldn't be a particularly nice way to handle it, since it forces >> every package to switch concurrently, rather than each maintainer >> getting to decide when to move from the Python 2 stack to the Python 3 >> stack for themselves (and that unilateral shift is already going to >> happen for unqualified dependency declarations when the virtual >> %python_provides macro moves from the Python 2 stack to the Python 3 >> stack) >> 3. Ideally, the recommended approach would work for arbitrary RHEL & >> CentOS based buildroots, not just those with the EPEL RPM macros >> available >> >> The most straightforward solution we came up with was for affected >> packages to define their own "%py_prefix" macro that selects the stack >> they want to use based on the Python version: >> >> ``` >> # The block below would become the conventional >> # "Python stack compatibility" dance for >> # EL6, EL7, and Fedora >> # Each package can decide for itself which version of >> # Fedora had a sufficiently complete Py3 stack for >> # them to be able to switch over >> >> # Current EL releases & older Fedora use "python-*" >> %if 0%{?el6} || 0%{?el7} || 0%{?fedora} < 25 >> %define py_prefix python >> %if 0%{?el6} || 0%{?el7} >> BuildRequires: python-devel >> %else >> BuildRequires: python2-devel >> %endif >> %else >> # Newer Fedora releases use "pythonX-*" >> # A Py2-only project would use "python2" here >> %define py_prefix python3 >> BuildRequires: python3-devel >> %endif >> >> >> # Dependency declarations use stack selected above >> BuildRequires: %{py_prefix}-builddep1 >> BuildRequires: %{py_prefix}-builddep2 >> Requires: %{py_prefix}-runtimedep1 >> Requires: %{py_prefix}-runtimedep2 >> ``` > > Erf, well, that seems kinda icky but manageable. Is it really better > than just coming up with some kind of script to at least add the > appropriate "Provides: python2-foo" to all packages (or at least a > large subset of the most commonly-used packages) in EL6/EL7/F24, > though? I mean, that doesn't seem beyond the bounds of possibility, to > just find everything that provides 'python-foo' and make it also > provide 'python2-foo'... Adding python2-foo is the second half of this proposal. We don't want to do that by *just* adding "Provides: python2-foo" to all packages, because if everyone did that, we'd need to ask them to change again in a few years. Would it be reasonable to do this in phases: first ask everyone to add python2- provides, then ask everyone to use them? Yes, and in fact, we already did that: the python2- prefix has been recommended in the guidelines for a quite a long time. But many packagers won't do the change until it starts being necessary. Rebuilding lots of packages in EL6 is practically impossible, actually. However, some subset of commonly-used packages do have the python2- prefixed provides even in EL6. The very example above is misleading: EL6's packages do provide python2-devel. -- Petr Viktorin

1 0

Re: Proposed Mass Bug Filing: Renaming "python-" binary packages to "python2-"
by Nick Coghlan 29 Jun '17

29 Jun '17

On 29 June 2017 at 11:39, Adam Williamson <adamwill(a)fedoraproject.org> wrote: > On Wed, 2017-06-28 at 16:21 +0200, Iryna Shcherbina wrote: >> 2) Using `python-` instead of `python2-` in the dependencies for the >> Python 2 binary RPM [2]. > > I'm not sure this list is terribly useful, because of the above. There > are thousands of packages that do this, because the 'python2-' provide > is not available on some older Fedora release, or on EPEL (and the > package is maintained for EPEL as well as Fedora). Sprinkling "if (some > release number condition) then Requires: python2-foo else Requires: > python-foo" all over your spec is a giant PITA and I for one am not > very interested in doing it. > > IMHO, if there is going to be some kind of requirement that all Python > requires be explicitly versioned, there needs to be a co-ordinated > effort to make sure the versioned Provides are available across at > least EL6, EL7, and all supported Fedora releases *first*. This was the key concern I raised in response to the initial email, and our conclusion at the time was: 1. This case does need to be addressed 2. Adding an opaque dependency on buildroot configuration settings wouldn't be a particularly nice way to handle it, since it forces every package to switch concurrently, rather than each maintainer getting to decide when to move from the Python 2 stack to the Python 3 stack for themselves (and that unilateral shift is already going to happen for unqualified dependency declarations when the virtual %python_provides macro moves from the Python 2 stack to the Python 3 stack) 3. Ideally, the recommended approach would work for arbitrary RHEL & CentOS based buildroots, not just those with the EPEL RPM macros available The most straightforward solution we came up with was for affected packages to define their own "%py_prefix" macro that selects the stack they want to use based on the Python version: ``` # The block below would become the conventional # "Python stack compatibility" dance for # EL6, EL7, and Fedora # Each package can decide for itself which version of # Fedora had a sufficiently complete Py3 stack for # them to be able to switch over # Current EL releases & older Fedora use "python-*" %if 0%{?el6} || 0%{?el7} || 0%{?fedora} < 25 %define py_prefix python %if 0%{?el6} || 0%{?el7} BuildRequires: python-devel %else BuildRequires: python2-devel %endif %else # Newer Fedora releases use "pythonX-*" # A Py2-only project would use "python2" here %define py_prefix python3 BuildRequires: python3-devel %endif # Dependency declarations use stack selected above BuildRequires: %{py_prefix}-builddep1 BuildRequires: %{py_prefix}-builddep2 Requires: %{py_prefix}-runtimedep1 Requires: %{py_prefix}-runtimedep2 ``` For dual stack libraries, the appropriate prefixes to define would be separate ones for each stack (%py2_prefix and %py3_prefix), and either leave the latter undefined for systems with no native Py3 stack, or else set it to rely on EPEL, IUS, or a suitable software collection. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Proposed Mass Bug Filing: Renaming "python-" binary packages to "python2-"
by Iryna Shcherbina 28 Jun '17

28 Jun '17

Hi all, Currently, in Fedora package names, "python" without a version number generally means Python 2. We would like to make it mean Python 3, but to make that switch, we first need to make sure nothing uses unqualified "python". All packages should use the "python2-" or "python3-" prefix. This has already been discussed in the packaging mailing list [0], which includes the technical details. Recently, there were two important changes to Python packaging guidelines. The first one bans using the ambiguous `python` prefix in binary RPM package names [1]. Thus you must explicitly define the Python version by using either `python2-` or `python3-`. The second change bans depending on anything using the ambiguous `python-` prefixed names in (Build)Requires [2]. The packages that violate the above-mentioned policies are being tracked in portingdb [3] and we plan to start filling bugs soon . Since th at's a lot of bugs to file (more than 1000) , we encourage all maintainers to fix the packages they maintain. The changes needed to rename the package s are documented here [4]. For your convenience, we have also implemented taskotron automated checks, which run whenever a package is built in Koji . Y ou can see the results in bodhi once you submit an update. The templates we are going to use for bugs' desctiptions can be found here [5]. Any feedback or suggestions are welcome ! [0] https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject… [1] https://fedoraproject.org/wiki/Packaging:Naming?rd=Packaging:NamingGuidelin… [2] https://fedoraproject.org/wiki/Packaging:Python #Dependencies [3] http://fedora.portingdb.xyz/namingpolicy/ [4] https://python-rpm-porting.readthedocs.io/en/latest/naming-scheme.html [5] https://fedoraproject.org/wiki/User:Ishcherb/Package_Naming_Bug_Filing Kind regards, Iryna Shcherbina Associate Software Engineer Red Hat

3 8

Re: Proposed Mass Bug Filing: Renaming "python-" binary packages to "python2-"
by Iryna Shcherbina 19 Jun '17

19 Jun '17

The reason behind the second change is that there are a lot of Python 2 only packages having `python-xxx` in Requires, which will start pulling Python 3 dependencies when we switch to Python 3 as default. This will most possibly cause a number of unexpected issues, which is what we are trying to avoid. If you expect your package to work when we switch to Python 3 as default, then you should already switch it to Python 3 or provide a Python 3 subpackage and depend on `python3-xxx`. Iryna On 06/19/2017 04:11 PM, Tom Hughes wrote: > On 19/06/17 14:59, Iryna Shcherbina wrote: > >> Recently, there were two important changes to Python packaging >> guidelines. The first one bans usingtheambiguous `python` prefix in >> binary RPM package names [1]. Thus you must explicitly >> definethePython version by using either `python2-` or `python3-`. The >> second change bans depending on anything using the ambiguous >> `python-` prefixed names in (Build)Requires [2]. > > I don't understand the logic behind the second change with regard to > Requires, or at least I think there are cases where it makes no sense. > > If I have a package with a python script using #!/usr/bin/python and > importing xxx then isn't python-xxx the correct thing to require so > that it will continue to behave correctly when we switch to python 3 > as default? > > If I change it to python2-xxx then I'll just have to change it again > when we switch the default? > > Tom >

1 0

Should Python 3 macros us UTF-8 locale?
by Miro Hrončok 18 Jun '17

18 Jun '17

Hi Pythonistas. Regarding our Python 3 C.UTF-8 locale coercing [1], aka PEP 538 [2]. As you probably know, we build RPM packages with the C locale. So everytime we use python3 in the spec file, the coercing message is shown. This can be more problematic than just spamming the build logs, see for example the related rpmlint bug [3][4]. Our macros, such as %{python3_sitelib}, %{python3_version} etc. all call python3 and generate the warning. Should we "fix" our macros to set the LANG to C.UTF-8? If we change the %{__python3} macro entirely, we might get rid of all of those warnings and we will workaround the fact that we build RPM packages with the C locale. On the other hand the packager would not be able to set a desired locale because it will always be overwritten: # The crazy test suite needs Czech locale LANG=cs_CZ.utf8 %{__python3} -m pytest Will become: LANG=cs_CZ.utf8 LANG=C.utf8 /usr/bin/python3 -m pytest So I would not do that. But we can change all other macros in /usr/lib/rpm/macros.d/macros.python3 to set the UTF-8 locale. Would that be wise? Desired? Any thoughts? [1] https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale [2] https://www.python.org/dev/peps/pep-0538/ [3] https://bugzilla.redhat.com/show_bug.cgi?id=1457786 [4] https://bugzilla.redhat.com/show_bug.cgi?id=1436345 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

6 7

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel June 2017