python-devel April 2017

python-devel@lists.fedoraproject.org

10 participants
9 discussions

What is your opinion on "sudo pip" fix for Fedora 27?
by Michal Cyprian 01 May '17

01 May '17

At the present time, running sudo pip3 in Fedora is not safe. Pip shares its installation directory with dnf, can remove dnf-managed files and generally break the Python 3 interpreter. Our first attempt to make sudo pip safe on Fedora [0] was based on using a different binary (/usr/libexec/system-python) for RPM packaging purposes and changing the behavior of /usr/bin/python3 (and pip that uses its settings) to install under /usr/local by default. Switching all the Python 3 packages to system-python turned out to be much more problematic than expected and we had to postpone the Change for F27. We decided to try a different approach. The main idea is to detect an ongoing RPM build and modify the behavior of the Python 3 executable only when in normal user environment so that RPM builds won't be affected at all. The adjustment of the behavior can be done on different levels. The first option is to set the sys.prefix variable to /usr/local when the interpreter is initialized. This will affect all the install methods, but the solution can cause some problems in applications that rely on the value of sys.prefix. A prototype of this implementation can be seen here [1]. The other possibility is to limit the pip install location change to distutils and pip [2]. This is the "safer" option, but does not cover all corner cases. For example, Python software built locally using cmake or similar tools will be installed into /usr/lib and can conflict with system tools. Debian chose to implement similar solution. I would like to ask which solution would work for your applications better, and what is in your opinion the right way to go. I will appreciate any costructive comments or ideas on how to improve these patches. If you want to check some specific use-case you can use the images docker.io/mcyprian/sudo_pip_sys_prefix and docker.io/mcyprian/sudo_pip_install_prefix to try out the two mentioned implementations. [0] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe?rd=User:Torsava… [1] https://github.com/mcyprian/python3/commit/6077c96460c03ddc334bae654fffcce4… [2] https://github.com/mcyprian/python3/commit/ac2f266a90871d67ba8d0b70d2b0ebe4… Michal Cyprian

4 9

Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan 28 Apr '17

28 Apr '17

On 27 April 2017 at 11:47, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote: > On Wed, Apr 26, 2017 at 7:13 AM, Charalampos Stratakis >> At the present time, running sudo pip3 in Fedora is not safe. >> Pip shares its installation directory with dnf, can remove >> dnf-managed files and generally break the Python 3 interpreter. > > This is true of every version of pip, not merely pip3. It was also > true of CPAN, and of many gradle, maven, and ant working environments > that semi-randomly collate the very latest versions of indeterminate > components and spray them on top of your system workspace with their > own quite distinct ideas about packaging and versioning. > > There is no solution to this problem, except to use tools like > "pyvenv" to set aside secluded workspaces for Python modules and their > dependencies assembly. So, for most developers, they *should not* use > "sudo pip". They should remain in user space, or possibly in shared > workspaces, to avoid overwriting system components. Nothing is changing in terms of *recommended* practices. This change proposal is entirely about harm mitigation for the cases where users *do* run "sudo pip ...", either because that's their instinctive reaction to a permissions error, or because some misguided installation instructions for a 3rd party package advised them to do it. Debian and derivatives already mitigate the potential harm for these cases by requiring the "--install-layout=deb" option to be passed to get distutils to install into the system directories rather than doing it by default: https://wiki.debian.org/Python#Deviations_from_upstream Their approach means that any harm caused by "sudo pip install X" can subsequently be fully reversed by doing "sudo pip uninstall X". At this moment, this is NOT true for Fedora and derivatives. Instead, the remediation step here is "sudo pip uninstall X && sudo dnf reinstall <something>" where you have to: 1. Figure out what "<something>" needs to be 2. Hope that whatever you broke didn't affect your ability to run "sudo dnf reinstall" >> Our first attempt to make sudo pip safe on Fedora [0] was >> based on using a different binary (/usr/libexec/system-python) >> for RPM packaging purposes and changing the behavior >> of /usr/bin/python3 (and pip that uses its settings) to install >> under /usr/local by default. Switching all the Python 3 packages > > Oh, Oh my. The extent to which this unworkable is... fascinating. Nico, the contemptuous tone you have adopted in this post is entirely inappropriate. Please give other list participants the courtesy of assuming expertise at least equivalent to your own, rather than engaging in attempted displays of assumed superiority. > You > will inevitably install incompatible future versions of critical > packages which are incompatible with the Fedora system packages. And > as soon as you install one that breaks RPM, if /usr/local/ if ahead of > /usr in your search path for modules, you are *screwed*. This isn't true, as long as the fix is as simple as using pip to uninstall whatever problematic package you just installed. At the moment, that isn't the case on Fedora and derivatives, since "sudo pip install ..." can actually *corrupt* a system package installation, rather than merely shadowing it at import time the way it does on Debian et al. > And if > /usr/local is first, well, an RPM update may introduce a component > that is incompatible with the pip installed modules at arbitrary > times. The goal of this change isn't to allow co-existence of arbitrary upstream and downstream versions as supported combinations - it's to make sure that forced installation of an incompatible upstream version is an easily reversible error, rather than a potentially system-breaking mistake. >> We decided to try a different approach. The main idea is to detect >> an ongoing RPM build and modify the behavior of the Python 3 >> executable only when in normal user environment so that RPM builds >> won't be affected at all. > > See https://xkcd.com/1172/ . You're inventing a workflow that isn't > supported by *anyone*. The problem is that the default behaviour needs to change (to mitigate the potential harm caused by a relatively common end user error), but unlike Debian package builds (which know they need to set "--install-layout=deb"), most RPM builds for Python packages currently rely on that default behaviour as part of the build process. Changing the default indiscriminately would thus be a breaking change that not only affects Fedora RPM builds, but also COPR and other third party RPM builds. So we need a solution that retains the existing behaviour for RPM builds, but changes the behaviour to be less potentially destructive when run interactively at a shell prompt, or as part of a configuration management script. >> The adjustment of the behavior can be done on different levels. >> The first option is to set the sys.prefix variable to /usr/local >> when the interpreter is initialized. This will affect >> all the install methods, but the solution can cause some >> problems in applications that rely on the value of sys.prefix. >> A prototype of this implementation can be seen here [1]. > > I'm sorry, but.... ouch. Please stop trying to implement "pip install" > as a system installation tool. That is not the goal of the change. > The effort would be better spent > upgrading and enhancing "py2pack". Improving the level of commonality and collaboration between Fedora & Mageia's pyp2rpm and openSUSE's py2pack would indeed be desirable, but it's entirely orthogonal to the harm mitigation exercise being discussed here. >> The other possibility is to limit the pip install location change >> to distutils and pip [2]. This is the "safer" option, but does >> not cover all corner cases. For example, Python software built >> locally using cmake or similar tools will be installed into >> /usr/lib and can conflict with system tools. Debian chose to >> implement similar solution. > > It's local environment specific, unstable software sensitive to > arbitrary and unmanaged upstream dependency updates, and extremely > sensitive to local system pip modules. It *does not belong* in > /usr/lib. Because the components are modular and bundled in a non-RPM > compatible fashion, it behooves developers to use a tool to segregate > the tools as much as feasible from the Fedora underlying > infrastructure. i.e., use pyvenv to build them in a contained > environment segregated from the system files. Helping users that are already doing the right thing isn't the problem at hand - this is about making it easier for users to recover from doing the *wrong* thing. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 3

Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan 27 Apr '17

27 Apr '17

On 27 April 2017 at 23:04, Daniel P. Berrange <berrange(a)redhat.com> wrote: > On Thu, Apr 27, 2017 at 04:32:09PM +1000, Nick Coghlan wrote: >> Their approach means that any harm caused by "sudo pip install X" can >> subsequently be fully reversed by doing "sudo pip uninstall X". >> >> At this moment, this is NOT true for Fedora and derivatives. Instead, >> the remediation step here is "sudo pip uninstall X && sudo dnf >> reinstall <something>" where you have to: >> >> 1. Figure out what "<something>" needs to be >> 2. Hope that whatever you broke didn't affect your ability to run >> "sudo dnf reinstall" > > Yep, recovering the system python install to a pristine state after > 'devstack' has done its stuff is very painful right now :-( > > devstack was originally written for Ubuntu systems, so I guess they > don't suffer as much due to the Debian specific change you describe > above. > > Which location takes priority in Debian if the same module is installed > in both places ? The DPKG location, or the Pip location ? Presumably it > would have to be the pip version that takes priority if you want it to > be usable for updating the (likely older) distro provided versions. Aye, /usr/local/lib has priority: $ docker run --rm ncoghlan/debian-python python3 -m site sys.path = [ '/', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages', ] USER_BASE: '/root/.local' (doesn't exist) USER_SITE: '/root/.local/lib/python3.4/site-packages' (doesn't exist) ENABLE_USER_SITE: True User level package installs take priority over system level ones (regardless of distro), so the current container build use case for root level pip installations can typically be met equally well by running as a non-root user inside the container and doing "pip install --user ...". Unfortunately, as with the venv option, there are a *lot* of container build scripts out there that don't do that :( Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Status update for PEP 538 (coercing the legacy C locale to C.UTF-8)
by Nick Coghlan 24 Apr '17

24 Apr '17

Hi folks, The Fedora 26 Alpha currently includes our downstream patch backporting PEP 538 to Python 3.6 - an update to CPython to implicitly coerce the legacy C locale to C.UTF-8 as the Python interpreter process starts up. I'm still aiming to have that PEP officially accepted for 3.7 upstream before the F26 beta deadline (currently May 16th), but hit a slight hitch recently when Barry Warsaw had to withdraw as BDFL-Delegate due to a lack of sufficient time to devote to the review process. Fortunately, Naoki Inada has agreed to take over the role, and Guido signed off on the updated delegation today: https://mail.python.org/pipermail/python-dev/2017-April/147796.html So with any luck, we should be able to get this change explicitly approved by upstream within the next couple of weeks. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1

Python 3 windows executables templates moved to python3-devel
by Charalampos Stratakis 20 Apr '17

20 Apr '17

Hello all, As of the python3 update for F26 [0] some windows executable files that reside at /usr/lib/python3.6/distutils/command/ have been moved to the python3-devel subpackage from the python3-libs [1]. These files are used for creating custom windows installers for python modules, so if you were using these templates for/during your workflow, you will have to install python3-devel to get them. [0] https://bodhi.fedoraproject.org/updates/FEDORA-2017-e1e5e717c6 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1426257 Regards, Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat

1 0

Troubles with Python 3.6.1 in Fedora 26
by Miro Hrončok 20 Apr '17

20 Apr '17

There is and update of Python to 3.6.1 coming to Fedora 26 [1]. Due to a bug [2][3] tho following scenario might happen: 0. user has python3 3.6.0 and python3-foo-0.1-1 1. python3 3.6.1 is pushed to Fedora 26 stable 2. maintainer of python3-foo rebuilds the package to python3-foo-0.1-2 (this happens against python3 3.6.1) and pushes it to updates 3. user updates to python3-foo-0.1-2, but does not update python3 4. if the Python foo module meets certain conditions, it is broken 5. user can `dnf update python3` to fix the issue Because Fedora 26 is not yet released, if we push Python 3.6.1 now, this problem can only affect the users who run Alpha and do partial dnf updates, and a full dnf update fixes the issue. So we decided to push Python 3.6.1 to Fedora anyway. Should I note this at the common bugs page [4]? I don't think the scenario is common enough, but I feel a need to inform about it somewhere. [1] https://bodhi.fedoraproject.org/updates/FEDORA-2017-e1e5e717c6 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1435135 [3] http://bugs.python.org/issue29943 [4] https://fedoraproject.org/wiki/Common_F26_bugs -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

self introduction
by Lumir Balhar 17 Apr '17

17 Apr '17

Hello. My name is Lumír Balhar and I am a member of Python maintenance team in Red Hat and Fedora contributor. I want to be a member of Python SIG to be able to help with important Python packages in Fedora. Have a nice day. Lumír

2 1

Anyone to maintain python2-ipython?
by Miro Hrončok 12 Apr '17

12 Apr '17

Hi, IPython is dropping support for Python 2 in it's 6.0.0 release [1], while the 5.x version remains long term supported with Python 2 support [2]. In order to update the ipython package in Fedora to 6.x, we'd need to move the python2 bits to a separate package (python2-ipython). Is there anyone in the SIG, who's willing to maintain the python2 version of ipython? Thanks Thomas Spura (current owner) and me - we are not interested, see the relevant BZ for more information [3]. [1] http://ipython.readthedocs.io/en/latest/ [2] http://blog.jupyter.org/2016/07/08/ipython-5-0-released/ [3] https://bugzilla.redhat.com/show_bug.cgi?id=1441004 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

ipykernel to recommend stuff?
by Miro Hrončok 09 Apr '17

09 Apr '17

Hi, I got this idea that the python{2,3}-ipykernel packages could recommend some modules useful to use in the Jupyter Notebook. Recommends: * numpy * matplotlib Not sure if should be recommended or suggested: * pillow * scikit-image * scikit-learn * scipy * statsmodels * sympy * pandas What do you think? It would mean that a user who dnf installs notebook gets some opt-outable things useful in the notebook. And who uses the Python 2 kernel, will get the Python 2 version. This would also drag the things to the Python Classroom Notebook which I think is good - it will make the Notebook much easier to use for teaching Python for science - without the need to start with virtualenvs first or install additional packages. But of course it would also make the image bigger. Also see [1] for relevant issue just for the Python Classroom Lab. [1] https://github.com/fedora-python/fedora-kickstarts-python-classroom/issues/1 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

5 8

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel April 2017