python-devel May 2017

python-devel@lists.fedoraproject.org

3 participants
7 discussions

Speculative idea: incorporating venv into our Python application packaging advice
by Nick Coghlan 23 May '17

23 May '17

My day job currently involves working on a Python CLI (and potentially a backing socket-activated service) that needs to run across Fedora/RHEL/CentOS/SCLs, *without* accidentally exposing a Python level API that we might inadvertently end up needing to support. (Note: this CLI is not being, and will likely never be, proposed for incorporation into Fedora itself - it's a tool to help migrate applications between different operating system versions without doing an in-place upgrade, so the update cycles need to be decoupled from those of the operating system itself) At the moment, we offer two different ways of installing that: 1. via pipsi, which uses the system Python, but has no access to system level Python libraries 2. via RPM, which has access to system level Python libraries, exposes the application's internal libraries for import by other applications (which we don't really want to do) and also requires that *all* dependencies be available as system packages Both approaches have significant downsides: * the pipsi based approach is *too* decoupled from the host OS, installing things into the virtual environment even when a perfectly acceptable version is already installed and maintained as a system package. It also means we can't benefit from distro level patches to packages like requests, so the app is decoupled from the system certificate store * the RPM based approach isn't decoupled from the OS *enough*, so we can't readily do things like selectively installing private copies of newer versions of dependencies on RHEL/CentOS, while using the system packages on Fedora. It also means the Python packages implementing the application itself are globally available for import rather than only being usable from within the application While we haven't implemented it yet, the approach I'm considering to tackle this problem [1] involves integrating creation of an app-specific private virtual environment into the definition of the application RPM, with the following details: * unlike pipsi, this virtual environment would be configured to allow access to the system site packages, giving us the best of both worlds: we'd use system packages if readily available, otherwise we'd stick our own pinned dependency in the virtual env and treat it as part of the application (and hence the app developers' responsibility to keep up to date) * we'd come up with some way of turning the Python level dependencies into additional entries in the RPM's Sources list, and then turn those into a local sdist index during the %prep phase. That way, we'd support offline builds automatically, and be well positioned to have pip autofill any gaps where system level dependencies didn't meet the needs of the application * we'd deliberately omit some of the packages injected into the virtual environment from the resulting RPM (most notably: we'd either remove pip, wheel, and setuptools, or else avoid installing them in the first place) Where I think this idea crosses over into being a suitable topic for the Fedora Python SIG relates to the current modularity initiatives and various problems we've faced over the years around separating the challenges of "provide an application that happens to be written in Python" and "provide a supported Python API as part of the system Python installation". Some examples: * the helper library for the "mock" CLI tool had to be renamed to "mockbuild" to fix a conflict with the upstream "mock" testing library * despite officially having no supported public API, people still write "import pip" instead of running the pip CLI in a subprocess * ditto for the yum CLI (and even for DNF, some non-trivial changes were recently needed to better separate the "supported for third party use with defined backwards compatibilty guarantees" APIs from the "for internal use by the DNF CLI and may change at any time" APIs All of those could have been avoided if the recommended structure for "applications that happen to be written in Python" included a virtual environment that isolated the "private to the application" Python modules (including the application's own source code) from the "intended for third party consumption" public APIs. In the near term, my own focus is going to be on figuring out the details of this structure specifically for LeApp, but I wanted to raise the notion here early so I didn't go down any paths that would later prove to be an absolute deal-breaker for updating the distro level recommendations. Cheers, Nick. [1] https://github.com/leapp-to/prototype/issues/126 -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1

[ANN] Matplotlib v2.0.2
by Thomas Caswell 10 May '17

10 May '17

Folks, Happy to announce Matplotlib 2.0.2 which fixes a number of critical bugs in 2.0.1. Wheels and source are available on pypi and conda packages from the conda-forge channel pip install --upgrade matplotlib conda install matplotlib -c conda-forge Tom

1 0

[REL] Matplotlib v2.0.1
by Thomas Caswell 02 May '17

02 May '17

Folks, Happy to announce the only slightly delayed release of 2.0.1, the first bug fix release for 2.0.x series. Source and wheels are an pypi and conda packages are available from the conda forge channel pip install --upgrade matplotlib or conda install matplotlib -c conda-forge This release includes bug fixes, documentation updates and one major change. API - Do not clip the linewidth used for dash pattern scaling. This is an API change, but is a major improvement. - Deprecate 'vega' color names in favor of 'tab'. Bug Fixes - Hatching color follows edge color again. - Fixes a critical bug with masked images. - Improved high-dpi support for Qt5. - Log ticking no never skip minor ticks - Do not skip points surrounded by nan/inf in vector outputs. Supported versions of Python - 2.7, 3.4, 3.5, 3.6 - Patches to improve compatibility with pypy Many thanks to everyone who helped with this release! Thanks to Mathew Brett and Christoph Gohlke for building the wheels. Tom

1 0

Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan 01 May '17

01 May '17

On 2 May 2017 at 09:36, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote: > On Mon, May 1, 2017 at 9:14 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> On 1 May 2017 at 22:47, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote: >>> On Sun, Apr 30, 2017 at 10:30 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >>>> If the intended benefit of this change remains unclear, it may help to >>>> focus on a specific concrete case, which would be that the following >>>> operations should be completely indistinguishable at the system level >>>> from not having done anything at all (except in the sudo logs): >>>> >>>> $ sudo pip3 install contextlib2 >>>> $ sudo pip3 uninstall contextlib2 >>> >>> And this successfully ignores *all* the dependencies which contextlib2 >>> may add at installation time. >> >> There's a reason I chose contextlib2 as my example rather than >> something more complex like ansible - I'm the maintainer of >> contextlib2, and I know it doesn't have any runtime dependencies (and >> likely never will, since it's a stdlib module backport). > > Heh: it seems as if you are cheating somewhat by choosing one of the > easiest components. I'm concerned that casual use of a new "pip3 > install" localized utility will replicate the problems I described > fairly quickly, and in ways that someone not as experienced with > dependency violations will have difficulty cleaning up. I'm > particularly thinking of "awscli" and the various Sphinx version > dependencies of its dependency tree. Nico, this suggests you're still missing the essential point of the proposal. Status quo: people run "sudo pip3 install <whatever>" and *corrupt their system (or SCL) packages*. Remediation in the case of system breakage will be some form of clearing out the pip installed components, and reinstalling the dnf/yum managed ones. Proposed change: people run "sudo pip3 install <whatever>" and only `/usr/local` (or the equivalent `/opt/...` directory for SCLs) gets modified. This will often "just work" (when there's no conflict with system packages), and even when it doesn't, remediation only involves removing the pip installed packages - the dnf/yum managed ones should still be in their original pristine state. Saying "This only mitigates the potential problems, it doesn't completely eliminate them" doesn't make sense as a response to a harm mitigation measure. That said, I re-read the original change proposal page, and that *does* give the impression that it completely solves the problem (since even the title uses words like "safe" rather than the more measured "safer"), so I've suggested on the Python SIG list that it be reworded to better explain the scope and potential benefits. >> Given the proposal at hand though, writing a >> `remove-pip-installed-modules` cleaner utility becomes a lot simpler >> that it is today, since it just needs to clear out any Python packages >> it finds in /usr/local (based on the Python level installation >> records), rather than needing to interact with the RPM database, >> figure out which system packages may have potentially been corrupted >> and reinstall them. > > That makes sense, at least to me. I continue to urge you to keep it > *out* of the default compiled in equivalents of PYTHONPATH. Perhaps > there needs to be something like a "pip3.local", to activate such such > modules in a better segregated space? That's what per-user and venv installs are for, as well as SCLs and containers at the OS level. One thing I would like to start doing is more strongly encouraging the use of `pipsi` ("pip script installer") when installing things like awscli directly from PyPI for personal or system wide use, since that automatically takes care of creating an application specific virtual environment for the tool's dependencies. However, there are still some issues with that around availability (pipsi isn't in the Python SCLs or EPEL at this point), as well as around readily upgrading components in the per-application venvs. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

What is your opinion on "sudo pip" fix for Fedora 27?
by Michal Cyprian 01 May '17

01 May '17

At the present time, running sudo pip3 in Fedora is not safe. Pip shares its installation directory with dnf, can remove dnf-managed files and generally break the Python 3 interpreter. Our first attempt to make sudo pip safe on Fedora [0] was based on using a different binary (/usr/libexec/system-python) for RPM packaging purposes and changing the behavior of /usr/bin/python3 (and pip that uses its settings) to install under /usr/local by default. Switching all the Python 3 packages to system-python turned out to be much more problematic than expected and we had to postpone the Change for F27. We decided to try a different approach. The main idea is to detect an ongoing RPM build and modify the behavior of the Python 3 executable only when in normal user environment so that RPM builds won't be affected at all. The adjustment of the behavior can be done on different levels. The first option is to set the sys.prefix variable to /usr/local when the interpreter is initialized. This will affect all the install methods, but the solution can cause some problems in applications that rely on the value of sys.prefix. A prototype of this implementation can be seen here [1]. The other possibility is to limit the pip install location change to distutils and pip [2]. This is the "safer" option, but does not cover all corner cases. For example, Python software built locally using cmake or similar tools will be installed into /usr/lib and can conflict with system tools. Debian chose to implement similar solution. I would like to ask which solution would work for your applications better, and what is in your opinion the right way to go. I will appreciate any costructive comments or ideas on how to improve these patches. If you want to check some specific use-case you can use the images docker.io/mcyprian/sudo_pip_sys_prefix and docker.io/mcyprian/sudo_pip_install_prefix to try out the two mentioned implementations. [0] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe?rd=User:Torsava… [1] https://github.com/mcyprian/python3/commit/6077c96460c03ddc334bae654fffcce4… [2] https://github.com/mcyprian/python3/commit/ac2f266a90871d67ba8d0b70d2b0ebe4… Michal Cyprian

4 9

Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan 01 May '17

01 May '17

On 1 May 2017 at 22:47, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote: > On Sun, Apr 30, 2017 at 10:30 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> If the intended benefit of this change remains unclear, it may help to >> focus on a specific concrete case, which would be that the following >> operations should be completely indistinguishable at the system level >> from not having done anything at all (except in the sudo logs): >> >> $ sudo pip3 install contextlib2 >> $ sudo pip3 uninstall contextlib2 > > And this successfully ignores *all* the dependencies which contextlib2 > may add at installation time. There's a reason I chose contextlib2 as my example rather than something more complex like ansible - I'm the maintainer of contextlib2, and I know it doesn't have any runtime dependencies (and likely never will, since it's a stdlib module backport). Given the proposal at hand though, writing a `remove-pip-installed-modules` cleaner utility becomes a lot simpler that it is today, since it just needs to clear out any Python packages it finds in /usr/local (based on the Python level installation records), rather than needing to interact with the RPM database, figure out which system packages may have potentially been corrupted and reinstall them. If the proposal is adjusted to also affect other installation directories (like the one for scripts), it will also have the significant benefit that any pip installed binaries will go into `/usr/local/bin`, so they'll appear on the default path for all regular users, but *won't* appear on the default path for "root". Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan 30 Apr '17

30 Apr '17

On 1 May 2017 at 09:59, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote: > Wouldn't it be more effective, more safe, and more efficient to > improve the existing support for doing "the right thing", rather than > trying to outsmart a build and deployment system that is essentially > outside your control? To encourage the use of "pyvenv" for local such > modules, in a local userspace, rather than as a default system > version? This is already happening, and is entirely independent of the discussion at hand (e.g. see [1], a presentation I did with Graham Dumpleton at Red Hat Summit/DevNation last year advocating for Red Hat's customers to adopt exactly this model). On Fedora's Developer Portal, virtual environments share the front page of the Python section with running Python itself: https://developer.fedoraproject.org/tech/languages/python/python-installati… Responding to "We want to make running 'sudo pip install' less harmful and easier to recover from when people do it by mistake" with "We should discourage the use of 'sudo pip install'" remains an unproductive non sequitur - we *know* it's inherently fragile, and requires deep knowledge of both Python packaging and RPM packaging to do reliably. However, Red Hat can't (and won't) answer a commercial support call from a distressed customer with "Ooh, you shouldn't have done that - sucks to be you" (although these supportability problems *are* one of the reasons we only ship "pip" commercially through Software Collections, and not as part of the system Python runtime). Fortunately, we have the ability to influence *all* of the moving parts (upstream pip, upstream CPython, downstream distros) here, and help to incrementally adjust their respective default behaviours so that the key risks involved in running "sudo pip install" are comparable to those involved in running "curl $REMOTE_SCRIPT | sudo sh". If the intended benefit of this change remains unclear, it may help to focus on a specific concrete case, which would be that the following operations should be completely indistinguishable at the system level from not having done anything at all (except in the sudo logs): $ sudo pip3 install contextlib2 $ sudo pip3 uninstall contextlib2 At the moment, that's *not* the case if you had previously done "sudo dnf install python3-contextlib2", as both of the above commands will mess with the system contextlib2 packages. With the proposed changes, contextlib2 may behave unexpectedly while the non-system version is installed, but removing the pip installed version will be sufficient to restore the affected machine to its previous state. (I chose contextlib2 as the example here, rather than something more complex like ansible, as pip installing components with dependencies makes the cleanup process more complicated. However, the /usr/local/ split helps there as well, by making the installed-via-pip versions easier to identify even without checking the PEP 376 installation metadata) Cheers, Nick. [1] https://www.slideshare.net/ncoghlan_dev/developing-in-python-on-red-hat-pla… -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel May 2017