Speculative idea: incorporating venv into our Python application
packaging advice
by Nick Coghlan
My day job currently involves working on a Python CLI (and potentially
a backing socket-activated service) that needs to run across
Fedora/RHEL/CentOS/SCLs, *without* accidentally exposing a Python
level API that we might inadvertently end up needing to support.
(Note: this CLI is not being, and will likely never be, proposed for
incorporation into Fedora itself - it's a tool to help migrate
applications between different operating system versions without doing
an in-place upgrade, so the update cycles need to be decoupled from
those of the operating system itself)
At the moment, we offer two different ways of installing that:
1. via pipsi, which uses the system Python, but has no access to
system level Python libraries
2. via RPM, which has access to system level Python libraries, exposes
the application's internal libraries for import by other applications
(which we don't really want to do) and also requires that *all*
dependencies be available as system packages
Both approaches have significant downsides:
* the pipsi based approach is *too* decoupled from the host OS,
installing things into the virtual environment even when a perfectly
acceptable version is already installed and maintained as a system
package. It also means we can't benefit from distro level patches to
packages like requests, so the app is decoupled from the system
certificate store
* the RPM based approach isn't decoupled from the OS *enough*, so we
can't readily do things like selectively installing private copies of
newer versions of dependencies on RHEL/CentOS, while using the system
packages on Fedora. It also means the Python packages implementing the
application itself are globally available for import rather than only
being usable from within the application
While we haven't implemented it yet, the approach I'm considering to
tackle this problem [1] involves integrating creation of an
app-specific private virtual environment into the definition of the
application RPM, with the following details:
* unlike pipsi, this virtual environment would be configured to allow
access to the system site packages, giving us the best of both worlds:
we'd use system packages if readily available, otherwise we'd stick
our own pinned dependency in the virtual env and treat it as part of
the application (and hence the app developers' responsibility to keep
up to date)
* we'd come up with some way of turning the Python level dependencies
into additional entries in the RPM's Sources list, and then turn those
into a local sdist index during the %prep phase. That way, we'd
support offline builds automatically, and be well positioned to have
pip autofill any gaps where system level dependencies didn't meet the
needs of the application
* we'd deliberately omit some of the packages injected into the
virtual environment from the resulting RPM (most notably: we'd either
remove pip, wheel, and setuptools, or else avoid installing them in
the first place)
Where I think this idea crosses over into being a suitable topic for
the Fedora Python SIG relates to the current modularity initiatives
and various problems we've faced over the years around separating the
challenges of "provide an application that happens to be written in
Python" and "provide a supported Python API as part of the system
Python installation".
Some examples:
* the helper library for the "mock" CLI tool had to be renamed to
"mockbuild" to fix a conflict with the upstream "mock" testing library
* despite officially having no supported public API, people still
write "import pip" instead of running the pip CLI in a subprocess
* ditto for the yum CLI (and even for DNF, some non-trivial changes
were recently needed to better separate the "supported for third party
use with defined backwards compatibilty guarantees" APIs from the "for
internal use by the DNF CLI and may change at any time" APIs
All of those could have been avoided if the recommended structure for
"applications that happen to be written in Python" included a virtual
environment that isolated the "private to the application" Python
modules (including the application's own source code) from the
"intended for third party consumption" public APIs.
In the near term, my own focus is going to be on figuring out the
details of this structure specifically for LeApp, but I wanted to
raise the notion here early so I didn't go down any paths that would
later prove to be an absolute deal-breaker for updating the distro
level recommendations.
Cheers,
Nick.
[1] https://github.com/leapp-to/prototype/issues/126
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 6 months
[ANN] Matplotlib v2.0.2
by Thomas Caswell
Folks,
Happy to announce Matplotlib 2.0.2 which fixes a number of critical bugs in
2.0.1.
Wheels and source are available on pypi and conda packages from the
conda-forge channel
pip install --upgrade matplotlib
conda install matplotlib -c conda-forge
Tom
6 years, 6 months
[REL] Matplotlib v2.0.1
by Thomas Caswell
Folks,
Happy to announce the only slightly delayed release of 2.0.1, the first bug
fix release for 2.0.x series. Source and wheels are an pypi and conda
packages are available from the conda forge channel
pip install --upgrade matplotlib
or
conda install matplotlib -c conda-forge
This release includes bug fixes, documentation updates and one major change.
API
- Do not clip the linewidth used for dash pattern scaling. This is
an API change, but is a major improvement.
- Deprecate 'vega' color names in favor of 'tab'.
Bug Fixes
- Hatching color follows edge color again.
- Fixes a critical bug with masked images.
- Improved high-dpi support for Qt5.
- Log ticking no never skip minor ticks
- Do not skip points surrounded by nan/inf in vector outputs.
Supported versions of Python
- 2.7, 3.4, 3.5, 3.6
- Patches to improve compatibility with pypy
Many thanks to everyone who helped with this release!
Thanks to Mathew Brett and Christoph Gohlke for building the wheels.
Tom
6 years, 7 months
Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan
On 2 May 2017 at 09:36, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote:
> On Mon, May 1, 2017 at 9:14 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote:
>> On 1 May 2017 at 22:47, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote:
>>> On Sun, Apr 30, 2017 at 10:30 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote:
>>>> If the intended benefit of this change remains unclear, it may help to
>>>> focus on a specific concrete case, which would be that the following
>>>> operations should be completely indistinguishable at the system level
>>>> from not having done anything at all (except in the sudo logs):
>>>>
>>>> $ sudo pip3 install contextlib2
>>>> $ sudo pip3 uninstall contextlib2
>>>
>>> And this successfully ignores *all* the dependencies which contextlib2
>>> may add at installation time.
>>
>> There's a reason I chose contextlib2 as my example rather than
>> something more complex like ansible - I'm the maintainer of
>> contextlib2, and I know it doesn't have any runtime dependencies (and
>> likely never will, since it's a stdlib module backport).
>
> Heh: it seems as if you are cheating somewhat by choosing one of the
> easiest components. I'm concerned that casual use of a new "pip3
> install" localized utility will replicate the problems I described
> fairly quickly, and in ways that someone not as experienced with
> dependency violations will have difficulty cleaning up. I'm
> particularly thinking of "awscli" and the various Sphinx version
> dependencies of its dependency tree.
Nico, this suggests you're still missing the essential point of the proposal.
Status quo: people run "sudo pip3 install <whatever>" and *corrupt
their system (or SCL) packages*. Remediation in the case of system
breakage will be some form of clearing out the pip installed
components, and reinstalling the dnf/yum managed ones.
Proposed change: people run "sudo pip3 install <whatever>" and only
`/usr/local` (or the equivalent `/opt/...` directory for SCLs) gets
modified. This will often "just work" (when there's no conflict with
system packages), and even when it doesn't, remediation only involves
removing the pip installed packages - the dnf/yum managed ones should
still be in their original pristine state.
Saying "This only mitigates the potential problems, it doesn't
completely eliminate them" doesn't make sense as a response to a harm
mitigation measure.
That said, I re-read the original change proposal page, and that
*does* give the impression that it completely solves the problem
(since even the title uses words like "safe" rather than the more
measured "safer"), so I've suggested on the Python SIG list that it be
reworded to better explain the scope and potential benefits.
>> Given the proposal at hand though, writing a
>> `remove-pip-installed-modules` cleaner utility becomes a lot simpler
>> that it is today, since it just needs to clear out any Python packages
>> it finds in /usr/local (based on the Python level installation
>> records), rather than needing to interact with the RPM database,
>> figure out which system packages may have potentially been corrupted
>> and reinstall them.
>
> That makes sense, at least to me. I continue to urge you to keep it
> *out* of the default compiled in equivalents of PYTHONPATH. Perhaps
> there needs to be something like a "pip3.local", to activate such such
> modules in a better segregated space?
That's what per-user and venv installs are for, as well as SCLs and
containers at the OS level.
One thing I would like to start doing is more strongly encouraging the
use of `pipsi` ("pip script installer") when installing things like
awscli directly from PyPI for personal or system wide use, since that
automatically takes care of creating an application specific virtual
environment for the tool's dependencies.
However, there are still some issues with that around availability
(pipsi isn't in the Python SCLs or EPEL at this point), as well as
around readily upgrading components in the per-application venvs.
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 7 months
What is your opinion on "sudo pip" fix for Fedora 27?
by Michal Cyprian
At the present time, running sudo pip3 in Fedora is not safe.
Pip shares its installation directory with dnf, can remove
dnf-managed files and generally break the Python 3 interpreter.
Our first attempt to make sudo pip safe on Fedora [0] was
based on using a different binary (/usr/libexec/system-python)
for RPM packaging purposes and changing the behavior
of /usr/bin/python3 (and pip that uses its settings) to install
under /usr/local by default. Switching all the Python 3 packages
to system-python turned out to be much more problematic than
expected and we had to postpone the Change for F27.
We decided to try a different approach. The main idea is to detect
an ongoing RPM build and modify the behavior of the Python 3
executable only when in normal user environment so that RPM builds
won't be affected at all.
The adjustment of the behavior can be done on different levels.
The first option is to set the sys.prefix variable to /usr/local
when the interpreter is initialized. This will affect
all the install methods, but the solution can cause some
problems in applications that rely on the value of sys.prefix.
A prototype of this implementation can be seen here [1].
The other possibility is to limit the pip install location change
to distutils and pip [2]. This is the "safer" option, but does
not cover all corner cases. For example, Python software built
locally using cmake or similar tools will be installed into
/usr/lib and can conflict with system tools. Debian chose to
implement similar solution.
I would like to ask which solution would work for your applications
better, and what is in your opinion the right way to go. I will
appreciate any costructive comments or ideas on how to improve these
patches. If you want to check some specific use-case you can use the images
docker.io/mcyprian/sudo_pip_sys_prefix and
docker.io/mcyprian/sudo_pip_install_prefix
to try out the two mentioned implementations.
[0] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe?rd=User:Torsa...
[1] https://github.com/mcyprian/python3/commit/6077c96460c03ddc334bae654fffcc...
[2] https://github.com/mcyprian/python3/commit/ac2f266a90871d67ba8d0b70d2b0eb...
Michal Cyprian
6 years, 7 months
Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan
On 1 May 2017 at 22:47, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote:
> On Sun, Apr 30, 2017 at 10:30 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote:
>> If the intended benefit of this change remains unclear, it may help to
>> focus on a specific concrete case, which would be that the following
>> operations should be completely indistinguishable at the system level
>> from not having done anything at all (except in the sudo logs):
>>
>> $ sudo pip3 install contextlib2
>> $ sudo pip3 uninstall contextlib2
>
> And this successfully ignores *all* the dependencies which contextlib2
> may add at installation time.
There's a reason I chose contextlib2 as my example rather than
something more complex like ansible - I'm the maintainer of
contextlib2, and I know it doesn't have any runtime dependencies (and
likely never will, since it's a stdlib module backport).
Given the proposal at hand though, writing a
`remove-pip-installed-modules` cleaner utility becomes a lot simpler
that it is today, since it just needs to clear out any Python packages
it finds in /usr/local (based on the Python level installation
records), rather than needing to interact with the RPM database,
figure out which system packages may have potentially been corrupted
and reinstall them.
If the proposal is adjusted to also affect other installation
directories (like the one for scripts), it will also have the
significant benefit that any pip installed binaries will go into
`/usr/local/bin`, so they'll appear on the default path for all
regular users, but *won't* appear on the default path for "root".
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 7 months
Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan
On 1 May 2017 at 09:59, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote:
> Wouldn't it be more effective, more safe, and more efficient to
> improve the existing support for doing "the right thing", rather than
> trying to outsmart a build and deployment system that is essentially
> outside your control? To encourage the use of "pyvenv" for local such
> modules, in a local userspace, rather than as a default system
> version?
This is already happening, and is entirely independent of the
discussion at hand (e.g. see [1], a presentation I did with Graham
Dumpleton at Red Hat Summit/DevNation last year advocating for Red
Hat's customers to adopt exactly this model). On Fedora's Developer
Portal, virtual environments share the front page of the Python
section with running Python itself:
https://developer.fedoraproject.org/tech/languages/python/python-installa...
Responding to "We want to make running 'sudo pip install' less harmful
and easier to recover from when people do it by mistake" with "We
should discourage the use of 'sudo pip install'" remains an
unproductive non sequitur - we *know* it's inherently fragile, and
requires deep knowledge of both Python packaging and RPM packaging to
do reliably. However, Red Hat can't (and won't) answer a commercial
support call from a distressed customer with "Ooh, you shouldn't have
done that - sucks to be you" (although these supportability problems
*are* one of the reasons we only ship "pip" commercially through
Software Collections, and not as part of the system Python runtime).
Fortunately, we have the ability to influence *all* of the moving
parts (upstream pip, upstream CPython, downstream distros) here, and
help to incrementally adjust their respective default behaviours so
that the key risks involved in running "sudo pip install" are
comparable to those involved in running "curl $REMOTE_SCRIPT | sudo
sh".
If the intended benefit of this change remains unclear, it may help to
focus on a specific concrete case, which would be that the following
operations should be completely indistinguishable at the system level
from not having done anything at all (except in the sudo logs):
$ sudo pip3 install contextlib2
$ sudo pip3 uninstall contextlib2
At the moment, that's *not* the case if you had previously done "sudo
dnf install python3-contextlib2", as both of the above commands will
mess with the system contextlib2 packages. With the proposed changes,
contextlib2 may behave unexpectedly while the non-system version is
installed, but removing the pip installed version will be sufficient
to restore the affected machine to its previous state. (I chose
contextlib2 as the example here, rather than something more complex
like ansible, as pip installing components with dependencies makes the
cleanup process more complicated. However, the /usr/local/ split helps
there as well, by making the installed-via-pip versions easier to
identify even without checking the PEP 376 installation metadata)
Cheers,
Nick.
[1] https://www.slideshare.net/ncoghlan_dev/developing-in-python-on-red-hat-p...
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
6 years, 7 months