What is your opinion on "sudo pip" fix for Fedora 27?
by Michal Cyprian
At the present time, running sudo pip3 in Fedora is not safe.
Pip shares its installation directory with dnf, can remove
dnf-managed files and generally break the Python 3 interpreter.
Our first attempt to make sudo pip safe on Fedora [0] was
based on using a different binary (/usr/libexec/system-python)
for RPM packaging purposes and changing the behavior
of /usr/bin/python3 (and pip that uses its settings) to install
under /usr/local by default. Switching all the Python 3 packages
to system-python turned out to be much more problematic than
expected and we had to postpone the Change for F27.
We decided to try a different approach. The main idea is to detect
an ongoing RPM build and modify the behavior of the Python 3
executable only when in normal user environment so that RPM builds
won't be affected at all.
The adjustment of the behavior can be done on different levels.
The first option is to set the sys.prefix variable to /usr/local
when the interpreter is initialized. This will affect
all the install methods, but the solution can cause some
problems in applications that rely on the value of sys.prefix.
A prototype of this implementation can be seen here [1].
The other possibility is to limit the pip install location change
to distutils and pip [2]. This is the "safer" option, but does
not cover all corner cases. For example, Python software built
locally using cmake or similar tools will be installed into
/usr/lib and can conflict with system tools. Debian chose to
implement similar solution.
I would like to ask which solution would work for your applications
better, and what is in your opinion the right way to go. I will
appreciate any costructive comments or ideas on how to improve these
patches. If you want to check some specific use-case you can use the images
docker.io/mcyprian/sudo_pip_sys_prefix and
docker.io/mcyprian/sudo_pip_install_prefix
to try out the two mentioned implementations.
[0] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe?rd=User:Torsa...
[1] https://github.com/mcyprian/python3/commit/6077c96460c03ddc334bae654fffcc...
[2] https://github.com/mcyprian/python3/commit/ac2f266a90871d67ba8d0b70d2b0eb...
Michal Cyprian
6 years, 12 months
Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan
On 27 April 2017 at 11:47, Nico Kadel-Garcia <nkadel(a)gmail.com> wrote:
> On Wed, Apr 26, 2017 at 7:13 AM, Charalampos Stratakis
>> At the present time, running sudo pip3 in Fedora is not safe.
>> Pip shares its installation directory with dnf, can remove
>> dnf-managed files and generally break the Python 3 interpreter.
>
> This is true of every version of pip, not merely pip3. It was also
> true of CPAN, and of many gradle, maven, and ant working environments
> that semi-randomly collate the very latest versions of indeterminate
> components and spray them on top of your system workspace with their
> own quite distinct ideas about packaging and versioning.
>
> There is no solution to this problem, except to use tools like
> "pyvenv" to set aside secluded workspaces for Python modules and their
> dependencies assembly. So, for most developers, they *should not* use
> "sudo pip". They should remain in user space, or possibly in shared
> workspaces, to avoid overwriting system components.
Nothing is changing in terms of *recommended* practices. This change
proposal is entirely about harm mitigation for the cases where users
*do* run "sudo pip ...", either because that's their instinctive
reaction to a permissions error, or because some misguided
installation instructions for a 3rd party package advised them to do
it.
Debian and derivatives already mitigate the potential harm for these
cases by requiring the "--install-layout=deb" option to be passed to
get distutils to install into the system directories rather than doing
it by default: https://wiki.debian.org/Python#Deviations_from_upstream
Their approach means that any harm caused by "sudo pip install X" can
subsequently be fully reversed by doing "sudo pip uninstall X".
At this moment, this is NOT true for Fedora and derivatives. Instead,
the remediation step here is "sudo pip uninstall X && sudo dnf
reinstall <something>" where you have to:
1. Figure out what "<something>" needs to be
2. Hope that whatever you broke didn't affect your ability to run
"sudo dnf reinstall"
>> Our first attempt to make sudo pip safe on Fedora [0] was
>> based on using a different binary (/usr/libexec/system-python)
>> for RPM packaging purposes and changing the behavior
>> of /usr/bin/python3 (and pip that uses its settings) to install
>> under /usr/local by default. Switching all the Python 3 packages
>
> Oh, Oh my. The extent to which this unworkable is... fascinating.
Nico, the contemptuous tone you have adopted in this post is entirely
inappropriate. Please give other list participants the courtesy of
assuming expertise at least equivalent to your own, rather than
engaging in attempted displays of assumed superiority.
> You
> will inevitably install incompatible future versions of critical
> packages which are incompatible with the Fedora system packages. And
> as soon as you install one that breaks RPM, if /usr/local/ if ahead of
> /usr in your search path for modules, you are *screwed*.
This isn't true, as long as the fix is as simple as using pip to
uninstall whatever problematic package you just installed.
At the moment, that isn't the case on Fedora and derivatives, since
"sudo pip install ..." can actually *corrupt* a system package
installation, rather than merely shadowing it at import time the way
it does on Debian et al.
> And if
> /usr/local is first, well, an RPM update may introduce a component
> that is incompatible with the pip installed modules at arbitrary
> times.
The goal of this change isn't to allow co-existence of arbitrary
upstream and downstream versions as supported combinations - it's to
make sure that forced installation of an incompatible upstream version
is an easily reversible error, rather than a potentially
system-breaking mistake.
>> We decided to try a different approach. The main idea is to detect
>> an ongoing RPM build and modify the behavior of the Python 3
>> executable only when in normal user environment so that RPM builds
>> won't be affected at all.
>
> See https://xkcd.com/1172/ . You're inventing a workflow that isn't
> supported by *anyone*.
The problem is that the default behaviour needs to change (to mitigate
the potential harm caused by a relatively common end user error), but
unlike Debian package builds (which know they need to set
"--install-layout=deb"), most RPM builds for Python packages currently
rely on that default behaviour as part of the build process. Changing
the default indiscriminately would thus be a breaking change that not
only affects Fedora RPM builds, but also COPR and other third party
RPM builds.
So we need a solution that retains the existing behaviour for RPM
builds, but changes the behaviour to be less potentially destructive
when run interactively at a shell prompt, or as part of a
configuration management script.
>> The adjustment of the behavior can be done on different levels.
>> The first option is to set the sys.prefix variable to /usr/local
>> when the interpreter is initialized. This will affect
>> all the install methods, but the solution can cause some
>> problems in applications that rely on the value of sys.prefix.
>> A prototype of this implementation can be seen here [1].
>
> I'm sorry, but.... ouch. Please stop trying to implement "pip install"
> as a system installation tool.
That is not the goal of the change.
> The effort would be better spent
> upgrading and enhancing "py2pack".
Improving the level of commonality and collaboration between Fedora &
Mageia's pyp2rpm and openSUSE's py2pack would indeed be desirable, but
it's entirely orthogonal to the harm mitigation exercise being
discussed here.
>> The other possibility is to limit the pip install location change
>> to distutils and pip [2]. This is the "safer" option, but does
>> not cover all corner cases. For example, Python software built
>> locally using cmake or similar tools will be installed into
>> /usr/lib and can conflict with system tools. Debian chose to
>> implement similar solution.
>
> It's local environment specific, unstable software sensitive to
> arbitrary and unmanaged upstream dependency updates, and extremely
> sensitive to local system pip modules. It *does not belong* in
> /usr/lib. Because the components are modular and bundled in a non-RPM
> compatible fashion, it behooves developers to use a tool to segregate
> the tools as much as feasible from the Fedora underlying
> infrastructure. i.e., use pyvenv to build them in a contained
> environment segregated from the system files.
Helping users that are already doing the right thing isn't the problem
at hand - this is about making it easier for users to recover from
doing the *wrong* thing.
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
7 years
Re: What is your opinion on "sudo pip" fix for Fedora 27?
by Nick Coghlan
On 27 April 2017 at 23:04, Daniel P. Berrange <berrange(a)redhat.com> wrote:
> On Thu, Apr 27, 2017 at 04:32:09PM +1000, Nick Coghlan wrote:
>> Their approach means that any harm caused by "sudo pip install X" can
>> subsequently be fully reversed by doing "sudo pip uninstall X".
>>
>> At this moment, this is NOT true for Fedora and derivatives. Instead,
>> the remediation step here is "sudo pip uninstall X && sudo dnf
>> reinstall <something>" where you have to:
>>
>> 1. Figure out what "<something>" needs to be
>> 2. Hope that whatever you broke didn't affect your ability to run
>> "sudo dnf reinstall"
>
> Yep, recovering the system python install to a pristine state after
> 'devstack' has done its stuff is very painful right now :-(
>
> devstack was originally written for Ubuntu systems, so I guess they
> don't suffer as much due to the Debian specific change you describe
> above.
>
> Which location takes priority in Debian if the same module is installed
> in both places ? The DPKG location, or the Pip location ? Presumably it
> would have to be the pip version that takes priority if you want it to
> be usable for updating the (likely older) distro provided versions.
Aye, /usr/local/lib has priority:
$ docker run --rm ncoghlan/debian-python python3 -m site
sys.path = [
'/',
'/usr/lib/python3.4',
'/usr/lib/python3.4/plat-x86_64-linux-gnu',
'/usr/lib/python3.4/lib-dynload',
'/usr/local/lib/python3.4/dist-packages',
'/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.4/site-packages' (doesn't exist)
ENABLE_USER_SITE: True
User level package installs take priority over system level ones
(regardless of distro), so the current container build use case for
root level pip installations can typically be met equally well by
running as a non-root user inside the container and doing "pip install
--user ...". Unfortunately, as with the venv option, there are a *lot*
of container build scripts out there that don't do that :(
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
7 years
Status update for PEP 538 (coercing the legacy C locale to C.UTF-8)
by Nick Coghlan
Hi folks,
The Fedora 26 Alpha currently includes our downstream patch
backporting PEP 538 to Python 3.6 - an update to CPython to implicitly
coerce the legacy C locale to C.UTF-8 as the Python interpreter
process starts up.
I'm still aiming to have that PEP officially accepted for 3.7 upstream
before the F26 beta deadline (currently May 16th), but hit a slight
hitch recently when Barry Warsaw had to withdraw as BDFL-Delegate due
to a lack of sufficient time to devote to the review process.
Fortunately, Naoki Inada has agreed to take over the role, and Guido
signed off on the updated delegation today:
https://mail.python.org/pipermail/python-dev/2017-April/147796.html
So with any luck, we should be able to get this change explicitly
approved by upstream within the next couple of weeks.
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
7 years
Troubles with Python 3.6.1 in Fedora 26
by Miro Hrončok
There is and update of Python to 3.6.1 coming to Fedora 26 [1].
Due to a bug [2][3] tho following scenario might happen:
0. user has python3 3.6.0 and python3-foo-0.1-1
1. python3 3.6.1 is pushed to Fedora 26 stable
2. maintainer of python3-foo rebuilds the package to python3-foo-0.1-2
(this happens against python3 3.6.1) and pushes it to updates
3. user updates to python3-foo-0.1-2, but does not update python3
4. if the Python foo module meets certain conditions, it is broken
5. user can `dnf update python3` to fix the issue
Because Fedora 26 is not yet released, if we push Python 3.6.1 now, this
problem can only affect the users who run Alpha and do partial dnf
updates, and a full dnf update fixes the issue. So we decided to push
Python 3.6.1 to Fedora anyway.
Should I note this at the common bugs page [4]? I don't think the
scenario is common enough, but I feel a need to inform about it somewhere.
[1] https://bodhi.fedoraproject.org/updates/FEDORA-2017-e1e5e717c6
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1435135
[3] http://bugs.python.org/issue29943
[4] https://fedoraproject.org/wiki/Common_F26_bugs
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
7 years
self introduction
by Lumir Balhar
Hello.
My name is Lumír Balhar and I am a member of Python maintenance team in
Red Hat and Fedora contributor.
I want to be a member of Python SIG to be able to help with important
Python packages in Fedora.
Have a nice day.
Lumír
7 years
ipykernel to recommend stuff?
by Miro Hrončok
Hi,
I got this idea that the python{2,3}-ipykernel packages could recommend
some modules useful to use in the Jupyter Notebook.
Recommends:
* numpy
* matplotlib
Not sure if should be recommended or suggested:
* pillow
* scikit-image
* scikit-learn
* scipy
* statsmodels
* sympy
* pandas
What do you think? It would mean that a user who dnf installs notebook
gets some opt-outable things useful in the notebook. And who uses the
Python 2 kernel, will get the Python 2 version.
This would also drag the things to the Python Classroom Notebook which I
think is good - it will make the Notebook much easier to use for
teaching Python for science - without the need to start with virtualenvs
first or install additional packages. But of course it would also make
the image bigger.
Also see [1] for relevant issue just for the Python Classroom Lab.
[1]
https://github.com/fedora-python/fedora-kickstarts-python-classroom/issues/1
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
7 years
