On 12/07/2016 01:56 PM, Neal Gompa wrote: I think there's been a slight miscommunication: system-python will be used instead of python3 only when building/installing packages inside spec files. The built packages themselves depend on will be run by python3.

On 12/07/2016 06:21 PM, Stephen John Smoogen wrote: System-python is a rather complex and evolving topic, so let my try to explain it a bit deeper (as I understand it): Up to this proposal, the system-python binary (/usr/libexec/system-python) has been simply a direct copy of the python3 executable (/usr/bin/python3{,.5}) [0]. The usefulness of system-python thus does not come from the executable you use, but from the `system-python` package that contains it. The difference between the `python3` and `system-python` packages is that the former requires the full Python 3 standard library (package `python3-libs`), whereas the `system-python` package requires only a smaller subset thereof (package `system-python-libs`). Thus saving space when Python is only needed for system tools. This proposal constitutes a second evolutionary step of system-python: We now want to slightly modify how the two executables behave. Python 3 will continue as it always has—you will be able to import modules whether you installed them through dnf or pip/setuptools. Contrary to that, system-python will become more limited: It will only recognize Python modules installed through dnf, and as a result, your system tolls (dnf et al.) will become impervious to problems stemming from non-functional/problematic modules installed using sudo pip, which up to now could simply overwrite crucial modules installed by dnf. To implement this modification, the locations of dnf-installed and sudo pip–installed modules need to be separated. The default location where Python decides to install new modules is contained within the executable, and thus Python 3's install location will be the one where sudo pip–installed modules belong, while system-python will install modules to a path where dnf-installed modules ought to go. Therefore if you run sudo pip, Python 3 executable will be invoked and pip will install modules to the appropriate place. And in spec files of Python RPM packages, the %{__python3} macro will be modified so that system-python is invoked instead, and the modules will be installed to the appropriate location for dnf-installed modules. A good question to ask at this point is: Is it ok to use system-python in spec files like this when it requires only a limited subset of the Python 3 standard library? The confusing part is that while system-python requires only a limited stdlib, it will use the full standard library if it is installed. And when it comes to Python RPM packages, it will always be installed because: 1. At build time all Python 3 packages have to use `BuildRequires: python3-devel` which brings in a full stdlib complement, and 2. at run time the resulting RPM packages will depend on the `python(abi) = 3.X`, which is only satisfied by the `python3` package and it's full stdlib. I hope this may shine a bit of light on the issue. [0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/python3.spec?id=...

On 12/07/2016 02:47 PM, Konstantin Zemlyak wrote: I'm not sure this would produce the desired effect, that is: - system-python: installs to and imports only from /usr/lib*/pythonX.Y - python3: installs to /usr/local/lib*/pythonX.Y but loads from both /usr/... and /usr/local/...

On 10 December 2016 at 03:09, Orion Poplawski <orion(a)cora.nwra.com&gt; wrote: This would be my main question as well, as tinkering with sys.prefix can have a host of unintended side effects. As far as I can see, the two main relevant pieces of the Debian solution are to: 1. Patch `site.getsitepackages()` to put a `/usr/local` path first: ['/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.4/dist-packages', '/usr/lib/dist-python'] 2. Patch the system level installation tools to support `--install-layout=deb`. We can ignore the site-packages -> dist-packages rename (as we're not worried about the "custom builds might pick up system packages on sys.path" concern that addresses), but the other two aspects should be something Fedora can adapt (although the exact spelling of the "I am building a Fedora system package" aspect may be quite different). I do like the notion that `system-python` won't see any additions made to `/usr/local`, just as it won't see any `--user` level installations. The other point I'd also raise is that yes, the *initial implementation* of "/usr/bin/system-python" will share sys.path with "/usr/bin/python3", and hence transparently pick up the full standard library if the latter is installed. However, given the work going on with Fedora modularity, I don't think we want to assume that that's necessarily *always* going to be the case - for example, we may some day be able to make "/usr/bin/python" and "/usr/bin/python3" end-user configurable (so sysadmins can choose their own "default Python") without risking any harm to core system utilities written in Python. Cheers, Nick. P.S. For folks wondering what the problem with "--user" is on Debian/Ubuntu, as far as I know it's mainly the fact that "~/.local/bin" isn't on PATH by default, so scripts installed via "--user" aren't automatically available. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

FTR: In Debian we already have --install-layout=deb: https://sources.debian.net/src/python-setuptools/28.7.1-1/debian/patches/... https://sources.debian.net/src/python3.5/3.5.2-8/debian/patches/distutils... https://sources.debian.net/src/python2.7/2.7.13~rc1-1/debian/patches/dist... (it also removes Python version from Egg directories and installs into dist-packages rather than site-packages) See also https://wiki.debian.org/Python#Deviations_from_upstream -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

On 11 December 2016 at 01:33, Donald Stufft <donald(a)stufft.io&gt; wrote: Ah, nice - that's very good to hear, as it removes one of the barriers to making "--user" the default behaviour upstream in pip. However, it also reminded me of our past discussions about defaulting to --user in pip, and re-reading the relevant issue brought me to: https://github.com/pypa/pip/issues/1668#issuecomment-107016411 The specific proposal there was to: 1. Add the `--global` flag to pip upstream, so folks can always force environment global installs by doing: pip3 install --global <install-spec> 2. Change the default install behavior in the Fedora system pip package to include "--user" (and require an explicit "--global" to override that) The burden would then be on Fedora as the distro to ensure that "pip3 install <arbitrary-package>" still offered a good user experience, such as making sure that the per-user bin directory was on PATH by default. Actually doing that would mean taking Donald's first pass at implementing the `--global` switch, bringing it up to merge-ready standards, and then incorporating the pip release containing the feature into Fedora: https://github.com/pypa/pip/pull/2418 The benefit of that approach is that it would not only solve the conflict between dnf and pip at the Fedora level, but also move the packaging user experience forward for the Python ecosystem as a whole (by making the `--global` switch available in anticipation of a future upstream switch to `--user` as the default). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

On 12/12/2016 05:39 AM, Nick Coghlan wrote: It would be a partial solution I think. People who first try sudo-less `pip3 install` would be covered, but there are great many people that already go with `sudo pip3 install` right away, as well as many install guides on the Internet that advise to do so. In that case pip would probably install it under /root, which wouldn't be accessible from user-run python. Tomas

I am one who was badly bitten by pip install into system in the past, and I have been using pip --user since. But there is one big usability issue here, there isn't a trivially simple way to make sure packages installed by pip are updated, much less automatically updated. I think it would be important for a change to --user by default to try to address this issue. On Sun, Dec 11, 2016 at 11:39 PM Nick Coghlan <ncoghlan(a)gmail.com&gt; wrote:

Currently, pip3 --user installed stuff can be updated with: pip3 install --upgrade --user $(pip3 list --user -o | cut -f 1 --delim=' ') But that isn't obvious/discoverable On Mon, Dec 12, 2016 at 8:48 AM Petr Viktorin <pviktori(a)redhat.com&gt; wrote:

This is not *entirely* true, it’ll just be *really* slow without it since we’ll need to download files to get their dependencies. The flip side is that It’s pretty easy for us to add it in right now for Wheels, but of course, we can’t for sdists, which I think is fine because it’s just another reason why all projects should strive to upload wheels when they can :) — Donald Stufft

Thanks for all suggestions and comments. Debian's patch is very complex compared to our solution. Our patch is 12 lines long including docstring and it doesn't break any tests. Behavior of /usr/bin/python3 executable with customized sys.prefix will be a bit similar to virtual environment created with option --system-site-packages. Early results of whole environment (system-python + patched python3) tests in mock are promising. Something similar to this proposal: could help us to enable `sudo pip`-installed extensions of DNF-installed programs. Unfortunately --executable doesn't affect scripts listed as 'entry-points' in setup.py script. We will probably have to patch setuptools to make this work. Michal Cyprian ----- Original Message ----- From: "Petr Viktorin" <pviktori(a)redhat.com&gt; To: python-devel(a)lists.fedoraproject.org, "Michal Cyprian" <mcyprian(a)redhat.com&gt; Sent: Monday, December 12, 2016 11:57:23 AM Subject: Re: Making sudo pip Safe On 12/09/2016 06:09 PM, Orion Poplawski wrote: > On 12/07/2016 05:53 AM, Michal Cyprian wrote: >> Hello, >> >> there is a long-standing problem that `sudo pip install` cannot be safely used in Fedora. Many users don't know about this and break python packages on theirs systems. Packages installed using this command can conflict and overwrite Python rpm packages. >> This is a major problem and we have seen several systems broken by people using "sudo pip". Unfortunately, telling people to not use it will not work: "sudo pip" appears in documentation of too many projects online. >> We plan to solve this in Fedora 26. A more precise description of the problem follows. >> >> Current behavior of python packages installation tools in Fedora: >> 1) sudo dnf install python3-foo # root installs foo from rpm to /usr/lib/pythonX.Y/site-packages >> 2) sudo pip3 install foo # root installs foo from PyPI to /usr/lib/pythonX.Y/site-packages >> 3) sudo pip3 install -t /usr/local/lib/pythonX.Y/site-packages foo # root install package to selected location >> 4) pip3 install --user foo # user install foo from PyPI to ~/.local/lib/python3.5/site-packages >> >> Using the --user option with pip would be the ideal solution. However, it is reportedly broken in some versions of Ubuntu, so it is hard to convince software authors to recommend it. >> >> Packages installed using `sudo pip` (2) under /usr can be overwritten or removed by dnf. `dnf install python3-foo` fails if foo was `sudo pip installed` before etc. This problem has been reported many times. [6] >> Another issue is that packages installed under usr/local (3) cannot be imported unless PYTHONPATH or sys.path is set explicitly. >> >> The default install location of pip/distutils-installed packages depends on the value of the >> sys.prefix variable [4]. >> There were several discussions on Bugzilla [1] and the pypa-dev mailing list [2]. >> Interesting solutions were conceived at the CPython level: >> - addition of sys.local_prefix [2] >> - simplification of CPython startup sequence [4] >> Unfortunately none of them were realized and both solutions require many changes of Python and Python Standard Library. >> >> We discussed all the possible solutions with colleagues from the Python maint team. >> It would be great to fix this upstream, but it won't happen in a reasonable time frame. >> We realized that System Python [3], which was announced in Fedora 24, can help us reach the goal. We came up with the following solution: >> - sys.prefix of the python3 executable will be set to /usr/local >> - sys.prefix of system-python will be set to /usr >> - all rpm python packages will be installed using system-python (value of rpm macro %{__python3} will be changed to point to system-python) >> - The path /usr/lib/.../site_packages will be included in sys.path of /usr/bin/python3 >> >> Note that no packages will be affected by other work going on around system-python, namely that packages can declare that they are only using a smaller subset of the Python stdlib. That option is still opt-in. >> >> Behavior of install methods after mentioned changes: >> 1) sudo dnf install foo # root installs foo from rpm to /usr/lib/pythonX.Y/site-packages >> 2) sudo pip3 install foo # root installs foo from PyPI to /usr/local/lib/pythonX.Y/site-packages >> - pip never touches /usr/lib/python3.5/site-packages >> - /usr/bin/python3 can import modules from >> - Python Standard Library >> - site-packages under /usr/local (priority) >> - site-packages under /usr >> - system-python (i.e. what all programs installed via DNF will use) is limited to site-packages under /usr, so DNF-installed software is unaffected by anything installed with pip >> > >> Michal Cyprian > > > I applaud this effort. My concern comes from the question of whether or not > one should be able to extend the available library for system tools. If I'm > using the dnf installed jupyter notebook and I want to use a newer version of > some other package or something that isn't packaged and I install it via "sudo > pip install foo" I can't import it without changing PYTHONPATH, which brings > us back to the problems with method #3 above. Or can we get around this by > not changing %__python3 but changing %py3_build to be: > > > And still having the installed shebangs reference /usr/bin/python3.X? Right, it makes sense for DNF-installed programs that expose a Python shell to the user or are otherwise extensible using pip should have access to `sudo pip`-installed packages, and there should probably be a nice macro to make that easy. Michal, could you make sure that case is covered in the proposal? Yes, it would make a lot of sense for system-python to always behave as if -I was specified. This diverges from upstream even more, and AFAICS the benefit it brings is a that if you compile Python from source with the default settings, it is isolated. With Python 3's built-in isolation between different *versions* of Python, I think there's no problem with separating packages installed with `sudo pip` from the system Python and from a Python compiled from source. -- Petr Viktorin