Hi all, I started working on updating python3 to 3.4.0. So far, this is only in a private branch "python3.4" in dist-git, since I'll need to get a Change approved for Fedora 21. I already created the Change [1], but I was told that it will get on FESCo schedule when Fedora 21 schedule and future is more clear, so for now I'll just work on Python 3.4.0. When Python 3.4.0 reaches beta1 (which means feature freeze, which also means that the bytecode format should no longer change) - November 24 [2] - I plan to create a repo in Copr so that everyone has access to built RPMs, and also I want to start rebuilding as many packages as possible to see how they work with Python 3.4. As for the current state of Python 3.4 in the dist-git branch: - It still seems a bit unclear what upstream will do with the sha3 implementation (although they will probably keep it), so I didn't rebase the fips patch yet, there is still plenty of time for that. - pep453 [3] (pip bootstraping) hasn't been implemented yet. The pep also contains recommendation for distro packagers, so here's what we should do. -- Make a circular dependency between python3 and python3-pip (ugly, will require bootstraping with every new Python minor version). -- Definitely delete bundled CA certificates and use system certificates instead. -- Maybe remove the bundled pip (although this goes against the pep) and use python3-pip. E.g. everything would work as expected, but under the wraps, python3-pip package would be used. So when doing security updates, we'd just fix python3-pip. It is however true that if Fedora's pip would be different from the upstream bundled one, users may experience some behaviour differencies. I'd like to hear your opinions on how we should approach this. - I'll be continuously working in the dist-git branch, so if you try it and something doesn't work, I'm just working on it or I plan to :) Patches are welcome.
Thanks, Slavek.
[1] https://fedoraproject.org/wiki/Changes/Python_3.4 [2] http://www.python.org/dev/peps/pep-0429/ [3] http://www.python.org/dev/peps/pep-0453/
On Tue, Nov 05, 2013 at 07:29:35AM -0500, Bohuslav Kabrda wrote:
Hi all, I started working on updating python3 to 3.4.0. So far, this is only in a private branch "python3.4" in dist-git, since I'll need to get a Change approved for Fedora 21. I already created the Change [1], but I was told that it will get on FESCo schedule when Fedora 21 schedule and future is more clear, so for now I'll just work on Python 3.4.0.
Hmmm.... Putting on a fesco hat, I would be more than willing to approve changes for F21 now.... We've already approved Python3 as Default which has both F21 and F22 goals.
I think that the F21 Schedule and Product desciptions are somewhat orthogonal to some features including this. The things I'm looking at that imply that are:
* Something that either is non-specific to one of our products * Something that is an upgrade of packages already in Fedora
So in my view -- submit the Change! :-)
When Python 3.4.0 reaches beta1 (which means feature freeze, which also means that the bytecode format should no longer change) - November 24 [2]
- I plan to create a repo in Copr so that everyone has access to built
RPMs, and also I want to start rebuilding as many packages as possible to see how they work with Python 3.4.
<nod> The rebuilding make me want to get this into the main repos early.
As for the current state of Python 3.4 in the dist-git branch:
- It still seems a bit unclear what upstream will do with the sha3 implementation (although they will probably keep it), so I didn't rebase the fips patch yet, there is still plenty of time for that.
I think Christian outlined what they are going to do with sha3 yesterday (but yeah, it depends on what is happening with the sha3 standard so there's a bit of uncertainty there still).
- pep453 [3] (pip bootstraping) hasn't been implemented yet. The pep also contains recommendation for distro packagers, so here's what we should do.
-- Make a circular dependency between python3 and python3-pip (ugly, will require bootstraping with every new Python minor version).
Do you mean minor version as in the "4" in 3.4 or as in the micro level ("0" in 3.4.0)?
We can get around bootstrapping at minor version revs if pip is only used at runtime, not at buildtime.
We can still get around bootstrapping at micro version if we are careful never to include pip's code in the package. Only figure out where to find pip's code.
-- Definitely delete bundled CA certificates and use system certificates instead.
+1
-- Maybe remove the bundled pip (although this goes against the pep) and use python3-pip. E.g. everything would work as expected, but under the wraps, python3-pip package would be used. So when doing security updates, we'd just fix python3-pip. It is however true that if Fedora's pip would be different from the upstream bundled one, users may experience some behaviour differencies. I'd like to hear your opinions on how we should approach this.
We should definitely do this. pip itself bundles code that we've had to unbundle locally -- maintaining that morass of software again (it's also bundled inside of virtualenv...) is going to lead to a lot of work anytime an update needs to happen.
However, talking with ncoghlan it may not be simple. The case that makes things tricky is pyvenv. Upstream python wants to protect venv's from changes to code on the system. ie: if a user installs a backwards incompatible pip on the system, that should not affect the pip inside the existing venvs. Equally, if there are security issues found with pip, those changes would not be propogated to the pip inside the venvs.
ncoghlan proposed that we create wheels in the pip package (and all the packages that pip bundles that we have to unbundle) and then when ensurepip installs into a venv, it copies those wheels in there.
I'm wondering if we could do a little better. Instead of keeping around two copies of all this software on the system, if we can reconstruct a wheel zip (or possibly just copy the files in) from the wheel metadata and files on the system. I know that sysadmins and developers will hotfix software on the system for issues that they care about. Having those changes propogate to newly installed venvs seems like a better plan than having the venvs get the old copies that were built at rpm build time.
-Toshio
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/06/2013 12:50 AM, Toshio Kuratomi wrote:
On Tue, Nov 05, 2013 at 07:29:35AM -0500, Bohuslav Kabrda wrote:
-- Maybe remove the bundled pip (although this goes against the pep) and use python3-pip. E.g. everything would work as expected, but under the wraps, python3-pip package would be used. So when doing security updates, we'd just fix python3-pip. It is however true that if Fedora's pip would be different from the upstream bundled one, users may experience some behaviour differencies. I'd like to hear your opinions on how we should approach this.
We should definitely do this. pip itself bundles code that we've had to unbundle locally -- maintaining that morass of software again (it's also bundled inside of virtualenv...) is going to lead to a lot of work anytime an update needs to happen.
However, talking with ncoghlan it may not be simple. The case that makes things tricky is pyvenv. Upstream python wants to protect venv's from changes to code on the system. ie: if a user installs a backwards incompatible pip on the system, that should not affect the pip inside the existing venvs. Equally, if there are security issues found with pip, those changes would not be propogated to the pip inside the venvs.
Right - the wording in the PEP is a little stronger than the real requirement of "don't break pyvenv installing pip into virtual environments", because we wanted to make it clear that providing an appropriate wheel file is the only upstream supported way to make pyvenv work as expected in 3.4+.
There are actually other ways to make it work, the approach in the PEP is just the one that we know for sure will work properly and will cover in the upstream test suite. (Leaving room for those alternative approaches is also why the PEP explicitly states that it is OK for system pip upgrades to affect later venv creation)
ncoghlan proposed that we create wheels in the pip package (and all the packages that pip bundles that we have to unbundle) and then when ensurepip installs into a venv, it copies those wheels in there.
Technically, ensurepip uses the wheel to install pip into the venv, rather than copying the wheel anywhere.
I'm wondering if we could do a little better. Instead of keeping around two copies of all this software on the system, if we can reconstruct a wheel zip (or possibly just copy the files in) from the wheel metadata and files on the system. I know that sysadmins and developers will hotfix software on the system for issues that they care about. Having those changes propogate to newly installed venvs seems like a better plan than having the venvs get the old copies that were built at rpm build time.
Yep, that should be entirely feasible - wheel installation is just a matter of copying files, so reversing that process should work fine. You could also potentially skip the interim wheel step and just copy files into the venv based on the pip and setuptools RECORD files. The one thing to watch out for will be the files *generated* at install time - you won't want to copy those back into the wheel file or virtual environment (that should just be the script files in /usr/bin, though).
If you get this working for Fedora, I could definitely see us accepting an upstream patch to make pyvenv work that way by default in 3.5+ (i.e. copying a system installed pip if available, and only fall back to using the bundled wheel if there's no system pip). We just didn't want to make blazing that trail (since it's not officially supported by pip at this point in time) a precondition for the PEP 453 implementation :)
Cheers, Nick.
- -- Nick Coghlan Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead Beaker Development Lead (http://beaker-project.org/)
----- Original Message -----
On Tue, Nov 05, 2013 at 07:29:35AM -0500, Bohuslav Kabrda wrote:
Hi all, I started working on updating python3 to 3.4.0. So far, this is only in a private branch "python3.4" in dist-git, since I'll need to get a Change approved for Fedora 21. I already created the Change [1], but I was told that it will get on FESCo schedule when Fedora 21 schedule and future is more clear, so for now I'll just work on Python 3.4.0.
<snip>
So in my view -- submit the Change! :-)
I already marked it ready for wrangler and I asked jreznik, who told me that the changes should be postponed. I'll try to talk to him again and see if there's any real problem about proposing this Change.
As for the current state of Python 3.4 in the dist-git branch:
- It still seems a bit unclear what upstream will do with the sha3 implementation (although they will probably keep it), so I didn't rebase the fips patch yet, there is still plenty of time for that.
I think Christian outlined what they are going to do with sha3 yesterday (but yeah, it depends on what is happening with the sha3 standard so there's a bit of uncertainty there still).
Yes, exactly what I mean. We still have plenty of time, so I didn't want to spend possibly unnecessary work on rebasing the fips patch.
- pep453 [3] (pip bootstraping) hasn't been implemented yet. The pep also contains recommendation for distro packagers, so here's what we should do.
-- Make a circular dependency between python3 and python3-pip (ugly, will require bootstraping with every new Python minor version).
Do you mean minor version as in the "4" in 3.4 or as in the micro level ("0" in 3.4.0)?
Minor as in "4". AFAIK the bytecode format only changes with major and minor version (e.g. from 3.X to 3.Y, but _not_ from 3.X.y to 3.X.z).
We can get around bootstrapping at minor version revs if pip is only used at runtime, not at buildtime.
We can still get around bootstrapping at micro version if we are careful never to include pip's code in the package. Only figure out where to find pip's code.
IMO we will need bootstrapping only for major and minor versions, not for micro versions. In other words, I think we will need bootstrapping only when bytecode format changes.
-- Maybe remove the bundled pip (although this goes against the pep) and use python3-pip. E.g. everything would work as expected, but under the wraps, python3-pip package would be used. So when doing security updates, we'd just fix python3-pip. It is however true that if Fedora's pip would be different from the upstream bundled one, users may experience some behaviour differencies. I'd like to hear your opinions on how we should approach this.
<snip>
I'm wondering if we could do a little better. Instead of keeping around two copies of all this software on the system, if we can reconstruct a wheel zip (or possibly just copy the files in) from the wheel metadata and files on the system. I know that sysadmins and developers will hotfix software on the system for issues that they care about. Having those changes propogate to newly installed venvs seems like a better plan than having the venvs get the old copies that were built at rpm build time.
Hmm, interesting idea. Once Python 3.4 test suite passes for me and Python is at Beta, I'll try to look into this :) The way that Nick is proposing in the other mail ("just copy files into the venv based on the pip and setuptools RECORD files") seems the best to me, I'll try to pursue this solution.
-Toshio
Thanks a lot, Slavek.
On 11/06/2013 06:19 PM, Bohuslav Kabrda wrote:
----- Original Message -----
Do you mean minor version as in the "4" in 3.4 or as in the micro level ("0" in 3.4.0)?
Minor as in "4". AFAIK the bytecode format only changes with major and minor version (e.g. from 3.X to 3.Y, but _not_ from 3.X.y to 3.X.z).
FWIW, this ambiguity is why I tend to refer to Python feature releases (X.Y) and maintenance releases (X.Y.Z) rather than major/minor (incorrect, since Python version numbering is major.minor.micro) or minor/micro (technically correct, but the major number changes so rarely that calling Python feature releases minor releases usually confuses people).
Cheers, Nick.
Update: - DONE: I've just created a copr repo for Python 3.4 [1] and built beta1 - IN PROGRESS: I'm now working on unbundling of pip and setuptools wheels, as mentioned in thread [2]. Progress: -- DONE: created python-wheels package for Fedora (I'll build it in copr ASAP) -- DONE: added logic to python-setuptools and python-pip so that they can be built as wheels (I'll build in copr ASAP) -- TODO: Now that I have an idea how to build wheel RPMs, I need to: --- wait to see if my patches to pip that enable this get accepted [3], [4] (and perhaps communicate with upstream to make them better) --- once the patches are accepted, I'd like to talk to FPC about allowing building wheel RPMs (most likely from Fedora 21 on) and how to do that --- once that is discussed, build setuptools and pip as wheels for F21 (this is really necessary, so even if Fedora as a whole distro doesn't accept wheels usage yet, I'll try to negotiate exception for these two packages) -- TODO: in the meanwhile, I'll start working on the actual ensurepip patch and I'll propose it upstream - TODO: also in the meanwhile, I'll be updating Python 3.4 to the newest prereleases, as they are released by upstream
If anyone wants to help, you can: - help push my patches to pip upstream - help with creating a proposal about building wheel RPMs for FPC (have a look at the SRPMs in the copr repo - once I've built them - to see how I'm currently doing it) - help with creating the ensurepip patch (if you want to work on any of these, please let everyone know, so that we don't duplicate our efforts)
Thanks a lot, Slavek.
python-devel@lists.fedoraproject.org
-
Bohuslav Kabrda -
Nick Coghlan -
Toshio Kuratomi