Python 3.7's Deterministic pycs
by Petr Viktorin
Hello!
The first beta for Python 3.7 is out. It will hopefully get into Fedora
soon as python37.
After it comes out of beta, we'll upgrade python3 to it.
The What's New list is at: https://docs.python.org/3.7/whatsnew/3.7.html
One thing that's interesting for packagers is PEP 552: Deterministic
pycs: https://www.python.org/dev/peps/pep-0552/
Let me summarize in my own words.
A new opt-in mode for byte-compilation makes .pyc (bytecode cache) files
depend only on the contents of the corresponding source file.
If we use this, it will slow down imports, because the whole source file
would need to be read and hashed in order to verify if a .pyc file is
valid. (Currently, metadata like the modification time and file size is
used.)
To speed things up, there's an option, UNCHECKED_HASH, which skips cache
validation entirely. Using this would mean that if you modify a .py
source file installed by RPM, the changes wouldn't take effect (the .py
contents would only be shown in tracebacks).
Modifying installed files in production is extremely bad practice, of
course, but it's quite useful for debugging on throw-away systems. If we
adopt UNCHECKED_HASH, anyone doing it will have to remember to remove
the corresponding .pyc file.
Honestly, I'm not sure we want to use this in Fedora. Is anyone here
into reproducible builds, to make a better argument for this?
--
Petr Viktorin
4 years, 2 months
Python Bundled Wheels package
by Miro Hrončok
Hi Pythonistas and Fedora packagers.
Recently I've realized we bundle too much wheels with our pythons +
virtualenv package. That is unfortunate:
* we don't build those. stricly seeking, it's just a zip with python
files in it, yet this is not permitted in Fedora
* we only sometimes list it as Provides: bundled(...) and when we do,
it is tedious
* we never list bundled deps in those bundled wheels (pip bundles a lot)
* we never adapt the license tag to include license of bundled wheels
(and bundled deps in those) - it would be even more tedious, as pip
License tag can be very complicated
* we duplicate those across packages
I went ahead and prepared a concept in:
https://bugzilla.redhat.com/show_bug.cgi?id=1605156
This is one package that builds all the required wheels. It might be a
bit weird that it's only one package, but I think it can lower the
maintenance burden. Also, we won't update any wheel package, we only add
or remove them, so there is no "life cycle". Later we can decide that
there are simply too many thing sin one package and split it).
This package makes sure the license tag is right and all the virtual
bundled provides are in place.
Even as one package, I think it's a big improvement comparing to current
state of things.
Could you please review the decisions made in the spec? Namely:
* naming (main package, subpackages)
* virtual provides
* that the spec is generated by a script and how that script works
* the method of usage described in the package review request
I've also decided not to run tests, as for them to mean something, we
would need to run them against all relevant Python versions. Also, it
would complicate the package a lot.
The package is approved thanks to Robert-André Mauchin, yet I won't
request the repo until it's settled that we want this.
Also, once we start using this, maybe we can stop doing rewheel and just
use wheels from here in the python3 package as well?
Thanks.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 1 month
HEADS UP: pip upgraded from 9 to 18 (rawhide only)
by Miro Hrončok
I've just built python-pip-18.0-1.fc29.
It has a lot of breaking changes, so please file bugs if you have
problems. We do not plan to upgrade pip in stable Fedora releases.
Release notes: https://pip.pypa.io/en/stable/news/
See notes for 18 and 10, as we skipped 10.
Note that upstream changed version scheme, nothing existed between 10
and 18.
Thanks,
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 1 month
Fwd: [Distutils]Introducing XAR - SquashFS based mountable
executables - Calling OS/Distro Maintainers
by Nick Coghlan
Hi folks,
Facebook just published an interesting bit of tech that uses squashfs
to create single-file executables that don't require a pre-installed
container engine to execute, but do place some more significant
demands on the FUSE subsystem, and also require a bootstrapping
executable to set up the filesystem mounts.
I recommend taking a look at their blog post, as xar seems like quite
an interesting capability to consider offering: whereas containers and
flatpaks give you both build time and runtime component isolation, xar
files aim to behave more like a statically linked ELF binary with no
supporting data files (no runtime isolation, but no direct
dependencies on the host filesystem either).
Cheers,
Nick.
---------- Forwarded message ----------
From: Cooper Ry Lees <me(a)cooperlees.com>
Date: 14 July 2018 at 06:51
Subject: [Distutils]Introducing XAR - SquashFS based mountable
executables - Calling OS/Distro Maintainers
To: distutils-sig(a)python.org
Hi distutils,
Today Facebook Open Sourced a competitor to PEX and other zip based
distribution methods for Python (and potentially other languages).
Basically it's claim to fame is start up time for large modules being
similar to regular on file system modules due to extracting on read
via SquashFS mounted executables. For more information read our blog
post:
- https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source...
The bdist_xar 'wheel' like plugin is also now in PyPI:
https://pypi.org/project/xar/
(https://github.com/facebookincubator/xar/).
Main reason I wanted to post here was I'd love to reach out to our OS
maintainers on list (e.g. Mr Stinner) and talk about getting the
components of XAR into the OS's package repos. The main components
that make sense are:
- xarexec_fuse: https://github.com/facebookincubator/xar/
- squashfuse: a newer version with squashfuse_ll (and optionally zstd
support) https://github.com/vasi/squashfuse
- squashfs-tools: a newer version with zstd support (made optional I guess)
Once we had this, it would make building XARs as easy as building
wheels, even more so when we can define OS level dependencies for it!
Please feel free to reach out to me directly with any questions etc. -
Also want to note, Sumanah suggested I floated this here, I apologize
if this is a misuse of the list. If so, please ignore. I was torn
wether or not this is appropriate.
Thanks,
Cooper
--
Distutils-SIG mailing list -- distutils-sig(a)python.org
To unsubscribe send an email to distutils-sig-leave(a)python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/messag...
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
5 years, 2 months
Python 3.7 on its way to rawhide
by Miro Hrončok
The merge of the f29-python side tag into rawhide has started.
Expect Python 3.7 in rawhide soon.
If your package will have broken dependencies (on Python 3.6), rebuild it.
If you cannot rebuild it due to dependencies not being rebuilt, (search
and) open bugs.
Make sure to block PYTHON37 [1] so we are aware of all the issues.
[1] http://bugzilla.redhat.com/show_bug.cgi?id=PYTHON37
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 2 months
Getting rid of python3- backports packages
by Miro Hrončok
During the 3.7 boostrapping of the interstitial sequence of Python 3
packages, I have noticed it includes a lot of packages that are only
intended as Python 3 stdlib backports for older Pythons.
https://pypi.org/project/contextlib2/ is a backport of the standard
library’s contextlib module to earlier Python versions
https://pypi.org/project/linecache2/ is a backport of linecache to older
supported Pythons.
https://pypi.org/project/pathlib2/ The goal of pathlib2 is to provide a
backport of standard pathlib module which tracks the standard library
module, so all the newest features of the standard pathlib can be used
also on older Python versions.
https://pypi.org/project/traceback2/ is a backport of traceback to older
supported Pythons.
https://pypi.org/project/unittest2/ is a backport of the new features
added to the unittest testing framework in Python 2.7 and onwards.
https://pypi.org/project/simplejson/ is the externally maintained
development version of the json library included with Python 2.6 and
Python 3.0, but maintains backwards compatibility with Python 2.5.
https://pypi.org/project/funcsigs/ is a backport of the PEP 362 function
signature features from Python 3.3
https://pypi.org/project/mock/ is now part of the Python standard
library, available as unittest.mock in Python 3.3 onwards.
Now I see fairly no real reason to have python3-contextlib2,
python3-linecache2 etc. packaged and maintained in Fedora. The only
reason there is that other packages actually depend on them.
I think we should actively change the packages not to depend on those
and eventually get rid of them. This involves changing how upstreams are
using those. Thoughts?
Here is some dependency info:
$ whatrequires python3-contextlib2
python3-futurist
python3-pytest-virtualenv
python3-vcrpy
python-pytest-shutil
python-traceback2
$ whatrequires python3-linecache2
python3-traceback2
$ whatrequires python3-pathlib2
pipenv
python3-spyder
python-testpath
$ whatrequires python3-traceback2
python3-testtools
python3-unittest2
$ whatrequires python3-unittest2
python3-testtools
ProDy
cloud-init
compose-utils
fedpkg
libabigail
pungi
pyephem
python-agate
python-agate-excel
python-autobahn
python-billiard
python-case
python-funcsigs
python-leather
python-linecache2
python-mimerender
python-mock
python-pyclipper
python-pykafka
python-ripozo
python-rsa
python-s3transfer
python-traceback2
python-tw2-core
python-unicodecsv
$ whatrequires python3-simplejson
copr-cli
copr-frontend
copr-rpmbuild
gnome-code-assistance
kismon
python2-pystache (huh?)
python3-canonicaljson
python3-cherrypy
python3-cinderclient
python3-cornice
python3-geojson
python3-jsonpickle
python3-manilaclient
python3-mwclient
python3-neutronclient
python3-novaclient
python3-octaviaclient
python3-osc-lib
python3-oslo-serialization-tests
python3-peewee
python3-rabbitvcs
python3-resultsdb_api
python3-scales
python3-tackerclient
python3-troveclient
python3-tw2-core
python3-tweepy
python3-twitter
python3-uri-templates
python3-x2go
translate-toolkit
pymol
python-anyjson
python-arrow
python-marshmallow
python-mygpoclient
python-openstackclient
python-sphinx
$ whatrequires python3-funcsigs
module-build-service
python3-debtcollector
python3-mock
python3-octaviaclient
python3-oslo-utils
python3-tinyrpc
python3-tinyrpc-tests
odcs
python-ZEO
python-msgpack
python-mwclient
python-ripe-atlas-cousteau
$ whatrequires python3-mock
copr-keygen
module-build-service
python3-ara-tests
python3-certbot
python3-certbot-dns-*
python3-conda
python3-congressclient-tests
python3-keystoneclient-tests
python3-kubernetes-tests
python3-magnumclient-tests
python3-music21
python3-netdiff
python3-octaviaclient-tests
python3-osc-lib-tests
python3-oslo-cache-tests
python3-oslo-log-tests
python3-oslo-middleware-tests
python3-oslo-policy-tests
python3-oslo-rootwrap-tests
python3-oslo-serialization-tests
python3-oslo-service-tests
python3-oslo-versionedobjects
python3-oslo-vmware-tests
python3-oslotest
python3-pytest-spec
python3-pywbem
python3-sphinx
python3-testfixtures-tests
python3-testify
python3-testinfra
python3-tinyrpc-tests
python3-vcrpy
waiverdb
...320 source packages buildrequire this one!
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 2 months