python-devel July 2018

python-devel@lists.fedoraproject.org

4 participants
6 discussions

Python 3.7's Deterministic pycs
by Petr Viktorin 15 Jul '19

15 Jul '19

Hello! The first beta for Python 3.7 is out. It will hopefully get into Fedora soon as python37. After it comes out of beta, we'll upgrade python3 to it. The What's New list is at: https://docs.python.org/3.7/whatsnew/3.7.html One thing that's interesting for packagers is PEP 552: Deterministic pycs: https://www.python.org/dev/peps/pep-0552/ Let me summarize in my own words. A new opt-in mode for byte-compilation makes .pyc (bytecode cache) files depend only on the contents of the corresponding source file. If we use this, it will slow down imports, because the whole source file would need to be read and hashed in order to verify if a .pyc file is valid. (Currently, metadata like the modification time and file size is used.) To speed things up, there's an option, UNCHECKED_HASH, which skips cache validation entirely. Using this would mean that if you modify a .py source file installed by RPM, the changes wouldn't take effect (the .py contents would only be shown in tracebacks). Modifying installed files in production is extremely bad practice, of course, but it's quite useful for debugging on throw-away systems. If we adopt UNCHECKED_HASH, anyone doing it will have to remember to remove the corresponding .pyc file. Honestly, I'm not sure we want to use this in Fedora. Is anyone here into reproducible builds, to make a better argument for this? -- Petr Viktorin

5 5

Python Bundled Wheels package
by Miro Hrončok 08 Aug '18

08 Aug '18

Hi Pythonistas and Fedora packagers. Recently I've realized we bundle too much wheels with our pythons + virtualenv package. That is unfortunate: * we don't build those. stricly seeking, it's just a zip with python files in it, yet this is not permitted in Fedora * we only sometimes list it as Provides: bundled(...) and when we do, it is tedious * we never list bundled deps in those bundled wheels (pip bundles a lot) * we never adapt the license tag to include license of bundled wheels (and bundled deps in those) - it would be even more tedious, as pip License tag can be very complicated * we duplicate those across packages I went ahead and prepared a concept in: https://bugzilla.redhat.com/show_bug.cgi?id=1605156 This is one package that builds all the required wheels. It might be a bit weird that it's only one package, but I think it can lower the maintenance burden. Also, we won't update any wheel package, we only add or remove them, so there is no "life cycle". Later we can decide that there are simply too many thing sin one package and split it). This package makes sure the license tag is right and all the virtual bundled provides are in place. Even as one package, I think it's a big improvement comparing to current state of things. Could you please review the decisions made in the spec? Namely: * naming (main package, subpackages) * virtual provides * that the spec is generated by a script and how that script works * the method of usage described in the package review request I've also decided not to run tests, as for them to mean something, we would need to run them against all relevant Python versions. Also, it would complicate the package a lot. The package is approved thanks to Robert-André Mauchin, yet I won't request the repo until it's settled that we want this. Also, once we start using this, maybe we can stop doing rewheel and just use wheels from here in the python3 package as well? Thanks. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 3

HEADS UP: pip upgraded from 9 to 18 (rawhide only)
by Miro Hrončok 31 Jul '18

31 Jul '18

I've just built python-pip-18.0-1.fc29. It has a lot of breaking changes, so please file bugs if you have problems. We do not plan to upgrade pip in stable Fedora releases. Release notes: https://pip.pypa.io/en/stable/news/ See notes for 18 and 10, as we skipped 10. Note that upstream changed version scheme, nothing existed between 10 and 18. Thanks, -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

Fwd: [Distutils]Introducing XAR - SquashFS based mountable executables - Calling OS/Distro Maintainers
by Nick Coghlan 15 Jul '18

15 Jul '18

Hi folks, Facebook just published an interesting bit of tech that uses squashfs to create single-file executables that don't require a pre-installed container engine to execute, but do place some more significant demands on the FUSE subsystem, and also require a bootstrapping executable to set up the filesystem mounts. I recommend taking a look at their blog post, as xar seems like quite an interesting capability to consider offering: whereas containers and flatpaks give you both build time and runtime component isolation, xar files aim to behave more like a statically linked ELF binary with no supporting data files (no runtime isolation, but no direct dependencies on the host filesystem either). Cheers, Nick. ---------- Forwarded message ---------- From: Cooper Ry Lees <me(a)cooperlees.com> Date: 14 July 2018 at 06:51 Subject: [Distutils]Introducing XAR - SquashFS based mountable executables - Calling OS/Distro Maintainers To: distutils-sig(a)python.org Hi distutils, Today Facebook Open Sourced a competitor to PEX and other zip based distribution methods for Python (and potentially other languages). Basically it's claim to fame is start up time for large modules being similar to regular on file system modules due to extracting on read via SquashFS mounted executables. For more information read our blog post: - https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-s… The bdist_xar 'wheel' like plugin is also now in PyPI: https://pypi.org/project/xar/ (https://github.com/facebookincubator/xar/) Main reason I wanted to post here was I'd love to reach out to our OS maintainers on list (e.g. Mr Stinner) and talk about getting the components of XAR into the OS's package repos. The main components that make sense are: - xarexec_fuse: https://github.com/facebookincubator/xar/ - squashfuse: a newer version with squashfuse_ll (and optionally zstd support) https://github.com/vasi/squashfuse - squashfs-tools: a newer version with zstd support (made optional I guess) Once we had this, it would make building XARs as easy as building wheels, even more so when we can define OS level dependencies for it! Please feel free to reach out to me directly with any questions etc. - Also want to note, Sumanah suggested I floated this here, I apologize if this is a misuse of the list. If so, please ignore. I was torn wether or not this is appropriate. Thanks, Cooper -- Distutils-SIG mailing list -- distutils-sig(a)python.org To unsubscribe send an email to distutils-sig-leave(a)python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/… -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1

Python 3.7 on its way to rawhide
by Miro Hrončok 02 Jul '18

02 Jul '18

The merge of the f29-python side tag into rawhide has started. Expect Python 3.7 in rawhide soon. If your package will have broken dependencies (on Python 3.6), rebuild it. If you cannot rebuild it due to dependencies not being rebuilt, (search and) open bugs. Make sure to block PYTHON37 [1] so we are aware of all the issues. [1] http://bugzilla.redhat.com/show_bug.cgi?id=PYTHON37 -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 1

Getting rid of python3- backports packages
by Miro Hrončok 01 Jul '18

01 Jul '18

During the 3.7 boostrapping of the interstitial sequence of Python 3 packages, I have noticed it includes a lot of packages that are only intended as Python 3 stdlib backports for older Pythons. https://pypi.org/project/contextlib2/ is a backport of the standard library’s contextlib module to earlier Python versions https://pypi.org/project/linecache2/ is a backport of linecache to older supported Pythons. https://pypi.org/project/pathlib2/ The goal of pathlib2 is to provide a backport of standard pathlib module which tracks the standard library module, so all the newest features of the standard pathlib can be used also on older Python versions. https://pypi.org/project/traceback2/ is a backport of traceback to older supported Pythons. https://pypi.org/project/unittest2/ is a backport of the new features added to the unittest testing framework in Python 2.7 and onwards. https://pypi.org/project/simplejson/ is the externally maintained development version of the json library included with Python 2.6 and Python 3.0, but maintains backwards compatibility with Python 2.5. https://pypi.org/project/funcsigs/ is a backport of the PEP 362 function signature features from Python 3.3 https://pypi.org/project/mock/ is now part of the Python standard library, available as unittest.mock in Python 3.3 onwards. Now I see fairly no real reason to have python3-contextlib2, python3-linecache2 etc. packaged and maintained in Fedora. The only reason there is that other packages actually depend on them. I think we should actively change the packages not to depend on those and eventually get rid of them. This involves changing how upstreams are using those. Thoughts? Here is some dependency info: $ whatrequires python3-contextlib2 python3-futurist python3-pytest-virtualenv python3-vcrpy python-pytest-shutil python-traceback2 $ whatrequires python3-linecache2 python3-traceback2 $ whatrequires python3-pathlib2 pipenv python3-spyder python-testpath $ whatrequires python3-traceback2 python3-testtools python3-unittest2 $ whatrequires python3-unittest2 python3-testtools ProDy cloud-init compose-utils fedpkg libabigail pungi pyephem python-agate python-agate-excel python-autobahn python-billiard python-case python-funcsigs python-leather python-linecache2 python-mimerender python-mock python-pyclipper python-pykafka python-ripozo python-rsa python-s3transfer python-traceback2 python-tw2-core python-unicodecsv $ whatrequires python3-simplejson copr-cli copr-frontend copr-rpmbuild gnome-code-assistance kismon python2-pystache (huh?) python3-canonicaljson python3-cherrypy python3-cinderclient python3-cornice python3-geojson python3-jsonpickle python3-manilaclient python3-mwclient python3-neutronclient python3-novaclient python3-octaviaclient python3-osc-lib python3-oslo-serialization-tests python3-peewee python3-rabbitvcs python3-resultsdb_api python3-scales python3-tackerclient python3-troveclient python3-tw2-core python3-tweepy python3-twitter python3-uri-templates python3-x2go translate-toolkit pymol python-anyjson python-arrow python-marshmallow python-mygpoclient python-openstackclient python-sphinx $ whatrequires python3-funcsigs module-build-service python3-debtcollector python3-mock python3-octaviaclient python3-oslo-utils python3-tinyrpc python3-tinyrpc-tests odcs python-ZEO python-msgpack python-mwclient python-ripe-atlas-cousteau $ whatrequires python3-mock copr-keygen module-build-service python3-ara-tests python3-certbot python3-certbot-dns-* python3-conda python3-congressclient-tests python3-keystoneclient-tests python3-kubernetes-tests python3-magnumclient-tests python3-music21 python3-netdiff python3-octaviaclient-tests python3-osc-lib-tests python3-oslo-cache-tests python3-oslo-log-tests python3-oslo-middleware-tests python3-oslo-policy-tests python3-oslo-rootwrap-tests python3-oslo-serialization-tests python3-oslo-service-tests python3-oslo-versionedobjects python3-oslo-vmware-tests python3-oslotest python3-pytest-spec python3-pywbem python3-sphinx python3-testfixtures-tests python3-testify python3-testinfra python3-tinyrpc-tests python3-vcrpy waiverdb ...320 source packages buildrequire this one! -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

3 4

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel July 2018