Python 3.7's Deterministic pycs
by Petr Viktorin
Hello!
The first beta for Python 3.7 is out. It will hopefully get into Fedora
soon as python37.
After it comes out of beta, we'll upgrade python3 to it.
The What's New list is at: https://docs.python.org/3.7/whatsnew/3.7.html
One thing that's interesting for packagers is PEP 552: Deterministic
pycs: https://www.python.org/dev/peps/pep-0552/
Let me summarize in my own words.
A new opt-in mode for byte-compilation makes .pyc (bytecode cache) files
depend only on the contents of the corresponding source file.
If we use this, it will slow down imports, because the whole source file
would need to be read and hashed in order to verify if a .pyc file is
valid. (Currently, metadata like the modification time and file size is
used.)
To speed things up, there's an option, UNCHECKED_HASH, which skips cache
validation entirely. Using this would mean that if you modify a .py
source file installed by RPM, the changes wouldn't take effect (the .py
contents would only be shown in tracebacks).
Modifying installed files in production is extremely bad practice, of
course, but it's quite useful for debugging on throw-away systems. If we
adopt UNCHECKED_HASH, anyone doing it will have to remember to remove
the corresponding .pyc file.
Honestly, I'm not sure we want to use this in Fedora. Is anyone here
into reproducible builds, to make a better argument for this?
--
Petr Viktorin
4 years, 2 months
Moving EPEL7 to python3.6
by Stephen John Smoogen
Hi,
EPEL is a set of packages which work for CentOS and RHEL versions 6
and 7. In the version 7, we are currently using python34 and would
like to move this to python36. In doing so, we need help in both our
packaging rules and in updating a lot of packages to work for
python36.
First problem: Packaging rules.
Because there could be updates of python versions from 3.4 to 3.6 or
3.8, we wanted to make clear what python was used for any particular
library. This would make it so someone needing python-bottle did not
end up with one packaged with python-3.6 installed on a python-3.4
system. So we wanted the names to be more specific than python3 and
went with naming all the sub packages python34 or python36.
However, this was decided a while ago and it may not be the best
convention to use or one that the current python sig would like us to
use. I would like to get a naming convention cleared up and documented
so when we do a mass rebuild that the packages come out with either a
python3-<foo> or python36-<foo>
Second problem: When to do this update
We had been looking to do this in October, but it may make more sense
to do this in November after Fedora29 has shipped so that people can
help focus on this versus anything F29 related. It also gives us some
lead time to write blogs/magazine items. How does 2018-11-14 sound?
Third problem: Updating and rebuilding packages to work with python36
Below are the list of packages I found which were making
python34-<libraryname> packages currently in EPEL-7. In updating to
python36, I would like to have a combined Virtual Fedora Activity Day
where we work together on IRC. First we would get any scripts ready
and then work with release engineering to change macros in epel-macros
to point to the correct versions of python and any name changes. We
would then do a mass release bump and rebuild all the packages against
python3.6. As problems are found during that day we would make
appropriate changes and fix.
This might take 2 gos.
autowrap-0.16.0-1.el7.src.rpm
clustershell-1.8-1.el7.src.rpm
debconf-1.5.69-1.el7.src.rpm
espresso-4.0.0-1.el7.src.rpm
fedfind-4.2.0-1.el7.src.rpm
future-0.16.0-6.el7.src.rpm
jpype-0.6.3-3.el7.src.rpm
lammps-20180822-1.el7.src.rpm
lensfun-0.3.2-13.el7.src.rpm
lhapdf-6.2.1-1.el7.src.rpm
libprelude-4.1.0-2.el7.src.rpm
libpreludedb-4.1.0-1.el7.src.rpm
lxc-1.0.11-1.el7.src.rpm
netcdf4-python-1.2.7-3.el7.src.rpm
nordugrid-arc-5.4.2-9.el7.src.rpm
petsc4py-3.9.1-3.el7.src.rpm
prelude-correlator-4.1.1-3.el7.src.rpm
py4j-0.10.7-3.el7.src.rpm
pycmd-1.2-4.el7.src.rpm
pyflakes-1.3.0-2.el7.src.rpm
pylint-1.6.5-4.el7.src.rpm
pythia8-8.2.35-4.el7.src.rpm
python-PyGithub-1.39-1.el7.src.rpm
python-PyMySQL-0.8.1-1.el7.src.rpm
python-aiosmtpd-1.0-2.el7.src.rpm
python-apsw-3.7.17.r1-3.el7.src.rpm
python-arrow-0.8.0-3.el7.src.rpm
python-astroid-1.4.9-2.el7.src.rpm
python-atpublic-0.5-1.el7.src.rpm
python-attrs-17.4.0-3.el7.src.rpm
python-backports_abc-0.5-1.el7.src.rpm
python-bitarray-0.8.3-1.el7.src.rpm
python-blessed-1.14.1-2.el7.src.rpm
python-bottle-0.12.13-1.el7.src.rpm
python-breathe-4.2.0-3.el7.src.rpm
python-cached_property-1.3.0-7.el7.src.rpm
python-chai-1.1.1-4.el7.src.rpm
python-click-6.7-6.el7.src.rpm
python-clyent-1.2.2-2.el7.src.rpm
python-collada-0.4-15.el7.src.rpm
python-colorclass-2.2.0-2.el7.src.rpm
python-contextlib2-0.5.1-2.el7.src.rpm
python-cookies-2.2.1-6.el7.src.rpm
python-cov-core-1.15.0-8.el7.src.rpm
python-crypto-2.6.1-13.el7.src.rpm
python-cytoolz-0.7.5-1.el7.src.rpm
python-ddt-1.1.3-1.el7.src.rpm
python-defusedxml-0.5.0-1.el7.src.rpm
python-distutils-extra-2.39-7.el7.src.rpm
python-dockerpty-0.4.1-9.el7.src.rpm
python-docopt-0.6.2-7.el7.src.rpm
python-easyargs-0.9.4-2.el7.src.rpm
python-easygui-0.96-19.el7.src.rpm
python-ecdsa-0.13-4.el7.src.rpm
python-execnet-1.2.0-5.el7.src.rpm
python-falcon-1.4.1-1.el7.src.rpm
python-flexmock-0.10.2-4.el7.src.rpm
python-flufl-bounce-2.3-3.el7.src.rpm
python-flufl-i18n-1.1.3-3.el7.src.rpm
python-flufl-lock-2.4.1-3.el7.src.rpm
python-flufl-testing-0.4-1.el7.src.rpm
python-freezegun-0.1.19-1.el7.src.rpm
python-gammu-2.11-2.el7.src.rpm
python-hexdump-3.4-0.2.20160818hg66325cb5fed8.el7.src.rpm
python-hypothesis-3.12.0-1.el7.src.rpm
python-idstools-0.6.3-1.el7.src.rpm
python-ipython_genutils-0.1.0-7.el7.src.rpm
python-iso8601-0.1.11-7.el7.src.rpm
python-isort-4.2.5-8.el7.src.rpm
python-ivi-0.14.9-6.el7.src.rpm
python-jaydebeapi-1.1.1-1.el7.src.rpm
python-jedi-0.10.2-3.el7.src.rpm
python-jsonschema-2.5.1-3.el7.src.rpm
python-jupyter-core-4.3.0-1.el7.src.rpm
python-keyring-5.0-3.el7.src.rpm
python-lazr-config-2.1-5.el7.src.rpm
python-lazr-delegates-2.0.3-5.el7.src.rpm
python-lazr-smtptest-2.0.3-6.el7.src.rpm
python-lazy-object-proxy-1.2.2-1.el7.src.rpm
python-libdiscid-0.4.1-11.el7.src.rpm
python-llfuse-1.0-1.el7.src.rpm
python-locket-0.2.0-2.el7.src.rpm
python-lz4-0.8.2-1.el7.src.rpm
python-markdown-2.4.1-2.el7.src.rpm
python-maxminddb-1.4.0-2.el7.src.rpm
python-mccabe-0.6.1-6.el7.src.rpm
python-mimeparse-1.6.0-4.el7.src.rpm
python-mock-1.0.1-9.el7.src.rpm
python-msgpack-0.5.6-4.el7.src.rpm
python-multilib-1.2-2.el7.src.rpm
python-nbformat-4.4.0-1.el7.src.rpm
python-neovim-0.2.6-2.el7.src.rpm
python-nose2-0.6.5-4.el7.src.rpm
python-numexpr-2.5.2-2.el7.src.rpm
python-olefile-0.45.1-1.el7.src.rpm
python-openqa_client-1.3.1-1.el7.src.rpm
python-pamela-0.3.0-1.el7.src.rpm
python-paramiko-2.1.1-0.4.el7.src.rpm
python-parso-0.3.1-1.el7.src.rpm
python-pathspec-0.5.3-1.el7.src.rpm
python-pdfminer-20160614-4.el7.src.rpm
python-pdir2-0.3.0-5.el7.src.rpm
python-pg8000-1.10.5-1.el7.src.rpm
python-pika-0.10.0-9.el7.src.rpm
python-pip-8.1.2-6.el7.src.rpm
python-pkgconfig-1.1.0-7.el7.src.rpm
python-pluginlib-0.6.1-1.el7.src.rpm
python-polib-1.0.7-9.el7.src.rpm
python-pretend-1.0.8-3.el7.src.rpm
python-process-tests-1.0.0-9.el7.src.rpm
python-productmd-1.16-1.el7.src.rpm
python-prompt_toolkit-1.0.15-1.el7.src.rpm
python-psutil-2.2.1-3.el7.src.rpm
python-ptyprocess-0.5.1-6.el7.src.rpm
python-pycodestyle-2.0.0-5.el7.src.rpm
python-pycosat-0.6.3-1.el7.src.rpm
python-pyev-0.9.0-0.1.20130610gite31d137.el7.src.rpm
python-pymilter-1.0.2-4.el7.src.rpm
python-pymssql-2.1.3-2.el7.src.rpm
python-pypandoc-1.3.3-2.el7.src.rpm
python-pyroute2-0.4.13-1.el7.src.rpm
python-pyside-1.2.4-1.el7.src.rpm
python-pysocks-1.6.7-1.el7.src.rpm
python-pytest-catchlog-1.2.2-2.el7.src.rpm
python-pytest-runner-3.0-1.el7.src.rpm
python-pytest-xdist-1.17.1-2.el7.src.rpm
python-pytoml-0.1.18-1.el7.src.rpm
python-pyxs-0.4.1-1.el7.src.rpm
python-regex-2018.07.11-1.el7.src.rpm
python-rencode-1.0.5-3.el7.src.rpm
python-responses-0.5.1-6.el7.src.rpm
python-rope-0.11.0-1.el7.src.rpm
python-ruamel-yaml-0.13.14-1.el7.src.rpm
python-scandir-1.8-1.el7.src.rpm
python-scripttest-1.3.0-9.el7.src.rpm
python-setuptools_scm-1.10.1-2.el7.src.rpm
python-sh-1.12.14-6.el7.src.rpm
python-simplejson-3.10.0-1.el7.src.rpm
python-simplepath-0.3.4-1.el7.src.rpm
python-sphinx-autobuild-0.7.1-1.el7.src.rpm
python-sphinx-theme-alabaster-0.7.9-1.el7.src.rpm
python-spur-0.3.17-1.el7.src.rpm
python-structlog-17.2.0-1.el7.src.rpm
python-terminado-0.6-2.el7.src.rpm
python-texttable-1.4.0-1.el7.src.rpm
python-tinydb-3.7.0-1.el7.src.rpm
python-toml-0.9.4-3.el7.src.rpm
python-toolz-0.7.4-1.el7.src.rpm
python-traitlets-4.3.2-2.el7.src.rpm
python-trololio-1.0-1.el7.src.rpm
python-typing-3.5.2.2-3.el7.src.rpm
python-unidiff-0.5.4-1.el7.src.rpm
python-vxi11-0.9-1.el7.src.rpm
python-wcwidth-0.1.7-2.el7.src.rpm
python-whoosh-2.7.4-3.el7.src.rpm
python-wrapt-1.10.4-7.el7.src.rpm
python-www-authenticate-0.9.2-3.el7.src.rpm
python-xapp-1.0.1-7.el7.src.rpm
python-xlrd-1.0.0-6.el7.src.rpm
python-yara-3.8.1-2.el7.src.rpm
python-zope-component-4.1.0-3.el7.src.rpm
python-zope-configuration-4.0.3-11.el7.src.rpm
python-zope-exceptions-4.0.5-4.el7.src.rpm
python-zope-i18nmessageid-4.0.3-2.el7.src.rpm
python-zope-schema-4.4.1-2.el7.src.rpm
python-zope-testing-4.1.2-2.el7.src.rpm
python3-Cython-0.23.5-1.el7.src.rpm
python3-PyYAML-3.11-2.el7.src.rpm
python3-backports-ssl_match_hostname-3.5.0.1-1.el7.src.rpm
python3-cairo-1.10.0-24.el7.src.rpm
python3-cffi-1.9.1-2.el7.src.rpm
python3-chardet-2.3.0-4.el7.src.rpm
python3-coverage-4.0.3-4.el7.src.rpm
python3-cryptography-1.7.2-4.el7.src.rpm
python3-cryptography-vectors-1.7.2-3.el7.src.rpm
python3-dateutil-2.4.2-3.el7.src.rpm
python3-dbus-1.2.4-2.el7.src.rpm
python3-decorator-4.0.11-1.el7.src.rpm
python3-dns-1.15.0-7.el7.src.rpm
python3-docker-2.6.1-1.el7.src.rpm
python3-docker-pycreds-0.2.1-1.el7.src.rpm
python3-docutils-0.12-0.8.20140510svn7747.el7.src.rpm
python3-gobject3-3.22.0-4.el7.src.rpm
python3-greenlet-0.4.12-1.el7.src.rpm
python3-gssapi-1.3.0-1.el7.src.rpm
python3-idna-2.7-1.el7.src.rpm
python3-jinja2-2.8-2.el7.src.rpm
python3-jwt-1.6.4-1.el7.src.rpm
python3-kitchen-1.2.5-1.el7.src.rpm
python3-markupsafe-0.23-1.el7.src.rpm
python3-mysql-1.3.12-1.el7.src.rpm
python3-netaddr-0.7.18-2.el7.src.rpm
python3-nose-1.3.7-3.el7.src.rpm
python3-numpy-1.10.4-6.el7.src.rpm
python3-ply-3.9-1.el7.src.rpm
python3-py-1.4.30-2.el7.src.rpm
python3-pyasn1-0.1.9-1.el7.src.rpm
python3-pycparser-2.14-1.el7.src.rpm
python3-pycurl-7.43.0-5.el7.src.rpm
python3-pygments-2.2.0-1.el7.src.rpm
python3-pytest-2.8.5-2.el7.src.rpm
python3-pytest-cov-2.5.1-1.el7.src.rpm
python3-pytz-2017.2-1.el7.src.rpm
python3-pyusb-1.0.2-2.el7.src.rpm
python3-requests-2.12.5-1.el7.src.rpm
python3-scipy-0.18.1-1.el7.src.rpm
python3-setproctitle-1.1.6-2.el7.src.rpm
python3-setuptools-19.6.2-3.el7.src.rpm
python3-simplepam-0.1.5-6.el7.src.rpm
python3-six-1.11.0-1.el7.src.rpm
python3-sqlalchemy-1.1.3-1.el7.src.rpm
python3-suds-0.6-1.el7.src.rpm
python3-tornado-4.4.2-1.el7.src.rpm
python3-urllib3-1.19.1-3.el7.src.rpm
python3-virtualenv-15.1.0-1.el7.src.rpm
python3-websocket-client-0.47.0-1.el7.src.rpm
python3-zope-event-4.2.0-1.el7.src.rpm
python3-zope-interface-4.3.3-1.el7.src.rpm
python34-3.4.9-1.el7.src.rpm
pyxdg-0.25-6.el7.src.rpm
root-6.14.04-1.el7.src.rpm
scapy-2.4.0-2.el7.src.rpm
spglib-1.10.3-1.el7.src.rpm
uwsgi-2.0.17.1-1.el7.src.rpm
xapps-1.0.4-12.el7.src.rpm
xrootd-4.8.4-1.el7.src.rpm
yubikey-piv-manager-1.4.2-5.el7.src.rpm
--
Stephen J Smoogen.
4 years, 11 months
EPEL: Python 3.4 will be EOL in March 2019
by Miro Hrončok
On 13.8.2018 11:49, Larry Hastings wrote on python-dev(a)python.org:
> We of the core dev community commit to supporting Python releases for
> five years. Releases get eighteen months of active bug fixes, followed
> by three and a half years of security fixes. Python 3.4 turns 5 next
> March--at which point we'll stop supporting it, and I'll retire as 3.4
> release manager.
>
> My plan is to make one final release on or around its fifth birthday
> containing the last round of security fixes. That's about seven
> months from now.
See also PEP 429 -- Python 3.4 Release Schedule [0].
We have python34 and python36 in EPEL7. python34 being the "main" one
and python36 the "other" one. The original plan [1] was that once the
ecosystem is adapted well enough, we can switch what "main" and "other"
is and eventually drop python34, creating room for maybe another python3
release to be the "other" next (python38 maybe?). Originally this was
supposed to happen for each release [2], but this was later abandoned
due to lack of manpower.
Do we want to ever switch "main" with "other"? The problem here is:
$ dnf repoquery --repo=epel7 --whatrequires 'python(abi) = 3.4' |
pkgname | sort | uniq | wc -l
243
$ dnf repoquery --repo=epel7 --whatrequires 'python(abi) = 3.6' |
pkgname | sort | uniq | wc -l
15
The original idea is that all packages would have the python3_other
macros in them [2] and all we need is rebuild everything in proper
order, maybe with some bootstrapping. However that never happened and
most of the packages I've seen lack the python3_other bits (no
statistics, just my impression).
Is this something we want? If so, are the packagers willing to adapt
their packages (as much as I'd like to do this, I lack the resources to
hack on 228 packages)?
The Python Maint team in Red Hat is here to help. However the drive
would need to come from the EPEL community, as now we are only guessing
what are the needs here.
[0] https://www.python.org/dev/peps/pep-0429/
[1]
https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproj...
[2] https://fedoraproject.org/wiki/User:Bkabrda/EPEL7_Python3
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4 years, 11 months
test files in a python flat package
by Jun Aruga
I noticed that the test files (/usr/lib64/python3.7/test) for python37
are occupying more than 50% in the package size, when I was cleaning
the files in my local machine.
If we actually do not use the test files, I do not want to include the
files in the python37 package.
Just want to separate the files as python37-test sub package.
Because of just reducing the file size.
How do you think? I want to hear your opinions.
Here is the bugzilla ticket.
Separate the test directory as python37-test
https://bugzilla.redhat.com/show_bug.cgi?id=1637959
--
Jun Aruga jaruga(a)redhat.com
4 years, 11 months
Mass Python 2 packages Removal
by Miro Hrončok
Hi, this is about:
https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal
We have ~300 bugz open and blocking
https://bugzilla.redhat.com/show_bug.cgi?id=PY2REMOVAL
Some of them are > 1 week old and hence anyone can go and remove the
mentioned (sub)packages. I've retired the packages that were for
retirement and I will retire more once they are unblocked (needs to go
to compose -> portingdb before the automation recognizes it).
I open more bugs as they are closed (I don't wan to deal with 1000 bugs
at once).
Retirement is easy. The problematic part is removing subpackages. I go
and manually choose the packages that appear not to be just simple "py
spec generates py2 and py3 subpackages" combo, but a bit more
complicated, and I leave the "simple" ones alone for now.
I'm theoretically able to remove a simple py2 subpackage in about a
minute, yet it is tedious and after ~20 packages I need to stop (before
I go crazy).
We have something in between 900 and 2000 packages to remove (there are
900 leaves to remove and we may make another 1100 into leaves as we go).
Is anyone willing to look into automating this? Write a spec depython2izer?
This usually means:
- removing python2- and python- BRs
- removing %package -n python2 and it's metadata, description & %files
- removing all %py2_build, %py2_install or their older manual variants
- removing python2 tests from %check
- (sometimes) removing %py3dir or python3 dir pushd/popd dance
- (sometimes) removing with_python3 conditionals as they make no sense
IIRC Zbyszek said at Flock that he might work on that. Is that still
true? If not, shall I draft something?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4 years, 11 months
