Why are we shipping debug builds of pythons?
by Miro Hrončok
Hi,
an interesting discussion came up in the Python Maint team recently,
about not shipping python3-debug and python2-debug.
On the Chesterton's fence principle [0], I'd would like to know why are
we building and shipping them before we have a discussion about their
removal to save build time and remove packaging cruft.
Anyone has an idea? Those packages are meant to debug Python, yet all
people I know who do that, build they own Python for that purpose (often
from the master branch).
I tracked down the introduction of the python-debug package in this
commit [1] by David Malcolm (CCed) @ 8 years ago, added in Fedora 14
shortly before upgrade to 2.7. Yet the commit message lacks rationale.
[0] https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence
[1]
https://src.fedoraproject.org/rpms/python/c/f020abd35954981b383884105dad4...
Thanks,
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 8 months
Python Bundled Wheels package
by Miro Hrončok
Hi Pythonistas and Fedora packagers.
Recently I've realized we bundle too much wheels with our pythons +
virtualenv package. That is unfortunate:
* we don't build those. stricly seeking, it's just a zip with python
files in it, yet this is not permitted in Fedora
* we only sometimes list it as Provides: bundled(...) and when we do,
it is tedious
* we never list bundled deps in those bundled wheels (pip bundles a lot)
* we never adapt the license tag to include license of bundled wheels
(and bundled deps in those) - it would be even more tedious, as pip
License tag can be very complicated
* we duplicate those across packages
I went ahead and prepared a concept in:
https://bugzilla.redhat.com/show_bug.cgi?id=1605156
This is one package that builds all the required wheels. It might be a
bit weird that it's only one package, but I think it can lower the
maintenance burden. Also, we won't update any wheel package, we only add
or remove them, so there is no "life cycle". Later we can decide that
there are simply too many thing sin one package and split it).
This package makes sure the license tag is right and all the virtual
bundled provides are in place.
Even as one package, I think it's a big improvement comparing to current
state of things.
Could you please review the decisions made in the spec? Namely:
* naming (main package, subpackages)
* virtual provides
* that the spec is generated by a script and how that script works
* the method of usage described in the package review request
I've also decided not to run tests, as for them to mean something, we
would need to run them against all relevant Python versions. Also, it
would complicate the package a lot.
The package is approved thanks to Robert-André Mauchin, yet I won't
request the repo until it's settled that we want this.
Also, once we start using this, maybe we can stop doing rewheel and just
use wheels from here in the python3 package as well?
Thanks.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 8 months
HEADS UP: /usr/bin/cython(ize) is now in python3-Cython
by Miro Hrončok
Cython-0.28.4-3.fc29 has landed in rawhide.
It moves all stuff in /usr/bin to pytyon3 package only, because the
functionality is the same on whatever Python version.
This might break your package if you BR Cython.
Here are some tips that should be Fedora/EPEL version agnostic:
If you need to use /usr/bin/cython (or /usr/bin/cythonize) during build,
BR /usr/bin/cython (or /usr/bin/cythonize).
If you need to use Cython from Python (from cython import...), BR
pytnonX-Cython or pythonXdist(cython) (for X={2,3} as needed).
If you would like to use Cython from the command line and explicitly use
a Python version (e.g. not to drag an unnecessary dependency on the
other Python version), you can use %{__pythonX} -m cython or pythonX -m
cython.
I'll give it some time to settle and later I will scratch build all
packages that BR Cython and see if we broke them, will apply fixes.
Packages that use Cython at build time:
Maintainers by package:
CheMPS2 talcite
PyYAML jeckersb
autowrap sagitter
blueman pwalter
borgbackup bpereto fschwarz
brltty jskarvad limb olysonek
bzr hno pstodulk vvitek
cassandra acaringi hhorak trepik
ceph branto dachary dmick ke4qqq kkeithle ktdreyer steve
stingray
compizconfig-python jskarvad raveit65
dionaea rebus
eclipse-pydev akurtakov jjohnstn
espresso deji junghans tomspur
git-fame fale ignatenkobrain
h5py stevetraylen terjeros
ipython churchyard cstratak dcantrel ignatenkobrain lbalhar
mrunge salimma tomspur
libfreenect jkastner kwizart rmattes
libmacaroons bbockelm
libplist pbrobinson
netcdf4-python orion
numpy cstratak jspaleta limb orion rdieter tomspur ttomecek
openfst jjames mikep
openms sagitter
petsc4py sagitter
pocketsphinx jjames mikep
printrun churchyard
pygame jkaluza jskarvad limb
pygrib jdekloe
pyliblo fab
pymilia sergiopr
pyproj jdekloe
pystatgrab fab heliocastro potty slankes ttorling
python-MDAnalysis rathann
python-admesh churchyard
python-afl mcyprian orphan
python-aiohttp fab ignatenkobrain
python-assimulo sagitter
python-astropy orion sergiopr
python-astropy-healpix lupinix
python-astroscrappy lupinix
python-basemap jspaleta limb
python-bintrees rkuska
python-cartopy qulogic
python-cassandra-driver filabrazilska lbalhar lkundrak praiskup
python-cffi brouhaha jdulaney
python-compreffor athoscr
python-cradox jpena
python-cypari2 pcpa
python-cysignals pcpa
python-djvulibre bstinson
python-falcon abompard carlwgeorge jamielennox
python-fiona qulogic
python-fpylll pcpa
python-gensim besser82
python-giacpy sagitter
python-gmpy2 jjames orion
python-gsd rathann
python-gssapi rharwood simo sross
python-healpy lupinix
python-hidapi besser82
python-httptools ignatenkobrain
python-hug hguemar
python-jnius raphgro
python-kmod grover
python-libdiscid ohaessler
python-line_profiler jacksonisaac
python-lupa dhanesh95
python-lxml fale ignatenkobrain mizdebsk
python-meliae lmacken tadej
python-mistune lupinix
python-msgpack dfateyev fab ktdreyer pjp sundaram
python-multidict athmane ignatenkobrain
python-nipy ignatenkobrain
python-numpy-stl churchyard
python-orderedset jjames
python-pandas kushal orion sergiopr wakko666
python-peewee cstratak mstuchli vkrizan
python-photutils sergiopr
python-plyvel dcallagh
python-pybloomfiltermmap athmane
python-pyclipper athoscr
python-pymssql melmorabity
python-pyopengl swt2c
python-pytaglib ignatenkobrain
python-pywt ignatenkobrain sergiopr
python-rasterio qulogic
python-rencode jgu orion patches
python-scikit-image orion sergiopr
python-scikit-learn besser82 ignatenkobrain lupinix sergiopr
python-shapely jcp volter
python-slixmpp fantom
python-smartcols ignatenkobrain
python-ssh2-python ignatenkobrain
python-statsmodels sergiopr
python-tables tnorth zbyszek
python-theano ignatenkobrain jjames
python-thriftpy ignatenkobrain
python-tinycss brouhaha
python-uvloop ignatenkobrain
python-yarl fab ignatenkobrain
python-zmq ralph tomspur
python2-astropy lupinix
python2-ipython lbalhar
python3-cangjie bochecha
sagemath pcpa
sphinxbase jjames mikep
sphinxtrain jjames mikep
sympy cbm jjames jussilehtola orion
urh jskarvad
xpra jgu sagitter sergiomb
Packages by maintainer:
abompard python-falcon
acaringi cassandra
akurtakov eclipse-pydev
athmane python-multidict python-pybloomfiltermmap
athoscr python-compreffor python-pyclipper
bbockelm libmacaroons
besser82 python-gensim python-hidapi python-scikit-learn
bochecha python3-cangjie
bpereto borgbackup
branto ceph
brouhaha python-cffi python-tinycss
bstinson python-djvulibre
carlwgeorge python-falcon
cbm sympy
churchyard ipython printrun python-admesh python-numpy-stl
cstratak ipython numpy python-peewee
dachary ceph
dcallagh python-plyvel
dcantrel ipython
deji espresso
dfateyev python-msgpack
dhanesh95 python-lupa
dmick ceph
fab pyliblo pystatgrab python-aiohttp python-msgpack python-yarl
fale git-fame python-lxml
fantom python-slixmpp
filabrazilska python-cassandra-driver
fschwarz borgbackup
grover python-kmod
heliocastro pystatgrab
hguemar python-hug
hhorak cassandra
hno bzr
ignatenkobrain git-fame ipython python-aiohttp python-httptools
python-lxml python-multidict python-nipy python-pytaglib python-pywt
python-scikit-learn python-smartcols python-ssh2-python python-theano
python-thriftpy python-uvloop python-yarl
jacksonisaac python-line_profiler
jamielennox python-falcon
jcp python-shapely
jdekloe pygrib pyproj
jdulaney python-cffi
jeckersb PyYAML
jgu python-rencode xpra
jjames openfst pocketsphinx python-gmpy2 python-orderedset
python-theano sphinxbase sphinxtrain sympy
jjohnstn eclipse-pydev
jkaluza pygame
jkastner libfreenect
jpena python-cradox
jskarvad brltty compizconfig-python pygame urh
jspaleta numpy python-basemap
junghans espresso
jussilehtola sympy
ke4qqq ceph
kkeithle ceph
ktdreyer ceph python-msgpack
kushal python-pandas
kwizart libfreenect
lbalhar ipython python-cassandra-driver python2-ipython
limb brltty numpy pygame python-basemap
lkundrak python-cassandra-driver
lmacken python-meliae
lupinix python-astropy-healpix python-astroscrappy python-healpy
python-mistune python-scikit-learn python2-astropy
mcyprian python-afl
melmorabity python-pymssql
mikep openfst pocketsphinx sphinxbase sphinxtrain
mizdebsk python-lxml
mrunge ipython
mstuchli python-peewee
ohaessler python-libdiscid
olysonek brltty
orion netcdf4-python numpy python-astropy python-gmpy2
python-pandas python-rencode python-scikit-image sympy
orphan python-afl
patches python-rencode
pbrobinson libplist
pcpa python-cypari2 python-cysignals python-fpylll sagemath
pjp python-msgpack
potty pystatgrab
praiskup python-cassandra-driver
pstodulk bzr
pwalter blueman
qulogic python-cartopy python-fiona python-rasterio
ralph python-zmq
raphgro python-jnius
rathann python-MDAnalysis python-gsd
raveit65 compizconfig-python
rdieter numpy
rebus dionaea
rharwood python-gssapi
rkuska python-bintrees
rmattes libfreenect
sagitter autowrap openms petsc4py python-assimulo python-giacpy xpra
salimma ipython
sergiomb xpra
sergiopr pymilia python-astropy python-pandas python-photutils
python-pywt python-scikit-image python-scikit-learn python-statsmodels
simo python-gssapi
slankes pystatgrab
sross python-gssapi
steve ceph
stevetraylen h5py
stingray ceph
sundaram python-msgpack
swt2c python-pyopengl
tadej python-meliae
talcite CheMPS2
terjeros h5py
tnorth python-tables
tomspur espresso ipython numpy python-zmq
trepik cassandra
ttomecek numpy
ttorling pystatgrab
vkrizan python-peewee
volter python-shapely
vvitek bzr
wakko666 python-pandas
zbyszek python-tables
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5 years, 8 months
Re: Why are we shipping debug builds of pythons?
by David Malcolm
On Mon, 2018-08-06 at 13:15 +0200, Jan Kratochvil wrote:
> On Mon, 06 Aug 2018 10:57:31 +0200, Petr Viktorin wrote:
> > On 08/05/18 14:01, Jan Kratochvil wrote:
> > > That is all together messy. This is why Microsoft has their debug
> > > build of
> > > their whole OS - Windows - called Checked Build:
> > > https://docs.microsoft.com/en-us/windows-hardware/drivers/devte
> > > st/checked-and-free-build-differences
> >
> > The main issue with --with-pydebug is that it changes the ABI, i.e.
> > all
> > extensions need to be re-built specifically for it (and debug
> > extensions
> > outside the standard library aren't usually packaged in Fedora).
> > That makes
> > it much less useful than if it just used less optimizations.
Petr: IMHO, it's much *more* useful: it checks for leaked objects,
which is the most painful issue to deal with when debugging Python
extensions. That was my motivation for adding it.
>
> That confirms the whole OS "Checked Build" variant would solve even
> this
> problem.
Jan: In my view, it's not a problem, it's a feature. An OS-wide
change wouldn't turn on "--with-pydebug" for Python, unless it was
directly coded into the specfile.
The change to the ABI is due to two new fields with the base struct for
python objects.
Dave
5 years, 8 months
Re: Why are we shipping debug builds of pythons?
by Petr Viktorin
On 08/05/18 14:01, Jan Kratochvil wrote:
> On Sun, 05 Aug 2018 13:39:58 +0200, Charalampos Stratakis wrote:
>> Here we are referring to the python2-debug or python3-debug builds, which
>> are just extra python builds that are compiled with the --with-pydebug flag
>
> Not just --with-pydebug:
>
> ------------------------------------------------------------------------------
> %if %{with debug_build}
> BuildPython debug \
> "--without-ensurepip --with-pydebug" \
> "-O0"
> %endif # with debug_build
>
> BuildPython optimized \
> "--without-ensurepip %{optimizations_flag}" \
> ""
> ------------------------------------------------------------------------------
>
> Many packages have their *-debug variants. And then some packages (like
> kernel) switch even their standard builds to debug mode but only in Rawhide.
> Despite all that the *-debug package one occasionally needs is then missing
> and one still has to build it locally.
>
> That is all together messy. This is why Microsoft has their debug build of
> their whole OS - Windows - called Checked Build:
> https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/checked...
The main issue with --with-pydebug is that it changes the ABI, i.e. all
extensions need to be re-built specifically for it (and debug extensions
outside the standard library aren't usually packaged in Fedora). That
makes it much less useful than if it just used less optimizations.
5 years, 8 months
