Debug Python stacks revisited - experimental build in Rawhide, targetting Fedora 14
by David Malcolm
(This is a scaled-back version of a proposal I sent to this list a
couple of months ago [1])
There are various configuration flags that can be used when building
Python.
Currently we have a configuration aimed at the typical use-case: as much
optimization as reasonable.
However, upstream Python supports a number of useful debug options which
use more RAM and CPU cycles, but make it easier to track down bugs [2]
Typically these are of use to people working on Python C extensions, for
example, for tracking down awkward reference-counting mistakes. I've
had at least three developers whose opinion I value very highly ask me
for these (for example John Palmieri is currently working on the PyGI
stack, and is running into difficult reference-counting issues).
Indeed, Debian and Ubuntu have had these alternate builds available for
a couple of years now. [3]
I've looked through Debian's patch [4], and come up with a somewhat
modified version that does mostly the same thing, though (I think)
somewhat better fitting our build process.
The python.spec now configures and builds, and installs the python
sources twice, once with the regular optimized settings, and again with
debug settings. (in most cases the files are identical between the two
installs, and for the files that are different, they get separate paths)
I've been testing with this on my machine and it works fine; I've also
been able to successfully use distutils to build extension modules
So I've decided to try this in Rawhide for F-14; the latest build is
here:
http://koji.fedoraproject.org/koji/buildinfo?buildID=174357
The relevant CVS commit is here:
http://cvs.fedoraproject.org/viewvc/rpms/python/devel/python.spec?r1=1.18...
http://cvs.fedoraproject.org/viewvc/rpms/python/devel/python-2.6.5-debug-...
and the specfile comment contains more detailed implementation notes.
The builds are set up so that they can share the same .py and .pyc files
- they have the same bytecode format.
However, they are incompatible at the machine-code level: the extra
debug-checking options change the layout of Python objects in memory, so
the configurations have different shared library ABIs. A compiled C
extension built for one will not work with the other.
The key to keeping the different module ABIs separate is that module
"foo.so" for the standard optimized build will instead be "foo_d.so"
i.e. gaining a "_d" suffix to the filename, and this is what the
"import" routine will look for. This convention is from the Debian
patch, and ultimately comes from the way the Windows build is set up in
the upstream build process.
Similarly, the optimized libpython2.6.so.1.0 now has a
libpython2.6_d.so.1.0 cousin for the debug build: all of the extension
modules are linked against the appropriate libpython, and there's
a /usr/include/python2.6-debug directory, parallel with
the /usr/include/python2.6 directory. There's a new "sys.pydebug"
boolean to distinguish the two configurations, and the distutils module
uses this to supply the appropriate header paths ,and linker flags when
building C extension modules.
Finally, the debug build's python binary is /usr/bin/python2.6-debug,
hardlinked as /usr/bin/python-debug (as opposed to /usr/bin/python2.6
and /usr/bin/python)
It's easy to spot the debug build: the interactive mode tells you the
total reference count of all live Python objects after each command:
[david@surprise devel]$ python-debug
Python 2.6.5 (r265:79063, May 19 2010, 18:20:14)
[GCC 4.4.3 20100422 (Red Hat 4.4.3-18)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print "hello world"
hello world
[28748 refs]
>>>
[28748 refs]
[15041 refs]
So the debug build shares _most_ of the files with the regular build
(.py/.pyc/.pyo files; directories; support data; documentation); the
only differences are the ELF files (binaries/shared libraries), and
infrastructure relating to configuration (Include files, Makefile,
python-config => python-debug-config, etc) that are different.
I've tested building the "coverage" module against both runtimes, and it
works; it installs shared .py/.pyc files and a pair of
tracer.so/tracer_d.so files.
I tried a few different ways of packaging the debug configuration: I
considered
(a) adding it to the python-devel subpackage, or
(b) to the python-debuginfo subpackage (Debian adds it to their
python-dbg packages, which are kind of the equivalent of our -debuginfo
rpms), alternatively
(c) building out a "debug" subpackage for each of the subpackages within
the python specfile, doubling the number of subpackages
The approach I favor (option (d), I guess), is to have a single
"python-debug" subpackage, holding everything to do with the debug
configuration: equivalent to all of the subpackaes from the regular
configuration, and requiring them all (since they leverage the
shared .py files, for instance). My reasoning here is that this feature
is aimed at advanced Python developers, and if you want some of it you
probably want all of it - so just one subpackage for simplicitly - but
you don't need it for regular builds or debugging, so it's seems better
to keep separate from the -devel and -debuginfo subpackages.
This is a scaled-back version of my earlier proposal (in which I
proposed entirely parallel stacks, and varying the unicode settings)
This is far simpler. In particular, the optimized build should be
unaffected: all of the paths and the ELF metadata for the standard build
should be unchanged compared to how they were before adding the debug
configuration.
I would like to build out some of our compiled extension modules so that
we can add -debug subpackages, in an analogous way to the core python
package, but I think it should purely be a voluntary thing: I don't want
to burden people packaging Python modules with additional work. Having
said that, if you do find yourself debugging a nasty reference counting
issue inside an extension module, you'll need a debug build of every C
extension module that your reproducer script uses, so the more the
better. For reference, Ubuntu do this for all of the Python code in a
typical GNOME desktop [3]. We should figure out sane RPM conventions
for packaging these (sorry: yes I want to change the python packaging
guidelines again, hopefully less invasive than the Python 3 change
though)
I'm tracking all of this work here:
https://fedoraproject.org/wiki/DaveMalcolm/DebugPythonStacks
I hope for it to be a Fedora 14 feature. It's debatable whether it
should be a feature: this is an area where we're somewhat behind other
distributions, so not so good from a marketing perspective - but a good
thing to get fixed.
I plan to work next on doing the same for our python3 src.rpm. I need
to try to get this upstream in some form as well.
Hope this seems sane - thoughts? (thanks for reading this far; I know
this email is too long)
Dave
[1]
http://lists.fedoraproject.org/pipermail/python-devel/2010-March/000213.html
[2] http://svn.python.org/projects/python/trunk/Misc/SpecialBuilds.txt
[3] https://wiki.ubuntu.com/PyDbgBuilds
[4]
http://patch-tracker.debian.org/patch/series/view/python2.6/2.6.5-2/debug... and http://patch-tracker.debian.org/patch/series/view/python2.6/2.6.5-2/pydeb...
13 years, 6 months
FYI: python-sphinx-1.0 coming to rawhide soon
by Toshio Kuratomi
This is just a heads up -- python-sphinx-1.0b1 has arrived. hircus or
I will be updating rawhide to the new version soon -- which may cause your
documentation to stop building. There are some incompatible changes in this
update, notably to the markup used for c functions and modules. There is
a compatibility plugin that you can enable to take care of some of the
issues or you can port to the new "domains" syntax.
we probably will not be pushing this back to older versions of Fedora but
we'll have to see what the response is. If we need to push out a compat
package for EPEL5 (python-sphinx1-1.0) then we'll have the possibility of
doing the same for older Fedora.
-Toshio
13 years, 6 months
TurboGears 2.1 beta in updates-testing
by Luke Macken
Hey all,
Last week I updated the entire TurboGears2/Pylons stack in rawhide, F13,
F12, and EL-5. Everything should currently be in in updates-testing, so
any feedback or karma you can provide would be appreciated.
I know there is a TG2 hackfest this Sunday, where we are trying to chip
away at the small blocker list for the TG2.1 release. I'm also still
trying to figure out a couple of bugs in Moksha & fedoracommunity due to
the latest stack, but I'm hoping to have those ironed out before or
shortly after F13.
I also updated the TurboGears resources on the wiki.
http://fedoraproject.org/wiki/TurboGears2
luke
13 years, 6 months
Some more Python 3: (was Re: rawhide report: 20100507 changes)
by David Malcolm
On Fri, 2010-05-07 at 12:25 +0000, Rawhide Report wrote:
Three more python 3 subpackages in today's "rawhide" heading for F14 -
I've gone ahead and added these to the wiki here:
https://fedoraproject.org/wiki/Features/Python3F13#Python_3_already_in_Fe...
Are we missing anything?
[snip]
> libsemanage-2.0.45-4.fc14
> -------------------------
> * Tue Apr 27 2010 David Malcolm <dmalcolm(a)redhat.com> - 2.0.45-4
> - add python3 subpackage
[snip]
> python-beaker-1.5.3-2.fc14
> --------------------------
> * Thu May 06 2010 Luke Macken <lmacken(a)redhat.com> - 1.5.3-2
> - Add a python3 subpackage
>
>
> python-mako-0.3.2-2.fc14
> ------------------------
> * Tue May 04 2010 David Malcolm <dmalcolm(a)redhat.com> - 0.3.2-2
> - add python3 subpackage
[snip]
13 years, 7 months
Trying to package "pypy" for F13 onwards
by David Malcolm
pypy is an alternate implementation of Python [1], implemented in Python
itself, together with a set of tools for compiling that implementation
down to C code. The resulting implementation can, in theory, be faster
than the regular "CPython" implementation we know as /usr/bin/python
I've managed to package up pypy for Fedora; I've opened up a review
request for it here:
https://bugzilla.redhat.com/show_bug.cgi?id=588941
Anyone feel like reviewing it?
You can download F-13 packages from the scratch build here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=2147291
Upstream pypy is working on implementing the C extension module API, so
in theory we could try recompiling our python extension modules against
pypy at some future point. I don't yet know if that's going to be
feasible in the F14 timescale, or if we're looking at further in the
future.
Hope this is useful
Dave
[1] http://pypy.org/
13 years, 7 months
