On Fri, Nov 20, 2009 at 12:37:00PM -0500, David Malcolm wrote:
Proposal: support multiple Python minor version, as one instance of
a
generalized way of supporting variant stacks.
Currently we support a single version of Python; we pick a specific
minor-release of Python (currently "2.6", in fact 2.6.4 in rawhide), and
all modules are assumed to be for that "2.MINOR" version of Python.
We plan to extend this further in Fedora 13, with the idea that there's
additionally a specific minor-release of Python 3 (currently "3.1", in
fact 3.1.1), and all python3-foo rpms are assumed to be for that
"3.MINOR" version of Python [1]
Some people want more: they want additional minor versions: for example,
when we want to upgrade "python" to mean "python 2.7", we will
rebuild
all our RPMs for 2.7, but some users may have code written to 2.6 that
they don't want to bother porting.
This makes more sense in an EPEL context: in EPEL5 we have the system
python as "2.4", but many people want a "2.6" stack, others want
"2.3",
2.5", "3.1", etc etc.
There are ways of doing this within the current Fedora/EPEL packaging
framework, but it would lead to an explosion of specfiles and CVS
directories. I dislike this: I think of specfiles as human-readable
source code for driving a reproducable build; I hate having many
almost-identical copies of a source file.
I had an idea for how this could be done whilst minimizing "specfile
explosion"; it allows for the generation of parallel families of RPMs
from one specfile, moving the complexity from the individual packager
into the build system, and onto release-engineering (sorry Jesse!) - and
eating up more disk space, build time, etc.
Caveat: I haven't looked into the details of whether Koji, package CVS,
and Makfile.common can actually cope with some of the ideas here, and
I've not tested the specifics of the rpm macros; the proposal is
independent of but compatible with the Python3 feature (I think).
PROPOSAL:
- Introduce the following new packaging macros:
- "pyprefix":
- defined as "python" for the system build of python
- defined as "pythonMN" for a build of python-M.N e.g.
"python26"
for "python-2.6"
- for use in the names of packages
- "pysuffix":
- defined as the empty string for the system build of python
- defined as "-M.N" for builds of python-M.N e.g "-32" for
"python-3.2"
- for use in /usr/bin when giving alternative versions of tools e.g.
"/usr/bin/pydoc-3.2" to be the python-3.2 version of "pydoc"
- Within python-foo module specfiles and the python module packaging
guidelines:
- Add default definitions of the macros:
%{!?pyprefix: %global pyprefix python}
%{!?pysuffix: %global pysuffix}
- Change:
Name: python-foo
to:
Name: %{pyprefix}-foo
and similarly for all dependencies, so that e.g.:
BuildRequires: python-devel python-setuptools
becomes:
BuildRequires: %{pyprefix}-devel %{pyprefix}-setuptools
- In %install, move scripts if necessary (to allow
parallel-installability), adding:
for script in foo bar baz ; do
mv %{buildroot}/usr/bin/$script mv
%{buildroot}/usr/bin/$script-%{pysuffix}
done
This gives us a single specfile, which can be used with different
definitions of these macros to give multiple builds, leading to multiple
SRPMs (the package changes name for the alternative builds).
Where do these macros gets expanded out?
Say in Koji's "dist-f14" we have python = python2.7 and python3 =
python3.2, and we want to support alternative python stacks.
Then for the dist-f14 build tags, introduce multiple new tags:
dist-f14-altbuild-python23
dist-f14-altbuild-python24
dist-f14-altbuild-python25
dist-f14-altbuild-python26
dist-f14-altbuild-python30
dist-f14-altbuild-python31
Each of these "altbuild" tags inherit from the main dist-f14 build tag,
adding a single RPM: "rpm-altbuild-macros-pythonMN-0.1-1.fc14" e.g.
"rpm-altbuild-macros-python26-0.1-1.fc14" which contains a single file:
/etc/rpm/altbuild-python26.macros
(changing the version as appropriate), defining the macros:
%global pyprefix python26
%global pysuffix -2.6
%global __python /usr/bin/python2.6
(need to ensure this last one actually gets used; need to move the
define from rpm to python-devel)
These Koji tags are in addition to the regular one ("dist-f14"), in
which "python" and "python3" have their default meanings.
All of these tags would deposit their built packages into dist-f14
Similarly, for EPEL5, extend "dist-5E-epel-testing-candidate",
introducing multiple child tags:
dist-5E-epel-testing-candidate-altbuild-python23
dist-5E-epel-testing-candidate-altbuild-python25
dist-5E-epel-testing-candidate-altbuild-python26
dist-5E-epel-testing-candidate-altbuild-python27
dist-5E-epel-testing-candidate-altbuild-python30
dist-5E-epel-testing-candidate-altbuild-python31
all of which would deposit their built packages into
dist-5E-epel-testing-candidate
For packaging CVS, add a new, optional file to a dist CVS subdirectory,
"build-variability". So for package "python-foo",
"python-foo/EL-5/build-variability" might contain these lines:
dist-5E-epel-testing-candidate-altbuild-python23
dist-5E-epel-testing-candidate # 2.4 is the default in EPEL5
dist-5E-epel-testing-candidate-altbuild-python25
dist-5E-epel-testing-candidate-altbuild-python26
dist-5E-epel-testing-candidate-altbuild-python27
dist-5E-epel-testing-candidate-altbuild-python30
dist-5E-epel-testing-candidate-altbuild-python31
If "build-variability" is present, it would change the meaning of "make
build": instead of triggering a job that does a single build into
"dist-5E-epel-testing-candidate", it would create a job that builds into
all of the build tags listed in the file. If any of these builds fail,
the whole build fails, and no RPMs reach he destination tag.
If a particular module only works with some minor-releases (perhaps it
doesn't work with python 3 yet, or needs python>=2.4 etc), then the file
would be edited appropriately.
I have no great love for "build-variability" as a name.
Caveat: I'm handwaving here; I haven't looked into the details of what
it would take on the Koji side and to Makefile.common
We'd be going from two dimensions of variability:
- build architecture (i386, x86_64 etc; currently expressed internally
in Koji)
- OS release (devel, F12, EPEL5 etc)
where we have a CVS directory per OS-release (one dimension)
up to three dimensions of variability:
- build architecture (i386, x86_64 etc)
- OS release (devel, F12, EPEL5 etc)
- Python minor release (2.3, 2.4, 2.5, 2.6, etc)
whilst keeping just one CVS directory per OS-release; the package
maintainer only has to edit a single specfile, and can attempt to ensure
that it builds across all minor-releases he/she cares about;
release-engineering would have a fair bit of extra complexity to deal
with though.
It may be necessary to add extra macros to the altbuild macro files so
that specfiles can somehow easily conditionalize the build for altbuild
variants matching some criteria (e.g. on altbuild for 2.6 and above,
need this extra Patch3")
I've tried to structure this so it can be reused for other language
runtimes where we might want to support multiple versions; perhaps the
Perl SIG would want a similar family of "altbuild" macros (I'm not a
perl person so maybe I'm missing something here).
The proposal only exposes a single extra dimension of variability;
higher dimensionality seems to lead to combinatorial explosion (e.g
sqlalchemy versions).
It also assumes (perhaps overoptimistically) than these other language
runtimes are independent, so that we never have collisions where e.g.
something wants to consume both multiple python stacks and multiple perl
stacks. This assumption fails for the case of e.g. multiple parallel
versions of say postgres or httpd, etc where you might also want
multiple versions of python, and multiple mod_pythons M * N
combinatorial explosion [2]
Thoughts? Total insanity?
Yes, insanity! :-)
But my objection is more along the implementation route taken rather than
the general idea of multiple versions of python2.
Right now, anyone can take an srpm, satisfy the BuildRequirements and a
static list of build dependencies listed in a buildsys package and generate
the builds. This proposal encodes some requirement information into the
buildsystem so that you can't just build the package on your own system.
Something that keeps all of the information in the SRPM is better.
Also, this scheme would leave us forcing maintainers to, some extent care
about multiple python versions. Even if, you only work with python3 and I
only work with python2, if there's a bug preventing builds on one python
version then the other person is partially blocked while that's being
figured out. If one python version is requiring rebuilds to fix runtime
bugs, users of the package on all python versions receive new packages
because the build triggers package rebuilding for all versions.
keeping binary rpms and srpms might be more difficult in this plan too. If
python24-foo was built from python-foo-1.0-1 but python-foo-1.0-2 only
builds python25-foo, do we reap the python24-foo and python-foo-1.0-1 SRPM
from the repositories or do we leave them?
I think if we do go for multiple python interpreter versions, having
separate srpms is going to provide a better experience for managing the
packages and for the end user.
-Toshio