On 21 August 2017 at 19:46, Petr Viktorin <pviktori(a)redhat.com&gt; wrote: Aye, this draft proposal was essentially me figuring out what I think we're going to want/need for Python before I pitched the related feature ideas to the Modularity WG :) The proposal then became something I could point to to say "This is the problem I'm trying to solve" while we discussed various possible ways of solving it. I finally had a discussion with Langdon about it today, and he really isn't a fan of my idea of trying to enhance the modularity tooling to natively support parallel installation of streams - he'd strongly prefer that we stick to the idea of "one active stream per module per system" (at least for now), and then rely on separate SRPMs and/or the Software Collections parallel installation layout to handle use cases like tox. (Note: I'll get the properly formatted proposal updated by tomorrow, so feel free to wait for that if the email version below is a bit hard to read) My current thinking based on that discussion is that we're actually going to need a module set that looks like this for F28+: * Platform Python (already planned for F27) - part of the Platform module - stream names match Fedora releases (f26, f27, etc) * Python Application Runtime Modules (already planned for F27) - one module for Python 2 and one for Python 3 - stream names match upstream Python releases (2.7, 3.6, etc) - Application Runtime Modules conflict with the corresponding Integrated Python Module - provide "/usr/bin/python2" and "/usr/bin/python3" respectively - also satisfy "python2-*" and "python3-*" RPM dependencies - only the Python 2 module satisfies "python-*" RPM dependencies (for now) * Integrated Python Modules - one module for Python 2 and one for Python 3 - stream names match Fedora releases (f26, f27, etc) - each depends on an *exact* version of the corresponding Python Application Runtime module * Default Python Module - provides /usr/bin/python (and nothing else) - stream names: no-default, python3-default, python2-default The scope for F27 would specifically be Platform Python and the Python Application Runtime Modules, with the separate Integrated Python Modules being defined for F28+ At least for now, the Python XY stacks for Fedora and EPEL, as well as the Python SCLs, would be treated as something generated downstream from the app runtime modules, rather than as something that the modularity tooling would necessarily handle natively. The general idea is that it would then be up to other modules to decide whether they specifically needed the Integrated Python Module with all the related system bindings (in which case they'd either depend on that module, or depend on another RPM that depended on it), or were prepared to cope with any installed version of Python 3 (in which case they'd just use normal RPM level "python3*" dependencies). Right now, the difference between the Integrated Python Modules and the Python Application Runtime Modules isn't particularly obvious, but it hopefully becomes clearer once we consider questions like "Who will decide when to rebase to Python 3.7?" and "When will a particular stream stop being updated?". For the Integrated Python Modules, those are Fedora level design decisions, as reflected in the stream names being based on Fedora versions. This means that for a system container, or a traditional mutable installation, you'd be able to continue to rely on a shared Python installation with all the operating system level bindings for things like the RPM database, without Fedora package maintainers having to provide prebuilt bindings for arbitrary Python versions. You'd only change streams when you changed base Fedora versions, and the update cycle for these streams would match the Fedora release cycle (i.e. each stream would receive updates for 13 months from the date of the corresponding Fedora release). By contrast, for the Python App Runtime Modules, when to rebase is an application developer level decision, as reflected in the stream names being based directly on Python versions. This means that for an application container image, you'd be able to select an arbitrary Python version of your choice, but in doing so, you'd potentially be giving up ready access to pre-built bindings for various system APIs (if your choice of stream didn't match the dependencies declared in the Integrated Python Module). For these streams, the update cycle would match that of the relevant upstream Python branch (i.e. full maintenance updates until shortly after the next Python feature release, then security updates until 5 years after the original X.Y.0 release). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

On 24 August 2017 at 19:02, Petr Viktorin <pviktori(a)redhat.com&gt; wrote: Oh, interesting, I had that backwards (I thought the planned F27 Python modules were the ones named after upstream Python releases). That means the current module definitions would be closer to what I'm calling the "Integrated Python Modules". Aye, I agree that will be a good way to tackle the parallel installable versions that *don't* define "/usr/bin/python3". However, for containers designed to run Python applications (web or otherwise), we *will* want mutually conflicting streams that define their symlinks and RPM provides based only on the Python major version. So lets make those extra modules their own distinct category: * Python Interoperability Testing Modules - one module per Python X.Y release - only one stream per module - conflict with the corresponding Application Runtime stream - provide "/usr/bin/pythonXY" and "/usr/bin/pythonX.Y" - do NOT satisfy "python2-*" and "python3-*" RPM dependencies And looking at their current implementation in Fedora, one notable difference between them and the Application Runtime streams is that these would just use their bundled pip and setuptools, rather than splitting those out the way the regular packages do. Sorry about that - when I started writing the email, my plan was to have them conflict with each other, but as I wrote that initial version up I went "Hang on, couldn't I use a stream dependency here and rely on *that* to trigger a conflict if you had an Integrated Python module installed and then tried to switch to an unsupported runtime stream, or vice-versa?" Pretty much, yeah. The "no-default" stream would be a bit different, as in that case it would install a shell script that printed out a custom error message. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

On 24 August 2017 at 21:28, Petr Viktorin <pviktori(a)redhat.com&gt; wrote: Oops, reading comprehension fail on my part :) These modules will probably only have one stream each, so I don't have a strong personal opinion on what that stream is called (although it's also possible they could each end up with two streams: one that actually installs the parallel installable version, and one that just depends on the corresponding application runtime stream). I think we can do better than that, and the relative timing of the F28 and Python 3.7 releases provides a good illustration of why I think we should aim to do so: * F28 code freeze is in March, with the release in May * 3.7 release candidate is in May, with the release in June Traditionally, that would have meant that we wouldn't get a Fedora based Python 3.7 container out the door until the release of Fedora 29 in October, and we'd never ship the F28+3.7 combination at all, even for folks that mainly just want the Python runtime, and then will use pip to install everything else they need. However, I think modularity gives us access to a more flexible approach: we can set up the F28 System Profile to *default* to the 3.6 Python App Runtime stream, allowing the sequence of updates to be: 1. In Fedora 28, we split the Python 3 runtime module into two pieces: - the App Runtime, with a 3.6 stream - the Integrated Runtime, with an f28 stream that depends on AppRuntime:3.6 2. After 3.7 hits beta in January 2018, we add a "3.7" stream to the App Runtime 3. After 3.7 is released, we start building a new F28+3.7 app development container - in this configuration, most system packages that need Python 3 can't be installed 4. In Fedora 29, we add an f29 stream to the integrated runtime that depends on AppRuntime:3.7 - now it would be the F29+3.6 configuration that prevents use of system packages that need Py3 Whether they're necessary or not depends on whether we enable the F28+3.7 configuration: if we allow that, then we need a way to make it clear that in that configuration, the only Python-3-based system packages you can install are the ones that rely on /usr/libexec/platform-python instead of /usr/bin/python3 (since the others ran their integration testing against the default 3.6 stack). I'm reasonably happy with App Runtime (since I stole that directly from "OpenShift Application Runtimes", which is the downstream use case I'm interested in making easier to maintain), and I think "Integrated Python" or "Integrated Runtime" accurately conveys the significance of the proposed stream mapping between Fedora versions and Python versions ("F28 integration testing uses Python 3.6", "F29 integration testing uses Python 3.7", etc). For the Testing Runtimes, we could likely just drop the "interoperability" qualifier, giving: - Platform Python: the private, always installed, potentially non-standards-compliant, Python that dnf uses - Integrated Python: an optional standards-compliant Python used for development and automated scripting *of* Fedora - Application Python: used to run Python apps *on* Fedora (but may not be integrated fully, depending on version) - Testing Python: sufficient for cross-version compatibility testing, but not recommended for app deployments Ugh, you're right - I completely overlooked that problem when I dreamed up that particular hack. That means providing a nicer error message than the default "interpreter not found" one would depend on either: - being able to "anti-register" a file in RPM and/or DNF (i.e. saying "I know I put a file there, but it's not a real implementation, so please don't actually use it to satisfy any dependencies") - providing some other way to customise the "interpreter not found" message that doesn't involve installing a stub interpreter implementation While the first of those actually sounds vaguely plausible, I'm not sure I dislike the current error message enough to put in *that* much effort into trying to improve it. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

On 29 August 2017 at 19:33, Petr Viktorin <pviktori(a)redhat.com&gt; wrote: My aim is to be able to define a "Fedora 28 + Python 3.7" container image that meets the following criteria: 1. Everything that's installed by default is owned by an RPM (no injecting extra symlinks as part of the container build process) 2. Invocations of "python3" and "/usr/bin/python3" (whether interactively or via shebang) run Python 3.7 3. I'm able to do this with just the Fedora Modularity tooling - no extra downstream tools like scl-utils If that's all they depend on, I guess they'll still be installable. However, if they depend on a particular version of the Python ABI, then the 3.7 package won't provide it. Originally that's what I had, but as I wrote it up in more detail (see https://github.com/fedora-python/fedora-python/commit/e76aa19870627059c9b...) I came to realise that it probably makes more sense to have a module level dependency from the Integrated Python branches to the corresponding Application Python branches and then have the "python3 symlink or not?" distinction exist between the Application module config and the Testing module configs. That way, the "Integrated Python" module shouldn't actually need to pull in any new source trees - it just becomes a way of mapping from Fedora versions to their integrated Python versions in a machine-readable way such that other Fedora modules can easily declare either "build for all Fedora Integrated Python streams" (by defining a module dependency on "python: []") *or* "build for all CPython version streams" (by depending on "cpython: []"). While exactly how the stream expansions are going to work is still a bit fuzzy (and I assume that's one of the things that will be resolved at Flock this week), my understanding based on the last fedora-devel thread about it is that we'll want to actively discourage folks from doing the latter without a compelling reason, as that's the approach that can lead to a combinatorial explosion of component builds: every version of Python on every active version of Fedora. While there are some cases where we'll actually want to do that (e.g. for the python-ecosystem modules), there are a lot of Fedora components that only need to be usable on the Python build that's integrated directly with the OS. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia