F31 System-Wide Change proposal: F31 Mass Python 2 Package Removal

https://fedoraproject.org/wiki/Changes/F31_Mass_Python_2_Package_Removal == Summary == Python 2 is reaching end of life, and the current maintainers would like to orphan it. To prevent massive breakage, and to plan help with porting to Python 3, we will systematically track and remove packages that still depend on Python 2. The Fedora 30 change, [[Changes/Mass Python 2 Package Removal]], continues in Fedora 31. We allow packagers to more easily abandon Python 2 parts of their packages (as an alternative to orphaning the whole package). We also start systematically collecting info on what remaining packages need to switch to Python 3 Finally, we'll use an expedited process to remove non-installable Python 2 packages from the distro. == Owner == * Name: [[User:Churchyard|Miro Hrončok]] * Name: [[User:Pviktori|Petr Viktorin]] * Name: [[User:Cstratak|Charalampos Stratakis]] * Name: [[User:Zbyszek|Zbigniew Jędrzejewski-Szmek]] (proposed) * Name: [[User:Ignatenkobrain|Igor Gnatenko]] * Name: [[User:Ngompa|Neal Gompa]] * Email: python-devel(a)lists.fedoraproject.org == Detailed Description == Over a thousand packages in Fedora still depend on Python 2, which will reach End of Life (EOL) on 2020-01-01, shortly after the expected Fedora 31 release date. Without upstream help, we (the current maintainers) cannot support Python 2 effectively, especially with such a vast ecosystem. There are also still too many packages to help porting them individually. We would ideally like to orphan and retire Python 2, but to avoid breaking the distro, we choose a more controlled path to removing it: * Python 2 subpackages can be abandoned easily, by a process similar to orphaning * Python 2 modules with no additional functionality will be removed * Non-essential requirements will be removed * We will collect information to help with "project management" * Non-installable Python 2 packages will be removed from the distro The Change describes several processes that, if approved, will be in force for Fedora 31 (until Final Freeze), or until this Change is reverted. (The processes are quite wordy and lawyery. Sorry! Unfortunately, with so many packages to handle, we need to formalize things.) This Change is just a more controlled version of orphaning the Python 2 package. For anyone who disagrees with the Change, there are several alternatives to following it: * Requesting an exception from the Python SIG (with reasons). * Suggesting changes to the process. * Taking over python2 now (with a better plan). * Expressing interest in maintaining Python 2, or just ignoring the Change. (When only these packages and their dependencies remain, the current Python maintainers will orphan python2 and many fundamental packages.) === Process for abandoning Python 2 subpackages === Many packages in Fedora contain Python 2 and Python 3 subpackages in the same component (SRPM). If a maintainer no longer wishes to maintain the Python 2 parts (e.g. a <code>python2-*</code> subpackage), it is currently easier to orphan/retire the entire package than to either: * remove only the Python 2 parts (which might break dependent packages, and so is not allowed), or * split the Python 2 parts out into a separate component (which would require formal review). We make abandoning Python 2 parts easier by introducing the following formal process. Both FESCo and FPC need to agree to this process, which will be in force for Fedora 31 (and may be extended to Fedrora 32). Highlights from the process: * The maintainer does not (have to) package the Python 2 parts, but must give a opportunity for others to do so. ** Prior to the split, Python 2 parts must be conditionalized (mainly to help others locate them). ** The package with the split-out Python 2 parts are granted several review exceptions. * If anything depends on the Python 2 parts, people are given a chance to claim them before they're removed. ** There is a 3-week period of waiting for responses to e-mails. (The normal time between oprhaning and retiring a package is 6 weeks.) ==== Removing Python 2 parts ==== # If the package has co-maintainers, discuss your intention with them. # Add <code>%bcond_without python2</code> to your specfile, and conditionalize all Python 2 parts using <code>%if %{with python2}</code>. (If the package already is conditionalized in some other way, you may keep the other way instead.) Commit the result to the master branch. # Check that your package builds and works without python2 (for example, by passing <code>--without python2</code> to <code>fedpkg mockbuild</code>). # Using a scratch build, check that both the Python 2 and Python 3 parts still build and work with default settings. # Invite people to claim the Python 2 parts. This process can be stopped if/when someone claims them: ## Announce on the fedora-devel mailing list that you wish to stop maintaining the Python 2 parts of your package, and invite others to claim them. ## If any packages depend on the functionality you are removing, then: ### After every 7 days, e-mail fedora-devel again. CC maintainers of all packages that depend on the parts you are removing (even transitively). Include a breakdown of the affected packages and maintainers. ### After 3 such e-mails (3 weeks since the first mail to fedora-devel), consider the Python 2 parts unclaimed and go on with the removal. ### If any *new* dependency on the functionality you are removing appears, restart the counter. (This should not happen often: python2 is deprecated, so adding new dependencies on it is generally against policy). # If anyone claimed the Python 2 parts, wait two weeks or until they are ready. See the following section for the claiming process. # Remove the Python 2 parts from Fedora Rawhide (i.e. the master branch). ## (The implementation is up to you: remove the conditionalized parts entirely, change the bcond to <code>%bcond_with python2</code>, or do something else to that effect.) # Scratch-build all dependent packages. If any fail to build from source, or to install, file a bug in Bugzilla (or comment on an existing bug). (If and when the Python 2 parts are claimed and packaged in Rawhide, you may backport the changes to released/branched Fedoras. The Bodhi update for such backport (if any) MUST contain both packages (Python 3 and Python 2).) ==== Claiming Python 2 parts of a package ==== If you wish to maintain Python 2 parts of a package which were (or are planned to be) removed through the above process: # Start with the spec file of the combined package # Remove everything that is Python 3-specific and/or would conflict with sub-packages of the Python3-only package. # Rename the package to start with "python2-" (instead of "python-", if applicable). Transform the python2- subpackage to the main package (if applicable). # Go through the [[Package_Review_Process]], noting to the reviewers that several exceptions apply (see below). Since this is a package split, any points where the Python 2 parts of the original subpackage did not meet the [https://docs.fedoraproject.org/en-US/packaging-guidelines/ Packaging Policy] or [[Packaging:ReviewGuidelines]] can be ignored for the new package, except: * The package MUST still be named according to the Package Naming Guidelines. * The source package for the software targeted at Python2 MUST take a <code>python2-</code> prefix. (This is stronger than the guidelines, which only say "should".) * Python2 binary packages MUST be named using a <code>python2-</code> prefix. And, of course: the new package MAY depend on <code>python2</code>, and/or any package that depends on <code>python2</code>, even if those are deprecated. ==== Notes ==== It is OK for a maintainer to split a package and keep maintaining both parts. The e-mail notifications can be skipped in this case. The package review exceptions still apply. It is OK for a maintainer to split a package, formally keep maintaining both parts, and immediately [[Orphaned_package_that_need_new_maintainers#Orphaning_Procedure|orphan]] the Python 2 parts after the split. Again, the e-mail notifications can be skipped in this case. We ask you to be nice and make your intentions clear. Note that leaf (not depended upon) packages only providing Python 2 modules are still being removed from Fedora. === Removing Modules === (Sub-)packages that only provide a python2 importable module, and are not required for other packages, will be removed. The details are specified in the Fedora 30 change, [[Changes/Mass Python 2 Package Removal]]. This effort continues in Fedora 31. === Removing Requirements === Requirements (Requires and BuildRequires) on Python2-only packages that bring little or no impact on features and quality of packages will be removed. This covers things like: * Tests of seldom-used features (for example, BuildRequiring a framework only to test integration with that framework). * Non-essential test-only functaionality (e.g. a test runner plugin for distributed multi-process testing). * Performance enhancements (e.g. compiled modules that are drop-in replacements for pure-Python modules). * Linters and code style checkers (these should be run in upstream CIs but, arguably, are not appropriate in <code>%check</code> at all). * Code and content generators (such as Cython, Sphinx, custom include file generators), especially if upstream ships pre-generated code/docs. Where possible, we will try to switch to the Python 3 versions of these dependencies rather than remove them. === Information on Remaining Packages === To help manage the removal, we ask maintainers of remaining Python2-only packages to provide some or all of the following information: * What is the reason for the Python2 dependency? (Is it software written in Python, or does it just provide Python bindings, or use Python in the build system or test runner?) * What are the upstream/community plans/timelines regarding Python 3? * What is the guidance for porting to Python 3? (Assuming that there is someone who generally knows how to port to Python 3, but doesn't know anything about the particular package, what are the next steps to take?) Generally, we will open Bugzillas asking for this information when needed, and we expect it to be provided and kept up to date. (Yes, mainaining a Python 2 package is a burden.) The bug may contain other questions as well. Every week (7 days), if the 3 questions are still unanswered, anyone may add a comment asking again for a response and linking to this process. After 3 such comments, the package can be orphaned (through a releng ticket, similar to the last step of the [[Policy_for_nonresponsive_package_maintainers]]). Weeks when the packager is on vacation (according to the [https://apps.fedoraproject.org/calendar/list/vacation/ calendar]) don't count. (If we think the packager is deliberately stalling, we file a FESCo ticket and ask to orphan the package immediately.) If the package is updated to remove the Python 2 dependency, this process immediately stops. This is more drastic version of the [[Policy_for_nonresponsive_package_maintainers]]: * Comments that don't answer the questions are ignored. * Instead of being taken over, the package is orphaned. * There are less steps and waiting. Both FESCo and FPC need to agree to this process, which will be in force for Fedora 31 (and may be extended to Fedora 32). (We will try to be nice and relax the process for packagers who just need more time and resources, but unfortunately we ultimately need a "big stick" to get things done.) === Removing non-installable packages from the distro === This process applies to packages that depend on Python 2. A similar distro-wide policy is suggested in https://pagure.io/fesco/issue/2109, but it has longer waiting times. If the distro-wide policy is accepted, this process may be adjusted (with FESCo approval) to better align with it. Both FESCo and FPC need to agree to this process, which will be in force for Fedora 31 (and may be extended to Fedora 32). If a package Fails To Build Or Install due to a missing Python 2 dependncy (for example becasue the maintainers ignored the announcements about removing a Python 2 subpackage) anyone can file a a bugzilla blocking the [https://bugzilla.redhat.com/show_bug.cgi?id=PY2FTBI PY2FTBI tracker bug]. After the bug reamins in the NEW state for a week, anyone can send a reminder with NEEDINFO. After 3 such reminders (and at least 3 weeks in the NEW state), the package gets orphaned, effectivelly short-circuiting the [[Policy_for_nonresponsive_package_maintainers]]. Packages still failing to install due to a missing Python 2 dependncy will be retired at Beta Freeze, assuming they have a Bugzilla bug open for at least 2 weeks. The package maintainer may pospone this retirement to the Final Freeze by promising to fix it until then. === Note about Deprecation === As a reminder, Python 2 is deprecated since Fedora 30 (see [[Packaging:Deprecating_Packages]]), which means that no package depends on it may be added to Fedora (though an exception might be granted by the packaging committee). This applies for indirect/transitive dependencies as well, so Python 2 packages generally don't need to be marked deprecated. == Benefit to Fedora == Python 2 will not be orphaned/retired suddenly, but in a controlled manner. == Scope == * Proposal owners: ** Track a list of packages to remove ** Track a graph of dependencies to break ** Retire python2-only packages ** Remove python2 subpackages from dual-python packages ** Break python2-only dependencies ** File bugs to request and track information * Other developers: ** Ask for exceptions if needed (with reasons) ** Help with the steps above (not needed, but helpful) ** Provide the requested information ** Optionaly remove Python 2 parts form their packages ** Optionaly mark remaining python2 packages deprecated * [[FPC]]: Formally agree with the processes above. * Release engineering: [https://pagure.io/releng/issue/8221 #8221] (a check of an impact with Release Engineering is needed) ** Releng may be asked to orphan some packages, according to the process above. Miro Hrončok will ask for permissions to orphan packages to not overwhelm releng. * Policies and guidelines: handled in the previous Change * Trademark approval: not needed for this Change == Upgrade/compatibility impact == Packages removed from Fedora repos will remain on the installed OS until explicitly removed by the user or obsoleted by the packagers. We'll only obsolete packages that would break upgrade (from fedora-obsolete-packages). == How To Test == Any program written in Python and any program that has Python plugins could potentially be influenced by this change. Testing is different for software that is still using Python 2 and for software that has been switched to Python 3. For any python2 programs, make sure those programs are still functional after the package removal. Since various packages that are no longer built will not be removed from an existing system, just upgrading and checking packages is not enough. Either a new installation should be used, or after a branching point, any packages which haven't been rebuilt for F31 (any packages with .fc30 or lower release suffix) should be removed from the system before testing. For the python3 programs, install all upgrades and check if the software works. Upgrade failures because of missing dependencies should be treated as bugs. Any removed python2 subpackages which break upgrades need to be added to Obsoletes in fedora-obsolete-packages. == User Experience == There will be less Python 2 RPMs in Fedora repos. Users are encouraged to switch to Python 3 and/or use Python 2 virtual environments and pip for development. == Dependencies == Ideally, all programs that use python2 would be switched to use python3. Although we don't expect everything to be switched over, as much as possible should be, so that the set of remaining python2 subpackages is as small as possible. == Contingency Plan == * Contingency mechanism: ** If for some reason not everything is removed, nothing happens, it can be removed later. If for some reason something breaks, some packages can be unretired or restored. ** If someone steps up to maintain Python 2 (including the full ecosystem of packages now in Fedora), they can decide to discontinue removing packages, revert the Change, or come up with another plan. (Note that in this case, current maintainers will most likely orphan many fundamental python2 packages.) * Contingency deadline: Final freeze * Blocks release? No * Blocks product? No == Documentation == This page is the main documentation. Also see: https://pythonclock.org -- Ben Cotton Fedora Program Manager TZ=America/Indiana/Indianapolis Pronouns: he/him