https://fedoraproject.org/wiki/Changes/Move_test.support_module_to_python... == Summary == Move <code>test.support</code> from <code>python3-libs</code> to <code>python3-test</code> subpackage. == Owner == * Name: [[User:lbalhar| Lumír Balhar]] * Email: [mailto:lbalhar@redhat.com| lbalhar(a)redhat.com] == Detailed Description == Python test modules should be used only for testing Python itself. However, some packages have buildtime or runtime dependency on parts of Python test modules. The aim of this change is to move the most popular Python test module <code>test.support</code> from <code>python3-libs</code> to <code>python3-test</code> subpackage which will help us discover packages which depend on it and also identify parts of the module which might be useful to move to standard library.

Also, [https://docs.python.org/3/library/test.html#module-test.support the Python documentation] says test.support is not a public module. Other things should not depend on it, and if something is useful outside the CPython test suite, we should ideally take it out of test.support. == Benefit to Fedora == The main benefit here is that moving it all to -test is less fragile and more consistent. It also helps us understand which parts of Python test suite are useful and which is worth to move to the standard library. There were also a few bugs caused by parts of the test module packaged in different subpackages and by differences between Python 2 and 3 - [https://bugzilla.redhat.com/show_bug.cgi?id=1651245 RHBZ#1651245], [https://bugzilla.redhat.com/show_bug.cgi?id=1528899 RHBZ#1528899], [https://bugzilla.redhat.com/show_bug.cgi?id=596258 RHBZ#596258] == Scope == * Proposal owners: ** Move <code>test.support</code> from <code>python3-libs</code> to <code>python3-test</code> subpackage. ** Identify as many as possible affected packages and try to fix their buildtime/runtime issues caused by the change. * Other developers: ** If a package depends on <code>test.support</code> module which we move to -test subpackage, change the build/runtime dependencies definition accordingly. * Release engineering: N/A (not a System Wide Change) * Policies and guidelines: N/A (not a System Wide Change) * Trademark approval: N/A (not needed for this Change) == Upgrade/compatibility impact == Test modules should not be used as a runtime dependency so there is no impact on an upgrade. == How To Test == N/A (not a System Wide Change) == User Experience == The user experience should not be affected. == Dependencies == N/A (not a System Wide Change) == Contingency Plan == * Contingency mechanism: Changes will be mostly done in Python specfile and they can be easily reverted in case of some unexpected issues. The second possibility is to set Provides to temporarily deactivate the impact of the change. * Contingency deadline: N/A (not a System Wide Change) * Blocks release? N/A (not a System Wide Change) * Blocks product? No. == Documentation == N/A (not a System Wide Change) == Release Notes == Since this affects only a few packages, no release notes are necessary.

On 24. 06. 19 22:06, Yaakov Selkowitz wrote: > On Mon, 2019-06-24 at 12:09 -0400, Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/Move_test.support_module_to_python... > > > > == Summary == > > > > Move <code>test.support</code> from <code>python3-libs</code> to > > <code>python3-test</code> subpackage. > > > > == Owner == > > * Name: [[User:lbalhar| Lumír Balhar]] > > * Email: [mailto:lbalhar@redhat.com| lbalhar(a)redhat.com] > > > > > > == Detailed Description == > > > > Python test modules should be used only for testing Python itself. > > However, some packages have buildtime or runtime dependency on parts > > of Python test modules. The aim of this change is to move the most > > popular Python test module <code>test.support</code> from > > <code>python3-libs</code> to <code>python3-test</code> subpackage > > which will help us discover packages which depend on it and also > > identify parts of the module which might be useful to move to standard > > library. > > The main reason for this change is to *experiment* and see what breaks > when it is moved? Why can't that be done in copr instead? The main reason for this change is to make packaging of Python easier (and more explicit) and less error prone in the future. Discovering packages that need test.support is a secondary goal that help upstream but does not help Fedora much. Should the change be reworded so this is more clear?

On 24. 06. 19 22:25, Yaakov Selkowitz wrote: > On Mon, 2019-06-24 at 22:10 +0200, Miro Hrončok wrote: > > On 24. 06. 19 22:06, Yaakov Selkowitz wrote: > > > On Mon, 2019-06-24 at 12:09 -0400, Ben Cotton wrote: > > > > https://fedoraproject.org/wiki/Changes/Move_test.support_module_to_python... > > > > > > > > == Summary == > > > > > > > > Move <code>test.support</code> from <code>python3-libs</code> to > > > > <code>python3-test</code> subpackage. > > > > > > > > == Owner == > > > > * Name: [[User:lbalhar| Lumír Balhar]] > > > > * Email: [mailto:lbalhar@redhat.com| lbalhar(a)redhat.com] > > > > > > > > > > > > == Detailed Description == > > > > > > > > Python test modules should be used only for testing Python itself. > > > > However, some packages have buildtime or runtime dependency on parts > > > > of Python test modules. The aim of this change is to move the most > > > > popular Python test module <code>test.support</code> from > > > > <code>python3-libs</code> to <code>python3-test</code> subpackage > > > > which will help us discover packages which depend on it and also > > > > identify parts of the module which might be useful to move to standard > > > > library. > > > > > > The main reason for this change is to *experiment* and see what breaks > > > when it is moved? Why can't that be done in copr instead? > > > > The main reason for this change is to make packaging of Python easier (and more > > explicit) and less error prone in the future. Discovering packages that need > > test.support is a secondary goal that help upstream but does not help Fedora much. > > > > Should the change be reworded so this is more clear? > > I would say the scoping should happen (e.g. in copr) first, then we'll > know what the scope (and hence feasibility) of this change truly is. What will the scoping actually tell us? If it tells us that X hundred packages need to add BR for python3-test, we'll just do that and report the list to upstream. We can do that during the mass rebuild.

On 25. 06. 19 0:18, Yaakov Selkowitz wrote: > On Mon, 2019-06-24 at 23:09 +0200, Miro Hrončok wrote: > > On 24. 06. 19 22:25, Yaakov Selkowitz wrote: > > > On Mon, 2019-06-24 at 22:10 +0200, Miro Hrončok wrote: > > > > On 24. 06. 19 22:06, Yaakov Selkowitz wrote: > > > > > On Mon, 2019-06-24 at 12:09 -0400, Ben Cotton wrote: > > > > > > https://fedoraproject.org/wiki/Changes/Move_test.support_module_to_python... > > > > > > > > > > > > == Summary == > > > > > > > > > > > > Move <code>test.support</code> from <code>python3-libs</code> to > > > > > > <code>python3-test</code> subpackage. > > > > > > > > > > > > == Owner == > > > > > > * Name: [[User:lbalhar| Lumír Balhar]] > > > > > > * Email: [mailto:lbalhar@redhat.com| lbalhar(a)redhat.com] > > > > > > > > > > > > > > > > > > == Detailed Description == > > > > > > > > > > > > Python test modules should be used only for testing Python itself. > > > > > > However, some packages have buildtime or runtime dependency on parts > > > > > > of Python test modules. The aim of this change is to move the most > > > > > > popular Python test module <code>test.support</code> from > > > > > > <code>python3-libs</code> to <code>python3-test</code> subpackage > > > > > > which will help us discover packages which depend on it and also > > > > > > identify parts of the module which might be useful to move to standard > > > > > > library. > > > > > > > > > > The main reason for this change is to *experiment* and see what breaks > > > > > when it is moved? Why can't that be done in copr instead? > > > > > > > > The main reason for this change is to make packaging of Python easier (and more > > > > explicit) and less error prone in the future. Discovering packages that need > > > > test.support is a secondary goal that help upstream but does not help Fedora much. > > > > > > > > Should the change be reworded so this is more clear? > > > > > > I would say the scoping should happen (e.g. in copr) first, then we'll > > > know what the scope (and hence feasibility) of this change truly is. > > > > What will the scoping actually tell us? If it tells us that X hundred packages > > need to add BR for python3-test, we'll just do that and report the list to > > upstream. We can do that during the mass rebuild. > > If there would indeed be X hundred packages affected, would this move > still make sense? The python3-test subpackage is even larger than most > of the rest of python3 combined, and the vast majority of it is > irrelevant to other packages. Such usage would indicate to me that > some work needs to be done within the ecosystem to respect its official > status as internal-only. (Perhaps, in that case, a move to python3- > devel could be considered instead.) Good questions. What number of affected packages do you think would be crucial? I say X hundred, because I don't anticipate it will go to thousands. However with 3000 Python packages, couple hundreds is IMHO not a big deal. Note that this is build dependency, not a runtime one.

>>>>> I would say the scoping should happen (e.g. in copr) first, then we'll >>>>> know what the scope (and hence feasibility) of this change truly is. >>>> >>>> What will the scoping actually tell us? If it tells us that X hundred packages >>>> need to add BR for python3-test, we'll just do that and report the list to >>>> upstream. We can do that during the mass rebuild. >>> >>> If there would indeed be X hundred packages affected, would this move >>> still make sense? The python3-test subpackage is even larger than most >>> of the rest of python3 combined, and the vast majority of it is >>> irrelevant to other packages. Such usage would indicate to me that >>> some work needs to be done within the ecosystem to respect its official >>> status as internal-only. (Perhaps, in that case, a move to python3- >>> devel could be considered instead.) >> >> Good questions. What number of affected packages do you think would be crucial? >> I say X hundred, because I don't anticipate it will go to thousands. However >> with 3000 Python packages, couple hundreds is IMHO not a big deal. Note that >> this is build dependency, not a runtime one. > > Nevertheless, there has been a great deal of effort recently to shrink > buildroots and speed up build times. Having to add BR: python3-test to > packages would mean adding a download of ~9MiB and an installation of > ~3K files to every single affected package's build time. IMO this > needs to be fully scoped before consideration. Wait a minute, adding a BR python3-test would mean an additional 9MiB ok, but currently these files are part of python3-libs which *every packages* get. In other words, for the packages that require python3-test the amount of data downloaded doesn't change while for all the packages that do *not* require python3-test, the amount of data is reduced. This sounds like a win to me :)

On Tue, Jun 25, 2019 at 10:48:42AM +0200, Miro Hrončok wrote: > On 25. 06. 19 9:50, Pierre-Yves Chibon wrote: >>>>>>> I would say the scoping should happen (e.g. in copr) first, then we'll >>>>>>> know what the scope (and hence feasibility) of this change truly is. >>>>>> >>>>>> What will the scoping actually tell us? If it tells us that X hundred packages >>>>>> need to add BR for python3-test, we'll just do that and report the list to >>>>>> upstream. We can do that during the mass rebuild. >>>>> >>>>> If there would indeed be X hundred packages affected, would this move >>>>> still make sense? The python3-test subpackage is even larger than most >>>>> of the rest of python3 combined, and the vast majority of it is >>>>> irrelevant to other packages. Such usage would indicate to me that >>>>> some work needs to be done within the ecosystem to respect its official >>>>> status as internal-only. (Perhaps, in that case, a move to python3- >>>>> devel could be considered instead.) >>>> >>>> Good questions. What number of affected packages do you think would be crucial? >>>> I say X hundred, because I don't anticipate it will go to thousands. However >>>> with 3000 Python packages, couple hundreds is IMHO not a big deal. Note that >>>> this is build dependency, not a runtime one. >>> >>> Nevertheless, there has been a great deal of effort recently to shrink >>> buildroots and speed up build times. Having to add BR: python3-test to >>> packages would mean adding a download of ~9MiB and an installation of >>> ~3K files to every single affected package's build time. IMO this >>> needs to be fully scoped before consideration. >> >> Wait a minute, adding a BR python3-test would mean an additional 9MiB ok, but >> currently these files are part of python3-libs which *every packages* get. >> In other words, for the packages that require python3-test the amount of data >> downloaded doesn't change while for all the packages that do *not* require >> python3-test, the amount of data is reduced. >> This sounds like a win to me :) > > Unfortunately, this is not correct. > > python3-test already is large. > > We just move a small bit of python3-libs (test.support) and we move it to > python3-test for consistency. > > test.support is 396K installed (for Python 3.8). Arf, then it's a lesser win than I thought. Have you considered doing a python3-test-support subpackage? This would allow the separate package and thus give you the information you're looking for as well as avoid downloading the entire python3-test for these few files.

On 6/25/19 12:25 PM, Miro Hrončok wrote: > On 25. 06. 19 12:07, Pierre-Yves Chibon wrote: > > On Tue, Jun 25, 2019 at 10:48:42AM +0200, Miro Hrončok wrote: > > > On 25. 06. 19 9:50, Pierre-Yves Chibon wrote: > > > > > > > > > I would say the scoping should happen (e.g. in copr) first, then we'll > > > > > > > > > know what the scope (and hence feasibility) of this change truly is. > > > > > > > > > > > > > > > > What will the scoping actually tell us? If it tells us that X hundred packages > > > > > > > > need to add BR for python3-test, we'll just do that and report the list to > > > > > > > > upstream. We can do that during the mass rebuild. > > > > > > > > > > > > > > If there would indeed be X hundred packages affected, would this move > > > > > > > still make sense? The python3-test subpackage is even larger than most > > > > > > > of the rest of python3 combined, and the vast majority of it is > > > > > > > irrelevant to other packages. Such usage would indicate to me that > > > > > > > some work needs to be done within the ecosystem to respect its official > > > > > > > status as internal-only. (Perhaps, in that case, a move to > > > > > > > python3-devel could be considered instead.) > > > > > > > > > > > > Good questions. What number of affected packages do you think would be crucial? > > > > > > I say X hundred, because I don't anticipate it will go to thousands. However > > > > > > with 3000 Python packages, couple hundreds is IMHO not a big deal. Note that > > > > > > this is build dependency, not a runtime one. > > > > > > > > > > Nevertheless, there has been a great deal of effort recently to shrink > > > > > buildroots and speed up build times. Having to add BR: python3-test to > > > > > packages would mean adding a download of ~9MiB and an installation of > > > > > ~3K files to every single affected package's build time. IMO this > > > > > needs to be fully scoped before consideration. > > > > > > > > Wait a minute, adding a BR python3-test would mean an additional 9MiB ok, but > > > > currently these files are part of python3-libs which *every packages* get. > > > > In other words, for the packages that require python3-test the amount of data > > > > downloaded doesn't change while for all the packages that do *not* require > > > > python3-test, the amount of data is reduced. > > > > This sounds like a win to me :) > > > > > > Unfortunately, this is not correct. > > > > > > python3-test already is large. > > > > > > We just move a small bit of python3-libs (test.support) and we move it to > > > python3-test for consistency. > > > > > > test.support is 396K installed (for Python 3.8). > > > > Arf, then it's a lesser win than I thought. > > Have you considered doing a python3-test-support subpackage? This would allow > > the separate package and thus give you the information you're looking for as > > well as avoid downloading the entire python3-test for these few files. > > The purpose of this change is to make packaging of Python easier and > avoid problems, when the test.support module cannot be properly used > without the rest of the test module. What you are proposing will make > packaging more complicated and would not eliminate the problems at all. > > I think we are talking about couple packages here. Would it make sense > to have a number and say that if more than X packages suddenly need to > add BR for python3-test, we revert the change and consider a different > approach? > > BTW python3-test is now buildrequired by 7 packages: > python-bz2file > python-honcho > python-iniparse > python-kitchen > python-typing-extensions > python-urwid > python-zodbpickle > I am bringing here some numbers which help us to understand that the change won't affect a lot of packages. I've prepared a special COPR [0] where test.support module is not available at all and then I rebuilt all Python packages. From 2952 packages 2769 built successfully and only 182 failed. Most failures were caused by missing dependencies (python2-), incompatible pytest or sphinx version, etc. and only 7 packages fail to build because of the missing test.support module. From those 7 packages, 4 already require python3-test so there will be need to change only 3 packages after we'll move test.support module from python3-libs to python3-test. [0] https://copr.fedorainfracloud.org/coprs/lbalhar/test.support_change/