Hi, I think it would be useful to have a standard way of disabling the running of tests during RPM build (in the %check section of a spec file).
I see a lot of packages already having %bcond's or other macro definitions to archieve this, but each package has their own way, there's no real standard. Thus you have to first look into the spec, locate the appropriate %bcond or macro name and only then you can disable the tests.
I would like to propose two approaches:
(a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to conditionalize the tests.
(b) Or, if that's too strong, mention in the guidelines the common methods that are being used (e.g. %bcond tests and %bcond check) so that new packagers have something to use.
What do you think? Tomas
On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
Hi, I think it would be useful to have a standard way of disabling the running of tests during RPM build (in the %check section of a spec file).
I see a lot of packages already having %bcond's or other macro definitions to archieve this, but each package has their own way, there's no real standard. Thus you have to first look into the spec, locate the appropriate %bcond or macro name and only then you can disable the tests.
I would like to propose two approaches:
(a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to conditionalize the tests.
(b) Or, if that's too strong, mention in the guidelines the common methods that are being used (e.g. %bcond tests and %bcond check) so that new packagers have something to use.
What's the motivation for disabling tests globally?
I have some packages where tests fail on particular architectures at particular times, and what I do there is (a) file a BZ (b) surround the %check section with %ifarch/%ifnarch and a comment linking to the bug, and this seems to me a practical and lightweight approach that requires no special support in the toolchain.
Also rpmbuild itself can happily disable tests, just add the --nocheck flag.
Rich.
On 05. 06. 20 16:26, Richard W.M. Jones wrote:
On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
Hi, I think it would be useful to have a standard way of disabling the running of tests during RPM build (in the %check section of a spec file).
I see a lot of packages already having %bcond's or other macro definitions to archieve this, but each package has their own way, there's no real standard. Thus you have to first look into the spec, locate the appropriate %bcond or macro name and only then you can disable the tests.
I would like to propose two approaches:
(a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to conditionalize the tests.
(b) Or, if that's too strong, mention in the guidelines the common methods that are being used (e.g. %bcond tests and %bcond check) so that new packagers have something to use.
What's the motivation for disabling tests globally?
Bootstrapping mostly.
I have some packages where tests fail on particular architectures at particular times, and what I do there is (a) file a BZ (b) surround the %check section with %ifarch/%ifnarch and a comment linking to the bug, and this seems to me a practical and lightweight approach that requires no special support in the toolchain.
Also rpmbuild itself can happily disable tests, just add the --nocheck flag.
However rpmbuild itself doesn't support "CheckRequires", so the bcond often looks like this:
BuildRequires: python3-devel BuildRequires: python3-setuptools %if %{with tests} BuildRequires: python3-pytest BuildRequires: python3-hypothesis %endif
...
%if %{with tests} %check %pytest %endif
On Fri, Jun 05, 2020 at 04:38:03PM +0200, Miro Hrončok wrote:
On 05. 06. 20 16:26, Richard W.M. Jones wrote:
On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
Hi, I think it would be useful to have a standard way of disabling the running of tests during RPM build (in the %check section of a spec file).
I see a lot of packages already having %bcond's or other macro definitions to archieve this, but each package has their own way, there's no real standard. Thus you have to first look into the spec, locate the appropriate %bcond or macro name and only then you can disable the tests.
I would like to propose two approaches:
(a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to conditionalize the tests.
(b) Or, if that's too strong, mention in the guidelines the common methods that are being used (e.g. %bcond tests and %bcond check) so that new packagers have something to use.
What's the motivation for disabling tests globally?
Bootstrapping mostly.
For the RISC-V bootstrap we used rpmbuild directly (before Koji and its dependencies had been ported), and added --nocheck. However once Koji was working we built packages properly with checks enabled.
How often do we bootstrap Fedora from scratch? Wholly new architectures are rare. Are there other events that require bootstrapping from scratch?
Rich.
On 6/5/20 4:46 PM, Richard W.M. Jones wrote:
On Fri, Jun 05, 2020 at 04:38:03PM +0200, Miro Hrončok wrote:
On 05. 06. 20 16:26, Richard W.M. Jones wrote:
On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
Hi, I think it would be useful to have a standard way of disabling the running of tests during RPM build (in the %check section of a spec file).
I see a lot of packages already having %bcond's or other macro definitions to archieve this, but each package has their own way, there's no real standard. Thus you have to first look into the spec, locate the appropriate %bcond or macro name and only then you can disable the tests.
I would like to propose two approaches:
(a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to conditionalize the tests.
(b) Or, if that's too strong, mention in the guidelines the common methods that are being used (e.g. %bcond tests and %bcond check) so that new packagers have something to use.
What's the motivation for disabling tests globally?
Bootstrapping mostly.
For the RISC-V bootstrap we used rpmbuild directly (before Koji and its dependencies had been ported), and added --nocheck. However once Koji was working we built packages properly with checks enabled.
How often do we bootstrap Fedora from scratch? Wholly new architectures are rare. Are there other events that require bootstrapping from scratch?
Not necessarily bootstrapping from scratch, this is useful for bootstrapping of anything in Fedora.
Fod example, Python now releases on a yearly schedule, and bootstrapping it is a huge undertaking involving thousands of components.
And most importantly, the reason tests are disabled during bootstrapping is missing dependencies. Those have to be conditionalized by some %bcond or macro, and `--nocheck` doesn't help.
Tomas
Dne 05. 06. 20 v 17:24 Tomas Orsava napsal(a):
On 6/5/20 4:46 PM, Richard W.M. Jones wrote:
On Fri, Jun 05, 2020 at 04:38:03PM +0200, Miro Hrončok wrote:
On 05. 06. 20 16:26, Richard W.M. Jones wrote:
On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
Hi, I think it would be useful to have a standard way of disabling the running of tests during RPM build (in the %check section of a spec file).
I see a lot of packages already having %bcond's or other macro definitions to archieve this, but each package has their own way, there's no real standard. Thus you have to first look into the spec, locate the appropriate %bcond or macro name and only then you can disable the tests.
I would like to propose two approaches:
(a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to conditionalize the tests.
(b) Or, if that's too strong, mention in the guidelines the common methods that are being used (e.g. %bcond tests and %bcond check) so that new packagers have something to use.
What's the motivation for disabling tests globally?
Bootstrapping mostly.
For the RISC-V bootstrap we used rpmbuild directly (before Koji and its dependencies had been ported), and added --nocheck. However once Koji was working we built packages properly with checks enabled.
How often do we bootstrap Fedora from scratch? Wholly new architectures are rare. Are there other events that require bootstrapping from scratch?
Not necessarily bootstrapping from scratch, this is useful for bootstrapping of anything in Fedora.
Just FTR, we have bootstrapping guidelines:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping
Vít
Fod example, Python now releases on a yearly schedule, and bootstrapping it is a huge undertaking involving thousands of components.
And most importantly, the reason tests are disabled during bootstrapping is missing dependencies. Those have to be conditionalized by some %bcond or macro, and `--nocheck` doesn't help.
Tomas _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
Le mardi 09 juin 2020 à 12:08 +0200, Vít Ondruch a écrit :
Just FTR, we have bootstrapping guidelines https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping
Those suffer from 1. the horrible bcond logic inversion that trips pretty much everyone all the time. 2. the fact you can not ask koji or mock for a bootstrapped build, you have to change the spec manually
Dne 09. 06. 20 v 12:12 Nicolas Mailhot napsal(a):
Le mardi 09 juin 2020 à 12:08 +0200, Vít Ondruch a écrit :
Just FTR, we have bootstrapping guidelines https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping
Those suffer from
- the horrible bcond logic inversion that trips pretty much everyone
all the time.
That won't be different for what was the original question here, i.e. conditionally disable tests. bconds are what we have for better or worse.
And really, this seems about bootstrapping not disabling tests, which are not completely different, but nobody can objects bootstrapping, while disabling tests might be good just to improve build speed and nothing else. That should never happen in production environment IMO.
- the fact you can not ask koji or mock for a bootstrapped build, you
have to change the spec manually
You can set them in modules and I think Koji can set them:
https://pagure.io/koji/issue/416
Not mentioning that there is almost always way to provide some macro file, e.g. there is no reason for python bootstrapping all the packages to not ship some macro in `/usr/lib/macros.d/macros.python-bootstrap`.
Vít
Le mardi 09 juin 2020 à 12:21 +0200, Vít Ondruch a écrit :
Dne 09. 06. 20 v 12:12 Nicolas Mailhot napsal(a):
Le mardi 09 juin 2020 à 12:08 +0200, Vít Ondruch a écrit :
Just FTR, we have bootstrapping guidelines
https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping
Those suffer from
- the horrible bcond logic inversion that trips pretty much
everyone all the time.
That won't be different for what was the original question here, i.e. conditionally disable tests. bconds are what we have for better or worse.
bconds had adoption problems from day one and will continue to have them as long as they are not fixed to use a human-friendly syntax
And really, this seems about bootstrapping not disabling tests, which are not completely different, but nobody can objects bootstrapping, while disabling tests might be good just to improve build speed and nothing else. That should never happen in production environment IMO.
That depends entirely on upstream’s test quality. FYI some upstream tests will attempt reconfiguring the system as root, or download random unchecked stuff drom the internet, or communicate with an internal server of the company that wrote the tets for example. There are many many shades or gray out there
You can set them in modules and I think Koji can set them: https://pagure.io/koji/issue/416
It would be nice if it has been fixed now
Not mentioning that there is almost always way to provide some macro file
That’s the kind of manual workaround that looks nice on paper for people who do not have to do it, and does not scale at all for people who actually have to do it for thousands of packages while rushing to meet release dealines
Regards,
On 09. 06. 20 12:21, Vít Ondruch wrote:
That won't be different for what was the original question here, i.e. conditionally disable tests. bconds are what we have for better or worse.
And really, this seems about bootstrapping not disabling tests, which are not completely different, but nobody can objects bootstrapping, while disabling tests might be good just to improve build speed and nothing else. That should never happen in production environment IMO.
FTR the discussion here is about packages that already have a bcond/macro to disable tests -- Tomáš proposed a common way of doing it. This discussion is not about adding new conditionals to packages that don't have them.
Whether or not disabling tests has legitimate use cases is out of scope here. It happens. We just want it to be more predictable when dealing with packaging in bulk.
As a metaphor (arguably not a very good one), imagine combustion motor vehicles. They pollute the environment. We are proposing to introduce colored emission stickers. You are discussing whether we should have such vehicles at all. While such discussion is certainly legitimate, it is out of scope. Sure, if we discard all gasoline- and diesel-powered cars and switch to electric or bicycles or perpetuum mobile, we don't have to put the energy into the emission stickers project. But how likely is that?
Dne 09. 06. 20 v 13:33 Miro Hrončok napsal(a):
On 09. 06. 20 12:21, Vít Ondruch wrote:
That won't be different for what was the original question here, i.e. conditionally disable tests. bconds are what we have for better or worse.
And really, this seems about bootstrapping not disabling tests, which are not completely different, but nobody can objects bootstrapping, while disabling tests might be good just to improve build speed and nothing else. That should never happen in production environment IMO.
FTR the discussion here is about packages that already have a bcond/macro to disable tests -- Tomáš proposed a common way of doing it. This discussion is not about adding new conditionals to packages that don't have them.
Whether or not disabling tests has legitimate use cases is out of scope here. It happens. We just want it to be more predictable when dealing with packaging in bulk.
As a metaphor (arguably not a very good one), imagine combustion motor vehicles. They pollute the environment. We are proposing to introduce colored emission stickers.
While we have already some other kind of stickers which could be reused.
The proposal was to optionally disable test. When somebody asked why, the answer was bootstrapping. But we know how to handle bootstrapping. So shouldn't somebody spend time changing the test conditionals to bootstrapping conditionals, because that seems to be the use case?
Or if you have different use case, then you probably want to explain it.
Vít
You are discussing whether we should have such vehicles at all. While such discussion is certainly legitimate, it is out of scope. Sure, if we discard all gasoline- and diesel-powered cars and switch to electric or bicycles or perpetuum mobile, we don't have to put the energy into the emission stickers project. But how likely is that?
Le mardi 09 juin 2020 à 14:35 +0200, Vít Ondruch a écrit :
The proposal was to optionally disable test. When somebody asked why, the answer was bootstrapping. But we know how to handle bootstrapping. So shouldn't somebody spend time changing the test conditionals to bootstrapping conditionals, because that seems to be the use case?
One use case is bootstrapping. Another is just getting things to build till you have the time to investigate if a new test failure is an actual problem or upstream being careless as usual. There are probably other use cases out there
Le mardi 09 juin 2020 à 14:56 +0200, Nicolas Mailhot a écrit :
Le mardi 09 juin 2020 à 14:35 +0200, Vít Ondruch a écrit :
The proposal was to optionally disable test. When somebody asked why, the answer was bootstrapping. But we know how to handle bootstrapping. So shouldn't somebody spend time changing the test conditionals to bootstrapping conditionals, because that seems to be the use case?
One use case is bootstrapping. Another is just getting things to build till you have the time to investigate if a new test failure is an actual problem or upstream being careless as usual. There are probably other use cases out there
Another fun case: someone broke the dep of a component used in unit tests. Fixing the component requires rebuilding the dep. Except, the dep uses the component itself in its own unit tests…
There are boundless possibilities for fun and profit there (well profit, not so sure actually)
On 6/9/20 3:01 PM, Nicolas Mailhot via devel wrote:
Le mardi 09 juin 2020 à 14:56 +0200, Nicolas Mailhot a écrit :
Le mardi 09 juin 2020 à 14:35 +0200, Vít Ondruch a écrit :
The proposal was to optionally disable test. When somebody asked why, the answer was bootstrapping. But we know how to handle bootstrapping. So shouldn't somebody spend time changing the test conditionals to bootstrapping conditionals, because that seems to be the use case?
One use case is bootstrapping. Another is just getting things to build till you have the time to investigate if a new test failure is an actual problem or upstream being careless as usual. There are probably other use cases out there
Another fun case: someone broke the dep of a component used in unit tests. Fixing the component requires rebuilding the dep. Except, the dep uses the component itself in its own unit tests…
There are boundless possibilities for fun and profit there (well profit, not so sure actually)
Another common one for me is rapid development in the spec file.
Overall, bootstrapping is definitely a common reason for disabling tests, but it's not the only one. Using bootstrapping conditional for non-bootstrapping purposes would be even more confusing than the status quo.
Therefore people will want to create macros that disable tests. I think we should follow the example of the bootstrapping macro, and recommend one macro that disables tests.
Tomas
Le vendredi 05 juin 2020 à 15:46 +0100, Richard W.M. Jones a écrit :
For the RISC-V bootstrap we used rpmbuild directly (before Koji and its dependencies had been ported), and added --nocheck. However once Koji was working we built packages properly with checks enabled.
How often do we bootstrap Fedora from scratch? Wholly new architectures are rare. Are there other events that require bootstrapping from scratch?
Some language ecosystems have very low quality unit tests, any mass rebuild (every release) basically works in two passes, make packages build with test disabled, then redo-it with tests and sift through test results to see what is actual breakage and what is broken testing code
The people who release poor unit tests also change their dep graph at high speed, poor unit tests go hand in hand with regular re-bootstraps
On 6/5/20 4:26 PM, Richard W.M. Jones wrote:
On Fri, Jun 05, 2020 at 04:10:20PM +0200, Tomas Orsava wrote:
Hi, I think it would be useful to have a standard way of disabling the running of tests during RPM build (in the %check section of a spec file).
I see a lot of packages already having %bcond's or other macro definitions to archieve this, but each package has their own way, there's no real standard. Thus you have to first look into the spec, locate the appropriate %bcond or macro name and only then you can disable the tests.
I would like to propose two approaches:
(a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to conditionalize the tests.
(b) Or, if that's too strong, mention in the guidelines the common methods that are being used (e.g. %bcond tests and %bcond check) so that new packagers have something to use.
What's the motivation for disabling tests globally?
For example bootstrapping.
And this doesn't only benefit us on a global level, it also lowers the cognitive load when you're working with a random package, for example doing a PR.
I have some packages where tests fail on particular architectures at particular times, and what I do there is (a) file a BZ (b) surround the %check section with %ifarch/%ifnarch and a comment linking to the bug, and this seems to me a practical and lightweight approach that requires no special support in the toolchain.
Also rpmbuild itself can happily disable tests, just add the --nocheck flag.
Indeed, but it's not supported by Koji, for example.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
On Fri, 2020-06-05 at 16:10 +0200, Tomas Orsava wrote:
Hi, I think it would be useful to have a standard way of disabling the running of tests during RPM build (in the %check section of a spec file).
I see a lot of packages already having %bcond's or other macro definitions to archieve this, but each package has their own way, there's no real standard. Thus you have to first look into the spec, locate the appropriate %bcond or macro name and only then you can disable the tests.
I would like to propose two approaches:
(a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to conditionalize the tests.
(b) Or, if that's too strong, mention in the guidelines the common methods that are being used (e.g. %bcond tests and %bcond check) so that new packagers have something to use.
What do you think?
I'd like to have this finally be implemented in https://github.com/rpm-software-management/rpm/issues/316. That way it would be simply rpmbuild --nocheck or define %_without_check 1 which would skip %check section entirely.
For now, all Rust crates just have `%bcond_without check` so using `-- without check` works just fine there.
Since this would be more generic thing to the RPM ecosystem, adding rpm-ecosystem@ to the copy.
Tomas _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
- -- Igor Raits ignatenkobrain@fedoraproject.org
packaging@lists.fedoraproject.org
-
Igor Raits -
Miro Hrončok -
Nicolas Mailhot -
Richard W.M. Jones -
Tomas Orsava -
Vít Ondruch