On Thu, 7 Feb 2019 at 12:24, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote: This should be "introducing a new requirement for tests which are not using Standard Test Roles". Sorry for the confusion. > psss... > > On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote: > > > > Hi! > > > > Currently the Standard Test Interface does not define how to > > distinguish test failure from an infrastructure error. When > > ansible playbook is run it usually does some environment > > preparation and only then executes tests. Upon a problem > > encountered during provisioning or configuration of the test > > environment ansible returns non-zero exit code but there is not > > an easy way how to detect such error from a successful test > > execution of test which failed (returns non-zero exit code as > > well). > > > > Let's make this part of the specification more clear so that > > tools can detect infrastructure errors and report them as such > > to the end user. This will help to direct problems (errors > > versus failures) to the right audience (infrastructure staff > > versus developer/tester). > > > > See the following ticket for proposed solution and share your > > thoughts there: > > > > https://pagure.io/fedora-ci/general/issue/18 > > > > Thanks. > > > > psss...

Hi! The proposed changes have been implemented in the standard-test-roles and will soon land in updates: https://bodhi.fedoraproject.org/updates/FEDORA-MODULAR-2019-ae9c582836 https://bodhi.fedoraproject.org/updates/FEDORA-MODULAR-2019-c28ebce16b Let us know if you encounter any problems. Thanks. psss... On Thu, 28 Feb 2019 at 15:52, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote: > > Hi, > > an update of the Standard Test Interface based on the changes > agreed in [1] is now available for review: > > https://pagure.io/fedora-ci/docs/pull-request/5 > > Review is appreciated because this change is introducing a > backward incompatible behaviour. Please, share your thoughts > and concerns directly in the pull request. Thanks. > > psss... > > [1] https://pagure.io/fedora-ci/general/issue/18 > > On Thu, 7 Feb 2019 at 12:24, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote: > > > > Hi, > > > > here's an update for those who were not involved in the issue: > > After a bit longer discussion we've come to the following > > agreement: > > > > * Introduce a new "results.yml" file which should contain > > results of individual tests in a defined format > > > > * Ansible playbook should return non-zero status on > > infrastructure error and not on test failures > > > > This should allow the pipeline to clearly detect infrastructure > > errors and report them accordingly. See the ticket for more > > details and the list of proposed changes: > > > > * https://pagure.io/fedora-ci/general/issue/18 > > > > The solution is introducing new requirement for tests which are > > not using Standard Test Roles, so please review this and let > > us know if you have any concerns. Thanks. > > > > psss... > > > > On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote: > > > > > > Hi! > > > > > > Currently the Standard Test Interface does not define how to > > > distinguish test failure from an infrastructure error. When > > > ansible playbook is run it usually does some environment > > > preparation and only then executes tests. Upon a problem > > > encountered during provisioning or configuration of the test > > > environment ansible returns non-zero exit code but there is not > > > an easy way how to detect such error from a successful test > > > execution of test which failed (returns non-zero exit code as > > > well). > > > > > > Let's make this part of the specification more clear so that > > > tools can detect infrastructure errors and report them as such > > > to the end user. This will help to direct problems (errors > > > versus failures) to the right audience (infrastructure staff > > > versus developer/tester). > > > > > > See the following ticket for proposed solution and share your > > > thoughts there: > > > > > > https://pagure.io/fedora-ci/general/issue/18 > > > > > > Thanks. > > > > > > psss...