Hi,
an update of the Standard Test Interface based on the changes
agreed in [1] is now available for review:
https://pagure.io/fedora-ci/docs/pull-request/5
Review is appreciated because this change is introducing a
backward incompatible behaviour. Please, share your thoughts
and concerns directly in the pull request. Thanks.
psss...
[1]
https://pagure.io/fedora-ci/general/issue/18
On Thu, 7 Feb 2019 at 12:24, Petr Šplíchal <psplicha(a)redhat.com> wrote:
>
> Hi,
>
> here's an update for those who were not involved in the issue:
> After a bit longer discussion we've come to the following
> agreement:
>
> * Introduce a new "results.yml" file which should contain
> results of individual tests in a defined format
>
> * Ansible playbook should return non-zero status on
> infrastructure error and not on test failures
>
> This should allow the pipeline to clearly detect infrastructure
> errors and report them accordingly. See the ticket for more
> details and the list of proposed changes:
>
> *
https://pagure.io/fedora-ci/general/issue/18
>
> The solution is introducing new requirement for tests which are
> not using Standard Test Roles, so please review this and let
> us know if you have any concerns. Thanks.
>
> psss...
>
> On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal <psplicha(a)redhat.com> wrote:
> >
> > Hi!
> >
> > Currently the Standard Test Interface does not define how to
> > distinguish test failure from an infrastructure error. When
> > ansible playbook is run it usually does some environment
> > preparation and only then executes tests. Upon a problem
> > encountered during provisioning or configuration of the test
> > environment ansible returns non-zero exit code but there is not
> > an easy way how to detect such error from a successful test
> > execution of test which failed (returns non-zero exit code as
> > well).
> >
> > Let's make this part of the specification more clear so that
> > tools can detect infrastructure errors and report them as such
> > to the end user. This will help to direct problems (errors
> > versus failures) to the right audience (infrastructure staff
> > versus developer/tester).
> >
> > See the following ticket for proposed solution and share your
> > thoughts there:
> >
> >
https://pagure.io/fedora-ci/general/issue/18
> >
> > Thanks.
> >
> > psss...