On Wed, Sep 6, 2017 at 7:18 PM, Merlin Mathesius <mmathesi(a)redhat.com>
wrote:
For what it's worth, the standard-test-beakerlib and
standard-test-rhts
included in the standard-test-roles package [1] currently write test.log as
a brief summary of the results of individual tests. For example,
PASS test-1
PASS test-2
FAIL test-3
...
Beyond that, standard-test-beakerlib captures a test.*test-name*.log
artifact containing the stdout+stderr for each individual test. (eg.,
test.test-1.log, test.test-2.log, etc.), whereas standard-test-rhts
includes the entire job.*XX* tree (generated from restraint) as an
artifact.
[1]
https://pagure.io/standard-test-roles
Merlin
Thanks Merlin. That's exactly what I wanted to avoid, each author
understanding the spec differently.
I think there are two major kinds of artifacts - an stdout+stderr output
used for debugging problems, and (optionally, but frequently) a result
summary. I wouldn't probably try to standardize the result summary file
name, it's not strictly required and the file type can vary a lot -
results.html, summary.txt, results.tap, results/index.html, etc. But it
would be useful to have a standard name for the stdout+stderr output, so
that you always know where to look when debugging problems (test.log is as
good name for it as any).
So if we go this way, some of the roles in standard-test-roles should get
updated.