On Wed, Sep 6, 2017 at 2:07 PM, Frederic Lepied <flepied@redhat.com> wrote:

On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral <kparal@redhat.com> wrote:

On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral <kparal@redhat.com> wrote:

Third, SI says:

The playbook and its test suite or test framework:
 MUST place the main readable output of the test suite into a test.log file in the artifacts variable folder. This MUST happen even if some of the test suites fail.

However, the default example doesn't contain any reference to test.log and uses output file instead. Shouldn't those examples above redirect to test.log instead?

It turns out I've been looking at old cached wiki content. After refreshing, I see that test.log is referenced instead.

 

Or is test.log not supposed to be an stdout+stderr output of the main test process? I don't exactly understand the definition from SI, what main readable output is supposed to be. When the test suite consists of X individual tests, are they supposed to concatenate their output to test.log?

These questions still stand, though.

 

You should see the test.log file as the main file to debug the run so it should contain the information to debug issues. So it's up to the test implementer to decide if it's stdout or stderr or both according to what he needs to debug. Does it make sense?

Fred

It does, thanks. Will you include this clarification in the spec? The main important information for me here is that the purpose of that file is to debug the execution, and most often it'll be stdout+stderr. Because "readable output", as it currently written, might be understood as human or machine readable, i.e. a json with detailed results, which is something completely different.