Multiple test suite playbooks
by Stef Walter
Lars from the standard-test-roles [0] project is using our test
specification for invoking tests. He brought up a good point (I paraphrase):
Many tests are destructive in that they change the state of the
VM or container that they are running in. However when using the
Ansible Dynamic Inventory means the entire tests.yml playbook is run
against the same VM or Docker image started from the test subject.
I would suggest our spec is amended in section "1.7.2 Invocation" to
have the first sentence replaced with:
"The testing system MUST run each playbook matching the glob
tests/tests*.yml in the dist-git repo. Each of these files constitutes a
test suite. Each test suite is invoked independently by the testing
system as follows."
What do you think?
Cheers,
Stef
6 years, 6 months
Allow multiple independent tests for each project
by Lars Karlitski
Hi,
The standard test interface defines `tests/tests.yml` as the sole
playbook to be run to test a repository. This makes it impossible to run
multiple independent tests on fresh instances of the same test subjects.
I noticed this while converting the tests for linux-system-roles to this
standard. We're running different ansible playbooks against virtual
machine images as test subjects and would like each playbook to start
from a fresh machine. We've already had a bug where one of the tests
didn't clean up correctly and influenced the result of the next test.
Thus, I suggest changing the "Invocation" section to say that the
testing system must independently run all playbooks `tests/tests*.yml`
in the dist-git repo. This is compatible with existing repositories.
If we decide to go this way, we'll have to decide how to handle
artifacts in this case. I see two reasonable choices:
1. Keep one artifacts directory and let the authors of the playbooks
take care that they don't overwrite each other's result. We couldn't run
the tests in parallel in this case, because tests might want to
concatenate a main`test.log`.
2. Have a separate artifacts directory for each test run.
What do you think?
Lars
6 years, 7 months
standard interface or standard test interface
by Kamil Paral
I wonder how set you are on the "standard interface" name. I find it too
generic, and confusing for most people not fully involved with it.
Interface for *what*?
When we talk about it in our team, we mostly call it "standard test
interface", because then it's clearer what it means, and it even has a nice
STI abbreviation when writing (while SI is too short and ambiguous).
Thoughts on calling it "standard test interface" instead? If you think this
is too much of a bikeshedding, I apologize, ignore me.
Kamil
6 years, 7 months
SI - multiple subjects concatenation
by Kamil Paral
SI says [1]:
Often only one *test subject* is passed in. However multiple subjects may
> be concatenated together in a shell escaped string. The playbook and
> inventory script split the string.
>
It does not say what the separator character is. It might be a space, it
might be any whitespace, it might be a comma, it might be a semicolon, etc.
This should be clarified.
I'm also confused what "shell escaped" means here. It's not like we'll pipe
this to bash, we'll process this with ansible. And even if we did, you
still want the raw value, not the escaped one, as the variable, don't you?
E.g. for "foo" and "bar" subjects, you want this:
$ export TEST_SUBJECTS='foo bar'
$ echo $TEST_SUBJECTS
foo bar
and not this:
$ export TEST_SUBJECTS='foo\ bar'
$ echo $TEST_SUBJECTS
foo\ bar
Correct?
But that brings another question - what if I have the separator character
inside the subject name? You might say that's not likely with current
fedora artfifacts, but a standard interface definition should be
forward-looking. Even now it is technically possible to have a space inside
an .iso/.qcow2 image name. It might be even more common for some future
test subject type. The spec should describe this as well.
Personally, I think we should get away from string parsing, if we want to
design a reliable system. String parsing and especially shell escaping is a
never ending story of unexpected failures and corner case issues. It would
be better to use a proper structure for this.
As it turns out, ansible can do this already [2]. You can provide a true
list to it via --extra-vars, by passing in a json, like this:
$ ansible-playbook tests.yml --extra-vars '{"subjects": ["foo","bar","even
a space"]}'
This way you don't need to do string parsing in an ansible playbook, you
can immediately use its native structures, like
module: do something for {{item}}
with_items: "{{subjects}}"
You also don't need to deal with character escaping (especially shell one),
all of that is handled by json. This makes writing tests.yml a lot easier,
I think.
There's of course a question what to do with TEST_SUBJECTS envvar if we
make "subjects" a native ansible variable. I'd personally use json for it
as well, so:
$ export TEST_SUBJECTS='["foo", "bar", "even a space"]'
There's a json parser for every possible language, and there are convenient
tools for bash as well [3].
But maybe I'd like to first understand why we require the envvars to be
present in the first place. I don't think they're needed. Is it just a
convenience measure?
Thanks,
Kamil
[1] https://fedoraproject.org/wiki/Changes/InvokingTests#Invocation
[2]
http://docs.ansible.com/ansible/latest/playbooks_variables.html#passing-v...
[3] https://stedolan.github.io/jq/
6 years, 7 months
a few questions regarding the example playbook for running tests
by Kamil Paral
Hello,
I'd like to get a few clarifications regarding the default test playbook
example here:
https://fedoraproject.org/wiki/CI/Tests#Wrapping_a_script_test
First, the example seems to suggest that all binaries should be placed in
/usr/local/bin:
- name: Create the folder where we will store the tests
action: file state=directory path={{ item }} owner=root group=root
with_items:
- /usr/local/bin
- name: Install the test files
copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755
with_items:
- {file: test-simple, dest: test-simple }
But I don't see such a requirement in the SI. Why shouldn't I run the test
executable from its current location (i.e. the git checkout directory)? And
if they can, why make the example unnecessarily complex?
Second, I'm a bit confused by the exec command in the test execution line:
- name: Execute the tests
shell: exec > {{ artifacts }}/output && /usr/local/bin/test-simple 2>&1
I've never seen this usage so I had to study what this does exactly in the
bash man page. I believe it's the same thing as this:
shell: /usr/local/bin/test-simple >{{ artifacts }}/output 2>&1
So why exactly don't we use this more readable and more often used syntax
to redirect a process output? An example should be as simple as possible.
Third, SI says:
> The playbook and its test suite or test framework:
> MUST place the main readable output of the test suite into a test.log
> file in the artifacts variable folder. This MUST happen even if some of the
> test suites fail.
>
However, the default example doesn't contain any reference to *test.log*
and uses *output* file instead. Shouldn't those examples above redirect to
*test.log* instead? Or is *test.log* not supposed to be an stdout+stderr
output of the main test process? I don't exactly understand the definition
from SI, what *main readable output* is supposed to be. When the test suite
consists of X individual tests, are they supposed to concatenate their
output to *test.log*?
It would be great to have these clarified.
Thanks,
Kamil
6 years, 7 months
SI - clarification about exit codes
by Kamil Paral
Hello,
the testing system is required to inspect the exit code of the test suite
[1]:
To invoke the playbook, the *testing system*:
> 6. MUST examine the exit code of the playbook. A zero exit code is
> successful *test result*, non-zero is failure.
>
However, the test suite is not required to return the correct exit code.
Under
> The playbook and its *test suite* or *test framework*:
>
there is no mention of returning the exit code. I think that should be
clarified, because it can be expected that people implementing SI will only
read those parts relevant for them. I believe this should be added:
> The playbook and its *test suite* or *test framework*:
>
> 6. MUST return a zero exit code of the playbook if the test result is a
> pass, or a non-zero exit code if the test result is a fail.
>
Thoughts?
[1] https://fedoraproject.org/wiki/Changes/InvokingTests
6 years, 7 months
