This is an initial take on stuff that was discussed in person during
Tim's
stay in Brno. Sending to the list for additional discussion/fine-tuning.
= What =
Talking rpmgrill-like checks, there will be a need to be able to facilitate
some kind of structure for representing that a check is composed of multiple
subchecks, for example:
check - FAILED
subcheck1 - PASSED
subcheck2 - PASSED
subcheck3 - FAILED
subcheck4 - PASSED
!IMPORTANT: ResultsDB will not be responsible for computing the result value
for an "upper level" Result from the subchecks - this is the check's
(check
developer's) responsibility.
I just realized this can go hand-in-hand with the new create_report directive that we
introduced lately. If the generated report contains all the reported results and links to
respective log files (if there are separate log files for individual subchecks, might or
might not be true), this way even if the user is notified about the subcheck3 result, he
can still click through and see the other check/subcheck results and their logs easily as
well.
This could (should?) be done on two levels:
* physicall nesting the Results as such in the database structure
* namespacing Testcases
For the start, we decided to go with the simplistic approach of nesting the
Testcases via a simple namespacing - thus allowing a frontend/query tool to
reconstruct the structure at least to some extent e.g. by relying on a
premise,
that Results that are a part of one Job can be converted to a tree-like
structure, based on the Testcase namespacing, at least to some extent, if
needed.
== Namespace structure ==
We'll be providing some top-level namespaces (list not yet final):
* app
This seems obvious at first, but the more I think about it, the more I'm not sure
whether it wouldn't be clearer to use something like qa.testdays instead of
app.testdays and team.cloud.tunir instead of app.tunir. Because at least it's obvious
who stands behind it and we don't need to decide which "app" is high-profile
enough to be put into "app" top namespace instead of the generic user/team
namespaces. But if we have a clear idea what goes here and what doesn't, I'm OK
with it.
* fedoraqa
I'd probably use just "qa". "fedora" seems superfluous. Or even
"team.qa", see below.
* package
Maybe just "pkg"?
* scratch (?)
I'd also add "fas"/"user" and "team". This way everyone
can use his/her own namespace (like user home on wiki) for experimenting or custom tasks,
and we also make team ownership clearer ("team.infra.task" instead of
"user.pingou.task"). With this, we might not even need "scratch",
because everyone can play is his/her own user namespace (but we might find some other
uses).
These will the further split to facilitate for a finer level of granularity,
e.g.:
app
testdays
powermanagement
pm-suspendr
fedoraqa
depcheck
rpmgrill
package
<pkgname>
unit
func
Everything below the top-level will be 100% user defined. We might have
recommendations for specific namespaces (like package.<pkgname>), but we
won't
be enforcing them.
By top-level you mean whole "pkg.<pkgname>" in this case, right? Because
it would be weird to allow people to arbitrarily create "pkg.foo" namespace if
there's no "foo" package in Fedora (and would introduce issues in the
future, if such package appeared). So "pkg.<pkgname>" would be the fixed
part, and anything below would be the flexible part, right? The same with
"user.<username>" and "team.<teamname>".
The structure will be implemented (at least in the initial implementation)
just
via the Testcase.name attribute in the DB, using dots as a separator. Later
on,
we can easily add an easy way of using wildcards for searching (e.g.
app.testdays.*.pm-suspendr)
!IMPORTANT: the namespaces are not to be used to represent "additional data"
about the underlying result such as architecture, item under test, etc.
This is what the Result's extra-data (ResultData) is there for.
NOTE: Although we do not encourage to store the results to the finest
granularity "just because" (e.g. individual results of a unittest testsuite),
we leave it to the check-developer's judgement. If there is a usecase for it,
let them do it, we don't care, as long as the DB is not extremely overloaded.
This is a very good point and in the documentation I'd like to stress out "please
don't create dozens of namespaces needlessly". We should explain when it makes
sense to use subcheck namespaces and when it does not. For example, if
"rpmgrill" executes many different checks and we consider some of them
release-critical and some of them not, which means we want to listen for them individually
over fedpkg (and package maintainers as well), it makes sense to have
qa.rpmgrill.rpm_sanity, qa.rpmgrill.rpmlint, qa.rpmgrill.security, etc. However, if
openssh test suite has 27 steps/test functions, it does not make sense to create
pkg.openssh.step1 through pkg.openssh.step27 namespaces, if we only care about the final
result, or if we process it in a single place only. If developers want the step separation
but they process it in a single place, it's much better to create a parseable
ResultYAML/JUnit file with a well-defined name as one of the artifacts (we can do that
automatically for every task), and they can download it from the output directory and
parse it. It's the same amount of work and it does not put 27 times heavier demand on
our database and fedmsg.
== Authentication/Authorization ==
We'll be continuing with the "expect no malice" approach we have right
now.
There will be just a simple limitation in libtaskotron:
check git clone
if cloned: only allow non-pkg namespace if __our__ repo
else: do whatever, don't care
in libtaskotron:
check the git checkout like listed above
have whitelisted napespace repos in config
!FIXME: the mechanism above is just copied from tflink's notes, I can't
remember the details :/
I'm not sure I understand this correctly, but I believe the idea was to give out
namespace write-access based on git source URL. So tasks from dist-git system would only
have write access to pkg.<pkgname>, where <pkgname> matches the package name
in their dist-git URL. Tasks from QA repos would have write-access to qa namespace. Etc.