On Thu, 2016-03-31 at 19:38 +0200, Giulio E. wrote:
Hi Adam.
I'd like to ask a question about this temporary part of release
validation testing.
Which one of the following scenario do you recommend?
Scenario 1) Sticking with Alpha 1.7, without performing updates. Fill
in the matrices with results coming from it.
Scenario 2) Running the test cases with an up to date F24 Alpha system.
Fill in the matrices with it.
Both of these have drawbacks:
Scenario 1) The wiki ends up with a mix of Alpha 1.7 and "updated
Alpha" results. So do the stats.
Scenario 2) If I remember right, Gnome 3.20 has been pushed stable
after the release of F24 Alpha. So, testing the version shipped with
Alpha 1.7, might not be so useful, when approaching Beta and Final test
cases.
In my opinion, the first scenario is not a big problem. Stats are
interesting, but a proper testing looks more important and nobody will
complain if we mess up some matrices. But, if we report an already
fixed problem, we waste time.
So... I'd rather test an up to date system.
What do you think?
So far as the matrices go, I personally prefer reporting results really
*for the compose in question* - it gets confusing if different people
report results for different updates to the same matrix pages, you have
'bob' saying GNOME works fine and 'anna' saying it's completely
broken
and it becomes a bit of a mess. But I'd say for non-install-related
tests it would be more valuable to test the up-to-date bits and just
don't report results to the wiki - just file bugs if you find any, and
nominate them as blockers if appropriate. We'll get a newer event soon
enough.
Always keep in mind that *ultimately* the goal of the testing is to
find bugs and fix them, not check boxes in wiki pages =) And you get
'Heroes of Fedora' credit for filing bugs as well as filling in wiki
boxes!
This is another reason it would be nice to have a more flexible test
result system than the wiki, of course. We'll get that off the todo
list one day!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net