Hi, folks! So, I quickly bashed out that draft F21 Test Plan I've been threatening to write for the last month or so.
https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan
So, what's the idea here?
Really, I'm just trying to come up with a starting point for all the decisions we need to make and planning we need to do for Fedora 21 testing, especially release validation testing, especially as regards Fedora.next. There's some mention of other stuff in there, but what I'm really trying to think about is what we need to do to extend our test processes to cover all the stuff that's coming as a part of .next.
I think the most interesting points are these:
* There are several practical implications from the test plan - just Work We Need To Do. Most obviously, we need to draw up release criteria and supporting test cases for the Fedora.next Products. We also will need to adjust the https://fedoraproject.org/wiki/QA/SOP_Release_Validation_Test_Event SOP to include those cases/matrices as they're created, and adjust the https://fedoraproject.org/wiki/Release_criteria page to refer to/include the Product-specific release criteria as *they're* created.
* Responsibilities! Particularly, in this *draft* Test Plan, I've suggested that creating the Product-specific criteria and test cases should be the responsibility of the relevant Working Groups, with assistance from the QA team. They would also be jointly responsible, along with the QA team, for conducting Product-specific testing, on the general principle that the more Product-specific a bit of testing is (and the more domain-specific expertise it needs), the more it's the WG's responsibility and the less it's QA's responsibility. Note that (again, as a suggestion in this draft) the QA team is still responsible for Fedora 21 testing *overall* - i.e. it's the QA team's responsibility to make sure the WGs do the stuff assigned to them in the plan. If that makes sense. Discussion welcome!
* Feasibility. This is, of course, one I really want to nail down. It will require, though, that the Product-specific release criteria and test cases / matrices get written. Once we have those, we can try and make a realistic judgement as to whether we as a project - the QA team and the WGs - actually have the resources to complete the minimal necessary level of testing within the scope of the F21 schedule. If we don't think that's going to be the case, we can take that to FESCo, and we can decide what to do about it that way. In the meantime I've provided a *very* vague 'Required resources' section just to sort of flag this up in a modest way.
I'd really, really appreciate feedback on this draft from all interested parties - QA folks, WG folks, FESCo, anyone at all reading this who has an opinion. Please, bikeshed away. It all helps.
I'll aim to kick off discussion between the WGs and QA about creating Product-specific release criteria and test cases next week. Obviously that's *somewhat* subject to feedback on this very draft, but I can't really envisage a world in which we aren't going to need those two things to happen, and whoever's job we ultimately decide it is, I don't *think* that me just kicking off an attempt to draft them can be a bad thing.
Thanks folks!
On Fri, May 23, 2014 at 06:02:07PM -0700, Adam Williamson wrote:
Hi, folks! So, I quickly bashed out that draft F21 Test Plan I've been threatening to write for the last month or so. https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan So, what's the idea here?
Psshh -- so much for your claim of not doing any real work. This looks great! I'll put discussing it on our next Cloud WG meeting.
- There are several practical implications from the test plan - just
Work We Need To Do. Most obviously, we need to draw up release criteria and supporting test cases for the Fedora.next Products. We also will need to adjust the
Mike has some drafted up for cloud at https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Test_Plan#Test_Pass.2FFail_Criteria. It'll need a bit more by way of specifics.
- Responsibilities! Particularly, in this *draft* Test Plan, I've
suggested that creating the Product-specific criteria and test cases should be the responsibility of the relevant Working Groups, with assistance from the QA team. They would also be jointly responsible,
This seems like the right way to me, too.
for Fedora 21 testing *overall* - i.e. it's the QA team's responsibility to make sure the WGs do the stuff assigned to them in the plan. If that makes sense. Discussion welcome!
Yeah. If something isn't working, do you think the best path is QA->WG directly, or should it be QA says something to FESCo, FESCo works with WG?
Since parts of this are new to all of us, and other parts new to a lot of us, maybe it'd be good to put suggested/reasonable deadlines on some parts of the responsibilities? That helps put in focus what needs to be done.
On Tue, May 27, 2014 at 7:58 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, May 23, 2014 at 06:02:07PM -0700, Adam Williamson wrote:
for Fedora 21 testing *overall* - i.e. it's the QA team's responsibility to make sure the WGs do the stuff assigned to them in the plan. If that makes sense. Discussion welcome!
Yeah. If something isn't working, do you think the best path is QA->WG directly, or should it be QA says something to FESCo, FESCo works with WG?
If you want a collaborative effort, then the WGs and QA should be working directly together. Encouraging one party to tattle to FESCo on the other (either way) isn't setting things up for success. This will be hard at times. The two groups need to work through those issues mostly on their own or people are going to be disenfranchised that the WGs are actually in control of their products at all.
josh
On Tue, 2014-05-27 at 07:58 -0400, Matthew Miller wrote:
On Fri, May 23, 2014 at 06:02:07PM -0700, Adam Williamson wrote:
Hi, folks! So, I quickly bashed out that draft F21 Test Plan I've been threatening to write for the last month or so. https://fedoraproject.org/wiki/User:Adamwill/Draft_Fedora_21_test_plan So, what's the idea here?
Psshh -- so much for your claim of not doing any real work. This looks great! I'll put discussing it on our next Cloud WG meeting.
Awesome, thanks.
- There are several practical implications from the test plan - just
Work We Need To Do. Most obviously, we need to draw up release criteria and supporting test cases for the Fedora.next Products. We also will need to adjust the
Mike has some drafted up for cloud at https://fedoraproject.org/wiki/User:Roshi/QA/Cloud_Docs/Test_Plan#Test_Pass.2FFail_Criteria. It'll need a bit more by way of specifics.
hah, I missed those somehow. That's a good start. My mental model was actually going to be that we'd keep the 'generic' release criteria as something shared between all Products and then just have additive Product-specific release criteria pages (they could also amend the base criteria slightly if necessary, as long as the delta didn't get too crazy), but the approach of having multiple 'complete' sets of release criteria may work out better, we'll have to think it through. I'm planning to kick off some threads about this work this week.
- Responsibilities! Particularly, in this *draft* Test Plan, I've
suggested that creating the Product-specific criteria and test cases should be the responsibility of the relevant Working Groups, with assistance from the QA team. They would also be jointly responsible,
This seems like the right way to me, too.
Cool, glad everyone seems to be agreeing on that.
for Fedora 21 testing *overall* - i.e. it's the QA team's responsibility to make sure the WGs do the stuff assigned to them in the plan. If that makes sense. Discussion welcome!
Yeah. If something isn't working, do you think the best path is QA->WG directly, or should it be QA says something to FESCo, FESCo works with WG?
As with all the others who've weighed in so far, I agree that the optimal path is just for us all to talk to each other. I generally prefer workflows that assume everyone's happy to work positively together to a consensus solution over ones that assume everything will be oppositional and must therefore go through a higher authority. So, yep, IMO we should just talk to each other, and FESCo should only be invoked to resolve irreconcilable disputes (of which we hope there won't be any).
Since parts of this are new to all of us, and other parts new to a lot of us, maybe it'd be good to put suggested/reasonable deadlines on some parts of the responsibilities? That helps put in focus what needs to be done.
Indeed, I didn't put it in the plan but it would obviously be helpful.
We should really aim to have the process nailed down and at least the Alpha criteria and test cases (and any required automated testing stuff) ready by the date of Alpha TC1. Ideally we'd have Beta and Final criteria and test cases prepared at that point too, but we _could_ delay those until those actual milestones if we have to.
The actual release process is heavily tied into the release criteria, validation testing, and blocker bug stuff; if we don't have that in place it's very difficult to work through a milestone release in any meaningful way which isn't just "let's chuck something over the wall and hope for the best".
Ideally I'd like to work fairly rapidly and get, say, the initial release criteria ready by the end of next week, and then get the required test cases and matrices written between then and branching (which is 'no earlier than July 8' on current schedule).
On Tue, May 27, 2014 at 12:58:12PM -0700, Adam Williamson wrote:
As with all the others who've weighed in so far, I agree that the optimal path is just for us all to talk to each other. I generally prefer workflows that assume everyone's happy to work positively together to a consensus solution over ones that assume everything will be oppositional and must therefore go through a higher authority. So, yep, IMO we should just talk to each other, and FESCo should only be invoked to resolve irreconcilable disputes (of which we hope there won't be any).
Yeah, and especially with the way Josh put it, this approach seems obviously better. Let's just pretend I didn't suggest the other way. :)
Ideally I'd like to work fairly rapidly and get, say, the initial release criteria ready by the end of next week, and then get the required test cases and matrices written between then and branching (which is 'no earlier than July 8' on current schedule).
As someone who is not actually volunteering to do much more than cheerlead for this particular bit of writing, that schedule sounds good to _me_....
server@lists.fedoraproject.org
-
Adam Williamson -
Josh Boyer -
Matthew Miller