Hi friends,
I was a little concerned that we were running some tests during the Go/No-Go last week, especially considering it seemed like we had most of the blockers wrapped up well ahead of when we normally do. Part of the answer, in my opinion, is to lean on some of the SIGs/WGs to take a more active role in the testing process for RCs and the period leading up to RCs.
In particular, I'd like to see the Server WG taking care of the Active Directory tests (which is a non-trivial thing to remove our dependence on sgallagh's setup, admittedly) and the Cloud SIG taking care of many of those tests. The fact that we didn't have any AWS AMIs uploaded until after the meeting started suggests there's a gap in our process. And of course, pwhalen and coremodule could always use more support in running ARM tests.
This isn't a criticism of the QA team, because y'all do a tremendous job. And I don't want to suggest that _all_ of the testing be pushed out onto other teams. But I'd like to start a conversation on how we can spread the responsibility out more. This should hopefully make everyone's life a little easier and make our Go/No-Go meetings more efficient to boot.
Thoughts?
Hi Ben,
Thanks for starting the discussion.
I agree with the intent. This has been a topic of discussion within the Fedora QA team for a while now and we have made some efforts to engage SIGs early in the cycle.
As far as the question of when the testing should be a focus from respective SIGs? In my humble opinion, it should be continuous and should be a priority in their respective upstream. SIGs start testing during RC will act as a helping hand but it won't prevent late findings of issue.
Cheers, Sudhir
On Tue, Mar 29, 2022 at 1:28 AM Ben Cotton bcotton@redhat.com wrote:
Hi friends,
I was a little concerned that we were running some tests during the Go/No-Go last week, especially considering it seemed like we had most of the blockers wrapped up well ahead of when we normally do. Part of the answer, in my opinion, is to lean on some of the SIGs/WGs to take a more active role in the testing process for RCs and the period leading up to RCs.
In particular, I'd like to see the Server WG taking care of the Active Directory tests (which is a non-trivial thing to remove our dependence on sgallagh's setup, admittedly) and the Cloud SIG taking care of many of those tests. The fact that we didn't have any AWS AMIs uploaded until after the meeting started suggests there's a gap in our process. And of course, pwhalen and coremodule could always use more support in running ARM tests.
This isn't a criticism of the QA team, because y'all do a tremendous job. And I don't want to suggest that _all_ of the testing be pushed out onto other teams. But I'd like to start a conversation on how we can spread the responsibility out more. This should hopefully make everyone's life a little easier and make our Go/No-Go meetings more efficient to boot.
Thoughts?
-- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Tue, Mar 29, 2022 at 12:05 AM Sudhir Dharanendraiah sdharane@redhat.com wrote:
I agree with the intent. This has been a topic of discussion within the Fedora QA team for a while now and we have made some efforts to engage SIGs early in the cycle.
Oh, great! I must have missed those conversations. If there's anything I can do to help from a program management perspective, please let me know.
As far as the question of when the testing should be a focus from respective SIGs? In my humble opinion, it should be continuous and should be a priority in their respective upstream. SIGs start testing during RC will act as a helping hand but it won't prevent late findings of issue.
I totally agree in theory. In practice, a lot of contributors are volunteers. I'd love to get to the point where the testing is continuous, but getting them to test the RCs more would be an improvement over where we are now.
On Wed, Mar 30, 2022 at 10:12 AM Kamil Paral kparal@redhat.com wrote:
I'm not sure if those mentioned parties read this list and will respond :-) But yes, in general, it would be nice to see more people from various SIGs to get involved.
I assume everyone has all of the mailing lists beamed directly into their brain. :-D But yeah, I wanted to start the conversation within the QA team to figure out how we can rope them in before we engage with the SIGs. For example, do the RC announcements (e.g. [1]) go to the right places? I see some SIGs receive it, but Cloud doesn't seem to. And there's no clear call-to-action for the SIGs. That could be a low-effort way to potentially improve engagement from SIGs.
[1] https://lists.fedoraproject.org/archives/list/test-announce@lists.fedoraproj...
-- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis
On Mon, Mar 28, 2022 at 9:59 PM Ben Cotton bcotton@redhat.com wrote:
In particular, I'd like to see the Server WG taking care of the Active Directory tests (which is a non-trivial thing to remove our dependence on sgallagh's setup, admittedly) and the Cloud SIG taking care of many of those tests. The fact that we didn't have any AWS AMIs uploaded until after the meeting started suggests there's a gap in our process. And of course, pwhalen and coremodule could always use more support in running ARM tests.
I'm not sure if those mentioned parties read this list and will respond :-) But yes, in general, it would be nice to see more people from various SIGs to get involved.
On Mon, 2022-03-28 at 15:57 -0400, Ben Cotton wrote:
Hi friends,
I was a little concerned that we were running some tests during the Go/No-Go last week, especially considering it seemed like we had most of the blockers wrapped up well ahead of when we normally do. Part of the answer, in my opinion, is to lean on some of the SIGs/WGs to take a more active role in the testing process for RCs and the period leading up to RCs.
In particular, I'd like to see the Server WG taking care of the Active Directory tests (which is a non-trivial thing to remove our dependence on sgallagh's setup, admittedly) and the Cloud SIG taking care of many of those tests. The fact that we didn't have any AWS AMIs uploaded until after the meeting started suggests there's a gap in our process. And of course, pwhalen and coremodule could always use more support in running ARM tests.
This isn't a criticism of the QA team, because y'all do a tremendous job. And I don't want to suggest that _all_ of the testing be pushed out onto other teams. But I'd like to start a conversation on how we can spread the responsibility out more. This should hopefully make everyone's life a little easier and make our Go/No-Go meetings more efficient to boot.
Thoughts?
I think what we had there is really two specific issues, rather than a more general responsibility issue.
It's worth noting that sgallagh doing the AD tests is *already* a case of a SIG/WG doing its own testing, because sgallagh does that with a "Server WG" hat on, not a "QA" hat. The issue there is more that a) Stephen is the only one with a setup for doing the tests, and b) doing them is a pain and Stephen's busy so he often doesn't get around to it until I ping him directly.
The fix there is really more "let's figure out the legal/practical issues and automate the tests" than "let's have another discussion about QA/SIG relationships", I think.
And on the Cloud tests - the major issue there was that AMI production/upload got broken so there were no AMIs to test. I think the Cloud folks are generally aware that testing needs doing and involved in doing it, but that was the stumbling block this time. We could've noticed it a bit earlier - that's mostly my fault for not going through the matrices the day before go/no-go, I was distracted by fixing Rawhide. But again, that seems more like a specific problem than a general one. I don't know if fedimg or whatever has been fixed yet, if anyone has the ticket handy please link it...
I can configure the bot that creates the events to email lists other than test-announce easily enough. IIRC it doesn't because long ago when I wrote it, the feedback from other groups was that they can easily subscribe to test-announce to get the announcements and they didn't want the CC spam. But we can certainly revisit that.
I work with AD integration in "real" life and can do such tests, did not notice request in time though, but can contribute on that front. Automating the test is of course better and is not so dependent on a specific person being available.
Jan Kuparinen fas: copperi (I work with several SIGs, among others QA and Server) ________________________________ From: Adam Williamson adamwill@fedoraproject.org Sent: Tuesday, March 29, 2022 7:09 PM To: For testing and quality assurance of Fedora releases test@lists.fedoraproject.org Subject: Re: Spreading test responsibilities to other teams
On Mon, 2022-03-28 at 15:57 -0400, Ben Cotton wrote:
Hi friends,
I was a little concerned that we were running some tests during the Go/No-Go last week, especially considering it seemed like we had most of the blockers wrapped up well ahead of when we normally do. Part of the answer, in my opinion, is to lean on some of the SIGs/WGs to take a more active role in the testing process for RCs and the period leading up to RCs.
In particular, I'd like to see the Server WG taking care of the Active Directory tests (which is a non-trivial thing to remove our dependence on sgallagh's setup, admittedly) and the Cloud SIG taking care of many of those tests. The fact that we didn't have any AWS AMIs uploaded until after the meeting started suggests there's a gap in our process. And of course, pwhalen and coremodule could always use more support in running ARM tests.
This isn't a criticism of the QA team, because y'all do a tremendous job. And I don't want to suggest that _all_ of the testing be pushed out onto other teams. But I'd like to start a conversation on how we can spread the responsibility out more. This should hopefully make everyone's life a little easier and make our Go/No-Go meetings more efficient to boot.
Thoughts?
I think what we had there is really two specific issues, rather than a more general responsibility issue.
It's worth noting that sgallagh doing the AD tests is *already* a case of a SIG/WG doing its own testing, because sgallagh does that with a "Server WG" hat on, not a "QA" hat. The issue there is more that a) Stephen is the only one with a setup for doing the tests, and b) doing them is a pain and Stephen's busy so he often doesn't get around to it until I ping him directly.
The fix there is really more "let's figure out the legal/practical issues and automate the tests" than "let's have another discussion about QA/SIG relationships", I think.
And on the Cloud tests - the major issue there was that AMI production/upload got broken so there were no AMIs to test. I think the Cloud folks are generally aware that testing needs doing and involved in doing it, but that was the stumbling block this time. We could've noticed it a bit earlier - that's mostly my fault for not going through the matrices the day before go/no-go, I was distracted by fixing Rawhide. But again, that seems more like a specific problem than a general one. I don't know if fedimg or whatever has been fixed yet, if anyone has the ticket handy please link it...
I can configure the bot that creates the events to email lists other than test-announce easily enough. IIRC it doesn't because long ago when I wrote it, the feedback from other groups was that they can easily subscribe to test-announce to get the announcements and they didn't want the CC spam. But we can certainly revisit that. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
_______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure