2:57 p.m.
Hi friends,
I was a little concerned that we were running some tests during the
Go/No-Go last week, especially considering it seemed like we had most
of the blockers wrapped up well ahead of when we normally do. Part of
the answer, in my opinion, is to lean on some of the SIGs/WGs to take
a more active role in the testing process for RCs and the period
leading up to RCs.
In particular, I'd like to see the Server WG taking care of the Active
Directory tests (which is a non-trivial thing to remove our dependence
on sgallagh's setup, admittedly) and the Cloud SIG taking care of many
of those tests. The fact that we didn't have any AWS AMIs uploaded
until after the meeting started suggests there's a gap in our process.
And of course, pwhalen and coremodule could always use more support in
running ARM tests.
This isn't a criticism of the QA team, because y'all do a tremendous
job. And I don't want to suggest that _all_ of the testing be pushed
out onto other teams. But I'd like to start a conversation on how we
can spread the responsibility out more. This should hopefully make
everyone's life a little easier and make our Go/No-Go meetings more
efficient to boot.
Thoughts?
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
11:03 p.m.
Hi Ben,
Thanks for starting the discussion.
I agree with the intent. This has been a topic of discussion within the
Fedora QA team for a while now and we have made some efforts to engage SIGs
early in the cycle.
As far as the question of when the testing should be a focus from
respective SIGs? In my humble opinion, it should be continuous and should
be a priority in their respective upstream. SIGs start testing during RC
will act as a helping hand but it won't prevent late findings of issue.
Cheers,
Sudhir
On Tue, Mar 29, 2022 at 1:28 AM Ben Cotton <bcotton(a)redhat.com> wrote:
Hi friends,
I was a little concerned that we were running some tests during the
Go/No-Go last week, especially considering it seemed like we had most
of the blockers wrapped up well ahead of when we normally do. Part of
the answer, in my opinion, is to lean on some of the SIGs/WGs to take
a more active role in the testing process for RCs and the period
leading up to RCs.
In particular, I'd like to see the Server WG taking care of the Active
Directory tests (which is a non-trivial thing to remove our dependence
on sgallagh's setup, admittedly) and the Cloud SIG taking care of many
of those tests. The fact that we didn't have any AWS AMIs uploaded
until after the meeting started suggests there's a gap in our process.
And of course, pwhalen and coremodule could always use more support in
running ARM tests.
This isn't a criticism of the QA team, because y'all do a tremendous
job. And I don't want to suggest that _all_ of the testing be pushed
out onto other teams. But I'd like to start a conversation on how we
can spread the responsibility out more. This should hopefully make
everyone's life a little easier and make our Go/No-Go meetings more
efficient to boot.
Thoughts?
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
_______________________________________________
test mailing list -- test(a)lists.fedoraproject.org
To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
--
S*UDHIR D*
SENIOR MANAGER, FEDORA QE,
9:10 a.m.
On Mon, Mar 28, 2022 at 9:59 PM Ben Cotton <bcotton(a)redhat.com> wrote:
In particular, I'd like to see the Server WG taking care of the
Active
Directory tests (which is a non-trivial thing to remove our dependence
on sgallagh's setup, admittedly) and the Cloud SIG taking care of many
of those tests. The fact that we didn't have any AWS AMIs uploaded
until after the meeting started suggests there's a gap in our process.
And of course, pwhalen and coremodule could always use more support in
running ARM tests.
I'm not sure if those mentioned parties read this list and will respond :-)
But yes, in general, it would be nice to see more people from various SIGs
to get involved.
10 a.m.
On Tue, Mar 29, 2022 at 12:05 AM Sudhir Dharanendraiah
<sdharane(a)redhat.com> wrote:
I agree with the intent. This has been a topic of discussion within the Fedora QA team
for a while now and we have made some efforts to engage SIGs early in the cycle.
Oh, great! I must have missed those conversations. If there's anything
I can do to help from a program management perspective, please let me
know.
As far as the question of when the testing should be a focus from
respective SIGs? In my humble opinion, it should be continuous and should be a priority in
their respective upstream. SIGs start testing during RC will act as a helping hand but it
won't prevent late findings of issue.
I totally agree in theory. In practice, a lot of contributors are
volunteers. I'd love to get to the point where the testing is
continuous, but getting them to test the RCs more would be an
improvement over where we are now.
On Wed, Mar 30, 2022 at 10:12 AM Kamil Paral <kparal(a)redhat.com> wrote:
I'm not sure if those mentioned parties read this list and will respond :-) But yes,
in general, it would be nice to see more people from various SIGs to get involved.
I assume everyone has all of the mailing lists beamed directly into
their brain. :-D But yeah, I wanted to start the conversation within
the QA team to figure out how we can rope them in before we engage
with the SIGs. For example, do the RC announcements (e.g. [1]) go to
the right places? I see some SIGs receive it, but Cloud doesn't seem
to. And there's no clear call-to-action for the SIGs. That could be a
low-effort way to potentially improve engagement from SIGs.
[1]
https://lists.fedoraproject.org/archives/list/test-announce@lists.fedorap...
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
5:24 p.m.
On Mon, 2022-03-28 at 15:57 -0400, Ben Cotton wrote:
Hi friends,
I was a little concerned that we were running some tests during the
Go/No-Go last week, especially considering it seemed like we had most
of the blockers wrapped up well ahead of when we normally do. Part of
the answer, in my opinion, is to lean on some of the SIGs/WGs to take
a more active role in the testing process for RCs and the period
leading up to RCs.
In particular, I'd like to see the Server WG taking care of the Active
Directory tests (which is a non-trivial thing to remove our dependence
on sgallagh's setup, admittedly) and the Cloud SIG taking care of many
of those tests. The fact that we didn't have any AWS AMIs uploaded
until after the meeting started suggests there's a gap in our process.
And of course, pwhalen and coremodule could always use more support in
running ARM tests.
This isn't a criticism of the QA team, because y'all do a tremendous
job. And I don't want to suggest that _all_ of the testing be pushed
out onto other teams. But I'd like to start a conversation on how we
can spread the responsibility out more. This should hopefully make
everyone's life a little easier and make our Go/No-Go meetings more
efficient to boot.
Thoughts?
I think what we had there is really two specific issues, rather than a
more general responsibility issue.
It's worth noting that sgallagh doing the AD tests is *already* a case
of a SIG/WG doing its own testing, because sgallagh does that with a
"Server WG" hat on, not a "QA" hat. The issue there is more that a)
Stephen is the only one with a setup for doing the tests, and b) doing
them is a pain and Stephen's busy so he often doesn't get around to it
until I ping him directly.
The fix there is really more "let's figure out the legal/practical
issues and automate the tests" than "let's have another discussion
about QA/SIG relationships", I think.
And on the Cloud tests - the major issue there was that AMI
production/upload got broken so there were no AMIs to test. I think the
Cloud folks are generally aware that testing needs doing and involved
in doing it, but that was the stumbling block this time. We could've
noticed it a bit earlier - that's mostly my fault for not going through
the matrices the day before go/no-go, I was distracted by fixing
Rawhide. But again, that seems more like a specific problem than a
general one. I don't know if fedimg or whatever has been fixed yet, if
anyone has the ticket handy please link it...
I can configure the bot that creates the events to email lists other
than test-announce easily enough. IIRC it doesn't because long ago when
I wrote it, the feedback from other groups was that they can easily
subscribe to test-announce to get the announcements and they didn't
want the CC spam. But we can certainly revisit that.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
10:38 a.m.
I work with AD integration in "real" life and can do such tests, did not notice
request in time though, but can contribute on that front. Automating the test is of
course better and is not so dependent on a specific person being available.
Jan Kuparinen
fas: copperi
(I work with several SIGs, among others QA and Server)
________________________________
From: Adam Williamson <adamwill(a)fedoraproject.org>
Sent: Tuesday, March 29, 2022 7:09 PM
To: For testing and quality assurance of Fedora releases
<test(a)lists.fedoraproject.org>
Subject: Re: Spreading test responsibilities to other teams
On Mon, 2022-03-28 at 15:57 -0400, Ben Cotton wrote:
Hi friends,
I was a little concerned that we were running some tests during the
Go/No-Go last week, especially considering it seemed like we had most
of the blockers wrapped up well ahead of when we normally do. Part of
the answer, in my opinion, is to lean on some of the SIGs/WGs to take
a more active role in the testing process for RCs and the period
leading up to RCs.
In particular, I'd like to see the Server WG taking care of the Active
Directory tests (which is a non-trivial thing to remove our dependence
on sgallagh's setup, admittedly) and the Cloud SIG taking care of many
of those tests. The fact that we didn't have any AWS AMIs uploaded
until after the meeting started suggests there's a gap in our process.
And of course, pwhalen and coremodule could always use more support in
running ARM tests.
This isn't a criticism of the QA team, because y'all do a tremendous
job. And I don't want to suggest that _all_ of the testing be pushed
out onto other teams. But I'd like to start a conversation on how we
can spread the responsibility out more. This should hopefully make
everyone's life a little easier and make our Go/No-Go meetings more
efficient to boot.
Thoughts?
I think what we had there is really two specific issues, rather than a
more general responsibility issue.
It's worth noting that sgallagh doing the AD tests is *already* a case
of a SIG/WG doing its own testing, because sgallagh does that with a
"Server WG" hat on, not a "QA" hat. The issue there is more that a)
Stephen is the only one with a setup for doing the tests, and b) doing
them is a pain and Stephen's busy so he often doesn't get around to it
until I ping him directly.
The fix there is really more "let's figure out the legal/practical
issues and automate the tests" than "let's have another discussion
about QA/SIG relationships", I think.
And on the Cloud tests - the major issue there was that AMI
production/upload got broken so there were no AMIs to test. I think the
Cloud folks are generally aware that testing needs doing and involved
in doing it, but that was the stumbling block this time. We could've
noticed it a bit earlier - that's mostly my fault for not going through
the matrices the day before go/no-go, I was distracted by fixing
Rawhide. But again, that seems more like a specific problem than a
general one. I don't know if fedimg or whatever has been fixed yet, if
anyone has the ticket handy please link it...
I can configure the bot that creates the events to email lists other
than test-announce easily enough. IIRC it doesn't because long ago when
I wrote it, the feedback from other groups was that they can easily
subscribe to test-announce to get the announcements and they didn't
want the CC spam. But we can certainly revisit that.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
_______________________________________________
test mailing list -- test(a)lists.fedoraproject.org
To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
757
days inactive
762
days old
5 comments
5 participants
participants (5)
-
Adam Williamson -
Ben Cotton -
Jan K -
Kamil Paral -
Sudhir Dharanendraiah