On Tue, Jun 14, 2022 at 6:00 AM <bcotton(a)redhat.com> wrote:
>
> You are kindly invited to the meeting:
> Prioritized bugs and issues on 2022-06-15 from 10:00:00 to 11:00:00 America/Indiana/Indianapolis
> At fedora-meeting-1(a)irc.libera.chat
>
> More information available at: https://docs.fedoraproject.org/en-US/program_management/prioritized_bugs/
We will review the following nominated bugs:
* Connecting/disconnecting an external monitor or VM resizing switches
GNOME to the login screen
- https://bugzilla.redhat.com/show_bug.cgi?id=2071394
* Get rid of multiple source path definitions
- https://bugzilla.redhat.com/show_bug.cgi?id=2079833
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
No missing expected images.
Soft failed openQA tests: 1/8 (x86_64), 1/8 (aarch64)
(Tests completed, but using a workaround for a known bug)
Old soft failures (same test soft failed in Fedora-Cloud-35-20220628.0):
ID: 1309357 Test: x86_64 Cloud_Base-qcow2-qcow2 cloud_autocloud
URL: https://openqa.fedoraproject.org/tests/1309357
ID: 1309361 Test: aarch64 Cloud_Base-qcow2-qcow2 cloud_autocloud@uefi
URL: https://openqa.fedoraproject.org/tests/1309361
Passed openQA tests: 7/8 (x86_64), 7/8 (aarch64)
--
Mail generated by check-compose:
https://pagure.io/fedora-qa/check-compose
No missing expected images.
Soft failed openQA tests: 1/8 (x86_64), 1/8 (aarch64)
(Tests completed, but using a workaround for a known bug)
Old soft failures (same test soft failed in Fedora-Cloud-36-20220628.0):
ID: 1309280 Test: x86_64 Cloud_Base-qcow2-qcow2 cloud_autocloud
URL: https://openqa.fedoraproject.org/tests/1309280
ID: 1309284 Test: aarch64 Cloud_Base-qcow2-qcow2 cloud_autocloud@uefi
URL: https://openqa.fedoraproject.org/tests/1309284
Passed openQA tests: 7/8 (x86_64), 7/8 (aarch64)
--
Mail generated by check-compose:
https://pagure.io/fedora-qa/check-compose
This is a conversation which has started a few times but not in anything close to a formal way. Let's fix that so we at least have an answer of why we're doing what we're doing :)
A test case management system (TCMS) is pretty much what it sounds like - a system which manages at least test cases for an organization. In general, a TCMS is going to also manage test plans and many also keep track of results. Fedora doesn't have a traditional TCMS - we use the wiki to store our test cases and the test matrices which roughly align to test cases, test plans and results.
The basic question is: "should we be using a more traditional TCMS instead of the wiki?". There aren't a ton of alternatives available to us, so the comparison list is pretty short. As far as I know, our options are:
1. Start using Kiwi TCMS
2. Create, deploy and maintain our own custom TCMS
3. Keep using the wiki for our TCMS
If anyone is aware of other options which don't sacrifice the functionality we use for release validation, are open source and aren't abandoned upstream, please reply to this thread. Just because these 3 are the only options I could find doesn't mean that there aren't others.
I've written up a brief summary of what I see of the advantages and disadvantages to the three approaches to start the conversation. What are peoples' thoughts on our options going forward? Should we look at moving on from our existing wiki-based TCMS? If so, do either of the alternatives appear promising enough to move forward with?
Thanks in advance for everyone's time and thoughts,
Tim
================================================================================
1. Start using Kiwi TCMS
================================================================================
Advantages:
- Has an existing user base, we wouldn't have to create/maintain an entire TCMS alone
- Uses a more structured format which can be easier to analyze
- Allows multiple people to report results at the same time
Disadvantages:
- Has strict relationships between products, test cases and test plans which don't align well to how Fedora QA currently does things
- Is currently missing some vital features (FAS integration)
- Would require us to deploy and maintain an instance
- Would likely require some custom patches/plutins that we'd have to keep working with any changes in upstream KiwiTCMS
- Does not support testcases in multiple languages
KiwiTCMS (https://kiwitcms.org/) is an open source (GPL2) TCMS which started life inside Red Hat managing test cases for RHEL. As far as I know, it's the only open source TCMS out there which is still maintained and is even close to our workflow. I have set up a demo instance for people to poke at, details are in a separate email to this list.
Kiwi is written in Django so that wouldn't be a huge hurdle for our currently available development skills but it is currently missing at least one feature before we could use it in production (FAS authentication). That being said, Kiwi is rather opinionated on what the relationship between testcases, test plans, builds and results should look like. We would have to make some non-trivial changes to how we do things like release validation and that's a non-trivial cost.
================================================================================
2. Create, deploy and maintain our own custom TCMS
================================================================================
Advantages:
- Could be whatever we wanted it to be
- Would integrate well with existing Fedora systems
Disadvantages:
- Writing our own, custom TCMS would be expensive and take time
- We'd have another system to maintain
There's always the option of writing our own TCMS. While it'd be great to have something that uses ResultsDB out of the box and works perfectly with our workflows, that would come at a cost. The question comes down to whether we'd gain enough from a custom TCMS to justify the work we'd have to put into the system.
================================================================================
3. Continue Using WikiTCMS
================================================================================
Advantages:
- Doesn't require any changes to how we're doing things
- Is very flexible - can do pretty much whatever we want to without a ton of code
- Doesn't require much maintenance or custom code beyond wikitcms
Disadvantages:
- More difficult to analyze data contained in the wiki
- Only one person can report results in a given matrix at a time
- QA is the only group still using the Fedora wiki
This kinda started a while back when CPE started asking about whether we still needed the wiki or not. The current Fedora wiki is a source of frustration for them due to the amount of spam and bot issues that they have to deal with. Seeing as how QA is one of the only (if not the only) groups left using the wiki, it's worth asking whether we really still need it or not. It does sound like most of CPE's concerns could be satisfied with a new wiki instance which could be locked down to only allow pages with a certain format but I don't believe that conversation has even been started with them.
A lot of what the wiki has going for it is quite simply "if it ain't broke, don't fix it". Is the wiki an ideal TCMS? No. Does it serve our needs for the moment? Yes. There are so many things that we could be putting development time into, I'm not convinced that it's worth the time and effort needed to set up a new TCMS and change our processes to work with that new system just to say we're using a "proper" TCMS.
No missing expected images.
Soft failed openQA tests: 1/8 (x86_64), 1/8 (aarch64)
(Tests completed, but using a workaround for a known bug)
Old soft failures (same test soft failed in Fedora-Cloud-35-20220627.0):
ID: 1308279 Test: x86_64 Cloud_Base-qcow2-qcow2 cloud_autocloud
URL: https://openqa.fedoraproject.org/tests/1308279
ID: 1308283 Test: aarch64 Cloud_Base-qcow2-qcow2 cloud_autocloud@uefi
URL: https://openqa.fedoraproject.org/tests/1308283
Passed openQA tests: 7/8 (x86_64), 7/8 (aarch64)
--
Mail generated by check-compose:
https://pagure.io/fedora-qa/check-compose