This is a conversation which has started a few times but not in anything close to a formal way. Let's fix that so we at least have an answer of why we're doing what we're doing :)
A test case management system (TCMS) is pretty much what it sounds like - a system which manages at least test cases for an organization. In general, a TCMS is going to also manage test plans and many also keep track of results. Fedora doesn't have a traditional TCMS - we use the wiki to store our test cases and the test matrices which roughly align to test cases, test plans and results.
The basic question is: "should we be using a more traditional TCMS instead of the wiki?". There aren't a ton of alternatives available to us, so the comparison list is pretty short. As far as I know, our options are:
1. Start using Kiwi TCMS 2. Create, deploy and maintain our own custom TCMS 3. Keep using the wiki for our TCMS
If anyone is aware of other options which don't sacrifice the functionality we use for release validation, are open source and aren't abandoned upstream, please reply to this thread. Just because these 3 are the only options I could find doesn't mean that there aren't others.
I've written up a brief summary of what I see of the advantages and disadvantages to the three approaches to start the conversation. What are peoples' thoughts on our options going forward? Should we look at moving on from our existing wiki-based TCMS? If so, do either of the alternatives appear promising enough to move forward with?
Thanks in advance for everyone's time and thoughts,
Tim
================================================================================ 1. Start using Kiwi TCMS ================================================================================
Advantages: - Has an existing user base, we wouldn't have to create/maintain an entire TCMS alone - Uses a more structured format which can be easier to analyze - Allows multiple people to report results at the same time
Disadvantages: - Has strict relationships between products, test cases and test plans which don't align well to how Fedora QA currently does things - Is currently missing some vital features (FAS integration) - Would require us to deploy and maintain an instance - Would likely require some custom patches/plutins that we'd have to keep working with any changes in upstream KiwiTCMS - Does not support testcases in multiple languages
KiwiTCMS (https://kiwitcms.org/) is an open source (GPL2) TCMS which started life inside Red Hat managing test cases for RHEL. As far as I know, it's the only open source TCMS out there which is still maintained and is even close to our workflow. I have set up a demo instance for people to poke at, details are in a separate email to this list.
Kiwi is written in Django so that wouldn't be a huge hurdle for our currently available development skills but it is currently missing at least one feature before we could use it in production (FAS authentication). That being said, Kiwi is rather opinionated on what the relationship between testcases, test plans, builds and results should look like. We would have to make some non-trivial changes to how we do things like release validation and that's a non-trivial cost.
================================================================================ 2. Create, deploy and maintain our own custom TCMS ================================================================================
Advantages: - Could be whatever we wanted it to be - Would integrate well with existing Fedora systems
Disadvantages: - Writing our own, custom TCMS would be expensive and take time - We'd have another system to maintain
There's always the option of writing our own TCMS. While it'd be great to have something that uses ResultsDB out of the box and works perfectly with our workflows, that would come at a cost. The question comes down to whether we'd gain enough from a custom TCMS to justify the work we'd have to put into the system.
================================================================================ 3. Continue Using WikiTCMS ================================================================================
Advantages: - Doesn't require any changes to how we're doing things - Is very flexible - can do pretty much whatever we want to without a ton of code - Doesn't require much maintenance or custom code beyond wikitcms
Disadvantages: - More difficult to analyze data contained in the wiki - Only one person can report results in a given matrix at a time - QA is the only group still using the Fedora wiki
This kinda started a while back when CPE started asking about whether we still needed the wiki or not. The current Fedora wiki is a source of frustration for them due to the amount of spam and bot issues that they have to deal with. Seeing as how QA is one of the only (if not the only) groups left using the wiki, it's worth asking whether we really still need it or not. It does sound like most of CPE's concerns could be satisfied with a new wiki instance which could be locked down to only allow pages with a certain format but I don't believe that conversation has even been started with them.
A lot of what the wiki has going for it is quite simply "if it ain't broke, don't fix it". Is the wiki an ideal TCMS? No. Does it serve our needs for the moment? Yes. There are so many things that we could be putting development time into, I'm not convinced that it's worth the time and effort needed to set up a new TCMS and change our processes to work with that new system just to say we're using a "proper" TCMS.
On Wed, Jun 22, 2022 at 08:57:04AM -0600, Tim Flink wrote:
================================================================================ 3. Continue Using WikiTCMS ================================================================================
Advantages:
- Doesn't require any changes to how we're doing things
- Is very flexible - can do pretty much whatever we want to without a ton of code
- Doesn't require much maintenance or custom code beyond wikitcms
Disadvantages:
- More difficult to analyze data contained in the wiki
- Only one person can report results in a given matrix at a time
- QA is the only group still using the Fedora wiki
From a user perspective, I discovered the QA test matrix pages long before I realized that they weren't actually meant to be used as wiki pages, and that they were actually managed by a tool. It's easy to mess things up. And the tool is somewhat klunky if you're not used to it.
On 6/22/22 09:55, Matthew Miller wrote:
On Wed, Jun 22, 2022 at 08:57:04AM -0600, Tim Flink wrote:
================================================================================ 3. Continue Using WikiTCMS ================================================================================
Advantages:
- Doesn't require any changes to how we're doing things
- Is very flexible - can do pretty much whatever we want to without a ton of code
- Doesn't require much maintenance or custom code beyond wikitcms
Disadvantages:
- More difficult to analyze data contained in the wiki
- Only one person can report results in a given matrix at a time
- QA is the only group still using the Fedora wiki
From a user perspective, I discovered the QA test matrix pages long before I realized that they weren't actually meant to be used as wiki pages, and that they were actually managed by a tool. It's easy to mess things up. And the tool is somewhat klunky if you're not used to it.
That's a good point. I guess I've been using the wiki matrices for so long that I don't think about it much anymore.
Do we have any reasonable way of estimating how many testers/results we lose because of the current interface? I suspect that anything we end up using is going to have a learning curve for new people but in this case, I'm not really sure which is the best/worst.
Tim
On Wed, 2022-06-22 at 11:55 -0400, Matthew Miller wrote:
On Wed, Jun 22, 2022 at 08:57:04AM -0600, Tim Flink wrote:
================================================================================ 3. Continue Using WikiTCMS ================================================================================
Advantages:
- Doesn't require any changes to how we're doing things
- Is very flexible - can do pretty much whatever we want to without a ton of code
- Doesn't require much maintenance or custom code beyond wikitcms
Disadvantages:
- More difficult to analyze data contained in the wiki
- Only one person can report results in a given matrix at a time
- QA is the only group still using the Fedora wiki
From a user perspective, I discovered the QA test matrix pages long before I realized that they weren't actually meant to be used as wiki pages, and that they were actually managed by a tool. It's easy to mess things up. And the tool is somewhat klunky if you're not used to it.
Well, hand editing them is still Officially Supported(tm). You're just supposed to *not* screw it up. If you do screw it up, one of two things will happen. If one of the tools that uses wikitcms noticeably chokes on it, I'll probably notice at some point, grumble a bit, and fix it. If I'm feeling super enthusiastic I might fix wikitcms to handle whatever you did wrong (it handles quite a lot of common human errors/inconsistencies). If nothing that uses wikitcms chokes *obviously* on it, we might never notice and it'll just be there forever!
I try to steer people to using `relval report-results` because it should be *somewhat* easier than hand-editing the pages, and it should not ever write things in a way wikitcms doesn't parse properly. But you are "allowed" to edit the pages by hand, too. (I actually do this myself quite a lot, still). Just please don't screw it up. :P
There are ways to improve on this that *don't* involve switching tools, of course. One option would be to enhance the testdays webapp quite a bit and use it for this too. We could also make `relval report-results` better, write a more graphical/web-y version of it, or something. I have never put *that* much work into it, honestly. It's not that far evolved beyond "just enough to do the job for the guy who wrote it". If someone felt inclined, you could improve its interface quite a bit.
If we got either of those approaches far enough along, we could consider locking out direct editing of the wiki pages somehow, if that's plausible in mediawiki, and only allowing tools/bots to edit them.
On Wed, 2022-06-22 at 08:57 -0600, Tim Flink wrote:
================================================================================ 3. Continue Using WikiTCMS ================================================================================
Advantages:
- Doesn't require any changes to how we're doing things
- Is very flexible - can do pretty much whatever we want to without a ton of code
- Doesn't require much maintenance or custom code beyond wikitcms
Another advantage of the current system that I'd like to highlight is that it allows us to synthesize manual and automated test results. That is, in the current wiki results pages, you see the results from openQA and human testers together.
I would definitely want any replacement for it to preserve this advantage - i.e., I would want it to be able to track results from both human testers and automated systems. Ideally, this would be achieved by resultsdb integration - it's a bit difficult to imagine technically how this could work, but it would be nice for results from both human testers and automated systems to be in resultsdb, and for any human- readable representation of the validation state to pull both sets of results from resultsdb.
wikitcms gives us the integration of the two types of result, but it does not currently forward results from human testers to resultsdb. So we only have the results from automated systems in resultsdb. openQA reports its results to *both* resultsdb and the wiki, so the wiki is the only canonical store of *both* human and bot results.
I've toyed in the past with the idea of having `relval report-results` also submit the result to resultsdb, but it requires solving authentication and we still wouldn't capture human test results submitted by editing the wiki directly. Another option would be a wikitcms-based bot which monitored result pages for edits and forwarded any 'new results' to resultsdb...
Disadvantages:
- More difficult to analyze data contained in the wiki
- Only one person can report results in a given matrix at a time
- QA is the only group still using the Fedora wiki
This kinda started a while back when CPE started asking about whether we still needed the wiki or not. The current Fedora wiki is a source of frustration for them due to the amount of spam and bot issues that they have to deal with. Seeing as how QA is one of the only (if not the only) groups left using the wiki, it's worth asking whether we really still need it or not. It does sound like most of CPE's concerns could be satisfied with a new wiki instance which could be locked down to only allow pages with a certain format but I don't believe that conversation has even been started with them.
If they do get to the point where they really, really want to turn the wiki off, I'm willing to try and handle this (setting up a reduced wiki instance specifically for this purpose, with restrictions to allowable page names and so on).
On 6/22/22 14:13, Adam Williamson wrote:
On Wed, 2022-06-22 at 08:57 -0600, Tim Flink wrote:
================================================================================ 3. Continue Using WikiTCMS ================================================================================
Advantages:
- Doesn't require any changes to how we're doing things
- Is very flexible - can do pretty much whatever we want to without a ton of code
- Doesn't require much maintenance or custom code beyond wikitcms
Another advantage of the current system that I'd like to highlight is that it allows us to synthesize manual and automated test results. That is, in the current wiki results pages, you see the results from openQA and human testers together.
I would definitely want any replacement for it to preserve this advantage - i.e., I would want it to be able to track results from both human testers and automated systems. Ideally, this would be achieved by resultsdb integration - it's a bit difficult to imagine technically how this could work, but it would be nice for results from both human testers and automated systems to be in resultsdb, and for any human- readable representation of the validation state to pull both sets of results from resultsdb.
For what it's worth, Kiwi does have a pretty comprehensive API when it comes to stuff like reading or posting test results.
When it comes to storing results for automated vs. manual testing, Kiwi does seem to focus more on having test cases which are marked as automated rather than having users marked as bots. As far as I know, there is no way to mark a Kiwi user as a bot.
Personally, I think there are arguments for both approaches but changing over to Kiwi would likely require a change to how we've been doing things.
wikitcms gives us the integration of the two types of result, but it does not currently forward results from human testers to resultsdb. So we only have the results from automated systems in resultsdb. openQA reports its results to *both* resultsdb and the wiki, so the wiki is the only canonical store of *both* human and bot results.
I've toyed in the past with the idea of having `relval report-results` also submit the result to resultsdb, but it requires solving authentication and we still wouldn't capture human test results submitted by editing the wiki directly. Another option would be a wikitcms-based bot which monitored result pages for edits and forwarded any 'new results' to resultsdb...
At some point, we'd end up with enough chewing gum and bailing wire that we might as well roll our own. I'm not sure we're there yet but if we start down the path of an alternate interface to post results and/or a bot to shuffle results between the wiki and resultsdb, I do think that would start approaching the point where we'd be better off writing our own solution.
Disadvantages:
- More difficult to analyze data contained in the wiki
- Only one person can report results in a given matrix at a time
- QA is the only group still using the Fedora wiki
This kinda started a while back when CPE started asking about whether we still needed the wiki or not. The current Fedora wiki is a source of frustration for them due to the amount of spam and bot issues that they have to deal with. Seeing as how QA is one of the only (if not the only) groups left using the wiki, it's worth asking whether we really still need it or not. It does sound like most of CPE's concerns could be satisfied with a new wiki instance which could be locked down to only allow pages with a certain format but I don't believe that conversation has even been started with them.
If they do get to the point where they really, really want to turn the wiki off, I'm willing to try and handle this (setting up a reduced wiki instance specifically for this purpose, with restrictions to allowable page names and so on).
I have played with https://kiwidemo.fedorainfracloud.org for a few days and I have to say I'm quite disappointed by it :-( I thought it would be closer to what we need, with some tweaking. But currently I'm hitting serious obstacles and the user experience for a regular tester is not great either. It seems that Kiwi is designed in a very rigid way where each person gets some area of responsibility and they work just there, or there's a smaller team of educated testers which can easily collaborate on a shared area. Other approaches feel cumbersome.
First, you need to create a test run, in order to test something. When creating the test run, you have to define which test cases will be included. You can populate it from a test plan, which we can prepare, but either we'll have many many smaller test plans with a couple of test cases, or a couple of large test plans with tens of test cases. It feels either constraining or overwhelming to work this way. You can't just simply pick one random test case which hasn't been tested yet, like with our wiki matrices. Moreover, searching for those test plans is tedious, always filling out Product, Version, etc. You're provided with a database list of test plans, without any logical layout, milestone priorities etc, as on our wiki. When creating a test run, you have to fill out some fields, like Build. Perhaps we can provide pre-created hyperlinks which would forward testers into a fully filled out page and they'd just click a button. After you create a test run, you have to start it, and after you're done, you have to end it. You don't actually need to, because you can populate results even if it hasn't started/has already ended, but certain views will display in-progress test runs and some won't (unless you display only in-progress test runs). This feels like bookkeeping feature more than anything else, and it's a nuisance for us. People will forget to end their test runs.
Instead of everyone creating their own test run (actually many test runs), it seems possible to create one single "shared" test run and distribute the link to it. Other people can fill out the results, even if they haven't created it. However, there are serious disadvantages: 1. Only a single result can be provided for a test case. So no longer we'd be able to see results from multiple people, or openqa and an additional manual tester, for one particular test case. As described above, this feels very "corporate style of work", but very different from what we need. 2. It is extremely easy to overwrite somebody else's result by accident. The test run page doesn't update automatically, you need to manually refresh it. So if you simply submit a result for a test case which looks untested, somebody else might have submitted the result already (a few hours earlier, possibly) and your result will simply overwrite it. There's no warning, nothing. On wiki, we at least get a warning if people edit the same page/section simultaneously and there's an edit collision. This is a complete no-go for us. I even wonder how corporate teams can avoid this situation if multiple people work on the same test run, unless they're sitting in the same office space talking to each other.
Because sharing test runs is a no-go, individual test runs is the way to go. But there's another big hurdle. How do you figure out what should you test, i.e. it hasn't been tested yet by somebody else? I only found a single solution to this, which is by looking into the "Status matrix". Unfortunately that page is probably the worst one I encountered in Kiwi. There's no dedicated Search button, it performs a search on any field change (and initially shows the whole database, if you wait long enough - ooof). There's no progress indication, so you never know if a search was initiated, if it is complete, if empty results page is actually what you should see or not. The search params are not placed into URL, so every time you want to refresh the page to see updated results, you have to fill out all search fields again. The page readability is decent if only a low number of test runs is displayed. But if people create multiple test runs (for each smaller test plan/test section, because of submitting results before lunch/after lunch, etc), the readability gets much worse and it's an exercise in horizontal scrolling. Our testcase_stats (e.g. [1]) do much better job in clearly showing what was run and what wasn't, what was the last tested build and where were some issues. Especially if you want to show the history, not just the current build. And of course the page again provides just a database list of test cases, but doesn't structure them in any way, so we can't e.g. display milestone priorities etc in that list. In essence, the more test runs are performed, the harder it is to figure out what wasn't run yet, any test case structuring is not possible, and you can't refresh the page to see the current state.
The Product/Version/Build organization is usable, I think, but currently conflicts with our way of using "environments" (different arches, bios vs uefi, etc). Tim tried to solve it by embedding the environments into Product, and also by populating the image names in there, but I'm not sure it can work this way. For example, Server tests which are run just once in general can be in the "Server" product. But if we want to distinguish between netinst and dvd, on either uefi or bios, on three different architectures, that would just explode the Product values. It makes managing the test plans and seeing the current test runs results a nightmare.
Kiwi supports some test case properties and defining applicable environments as an experimental feature. It is not documented in their docs, but by some trial and error I managed to define "firmware=bios, firmware=uefi" on a test case, and then it got split up into two different test case executions (one with bios, one with uefi) when a test run is created. I find it a bit weird that those properties are defined in a test case rather than in a test plan, but it would work for us, I think. Unfortunately currently even when you can submit the results for these "environment-defined test cases", they are not displayed in the Status matrix. When a test case is split into two, it shows up as a single result there, with whatever result was submitted for the first half of it. So while this could solve some of our needs, currently it doesn't seem finished (unsurprisingly, since it's marked experimental).
So, overall, the user experience of submitting individual test case results is, I think, good, but everything around it (determining which test cases haven't been tested yet, getting some guidance what to focus on, creating and managing your test run, navigating the system, getting an overall picture of testing) is worse than with our wiki, sometimes considerably. (Of course I could have misunderstood many Kiwi's features, I never worked with it before. I'll be happy to be corrected). I think that we could work around the issues by creating: a) a startup page, where people would see our test cases logically separated (installation/desktop, Basic/Beta/Final, storage/package management/etc), and just click pre-created hyperlinks leading to Kiwi forms with all necessary input fields filled out already (assuming Kiwi supports that). This would save people from searching in endless drop-down menus, possibly selecting wrong things. It would also guide them on what to do ("this section is more important than that one because it's Beta and the other is Final"). b) an overall results page of the compose. This would help people figure out what test cases need attention (not tested yet, marked as failed, etc) and what the overall status of the compose is. This can be combined with a), of course. c) an updated testcase_stats page to see the historical progress Pages b) and c) would pull from Kiwi and auto-generate the current picture. The problem of environments would be solved graphically by our own pages, and in the Kiwi internals, we'd probably do some ugly duplication or explosion of fields like Product.
I'm honestly not sure if this is a good idea. It's a lot of work, and the end result doesn't seem to be much better than what we currently have. Yes, there is a structured results storage and a nice results submission page. On the other hand, we'd need to maintain a hodge-podge of integration code which can easily break, and yet-another-tool instance. Not to mention we could create those part ourselves (use resultsdb and some simple html pages for submission), if we wanted to invest time into it. Speaking as someone who really dislikes the NIH syndrome and tries to use existing projects as much as possible, it seems we'd gut Kiwi too much to make it worth it. I was hoping we would be able to use it with just some slight tweaks, but so far, from what I've seen, it doesn't seem that way.
I'll be happy if more people spend some time in it and post their thoughts/impressions.
Kamil
[1] https://openqa.fedoraproject.org/testcase_stats/36/Installation.html
-
Adam Williamson -
Kamil Paral -
Matthew Miller -
Tim Flink