Hello everyone,
on behalf of the Fedora QA team I'd like inform you about some useful QA
processes and links that can help you with your Fedora CoreOS releases.
Here we go.
= Test Days =
When you have something new and interesting, or perhaps something you'd
like to get tested by different people/workflows/hardware configurations,
it's a good idea to ask the community to participate in a test day. You
provide the thing to be tested (an image, package, etc) and the test cases,
and we'll try to spread the word and support attendees.
You can see the schedule, the instructions and other important info at:
https://fedoraproject.org/wiki/QA/Test_Days
= Blockers and freeze exceptions =
Blocker process applies to products that are blocking Fedora release (which
is not CoreOS's case at this moment), but the freeze exception process can
be used by anyone. Before the major release milestones (Beta, Final), the
whole Fedora release goes into a freeze (see schedule [1]). Nothing can
enter stable updates except for builds that either fix an accepted blocker
bug, or have been granted a freeze exception.
When you have an update that you really need to get to stable updates, you
can ask for an exception. You explain why you need it, and during a blocker
bug meeting the importance of your use case will be weighted against the
risk of breaking something in our release-blocking products. A freeze
exception is then granted or denied, and the package is pushed stable (or
stays in testing until the freeze is over).
The process is described in detail at:
https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process
We have a webapp that can be used to propose the freeze exception:
https://qa.fedoraproject.org/blockerbugs/ (the Propose button)
or it can be done manually in bugzilla (see the SOP).
[1]
https://fedoraproject.org/wiki/Releases/29/Schedule
= Common Bugs =
When there's an important or perhaps just an uncomfortable bug in your
product, it's good to let your users know. One of the more visible
locations that gets referenced in release announcements is the "common
bugs" wiki page. It's used both pre-release (important bugs we're trying to
fix) and post-release (bugs we failed to fix and were not critical enough
to block the release, or bugs we found out too late).
F29 common bugs will be documented at (wiki page not yet created):
https://fedoraproject.org/wiki/Common_F29_bugs
Here are F28 common bugs as an example:
https://fedoraproject.org/wiki/Common_F28_bugs
The process of including a new entry is documented on the same wiki page.
You don't need anyone's permission, simply follow the process and edit the
page. Alternatively, you can mark it in bugzilla, and someone from QA will
get to it eventually and document it for you.
Usually we try to provide a short and understandable description of the bug
on that wiki page, so that even end users without deep technical knowledge
can read and understand it, and document a workaround or some other
solution when possible.
= Release validation matrices =
We have a process of using wiki matrices for tracking release validation
testing. It contains the test cases and their results for a particular
compose. The results are partly provided from manual testing and partly
from automated systems. Here's an example of F28 final compose summary page
(containing all matrices grouped into a single view):
https://fedoraproject.org/wiki/Test_Results:Fedora_28_RC_1.1_Summary
As I learned, you give a heavy emphasis on automated testing and don't
intend to have manual test cases at the moment. Therefore we probably won't
extend our matrices with FCOS test cases and results (at least not yet,
might change in the future, e.g. if it changed to be release-blocking). All
you need to do is to pay attention to your automated test suite results. It
would be nice if Fedora QA had access to those results as well, and we've
been discussing that on your github issue tracker. One of the options is to
make your test runner public, another (possibly combined) is to send at
least the overall results to our ResultsDB instance [1], so that it's
stored in the same place as the rest of our automation results and we can
check on it in a programmatic way.
So, this might not be immediately useful to you, but we'll try to work on
it in the future.
[1]
https://taskotron.fedoraproject.org/resultsdb/
= Communication =
I'm not sure whether I included all that is relevant or forgot something.
Either way, if you have any questions or requests, please contact us. You
can either ask here or on IRC (I and coremodule will be following your
channels), or you can use our QA channels:
https://fedoraproject.org/wiki/QA#Communicate
Thanks,
Kamil
Fedora QA