4:06 a.m.
Hello KDE and XFCE teams,
I am writing to you on behalf of the Fedora QA team, because you are the
stakeholders of release-blocking desktop environments in Fedora.
We believe that there is need to revisit and review some of the approaches
we
have in desktop testing. I hope that I can clarify in the text.
*How do we test desktop applications?*
Currently, there are three desktop environments, which block the Fedora
releases: GNOME, KDE, and XFCE (on ARM). Among others, there is a test case
called “Desktop Menus” [
https://fedoraproject.org/wiki/QA:Testcase_desktop_menus ] which is used
for all three desktop environments.
*What is the purpose of the Desktop Menus test case?*
The purpose of the “Desktop Menus” test case is to test that applications
listed
in the menu do generally work. That means that each application in the
default
system menu must launch without crashing and withstand basic usage. [2]
*What problems can arise when trying to follow the test case?*
This approach leaves a lot of space for variability, because the terms are
not
clearly given, especially when talking about “basic functionality” of the
desktop applications. In the past, there have been some disputes during
Blocker
Review meetings, or during Go/No go meetings whether basic functionality is
broken or not. It's usually up to group judgement to decide what is a basic
functionality and what it beyond that, which means as testers we need to
err on
the side of doing more testing rather than less.
Testing the “Desktop Menus” test case is one of the most time consuming test
case we currently have. In GNOME, there are currently about 40 applications
in
the default menu and all need to be started and their basic functionality
tested. In KDE, the number of applications is even higher, and some
application
types (like web browsers) are even present as several different
applications.
The scope of this test case also covers testing the subcomponents in the
Control
Center (or a similar equivalent) and, usually, such subcomponents are
numerous.
Testing XFCE on ARM is also rather time consuming, because the ARM devices
we
can use for testing are very slow and there are some application that make
little sense on an ARM motherboard (e.g. a CD writing software). Thus,
doing a
thorough Desktop Menus test case requires a rather longish time frame and we
lose time we could devote to different problems.
Besides that, the test case is also very boring, so it usually gets tested
by
Fedora QA members only, with few inputs from the community. So, before the
Beta
and Final release we are able to do such tests a couple of times, but the
overall coverage is not big.
*What could we do about it?*
At Fedora QA, we have been trying to come up with some ideas to make the
situation a little better than it is now. The goal is that the important
areas
get enough attention and testing they deserve, and we avoid shipping broken
and
under-tested software. See the ideas below and please tell us if you could
adopt
all/some of them, or if you have different suggestions to improve the
current
state, we'd love to hear them too.
- The list of pre-installed applications could be pruned of those which
don't make much sense for that particular purpose (e.g. CD writing software
on ARM, or perhaps on any system nowadays).
- Pre-installed applications could be deduplicated where possible (e.g.
not having several web browsers pre-installed by default).
- We could define a list of "important" applications that would be
tested thoroughly and the basic functionality criterion would apply just to
them. That would tell QA where to focus. (This would obviously contain
applications like system settings, file browser, text editor, browser,
terminal emulator, etc. But e.g. a screenshot utility would probably not
fall into the list. Also, in case of duplicated apps, only one of those
could be marked as "important", and the others would* not). The rest of the
applications would be just best effort in terms of QA, and wouldn't block
the release.
- Last but not least, you could help us more with release validation
before relevant milestones (Beta, Final) :-) If there's anything that
should be improved on our part (communication, instructions, etc), we'd
love to hear that.
Please, let me know what you think about the ideas proposed above. Do you
have
any other ideas that could help us improve the quality (with the same
amount of
testing force)?
Best regards,
--
Lukáš Růžička
FEDORA QE, RHCE
Red Hat
<https://www.redhat.com>
Purkyňova 115
612 45 Brno - Královo Pole
lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>
4:17 a.m.
On Tue, Jan 8, 2019 at 11:08 AM Lukas Ruzicka <lruzicka(a)redhat.com> wrote:
Hello KDE and XFCE teams,
I am writing to you on behalf of the Fedora QA team, because you are the
stakeholders of release-blocking desktop environments in Fedora.
We believe that there is need to revisit and review some of the approaches
we
have in desktop testing. I hope that I can clarify in the text.
*How do we test desktop applications?*
Currently, there are three desktop environments, which block the Fedora
releases: GNOME, KDE, and XFCE (on ARM). Among others, there is a test case
called “Desktop Menus” [
https://fedoraproject.org/wiki/QA:Testcase_desktop_menus ] which is used
for all three desktop environments.
Why is Xfce tested only on ARM devices? I mean if that's one of the main
reasons it's slow and tedious, why not change the architecture?
Additionally, it would be probably worthwhile to create two different sets
of programs, one group has to start and thus needs to be tested and the
other is a nice
to have, but doesn't need to start. At least the criteria could be a little
bit less strict.
Johannes
*What is the purpose of the Desktop Menus test case?*
The purpose of the “Desktop Menus” test case is to test that applications
listed
in the menu do generally work. That means that each application in the
default
system menu must launch without crashing and withstand basic usage. [2]
*What problems can arise when trying to follow the test case?*
This approach leaves a lot of space for variability, because the terms are
not
clearly given, especially when talking about “basic functionality” of the
desktop applications. In the past, there have been some disputes during
Blocker
Review meetings, or during Go/No go meetings whether basic functionality is
broken or not. It's usually up to group judgement to decide what is a basic
functionality and what it beyond that, which means as testers we need to
err on
the side of doing more testing rather than less.
Testing the “Desktop Menus” test case is one of the most time consuming
test
case we currently have. In GNOME, there are currently about 40
applications in
the default menu and all need to be started and their basic functionality
tested. In KDE, the number of applications is even higher, and some
application
types (like web browsers) are even present as several different
applications.
The scope of this test case also covers testing the subcomponents in the
Control
Center (or a similar equivalent) and, usually, such subcomponents are
numerous.
Testing XFCE on ARM is also rather time consuming, because the ARM devices
we
can use for testing are very slow and there are some application that make
little sense on an ARM motherboard (e.g. a CD writing software). Thus,
doing a
thorough Desktop Menus test case requires a rather longish time frame and
we
lose time we could devote to different problems.
Besides that, the test case is also very boring, so it usually gets tested
by
Fedora QA members only, with few inputs from the community. So, before the
Beta
and Final release we are able to do such tests a couple of times, but the
overall coverage is not big.
*What could we do about it?*
At Fedora QA, we have been trying to come up with some ideas to make the
situation a little better than it is now. The goal is that the important
areas
get enough attention and testing they deserve, and we avoid shipping
broken and
under-tested software. See the ideas below and please tell us if you could
adopt
all/some of them, or if you have different suggestions to improve the
current
state, we'd love to hear them too.
- The list of pre-installed applications could be pruned of those
which don't make much sense for that particular purpose (e.g. CD writing
software on ARM, or perhaps on any system nowadays).
- Pre-installed applications could be deduplicated where possible
(e.g. not having several web browsers pre-installed by default).
- We could define a list of "important" applications that would be
tested thoroughly and the basic functionality criterion would apply just to
them. That would tell QA where to focus. (This would obviously contain
applications like system settings, file browser, text editor, browser,
terminal emulator, etc. But e.g. a screenshot utility would probably not
fall into the list. Also, in case of duplicated apps, only one of those
could be marked as "important", and the others would* not). The rest of the
applications would be just best effort in terms of QA, and wouldn't block
the release.
- Last but not least, you could help us more with release validation
before relevant milestones (Beta, Final) :-) If there's anything that
should be improved on our part (communication, instructions, etc), we'd
love to hear that.
Please, let me know what you think about the ideas proposed above. Do you
have
any other ideas that could help us improve the quality (with the same
amount of
testing force)?
Best regards,
--
Lukáš Růžička
FEDORA QE, RHCE
Red Hat
<https://www.redhat.com>
Purkyňova 115
612 45 Brno - Královo Pole
lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>
_______________________________________________
xfce mailing list -- xfce(a)lists.fedoraproject.org
To unsubscribe send an email to xfce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/xfce@lists.fedoraproject.org
4:19 a.m.
Perhaps automation needs to be implemented. CC-ing Pavlin that I think
can advise about automation strategies.
Johannes Lips wrote on 1/8/19 12:17 PM:
>
>
> On Tue, Jan 8, 2019 at 11:08 AM Lukas Ruzicka <lruzicka(a)redhat.com
> <mailto:lruzicka@redhat.com>> wrote:
>
> Hello KDE and XFCE teams,
>
> I am writing to you on behalf of the Fedora QA team, because you are the
> stakeholders of release-blocking desktop environments in Fedora.
>
> We believe that there is need to revisit and review some of the
> approaches we
> have in desktop testing. I hope that I can clarify in the text.
>
> *How do we test desktop applications?*
>
> Currently, there are three desktop environments, which block the Fedora
> releases: GNOME, KDE, and XFCE (on ARM). Among others, there is a
> test case
> called “Desktop Menus” [
> https://fedoraproject.org/wiki/QA:Testcase_desktop_menus ] which is
> used for all three desktop environments.
>
> Why is Xfce tested only on ARM devices? I mean if that's one of the main
> reasons it's slow and tedious, why not change the architecture?
> Additionally, it would be probably worthwhile to create two different
> sets of programs, one group has to start and thus needs to be tested and
> the other is a nice
> to have, but doesn't need to start. At least the criteria could be a
> little bit less strict.
>
> Johannes
>
> *What is the purpose of the Desktop Menus test case?*
>
> The purpose of the “Desktop Menus” test case is to test that
> applications listed
> in the menu do generally work. That means that each application in
> the default
> system menu must launch without crashing and withstand basic usage. [2]
>
> *What problems can arise when trying to follow the test case?*
>
> This approach leaves a lot of space for variability, because the
> terms are not
> clearly given, especially when talking about “basic functionality”
> of the
> desktop applications. In the past, there have been some disputes
> during Blocker
> Review meetings, or during Go/No go meetings whether basic
> functionality is
> broken or not. It's usually up to group judgement to decide what is
> a basic
> functionality and what it beyond that, which means as testers we
> need to err on
> the side of doing more testing rather than less.
>
> Testing the “Desktop Menus” test case is one of the most time
> consuming test
> case we currently have. In GNOME, there are currently about 40
> applications in
> the default menu and all need to be started and their basic
> functionality
> tested. In KDE, the number of applications is even higher, and some
> application
> types (like web browsers) are even present as several different
> applications.
> The scope of this test case also covers testing the subcomponents in
> the Control
> Center (or a similar equivalent) and, usually, such subcomponents
> are numerous.
>
> Testing XFCE on ARM is also rather time consuming, because the ARM
> devices we
> can use for testing are very slow and there are some application
> that make
> little sense on an ARM motherboard (e.g. a CD writing software).
> Thus, doing a
> thorough Desktop Menus test case requires a rather longish time
> frame and we
> lose time we could devote to different problems.
>
> Besides that, the test case is also very boring, so it usually gets
> tested by
> Fedora QA members only, with few inputs from the community. So,
> before the Beta
> and Final release we are able to do such tests a couple of times,
> but the
> overall coverage is not big.
>
> *What could we do about it?*
>
> At Fedora QA, we have been trying to come up with some ideas to make the
> situation a little better than it is now. The goal is that the
> important areas
> get enough attention and testing they deserve, and we avoid shipping
> broken and
> under-tested software. See the ideas below and please tell us if you
> could adopt
> all/some of them, or if you have different suggestions to improve
> the current
> state, we'd love to hear them too.
>
>
> * The list of pre-installed applications could be pruned of those
> which don't make much sense for that particular purpose (e.g. CD
> writing software on ARM, or perhaps on any system nowadays).
> * Pre-installed applications could be deduplicated where possible
> (e.g. not having several web browsers pre-installed by default).
> * We could define a list of "important" applications that would be
> tested thoroughly and the basic functionality criterion would
> apply just to them. That would tell QA where to focus. (This
> would obviously contain applications like system settings, file
> browser, text editor, browser, terminal emulator, etc. But e.g.
> a screenshot utility would probably not fall into the list.
> Also, in case of duplicated apps, only one of those could be
> marked as "important", and the others would* not). The rest of
> the applications would be just best effort in terms of QA, and
> wouldn't block the release.
> * Last but not least, you could help us more with release
> validation before relevant milestones (Beta, Final) :-) If
> there's anything that should be improved on our part
> (communication, instructions, etc), we'd love to hear that.
>
> Please, let me know what you think about the ideas proposed above.
> Do you have
> any other ideas that could help us improve the quality (with the
> same amount of
> testing force)?
>
>
> Best regards,
>
> --
>
> Lukáš Růžička
>
> FEDORA QE, RHCE
>
> Red Hat
>
> <https://www.redhat.com>
>
> Purkyňova 115
>
> 612 45 Brno - Královo Pole
>
> lruzicka(a)redhat.com <mailto:lruzicka@redhat.com>
>
>
> TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>
>
> _______________________________________________
> xfce mailing list -- xfce(a)lists.fedoraproject.org
> <mailto:xfce@lists.fedoraproject.org>
> To unsubscribe send an email to xfce-leave(a)lists.fedoraproject.org
> <mailto:xfce-leave@lists.fedoraproject.org>
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/xfce@lists.fedoraproject.org
>
>
> _______________________________________________
> xfce mailing list -- xfce(a)lists.fedoraproject.org
> To unsubscribe send an email to xfce-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/xfce@lists.fedoraproject.org
>
12:37 p.m.
Sending email from Pavlin because he is not a member of the group and
message is rejected.
--
QUESTIONS
How do we test desktop applications?
On what architectures do we test?
What problems can arise when trying to follow the test case?
I will provide you with brief answer.
WHAT DO WE TEST?
We do automated testing of GNOME environment ONLY.
GUI applications from GNOME fall into our domain of tests.
Our team do not develop automated tests for other GUI environments.
WHERE DO WE TEST?
We test GNOME GUI on several CPU platforms:
1. x86_64
2. PPC64
3. PPC64LE
4. Aarch64 (ARM)
5. s390x
The above statement is valid for RHEL 7.
In the case of RHEL 8, those platforms are reduced to:
1. x86_64
2. PPC64LE
In all cases we test on machines available in Beaker.
For all tests without specific hardware requirements we use VM.
For testing Wi-Fi and displays, we use physical hosts.
HOW DO WE TEST?
We create automated tests which have several major parts:
1) define the logic of a test
2) prepare conditions for testing
3) define functions and algorithms
4) execute GUI testing
5) check for results
(1)
To describe the logic of a test we use English sentences in plain text
files.
This language is called /*Behave*/ (Gherkin language).
Each sentence is connected to a programming function.
(2)
To prepare initial conditions for tests we use:
1. YAML file - contains a list of needed packages
2. a bash script - to set initial conditions for all tests
3. a python module - to set specific conditions for some tests
This preparation includes but is not limited to:
* install needed RPM packages
* install needed Python modules
* set up system parameters
* download security certificates
* add polkit privilege to allow passwordless setup in some components
* grant sudo privilege to users
* change passwords
* restart system services
* set up printers
* set up network interfaces and connections
* set up VPN connections
* set up 802.1x security
(3)
Functions that perform GUI testing are written in Python.
Each sentence from the logical language Behave has its corresponding
function in Python.
This correspondence is known as "/*step*/". It's the "link area"
between
Python and Behave.
Every parameter from an English sentence (in Behave) becomes a function
parameter in Python.
Thus, a sentence from the human language is transformed into programming
code.
To test RHEL 7 we use Python 2.7.
To test RHEL 8 we use Python 3.6.
(4)
To make automated tests manage GUI we use:
1. Accessibility Technology (A11Y)
2. dogtail
3. Python
Combining those we are able to perform various tasks in the GUI.
An automated test clicks/double clicks/drags/types text in GUI components.
Most frequently used components:
* frame windows
* dialog windows
* text labels
* edit fields
* buttons
* combo boxes
* check boxes
* tables
Since the GUI hierarchy is based on parent-child relationship, we use
that relationship for testing.
/*dogtail*/ "sees" a GUI element on the screen via /*A11Y*/ technology.
dogtail performs various activities on different GUI elements in the
same ways as a human user would do.
(5)
After the testing of GUI components is complete
and modifying of system settings is done
we check for results.
The expects results are found in the modified system state.
We check a specific area of RHEL by using shell command or a function in
Python.
*Examples*
1. An automated test starts an application then it checks if it's in
the output of "ps ax".
2. An automated test makes a change in GNOME environment then it
searches for that change in GSettings.
3. An automated test changes the user's password then it tries to log
in with those credentials.
What problems can arise when trying to follow the test case?
Most common issues in our automated tests:
1. Initial conditions for tests are not prepared.
2. The tested GUI application has new interface, different placement of
GUI elements. The current automated test does not work.
3. The tested GUI application has new behaviour. The current automated
test does not work.
4. Unexpected side effects come out which block the execution. For
instance delays, clicking before a GUI form appears, unable to click
on a GUI element, unable to make changes, unable to scroll down the
GUI, etc.
5. A bug has appeared in the tested application. In that case testing
can continue after bug resolution.
6. A bug has appeared in some of the dependent packages. Similarly, the
automated test works after the bug is fixed.
7. The sequence of steps is incorrect. It must be changed, steps should
be added or replaced to achieve desired result.
8. The environment (the testing machines) does not provides hardware
resources for the automated test.
If you would like more detailed description, please let me know.
Pavlin
10:09 a.m.
On Tue, 2019-01-08 at 12:19 +0200, Aleksandar Kostadinov wrote:
Perhaps automation needs to be implemented. CC-ing Pavlin that I
think
can advise about automation strategies.
Automation is something we can do, but it's not a magic bullet: if you
automate the tests, instead of spending your time running them, you get
to spend your time managing the automation :) So it is still a problem
to have a huge set of apps to test, because then you will have a huge
set of automated tests to manage, and GUI apps tend to change in ways
that break automation quite often.
So whether the tests are manual or automated, it is still valuable to
consider the questions Lukas presented.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
10:13 a.m.
On Tue, 2019-01-08 at 11:17 +0100, Johannes Lips wrote:
Why is Xfce tested only on ARM devices? I mean if that's one of
the main
reasons it's slow and tedious, why not change the architecture?
Additionally, it would be probably worthwhile to create two different sets
of programs, one group has to start and thus needs to be tested and the
other is a nice
to have, but doesn't need to start. At least the criteria could be a little
bit less strict.
This is because it is only a release-blocking desktop for ARM. It's an
awkward situation: GNOME and KDE are kind of "grandfathered in" as
release blocking desktops, but neither works well on ARM (we tried
having KDE as the release-blocking ARM desktop for a bit, but it didn't
go really well). So if we want to have desktop ARM block the release at
all, we have to pick another desktop, and Xfce is the one we pick.
Since it's explicitly *for ARM* that Xfce is a blocking desktop, it
would feel a bit odd to test it only on x86_64 (where it is *not* a
blocking desktop).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
4:21 p.m.
On 1/8/19 5:06 AM, Lukas Ruzicka wrote:
* The list of pre-installed applications could be pruned of those
which don't make much sense for that particular purpose (e.g. CD
writing software on ARM, or perhaps on any system nowadays).
Certainly a good thing to do.
* Pre-installed applications could be deduplicated where possible
(e.g. not having several web browsers pre-installed by default).
* We could define a list of "important" applications that would be
tested thoroughly and the basic functionality criterion would apply
just to them. That would tell QA where to focus. (This would
obviously contain applications like system settings, file browser,
text editor, browser, terminal emulator, etc. But e.g. a screenshot
utility would probably not fall into the list. Also, in case of
duplicated apps, only one of those could be marked as "important",
and the others would* not). The rest of the applications would be
just best effort in terms of QA, and wouldn't block the release.
Addressing both points together -
I believe we are decent with not having too many duplicates on Xfce (not
sure about other DEs). One approach I can think of is testing only those
in the primary group in comps. In the case of Xfce, the live ISO has
packages from both xfce-desktop and xfce-apps group. Perhaps, we can
limit QA testing to only the primary group.
Of course, this would need bit more organization from the spin
maintainers side since some "important" applications are in xfce-apps
(e.g. firefox) but we (xfce maintainers) can do that.
* Last but not least, you could help us more with release
validation
before relevant milestones (Beta, Final) :-) If there's anything
that should be improved on our part (communication, instructions,
etc), we'd love to hear that.
We (to be clear - I am not blaming anyone) did drop the ball on Xfce
testing in F29 cycle. Hopefully, we can avoid this moving forward.
My two cents.
Mukundan.
1915
days inactive
1935
days old
6 comments
5 participants
participants (5)
-
Adam Williamson -
Aleksandar Kostadinov -
Johannes Lips -
Lukas Ruzicka -
Mukundan Ragavan