Automated Fedora update testing
by Adam Williamson
Hi folks! I thought this might be of interest to the Cockpit community,
so I thought I'd write it up here in case anyone missed it elsewhere.
I work on the Fedora QA team, and we have been using the openQA
automated test system (developed by our friends at SUSE) to run various
functional tests on Fedora composes for the last couple of years.
As Cockpit is considered a critical part of Fedora Server, we run a few
tests that exercise Cockpit. They're nowhere near as extensive as
Cockpit's own test suite, but they run in a somewhat different context:
they run on the Fedora packaged Cockpit, against the rest of a Fedora
environment. They also test in a somewhat different way from Cockpit's
own test suite: openQA tests a lot like a human, by booting up a
virtual machine and then effectively "looking at the screen" and
sending mouse and keyboard input events (via VNC). The test boots a
full Fedora system, runs Firefox, and interacts with Cockpit like a
human being would. For both of these reasons, it can sometimes catch
problems Cockpit's own tests would not (it also sometimes goes wonky in
ways that Selenium-based tests don't, but never mind, that's my job to
deal with :>)
Until recently we ran these tests only on Fedora's nightly development
release distribution composes. Recently, though, we deployed some
enhancements to our openQA setup that let us run tests on Fedora
distribution updates as well, and have the results made visible through
the Fedora update system (Bodhi). I have just made it so that, from now
on, these tests will run on any update that contains the 'cockpit'
package. They also run on any critical path update.
This means that for any Fedora update containing cockpit or any
critical path package, Fedora's openQA cockpit tests should run, and
you should see the results in the Fedora update system (Bodhi). You can
see the results in Bodhi by clicking the Automated Updates tab for any
update. For instance, here's a recent Firefox update for Fedora 25:
https://bodhi.fedoraproject.org/updates/FEDORA-2017-31c64a0bbf
If you look at the Automated Tests tab, you can see passes for:
update.server_cockpit_basic
update.server_cockpit_default
update.realmd_join_cockpit
indicating that this Firefox update didn't cause any problems for
Cockpit. Clicking on any test result will take you to the openQA page
for the test, where you can diagnose failures and so on (explaining how
to do this is a bit beyond the scope of this mail, please do ask me if
you're interested!)
I hope this stuff will help us avoid shipping updates that break
Cockpit (and other key components). If you have any questions,
concerns, comments, or suggestions, please do ask!
To anticipate one question: you can cause *all* the tests for an update
to be re-run by editing the update in any way (you don't have to change
the package loadout, just changing a single character in the
description or something will do). If you think just one test result is
bogus and want it re-run, currently, you'll have to ask someone with
the necessary power - either me or Jan Sedlak (garretraziel on IRC).
I'm in North America and he's in Europe, so we should have most
timezones covered between us. We're hoping to set up a better mechanism
for this in future.
Note, if you're interested in the results for the nightly Fedora
distribution composes, an email summary of the results for those is
sent each time they're run to the Fedora test@ and devel@ lists, look
for mails with "compose check report" in the subject. Any time any of
the cockpit tests fails, the failure will be listed in the mail (passed
tests are not specifically listed, just a count of them). I usually
keep an eye on those results and analyze failures and file bugs,
though.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
6 years, 11 months
Assumed security between nested login sessions
by Stef Walter
Cockpit is a Linux session.
Normally when you connect from one login session to a second one, the
second login session does not access to the first ... although the does
not hold.
For example if you SSH into machineA ... and then from machineA SSH into
machineB ... it is assumed (and typically the case) that the login
session on machineB does not have access to the login session on machineA.
In Cockpit this has not been the case. When you start a Cockpit login
session on machineA ... and then in Cockpit on machineA you add machineB
as a second server. The Cockpit login session tasks/code for each of
those have equal access to one another via the browser.
I worked with Muelli over the weekend to try and fix this and here's a
basic pull request:
https://github.com/cockpit-project/cockpit/pull/6260
We can ask the browser to apply the Same Origin policy and keep the
sessions sandboxed. There's several places where some follow up work is
needed, as seen by the dependencies of that pull request.
But also it's impossible to use window.localStorage from such a browser
sandbox. So if we put this in, we would likely need to implement a new
window.localStorage that uses window.postMessage with "hint" messages to
store/retrieve contents.
Once this is done, we can enforce the separation of concerns between
stuff running from different machines and start to mirror this assumed
behavior of Linux login sessions.
Cheers,
Stef
7 years
