On 26.02.2016 15:28, Daniel Mack wrote:
On 02/26/2016 02:29 PM, Stef Walter wrote:
> On 26.02.2016 13:28, Daniel Mack wrote:
>> Hmm, but the machines I'm running this one are more or less stateless,
>> and the scripts literally download gigabytes of images, which also takes
>> a long time.
>
> Typically the runners that run these things cache the images, and only
> new ones get downloaded. There's a $TEST_DATA environment variable that
> htelps with this. On a test runner typically pointed to a volume that's
> mounted persistently.
I have to ask whether such persistent storage is available.
> Obviously if there's non-sensical complexity, we want to get that out of
> the way. I'll do a bit of cleanup to help. But as to "why":
>
> Cockpit is an interface for complex and messy operating systems. It
> interacts directly with tons different projects and they all break,
> regress, remove API and on and on.
>
> So the integration tests have to actually integrate and boot real
> operating systems. It ends up being very close to how users would use
> Cockpit. In most cases they use the exact same build scripts (like spec
> files) to build cockpit as ends up being built on that operating system
> ... and so on.
>
> Without the integration testing that we do, the Cockpit project would be
> dead ... or have a year long stabilization process before a release (ie:
> as good as dead). But because the tests boot real operating systems,
> that users are actually running, every week we can release something
> that works, and catch regressions before they pile up, etc.
>
> Obviously, I said above, you don't have to use the same operating system
> images we do. If you want to generate your own test machine image,
> that's possible.
Would you expect CentOS 7 to deviate that much that we couldn't rely on
the rest of the system (everything except systemd) to remain stable?
The machine my script provisions is a clean CentOS 7 install which can
be rebooted and accessed from a control host or other test machine
instances. Would it be feasible to install a new version of system and a
stable release of Cockpit on one those, and then run the test suite
(which would purely act as remote control for Cockpit) on another? Both
machines would have the same version of Cockpit checked out, of course.
That would be something I could set up.
Yup, should be possible. The image will need to have the right SSH keys,
and then a few other things need to happen in the test suite (see below).
You'll probably end up select certain tests to run, rather than running
a whole test suite. That's since some of the tests do destructive things
to the system, or change the state in unexpected ways. But lets get this
working, and then you can pick and choose.
But to start needs the following work:
1. Make verify/check-xxxx take an address that they use to test
against rather than starting a VM. Some work on that done here:
https://github.com/cockpit-project/cockpit/pull/3848
2. Get the Cockpit testsuite running with CentOS ... talked about this
at devconf. Should mostly work, but every operating system wants
to be different.
Also here's a bit of work to clean up the test/ directory. Lots of folks
have added test suites and other stuff there recently, so time for a
spring cleaning.
https://github.com/cockpit-project/cockpit/pull/3846
> phantomjs is what we use as the remote-control. And you could
generate a
> test machine that works with the given tests. I know that Jan Scotka
> does. And it goes without saying, that if there's issues that get in the
> way of rolling your own test machine, then we'd be happy to work with
> you to change things.
That's good to know, thanks for that. But still, the entire qcow images
dance sounds like a layer that might not be necessary in the CentOS CI,
and if possible, I'd like to skip it entirely for this particular use case.
Well try it out. You may find that it's simpler to use the images for
tests that want to reset the state between each test, bring up multiple
machines etc.... Or you might find those are not relevant tests. Lets see.
Stef