7:07 p.m.
Hi folks!
So, since we've been getting Workstation ostree installer images in the
last few Rawhide composes, I've been setting up openQA to test it. This
is not at present deployed to production openQA, but it *is* deployed
to staging openQA. Unfortunately the last few Rawhide composes have
contained a broken anaconda so the image has not been built, but the
20170615.n.0 compose *did* feature a working image, and you can see the
test results there:
https://openqa.stg.fedoraproject.org/tests/overview?distri=fedora&ver...
(note that *right now* there's an infra outage going on so as I write
this the page is inaccessible, but all should be back to normal
shortly).
Look for the Workstation dvd-ostree 'Flavor'. As you can see, for now I
just have it running a subset of the regular Workstation tests, with
the update-related tests - which assume an RPM-based system - left out,
for now. Everything mostly appears to work well, except for the known
bug https://bugzilla.redhat.com/show_bug.cgi?id=1193590 (that's the
cause of most of the tests showing up as 'soft failures', the 'orange'
state, rather than 'passes' / green).
So I have two purposes here: firstly just to let you know that this
testing is now happening (they will be run for any Rawhide or Branched
compose which contains a Workstation ostree installer image), and
secondly to ask about any other testing that would be useful. Is there
any practical way we can actually test the ostree update process, for
instance? For testing RPM-based updates, what we do is install a
'dummy' python3-kickstart package, with a NEVR known to be lower than
any 'real' version of the package, so we know for sure that an update
will be available from the official repositories, and then we go ahead
and run an update in the usual fashion. Would there be any similar kind
of possibility for testing ostree updates? The goals of openQA testing
are to test as 'realistically' as possible, so we only test unmodified
images, and want to have the test behave as similarly as possible to
how the 'real' operation would behave on an end user's system.
And besides updates, can anyone think of any other test that would be
particularly useful to run?
Of course, to be clear, this is mostly 'advisory' testing for now: this
image is still not a release-blocking image, so failures of the tests
will not automatically trigger any kind of special attention or even
bugs being filed, it will require people to be interested in looking at
them and filing bugs. I will try and do so, of course.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
9:13 a.m.
On Wed, Jun 21, 2017 at 05:07:12PM -0700, Adam Williamson wrote:
'dummy' python3-kickstart package, with a NEVR known to be
lower than
any 'real' version of the package, so we know for sure that an update
will be available from the official repositories, and then we go ahead
and run an update in the usual fashion. Would there be any similar kind
of possibility for testing ostree updates? The goals of openQA testing
are to test as 'realistically' as possible, so we only test unmodified
images, and want to have the test behave as similarly as possible to
how the 'real' operation would behave on an end user's system.
You should be able to go back and forth between various ostree commits,
right? Possibly in addition to a "do full install, test" thing, there
could just be a "update from previous to new, test" image. Or, am I
misunderstanding?
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
8:54 a.m.
On Wed, Jun 21, 2017, at 08:07 PM, Adam Williamson wrote:
Look for the Workstation dvd-ostree 'Flavor'. As you can see, for now I
just have it running a subset of the regular Workstation tests, with
the update-related tests - which assume an RPM-based system -
rpm-ostree is also an RPM-based system; it's even literally in the name of
the project! It's *also* an ostree-based system. Just like hybrid cars
are both gasoline and electric.
That said of course, things are different since again it's a hybrid and
one needs to consider the base tree vs layered packages.
The most realistic test is to test updates from "previous stable
image/installer".
to "latest tree".
(As well of course to test layered package updates, but
that runs into
https://github.com/projectatomic/rpm-ostree/issues/391 )
There's a variety of rpm-ostree tests that are used for Atomic Host here:
https://github.com/projectatomic/atomic-host-tests/tree/master/roles
That all said I'm still not sure about doing these types of tests in AutoQA
versus something more like Ansible as we already use for a-h-t.
Though testing gnome-software's forthcoming rpm-ostree backend would
be back in the realm of GUI testing.
2:38 p.m.
On Fri, 2017-06-23 at 09:54 -0400, Colin Walters wrote:
On Wed, Jun 21, 2017, at 08:07 PM, Adam Williamson wrote:
>
> Look for the Workstation dvd-ostree 'Flavor'. As you can see, for now I
> just have it running a subset of the regular Workstation tests, with
> the update-related tests - which assume an RPM-based system -
rpm-ostree is also an RPM-based system; it's even literally in the name of
the project! It's *also* an ostree-based system. Just like hybrid cars
are both gasoline and electric.
Well, yes, but I'm usually talking from the *user* perspective. So far
as the user is concerned, it's not an RPM-based system: we can't test
updates by fiddling around with dnf, is the pertinent point here.
That said of course, things are different since again it's a
hybrid and
one needs to consider the base tree vs layered packages.
The most realistic test is to test updates from "previous stable
image/installer".
to "latest tree".
(As well of course to test layered package updates, but
that runs into
https://github.com/projectatomic/rpm-ostree/issues/391 )
There's a variety of rpm-ostree tests that are used for Atomic Host here:
https://github.com/projectatomic/atomic-host-tests/tree/master/roles
That all said I'm still not sure about doing these types of tests in AutoQA
versus something more like Ansible as we already use for a-h-t.
openQA, not AutoQA.
If we have another system where we can test the update experience
exactly as it works for a real end user - starting from deploying
ostree Workstation in the way we actually expect a user to do so,
starting from our actual nightly / candidate images - then that's
great, let's do that. But if we don't, openQA is probably a practical
way to do it. What I'm really kinda looking for is more specific
details on the level of "here is approximately what you could do to
reliably put a Workstation ostree system into a state where an update
for it would be available through the usual mechanism".
Though testing gnome-software's forthcoming rpm-ostree backend
would
be back in the realm of GUI testing.
So, updating via GNOME Software is not yet possible? That's useful
information, thanks.
Still, thinking about it, for ostree Workstation we really need to test
two *separate* things, yes? Updating the base system, and then
deploying and updating software on top of the base system. Again, more
details on how that process is expected to work would be useful; are
Flatpaks the only expected deployment method for apps on ostree
Workstation, for instance?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2:56 p.m.
On Fri, Jun 23, 2017, at 03:38 PM, Adam Williamson wrote:
Well, yes, but I'm usually talking from the *user* perspective.
So far
as the user is concerned, it's not an RPM-based system: we can't test
updates by fiddling around with dnf, is the pertinent point here.
But one can (and is definitely expected to in many general cases)
to use layered packages. A lot (but not all) of the functionality of dnf
is also possible with rpm-ostree - increasingly so, for example we're
working on supporting "removing" things from the base tree:
https://github.com/projectatomic/rpm-ostree/pull/797
If we have another system where we can test the update experience
exactly as it works for a real end user - starting from deploying
ostree Workstation in the way we actually expect a user to do so,
starting from our actual nightly / candidate images - then that's
great, let's do that. But if we don't, openQA is probably a practical
way to do it. What I'm really kinda looking for is more specific
details on the level of "here is approximately what you could do to
reliably put a Workstation ostree system into a state where an update
for it would be available through the usual mechanism".
We already indeed do these types of things in CentOS CI.
Still, thinking about it, for ostree Workstation we really need to
test
two *separate* things, yes? Updating the base system, and then
deploying and updating software on top of the base system.
Right!
Again, more
details on how that process is expected to work would be useful; are
Flatpaks the only expected deployment method for apps on ostree
Workstation, for instance?
No. There's lots of stuff that isn't yet in flatpak form. And further,
not everything is a desktop app.
I have `emacs flatpak-builder opensc pcsc-lite-ccid keepassx powerline tmux
vagrant-libvirt virt-manager ykclient`
layered here for example; about half of those are desktop apps I haven't
bothered to try to flatpak, and most of them are privileged anyways. Half are utilities
or userspace driver-type things like ykclient.
Now things like `fedpkg` live in my tools container.
I also have `oc cluster up` going for server side development.
There's a bit more here: https://fedoraproject.org/wiki/Workstation/AtomicWorkstation
(Though origin-clients isn't yet in "workstation-ostree" partly due to this
ongoing
slow moving debate about whether it's "the same workstation, just with
rpm-ostree"
or whether it's something more different than that where e.g. devel tools are always
in a container, one uses openshift for local server development etc)
3:25 p.m.
On Fri, 2017-06-23 at 15:56 -0400, Colin Walters wrote:
On Fri, Jun 23, 2017, at 03:38 PM, Adam Williamson wrote:
> Well, yes, but I'm usually talking from the *user* perspective. So far
> as the user is concerned, it's not an RPM-based system: we can't test
> updates by fiddling around with dnf, is the pertinent point here.
But one can (and is definitely expected to in many general cases)
to use layered packages. A lot (but not all) of the functionality of dnf
is also possible with rpm-ostree - increasingly so, for example we're
working on supporting "removing" things from the base tree:
https://github.com/projectatomic/rpm-ostree/pull/797
Ah, I see, thanks for the clarification; so, long story short, we
*could* actually test using dnf for install / remove / update of
packages on Workstation-ostree? But is this also something that's
already tested in CentOS CI? If so, probably not worth duplicating it.
> If we have another system where we can test the update
experience
> exactly as it works for a real end user - starting from deploying
> ostree Workstation in the way we actually expect a user to do so,
> starting from our actual nightly / candidate images - then that's
> great, let's do that. But if we don't, openQA is probably a practical
> way to do it. What I'm really kinda looking for is more specific
> details on the level of "here is approximately what you could do to
> reliably put a Workstation ostree system into a state where an update
> for it would be available through the usual mechanism".
We already indeed do these types of things in CentOS CI.
Great, one less thing for me to worry about, then. Well. I suppose
there's one more thing to ask: do you test install from the *installer
image* in CentOS CI? If not, how significant are the chances (do you
think) that a system installed from the installer image might differ in
its behaviour from one deployed via a filesystem image or however you
get to the starting state of the CentOS CI tests?
> Still, thinking about it, for ostree Workstation we really need
to test
> two *separate* things, yes? Updating the base system, and then
> deploying and updating software on top of the base system.
Right!
> Again, more
> details on how that process is expected to work would be useful; are
> Flatpaks the only expected deployment method for apps on ostree
> Workstation, for instance?
No. There's lots of stuff that isn't yet in flatpak form. And further,
not everything is a desktop app.
OK, so, to summarize, you expect people to deploy stuff on top of
ostree Workstation:
* Via layered packages (which is basically just 'dnf install foo' so
far as the end user is concerned, right?)
* As Flatpaks, installed via GNOME Software
* As containers
Does that sound about right? Thanks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
7:22 a.m.
On Fri, 2017-06-23 at 13:25 -0700, Adam Williamson wrote:
OK, so, to summarize, you expect people to deploy stuff on top of
ostree Workstation:
* Via layered packages (which is basically just 'dnf install foo' so
far as the end user is concerned, right?)
* As Flatpaks, installed via GNOME Software
* As containers
Does that sound about right? Thanks!
If you want a full test matrix, things that need to work are:
* Updating or downgrading the base image, using commandline tools or
gnome-software
* The same, in the presence of layered packages
* Installing, updating and removing flatpaks, using commandline tools
of gnome-softare
* Installing, updating and removing layered packages, using commandline
tools
* Installing, updating and removing containers, using commandline tools
4:07 p.m.
On Fri, Jun 23, 2017 at 03:56:16PM -0400, Colin Walters wrote:
But one can (and is definitely expected to in many general cases)
to use layered packages. A lot (but not all) of the functionality of dnf
is also possible with rpm-ostree - increasingly so, for example we're
working on supporting "removing" things from the base tree:
https://github.com/projectatomic/rpm-ostree/pull/797
I think we'll want to test _both_ this and ostree updates (and
rollbacks), right?
> way to do it. What I'm really kinda looking for is more
specific
> details on the level of "here is approximately what you could do to
> reliably put a Workstation ostree system into a state where an update
> for it would be available through the usual mechanism".
We already indeed do these types of things in CentOS CI.
Do the results of that get put out to fedmsg?
(Though origin-clients isn't yet in
"workstation-ostree" partly due
to this ongoing slow moving debate about whether it's "the same
workstation, just with rpm-ostree" or whether it's something more
different than that where e.g. devel tools are always in a
container, one uses openshift for local server development etc)
FWIW, I'm in favor of "more different". Let's make it interesting and
stand out.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
8:46 p.m.
On Fri, 2017-06-23 at 17:07 -0400, Matthew Miller wrote:
On Fri, Jun 23, 2017 at 03:56:16PM -0400, Colin Walters wrote:
> But one can (and is definitely expected to in many general cases)
> to use layered packages. A lot (but not all) of the functionality of dnf
> is also possible with rpm-ostree - increasingly so, for example we're
> working on supporting "removing" things from the base tree:
> https://github.com/projectatomic/rpm-ostree/pull/797
I think we'll want to test _both_ this and ostree updates (and
rollbacks), right?
> > way to do it. What I'm really kinda looking for is more specific
> > details on the level of "here is approximately what you could do to
> > reliably put a Workstation ostree system into a state where an update
> > for it would be available through the usual mechanism".
>
> We already indeed do these types of things in CentOS CI.
Do the results of that get put out to fedmsg?
It'd be great if they could be forwarded to ResultsDB; that would give
us fedmsgs for free, I believe.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
2496
days inactive
2498
days old
desktop@lists.fedoraproject.org
8 comments
4 participants
participants (4)
-
Adam Williamson -
Colin Walters -
Matthew Miller -
Matthias Clasen