9:52 a.m.
Hi folks,
this message sparked an interest:
https://lists.fedorahosted.org/pipermail/python-bugzilla/2013-December/00...
In addition to that I've inspected around 30 packages which seem to be missing
an upstream test suite (a few have one but it is not automatically executed in
%check section in the spec file).
I'm pretty sure many more packages are like this, I just didn't have the time to
investigate all several thousands of them.
My idea is simple - starting after the holidays to call for help in writing test
suites (or more test cases) for packages. This can be coupled with settings to
execute them in Travis CI or another CI system of choice.
My questions are:
* What is the general feeling of using Travis CI in Fedora? It is well
established in Ruby and Python circles but I know we like to keep dependency on
external services to minumum.
Does Fedora have its own CI infrastructure coupled with Koji ?
Maybe deploy our own instance or contribute to Travis with a pool of systems
sponsored by Fedora?
What to do with packages whose test suite is not suitable to be executed during
build (e.g. due to requirements or limitations on the build servers) ?
What's your take ?
( Adding Tim Flink to CC to answer from the infrastructure side. )
* Are there any volunteers to join me in planning and coordinating this project?
We need to somehow prioritize which packages need inspection and working on,
count the available test cases, report bugs if missing, produce patches, etc. It
will be a long run one and needs lots of work just because the great number of
packages.
* Who else should I be talking to ?
Regards,
Alex
--
http://atodorov.org
10:16 a.m.
On mið 18.des 2013 15:52, Alexander Todorov wrote:
* Are there any volunteers to join me in planning and coordinating
this project? We need to somehow prioritize which packages need
inspection and working on, count the available test cases, report bugs
if missing, produce patches, etc. It will be a long run one and needs
lots of work just because the great number of packages.
Count me it inn
JBG
10:28 a.m.
On Wed, 18 Dec 2013 17:52:59 +0200
Alexander Todorov <atodorov(a)redhat.com> wrote:
Hi folks,
this message sparked an interest:
https://lists.fedorahosted.org/pipermail/python-bugzilla/2013-December/00...
In addition to that I've inspected around 30 packages which seem to
be missing an upstream test suite (a few have one but it is not
automatically executed in %check section in the spec file).
I'm pretty sure many more packages are like this, I just didn't have
the time to investigate all several thousands of them.
My idea is simple - starting after the holidays to call for help in
writing test suites (or more test cases) for packages. This can be
coupled with settings to execute them in Travis CI or another CI
system of choice.
Sounds interesting. We couldn't run travis in %check... so that would
be something run async somewhere?
I think adding %checks would be longer term more handy, but would also
be more work (you would involve upstreams of projects and work with
them to create checks,etc)
My questions are:
* What is the general feeling of using Travis CI in Fedora? It is
well established in Ruby and Python circles but I know we like to
keep dependency on external services to minumum.
I guess that somewhat depends on whats done with the information. Would
a failure there block updating a Fedora package? Or would it simply be
informational?
Would we then contribute the travis info to upstream to use, or this
is tied to the Fedora package version?
Does Fedora have its own CI infrastructure coupled with Koji ?
No.
Maybe deploy our own instance or contribute to Travis with a pool of
systems sponsored by Fedora?
In order for us to deploy it, it would need to be packaged up (looks
like it's a lot of moving parts) and you would need to get enough
folks who know ruby/jruby/etc to commit to maintain it long term.
What to do with packages whose test suite is not suitable to be
executed during build (e.g. due to requirements or limitations on the
build servers) ?
Note also that builds allow no network, so you can't depend on remote
resources. There's various ways to test things even so, depending on
what you are testing.
What's your take ?
I think more testing at this level is great, but might be better to
look at working with upstreams to add tests that could run from
%check...
( Adding Tim Flink to CC to answer from the infrastructure side. )
* Are there any volunteers to join me in planning and coordinating
this project? We need to somehow prioritize which packages need
inspection and working on, count the available test cases, report
bugs if missing, produce patches, etc. It will be a long run one and
needs lots of work just because the great number of packages.
* Who else should I be talking to ?
If you want to involve fedora infrastructure, the infrastructure list
might be good...
kevin
12:09 p.m.
On Wed, 18 Dec 2013 17:52:59 +0200
Alexander Todorov <atodorov(a)redhat.com> wrote:
Hi folks,
this message sparked an interest:
https://lists.fedorahosted.org/pipermail/python-bugzilla/2013-December/00...
In addition to that I've inspected around 30 packages which seem to
be missing an upstream test suite (a few have one but it is not
automatically executed in %check section in the spec file).
I'm pretty sure many more packages are like this, I just didn't have
the time to investigate all several thousands of them.
On a side note, it might be interesting to find out what percentage of
packages are running things in %check. I don't know what we would do
with that metric, but I think it would be interesting :)
My idea is simple - starting after the holidays to call for help in
writing test suites (or more test cases) for packages. This can be
coupled with settings to execute them in Travis CI or another CI
system of choice.
My questions are:
* What is the general feeling of using Travis CI in Fedora? It is
well established in Ruby and Python circles but I know we like to
keep dependency on external services to minumum.
I've not used it personally, so I can't speak much about Travis in
particular. Depending on what you're looking to do, I would hesitate to
add yet another CI system to Fedora - infra already has a jenkins
instance set up and qa-devel is starting to use buildbot for both CI
and as a part of taskotron (the project formerly known as taskbot).
Does Fedora have its own CI infrastructure coupled with Koji ?
That depends on what you consider CI to be, I guess. I consider both
autoqa and taskotron to be forms of CI but not the exact same thing as
travis or jenkins.
Both autoqa and taskotron are designed to run various things based on a
package's state in either koji or bodhi.
Maybe deploy our own instance or contribute to Travis with a pool of
systems sponsored by Fedora?
What to do with packages whose test suite is not suitable to be
executed during build (e.g. due to requirements or limitations on the
build servers) ?
What's your take ?
( Adding Tim Flink to CC to answer from the infrastructure side. )
I'm still unclear on what you're looking to do. Are you talking about
looking for test suites in package upstreams and running those tests,
regardless of whether they're run in %check during build? Are you
talking about creating and maintaining an out-of-band set of unit tests
for packages without an upstream unit test suite? Are you talking about
creating a set of package-specific functional/integration test suites
that are run when packages are built?
* Are there any volunteers to join me in planning and coordinating
this project? We need to somehow prioritize which packages need
inspection and working on, count the available test cases, report
bugs if missing, produce patches, etc. It will be a long run one and
needs lots of work just because the great number of packages.
* Who else should I be talking to ?
Kevin already mentioned this, but definitely infra.
Tim
4:38 a.m.
На 18.12.2013 20:09, Tim Flink написа:
On a side note, it might be interesting to find out what percentage
of
packages are running things in %check. I don't know what we would do
with that metric, but I think it would be interesting :)
I have something in the works which can be modified to extract this info. It
should be fairly easy to make a good guesstimate of the presence/non presence of
test suites as well. I think I'll start from here.
> Does Fedora have its own CI infrastructure coupled with Koji ?
That depends on what you consider CI to be, I guess. I consider both
autoqa and taskotron to be forms of CI but not the exact same thing as
travis or jenkins.
Both autoqa and taskotron are designed to run various things based on a
package's state in either koji or bodhi.
Thanks, I have to look into these. I gave Travis as example because it is well
known and because somebody else decided to make use of it in python-bugzilla.
I'm still unclear on what you're looking to do. Are you talking about
looking for test suites in package upstreams and running those tests,
regardless of whether they're run in %check during build? Are you
talking about creating and maintaining an out-of-band set of unit tests
for packages without an upstream unit test suite? Are you talking about
creating a set of package-specific functional/integration test suites
that are run when packages are built?
Pretty much all of these in the long term but I wanted to get a general feeling
so I can concentrate the effort on something specific.
* For packages which have upstream test suites (e.g. parted) - contribute more
tests where necessary; Ideally covering bugs reported against Fedora or RHEL for
which there isn't a test case.
* For packages without upstream test suite my preference is to create one and
contribute it back upstream. If upstream doesn't accept it and the package is
critical enough maybe maintain that in the Fedora branch.
* Functional/integration test suites (e.g. using Beaker tasks) are less priority
goal for the moment but also need to be considered. For some packages these are
more suitable (e.g. anaconda), for others maybe not (e.g. pykickstart seems to
have fairly good amount of unit tests).
Whatever the test suite(s) come out to be we need an environment(or
environments) where to execute them and a method of starting the suite. I can
see how some test cases are part of %check and can halt the build process and
others are started async after the build and are left for Fedora QA and devel to
investigate.
In the case of Travis, test runs are triggered by git commits and Travis is not
connected to any build infrastructure from what I know. It provides the results
to whoever is interested in them.
10:04 p.m.
Well, I think travis can just be a supplemental part of %check, like
we can run it after package building, and paste its results as "for
reference only".
12:40 a.m.
На 18.12.2013 20:09, Tim Flink написа:
On a side note, it might be interesting to find out what percentage
of
packages are running things in %check. I don't know what we would do
with that metric, but I think it would be interesting :)
Hi Tim,
I did a quick measurement and the number of packages which execute something in
%check is around 30%. 35% is the number of packages which appear to have a test
suite at all. More details here:
http://atodorov.org/blog/2013/12/24/upstream-test-suite-status-of-fedora-20/
Happy holidays,
Alex
3802
days inactive
3808
days old
6 comments
5 participants
participants (5)
-
Alexander Todorov -
Christopher Meng -
Jóhann B. Guðmundsson -
Kevin Fenzi -
Tim Flink