rpmgrill in taskotron
by Kamil Paral
Hey Ralph (ccing qa-devel),
just to let you know, I added some code to Koji directive to download build logs. The patch is posted here:
https://phab.qadevel.cloud.fedoraproject.org/D916
(you can use 'arch patch D916' to apply the diff in your libtaskotron checkout)
The corresponding changes for task-rpmgrill are in feature/buildlog branch:
https://bitbucket.org/fedoraqa/task-rpmgrill/branch/feature/buildlog
I could have implemented shuffling the build.log files around in run_rpmgrill.py, but it seemed easier to have it in a bash script. I moved all the rpmgrill* commands in there as well, and used our new bash directive to execute it (it still has rough edges, we're working on that). But having it in a bash script is not necessary, if you prefer to do it all in python, we definitely don't have to merge the code as it is :)
I also removed your failsafe decorator, but again, it's just a suggestion. The first reason was that since I moved most of the setup code and rpmgrill execution into the bash directive, the decorator didn't have any affect on it anyway. The second reason is that we seem to be of an opinion that ResultsDB should only contain real results (passed, failed), and crashes and similar execution statuses should be only present in ExecDB [1] (which, however, cannot be searched yet). This is all up to discussion, and our current solutions have a lot of rough edges, so if you want to continue sending CRASHED to ResultsDB in case of errors, we have no problem with it, and we can convert task-rpmgrill once we're 100% sure how we want things to look like and have it implemented properly. Please note, though, that we currently receive emails for all tests that crash hard, so we will be able to spot any task-rpmgrill issues even if not wrapped in the failsafe decorator.
If you have any questions or concerns, please ask :)
Cheers,
Kamil
[1] https://taskotron.fedoraproject.org/execdb/
7 years, 9 months
Automatic ABI checks for new package updates in bodhi using taskotron
by Sinny Kumari
Hi Kamil,
Thank you for the reply.
On Wed, Apr 13, 2016 at 7:34 PM, Kamil Paral <kparal(a)redhat.com> wrote:
>
> Hello Sinny,
> thanks for your work and sorry for late response. I'll review your
> taskotron task and let you know if there's something that should be changed
> or not. Afterwards, we will start mirroring your git repo on our taskotron
> servers, and patch our taskotron-trigger to know about task-libabigail. We
> can then execute it on every new Koji build (or Bodhi update, your choice).
> Your task will probably be the first one that we will execute regularly
> while not being written and maintained directly by us, so if there are any
> rough edges in the process, I apologize in advance :-)
>
Okay, sounds good to me.
>
> You'll need to have two branches in your git:
> master - this will be used on our production server
> https://taskotron.fedoraproject.org/
> develop - this will be used on our dev and staging server
> http://taskotron-dev.fedoraproject.org/ and
> https://taskotron.stg.fedoraproject.org/
Okay, works for me.
You need to decide whether it is better to run libabigain against every new
> Koji build, or just against every new Bodhi update. From a quick look, I
> think it makes more sense to run libabigail on every new Koji build, so
> that people can see the results even before creating the update (that
> requires looking into ResultsDB manually at the moment). If we run it on
> every Koji build, the results will still show up in Bodhi - Bodhi should
> query ResultsDB and show the results for those particular builds present in
> the update. (We might need to teach Bodhi about libabigail existence, I'm
> not sure). Ultimately it's your choice, what makes more sense for your
> check.
>
I believe that when we say every new Koji build, we are talking about
non-scratch build which doesn't include scratch build done by anyone. If my
assumption is right, then yes running libabigail task on each koji build
will be good. It is possible to do that with current implementation since
libabigail task look for a koji build-id to download corresponding rpms.
> Please also create a wiki page at
> https://fedoraproject.org/wiki/Taskotron/Tasks/libabigail similar to these
> https://fedoraproject.org/wiki/Taskotron/Tasks/depcheck
> https://fedoraproject.org/wiki/Taskotron/Tasks/upgradepath
> linked from https://fedoraproject.org/wiki/Taskotron/Tasks .
>
Sure, I will create one.
> We try to have at least some basic documentation and FAQ for our checks in
> there. Currently it's not very discoverable (we should link to it at least
> from ResultsDB, which we currently don't) and the location can change, but
> at least it's a link we can give to people when asking basic questions
> about one of our tasks. Also, since you're going to maintain the task and
> not us, please include some "Contact" section where to post feedback or
> report bugs (e.g. github issues page). If people ask us about the task and
> we don't know the answer, we will point them to that wiki page.
>
Will add contact section in wiki page.
I wonder if it is better to have this included with other fedora qa tasks?
> Can we please continue this discussion in qa-devel [1] mailing list? We
> can discuss more implementation details in there, and I'll post my review
> findings in there as well.
>
done!
--
http://sinny.io/
7 years, 10 months
openQA upgrade plans
by Adam Williamson
Hi, folks! Just wanted to fill everyone in on openQA upgrade plans.
I have upgraded the staging server and worker boxes to Fedora 24. prod
is still on Fedora 23 for now.
I have built current git os-autoinst and openqa, plus latest upstream
perl-Mojolicious-Plugin-AssetPack (which is needed for new openqa), for
Rawhide. More-or-less identical builds for F24 are available in this
COPR:
https://copr.fedorainfracloud.org/coprs/adamwill/openQA-stg/
My pet openqa - https://openqa.happyassassin.net - is currently running
those packages, as a first step in testing if they work OK. So far it
seems to be alright.
This week I'm planning - if Emmanuel Seyman from the perl SIG is OK
with it - to send out the same three builds to updates-testing for F24,
and update staging to the new openqa and os-autoinst. If that all
appears to go OK, I'll move on to updating prod to F24 and the new
openqa/os-autoinst.
The changes to AssetPack are a bit unfortunate; basically, openQA is
now pretty tied to a design where the web assets (CSS and JavaScript)
that it uses have to be pulled from remote servers and cached at some
point, which is awkward for packaging. On the plus side, the cached
assets no longer change every time openQA or AssetPack changes, but
only when openQA does. After thinking it through I've gone for an
approach similar to SUSE's, by adding a script to the package git repo
which generates the asset cache and stuffs it into a tarball, which is
then included as a Source1 for the package, so the final packages just
include a pre-generated copy of the asset cache. Whoever's doing the
packaging (so, uh, me...) has to run the script to re-generate the
asset cache any time we bump to a new upstream openQA checkout or
release.
Please do check out the new packages and the deployment on
happyassassin and yell if you see any problems. Note that some tests
are failing on happyassassin because I haven't built all the hard disk
images (that box is limited in disk space) and for some reason tests
that try to upload the disk image after they finish tend to fail on
that deployment (I've never managed to work out why).
The most obvious change is that the web UI has been rather revamped
(and now uses the distro-neutral icon that the Fedora design team came
up with, yay!) but there's other stuff as well, you can dig through the
openqa and os-autoinst git changelogs to see what...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
7 years, 10 months
2016-06-06 Fedora QA Devel Meeting Minutes
by Tim Flink
Despite the lack of a meeting announcement, we were productive and seem
to have mostly figured out the remaining namespace issues.
That thread will be updated separately with the proposal that we came
up with.
=================================
#fedora-meeting-1: fedora-qadevel
=================================
Minutes: https://meetbot.fedoraproject.org/fedora-meeting-1/2016-06-06/fedora-qade...
Minutes (text): https://meetbot.fedoraproject.org/fedora-meeting-1/2016-06-06/fedora-qade...
Log: https://meetbot.fedoraproject.org/fedora-meeting-1/2016-06-06/fedora-qade...
Meeting summary
---------------
* Roll Call (tflink, 14:01:35)
* Announcements and Information (tflink, 14:03:55)
* taskotron-trigger sent for package review (mkrizek, 14:04:18)
* taskotron-trigger sent for package review (mkrizek, 14:05:44)
* Task Namespaces (tflink, 14:10:10)
* ACTION: tflink to make qa-devel@ archives public (tflink, 14:14:57)
* LINK:
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraprojec...
(tflink, 14:15:28)
* AGREED: the items 1-6 above seem reasonable at this time and is good
enough to start working on but a more detailed document should be
sent out to qa-devel@ for final approval (tflink, 15:17:49)
* Open Floor (tflink, 15:19:13)
Meeting ended at 15:21:55 UTC.
Action Items
------------
* tflink to make qa-devel@ archives public
Action Items, by person
-----------------------
* tflink
* tflink to make qa-devel@ archives public
* **UNASSIGNED**
* (none)
People Present (lines said)
---------------------------
* tflink (148)
* kparal (79)
* Dodji (43)
* mkrizek (8)
* zodbot (5)
* garretraziel (4)
* omiday (2)
* coremodule (1)
* lbrabec (1)
Generated by `MeetBot`_ 0.1.4
.. _`MeetBot`: http://wiki.debian.org/MeetBot
7 years, 10 months
Task Result Namespaces
by Martin Krizek
Hi,
we have deployed task result namespaces support a while ago and put
our checks (depcheck, upgradepath, rpmlint) into the 'qa' namespace.
With newly added tasks like task-abicheck and task-dockerautotest,
we weren't sure where to put them and so we used the scratch
namespace which is supposed to be used for testing purposes. Now
with those "3rd party" tasks deployed to production, we need to
decide what namespaces should they and other future tasks belong in.
The starting namespaces, and maintainers of tasks belonging to that
namespace, follows:
qa.* - Fedora QA
pkgs.<pkgname>.* - <pkgname> maintainers
fas.<username>.* - <username>
fasgroup.<groupname>.* - fas members belonging to <groupname>
scratch.* - anyone
Now, since we maintain task-dockerautotest, it should go into qa.*,
I am not sure though where does task-abicheck belong. I see three
options here:
1. we can create fas group and put it there,
2. create additional dist.* namespace where tasks like abicheck
and/or rpmgrill would be, or
3. change semantics of qa.* to 'anything Fedora QA maintains or
important, not package-specific, tasks".
I don't really have strong feelings about either to be honest.
Thoughts? Other proposals?
Thanks,
Martin
7 years, 10 months
PoC of "configurable trigger"
by Josef Skladanka
Source: https://bitbucket.org/fedoraqa/taskotron-trigger/branch/pony
Diff: https://phab.qadevel.cloud.fedoraproject.org/D872
This started as simple bike-shedding to make more sense in naming (so
everything is not named "Trigger"), but it went further :D
The main change here is, what I call "configurable trigger" - at the moment,
every time we want to add support for even the most basic new task (like the
package-specific task for docker), changes are needed in the trigger's source
code.
These changes add a concept of a "rules engine", that decides what tasks to
schedule based on data extracted from the received FedMessage, and a
set of rules.
The rules-engine is YAML, in a format like this::
```
- do:
- {tasks: [depcheck, upgradepath]}
when: {message_type: KojiTagChanged}
- do:
- {tasks: [dockerautotest]}
when: {message_type: KojiBuildCompleted, name: docker}
- do:
- {tasks: [abicheck]}
when:
message_type: KojiBuildCompleted
name:
$in: ${critpath_pkgs}
$nin: ['docker'] # critpath excludes
```
The rules are split in two parts `when` and `do`, the `when` clause is
a mongo query that will get evaluated against the dataset provided by
the FedMsg consumer. For example, the KojiBuildCompletedJobTrigger now
publishes this (values are fake, to make it more descriptive::
message_data = {
"_msg": {...snipped...},
"message_type": "KojiBuildCompleted",
"item": "docker-1.9.1-6.git6ec29ef.fc23",
"item_type": "koji_build",
"name": "docker",
"version": "1.9.1-6.git6ec29ef",
"release": "fc23",
"critpath_pkgs": [..., "docker", ...]
"distgit_branch": "f23",
}
So taking the rules, and the data, going from the top:
# First rule's `when` is `False` as `message_type` is not `KojiTagChanged`
# Second rule is `True` because both the `message_type` and name in the
`when` clause match the data
# Third rule does _not_ schedule anything, because even though `docker` is
in `critpath_pkgs`, it also is part of the critpath excludes list, and
so the rule is ignored
The `when` clauses are in fact mongo queries
<https://docs.mongodb.com/manual/tutorial/query-documents/>,
evaluated using a Python library that implements it for querying Python objects.
The rules engine then takes the `do` clauses of the 'passed' rules, and
produces arguments for the `trigger_tasks()` calls. By default, `item`, and
`item_type` are taken from the `message_data`, `arches` is set to
`config.valid_arches`, and then all the key/values from the `do`'s body are
added on top. This means, that we can have a task, that for example forces
an architecture different than default::
```
- do:
- {tasks: [awesome_arm_check], arches: [armhfp]}
when: {message_type: KojiBuildCompleted}
```
The `do` clause can have multiple items in it, so something like this is
possible::
```
- do:
- {tasks: [rpmlint]}
- {tasks: [awesome_arm_check], arches: [armhfp]}
when: {message_type: KojiBuildCompleted}
```
Triggering `rpmlint` on the default architectures, and `awesome_arm_check`
on `armhfp` for each package built in Koji.
This means, that when we want to trigger new (somewhat specific) tasks,
no changes are needed in the trigger's code, but just in the configuration,
to alter the rules. If we come to the point where more functionality is
needed, than it obviously calls for changes in the underlying code, in order
to add more key/values to the data provided by the Fedmsg consumer, or
adding more general functionality overall.
A good example of this is the dist-git style tasks problem. To solve it
I have added a new command (`$discover`)to the `do` section, that crawls the
provided git repo/branch, and schedules jobs for all `runtask.yml`'s found::
```
- do:
- {$discover: {repo:
'http://pkgs.fedoraproject.org/git/rpms-checks/${name}.git', branch:
'${distgit_branch}'}}
when: {message_type: KojiBuildCompleted}
```
In the bigger picture, this 'rules engine' functionality can be used to
make (for example) a web interface, that allows creating/altering the rules,
instead of changing the config file (the rules can as easily be taken from
a database, as from the config file), or even to provide a per-user triggering
capability - we could add a piece of code, that checks (selected) users'
Fedorapeople profile for a file, that contains rules in this format, and
then could simply run the engine on those rules+data from Fedmsg to decide
whether the user-defined tasks should be run.
It also somewhat reduces the tight bond between the trigger and FedMessage,
as the rules engine does not really care where did the data (used to evaluate
the rules) come from.
This is by no means final, but it IMO shows quite an interesting PoW/idea, that
was not that complicated to implement, and made the trigger lot better at what
it can do.
Please, share thoughts/comments/rants!
7 years, 10 months