upgradepath and updates-testing - summary of problems
by Kamil Paral
This is a summary of all problems related to upgradepath test and the
updates-testing repo.
First of all, upgrade path requirement is not properly defined anywhere.
The best what I could find is this:
https://fedoraproject.org/wiki/Package_maintainer_responsibilities#Miscel...
That means we work mainly on assumptions.
For base and -updates repo, this is not a big problem, the task is pretty
straightforward:
Algorithm example
-----------------
Repo set RS = base + updates
For every Fedora release F:
For every package P in RS of F:
NVR of P <= NVR of P in RS of F+1
We must check this constraint for every pending package update, right before
it is pushed.
We have this already implemented, and when it is enforced, we will guard
all the common users (common user == using just base + updates repo set)
against upgrade path problems.
For -updates-testing repo however, things are not as easy.
Problem #1: What is the desired repo set?
-----------------------------------------
In the algorithm example, repo set of F and repo set of F+1 were equal (base +
updates repos). How does repo set look like when -updates-testing is in place?
Do we want to check it this way?
a) F base + updates + updates-testing => F+1 base + updates + updates-testing
or this way?
b) F base + updates + updates-testing => F+1 base + updates
Option a) is simple, we use the same approach as in the algorithm example. But
it may not really guard the user, see following problems.
Option b) guards the user well, but it has some serious drawbacks. It basically
says that you must first push into F+1-updates before you can push to
F-updates-testing. That can have a large impact on package maintainers workflow.
If you consider that it may take one or two weeks before your update is pushed
into -updates, then it may a whole month or more before your update propagates
from Rawhide to Rawhide-3.
Example:
1. In all repos there is foo-2.0.
2. I want to push foo-3.0 into Rawhide, F14, F13 and F12.
3. First I must push to Rawhide.
4. Once it is in Rawhide, I can push to F14-updates-testing.
5. Now I must wait until testers give me enough karma, and then push it into
F14-updates.
6. Only now (not before) I can push F13-updates-testing.
7. Now I must wait until testers give me enough karma, and then push it into
F13-updates.
8. Only now (not before) I can push F12-updates-testing.
...
Only FESCo may say whether this is the desired workflow for package
maintainers. It is certainly different from current practice.
Problem #2: Removing repositories
---------------------------------
Option a) in Problem #1 sounded nice, didn't it? Well, not so much.
A problem arises when you disable -updates-testing repo. Either by hand
(a power-user downloaded a few packages from this repo because he wanted
to test new versions of his favorite software or test some bugfix, and then
he disabled this repo again) or automatically (anaconda disabled
-updates-testing on distribution upgrade - I don't know whether this is
actually done, but anaconda developers don't know either:)).
After you disable this repo, you still might have some packages installed
that are newer than those in F+1 base+updates. That is a problem, we haven't
saved this user from upgrade path problems in this case.
Problem #3: Unpushing packages
------------------------------
Even more juicy problem exists - unpushing packages from -updates-testing.
If we use Option a) in Problem #1, we still can't have any confidence in
our results. Because our results will apply only in that precise moment when
we check it. An hour later the corresponding package may be unpushed from
F+1 updates-testing, but left intact in F updates-testing (or you already
downloaded it, that's the same). And voilà, we have an upgrade path problem.
Example:
1. foo-3.0 is pushed into F12-updates-testing and F13-updates-testing.
2. User installs foo-3.0 from F12-updates-testing.
3. foo-3.0 is unpushed from F13-updates-testing.
4. Upgrade path problem.
Conclusion
----------
1. We don't know exactly how upgradepath constraint is defined.
2. We have no idea what -updates-testing should be tackled.
3. We should ask FESCo what's the preferred approach.
4. Current proposed solution is: "Updates-testing is for power-users, they
will handle possible problems. After we receive some info how to proceed
in this matter, we can extend our code."
5. Basic base+updates repo checking seems simple and is already implemented.
All common users should be safe.
If you see some logical mistakes or gaps in this email, please correct it.
There might be some further issues that didn't occur to me.
Thanks,
Kamil
13 years, 7 months
[AutoQA] #226: upgradepath: crash for -updates-testing koji tag
by fedora-badges
#226: upgradepath: crash for -updates-testing koji tag
--------------------+-------------------------------------------------------
Reporter: kparal | Owner: kparal
Type: defect | Status: new
Priority: major | Milestone: Package Update Acceptance Test Plan
Component: tests | Version: 1.0
Keywords: |
--------------------+-------------------------------------------------------
It seems that upgradepath crashes when we try to give it *-updates-testing
koji tag.
{{{
[root@aqd autoqa]# autoqa post-bodhi-update --title freeciv --target-tag
dist-f13-updates-testing freeciv-3.0-1.fc13 --local -t upgradepath
...
11:35:12 INFO | ERROR: Entered bad kojitag
11:35:12 ERROR| Exception escaping from test:
Traceback (most recent call last):
File "/usr/share/autotest/client/common_lib/test.py", line 384, in _exec
_call_test_function(self.execute, *p_args, **p_dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 577, in
_call_test_function
raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled UnboundLocalError: local variable
'matching_build' referenced before assignment
Traceback (most recent call last):
File "/usr/share/autotest/client/common_lib/test.py", line 570, in
_call_test_function
return func(*args, **dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 279, in
execute
postprocess_profiled_run, args, dargs)
File "/usr/share/autotest/client/common_lib/test.py", line 201, in
_call_run_once
self.run_once(*args, **dargs)
File "/usr/lib/python2.6/site-packages/autoqa/decorators.py", line 71,
in newf
f_result = f(*args, **kwargs) #call the decorated function
File "/usr/share/autotest/client/site_tests/upgradepath/upgradepath.py",
line 92, in run_once
self.envr_list.add(matching_build['envr'])
UnboundLocalError: local variable 'matching_build' referenced before
assignment
}}}
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/226>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years, 7 months
[AutoQA] #134: ResultsDB: create resultsdb prototype
by fedora-badges
#134: ResultsDB: create resultsdb prototype
----------------------+-----------------------------------------------------
Reporter: jskladan | Owner:
Type: task | Status: new
Priority: major | Milestone: Resultdb
Component: tests | Version: 1.0
Keywords: |
----------------------+-----------------------------------------------------
Create prototype of database and software which will implement dbSchema
from #133 and API from #131
This prototype should also include some test suite, which will be used to
determine, if all the functionality is OK.
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/134>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
13 years, 7 months
Re: rpmlint improvements
by Kamil Paral
----- "Alexander Todorov" <atodorov(a)redhat.com> wrote:
> Hello folks,
> I want to discuss topics in tickets:
> https://fedorahosted.org/autoqa/ticket/149
> https://fedorahosted.org/autoqa/ticket/150
>
> I did a test run of rpmlint against the latest RHEL6 tree and filed
> some bugs.
> Some were legitimate but others turned out to be false negatives and
> developers
> got angry.
>
> Then it turned out that rpmlint needs more features in order to be
> used reliably
> without making developers unhappy. So I talked to twoerner who is
> rpmlint
> maintainer for RHEL6 and exchanged some ideas.
>
> What we think is a good idea and needs to be present in upstream is
> having a
> rpmlint-data package which will provide rules.$pkg-name files for
> packages that
> need special handling. Then rpmlint will take into account those files
> and act
> accordingly. The rules files will be maintained by a separate
> maintainer and not
> by the packagers. Also any change into those rules/exceptions needs to
> be
> approved with a review process to avoid pkg maintainers manipulating
> the rules
> simply to silence out rpmlint output. This will guarantee accurate
> results.
>
> This data package will be optional to rpmlint and not affect core
> functionality.
> It will most likely be different for Fedora and RHEL as well. This is
> more or
> less the same as proposed in ticket 149 with lintian tool for Debian.
>
> What do you think? Any drawbacks/gotchas you may think of?
Hello Alexander,
thanks for looking into this and sorry for late response.
The big difference between lintian and your proposed solution is that
lintian allows to specify "overrides" inside the affected packages [1]
(files located at /usr/share/lintian/overrides/<package> or at
debian/source/lintian-overrides), but your solution centralizes all those
files in a single package rpmlint-data.
I wonder what the better approach is. Per-package solution obviously gives
more freedom (and trust) to individual package maintainers and is easier
to manage (for them). Centralized approach is stricter (you have to ask
for inclusion, review process can be easily enforced), therefore quality
may be higher, on the other hand it may annoy and discourage a lot of people.
("Just another set of policies to follow.")
Originally I was thinking about the per-package (lintian-like) approach.
My idea is that we can still apply some policies on top of it, but it is
much more convenient for package maintainers in the rest of the cases.
The policy might look like: Every line whitelisting an Error message must
be sponsored by some different package maintainer. On the other hand,
Warning and Info messages might be whitelisted without restrictions.
That's only one example of a possible policy, certainly there is plenty
of solutions.
As Michal Scherer mentioned [2], rpmlint already supports "config files"
in which we can filter out some output (based on regexp I think). So we are
already able to do this inside autoqa - just read the per-package definition
file, create custom rpmlint config and run rpmlint with this config.
Of course, having this feature upstream would be much better for us. The
package maintainers should see the same output when they run rpmlint locally
as they see when rpmlint is run inside autoqa. That is possible only when
rpmlint upstream supports these whitelists definitions directly.
[1] http://lintian.debian.org/manual/ch2.html#s2.4
[2] https://www.zarb.org/pipermail/rpmlint-discuss/2010-September/000861.html
>
>
> The rules files format need to support the following (as identified so
> far):
>
> * List file paths/directories with special permissions and uid/gid
> settings. For
> example CUPS has such. audit also has such (in RHEL for certification
> purposes)
An important note here - our proposed solution talked just about whitelisting
some warning/error messages. Also lintian overrides files provide just
whitelisting of output messages.
What you talk about here (and what you sent in your patch [3]) is a little
different. You just don't whitelist some output, you "fix" (override) the
input. For example you don't just define "don't show me the line that warns
me about incorrect permissions on /etc/passwd", but you also define *what
the correct permissions are*. Per package.
That is a little different (and somewhat harder task). I don't know whether
rpmlint developers will be inclined to so big change (redefining what the
correct input is). But allowing output whitelisting per-package could be
easier to convince them to adopt.
>From my point of view, I would rather start with simple whitelisting too.
Your approach seems to bring more challenging tasks and I think it's better
to start small.
What do you think?
[3] https://www.zarb.org/pipermail/rpmlint-discuss/2010-September/000860.html
>
> * Ignore some test cases for particular package(or file). For
> example:
> gcc.x86_64: E: devel-dependency glibc-devel
> gcc.x86_64: W: devel-file-in-non-devel-package
>
> This looks normal since gcc is a compiler but will be an error for
> another
> package (e.g. an application).
>
> jlaska told me that Fedora QA already uses rpmlint. Have you guys
> identified
> other areas/tests that need special handling or exclusion ? What are
> those?
>
> Have you thought about the file format? It needs to be easy to
> maintain and easy
> to read - probably just plain text. It also needs to be flexible so
> that we can
> add to it without breaking compatibility. We can probably have a
> separate file
> (format) for every test/package that rpmlint performs.
>
> Just to let you know that I'm happy to work on the rpmlint code and
> push it
> upstream if noone else has spare cycles.
Your effort is greatly appreciated, thanks.
>
> Regards,
> Alexander.
>
>
> _______________________________________________
> autoqa-devel mailing list
> autoqa-devel(a)lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/autoqa-devel
13 years, 7 months
Several small fixes
by James Laska
Greetings,
Nothing urgent, just a few small patches I'd like to apply to master. Nothing
noteworthy with the first two, but the third might be more interesting given
recent upgradepath discussion. Kamil might already have this in his pending
upgradepath changes.
Thoughts/concerns welcome.
> 0001-Tell-post-tree-compose-that-composes-are-enabled-for.patch
The repoinfo.conf configuration value 'composes' was enabled so that
post-tree-compose would know which entries to monitor for installation images.
This patch turns on composes for the Branched release (F14). I've also updated
the wiki documentation as well
(https://fedoraproject.org/wiki/How_to_update_AutoQA_repoinfo.conf).
> 0002-Correct-missing-config_loader-import.patch
Not sure why this never surfaced until now, but this patch corrects a missing
import from the rats_install.py script.
> 0003-Enable-owner-fedoraproject.org-cc-for-upgradepath-re.patch
I think this might make more sense to move out of the tests, and into
lib/python/test.py, but until we do that, this patch turns on cc of test
results for upgradepath. Josef also pointed out that we may want a way for
tests to enable/disable sending output to maintainers. I don't have any great
ideas on that, so for now, this patch just mimics our rpmlint and rpmguard by
including $package-owner(a)fedoraproject.org.
Thanks,
James
13 years, 7 months
Re: [PATCH] tidy up upgradepath test code
by Kamil Paral
----- "Kamil Páral" <kparal(a)redhat.com> wrote:
> This patch introduces no new features and (hopefully) no new bugs. It
> just reformats the code to be more readable and maintainable - now
> the
> semantics and variables are very similar to those used in rpmlint and
> rpmguard tests, so it's easy to read through the code.
> ---
> tests/upgradepath/upgradepath.py | 99
> ++++++++++++++++++++++++++------------
> 1 files changed, 68 insertions(+), 31 deletions(-)
Although this patch looks large, it is really just a reformatting of
a few variables. If there are no objections against it, I'll push it
tomorrow into master.
Thanks,
Kamil
13 years, 7 months
timing-critical tests - depcheck, upgradepath
by Kamil Paral
This ticket:
https://fedorahosted.org/autoqa/ticket/228
and this patch:
https://fedorahosted.org/pipermail/autoqa-devel/2010-September/001195.html
forced me to do some thinking.
We have some tests that are really timing-critical - depcheck and upgradepath.
We can't run them in an arbitrary time as we can do with rpmlint and similar.
We must run them just before the packages are pushed - not earlier, not later.
In other words, we are given set of packages to examine and a repository
in some well-defined state. We examine the packages and the repo and say
"OK" or "not OK". Someone (Bodhi) must guarantee that neither the repo nor
the set of packages change between our examination and actual push of those
packages. There must be no steps in between. Otherwise our result would be
misleading.
For upgradepath we have even stricter requirements - *no* repo must change
between our examination and actual push.
Now that we know the requirements, we can ask some questions:
As for the ticket:
Can we really rely on checking koji -pending tags, instead of communicating
with Bodhi? I don't know how the -pending tags are implemented. But can we
be sure that Bodhi takes exactly same set of packages in the exactly same
order as we do, when it tries to push them?
As for the patch:
I have a feeling that for these aforementioned tests there's not really any
sense in sending email reports. These tests either pass or fail, just before
push. If our test fails, then the push fails and maintainers will be probably
notified by Bodhi anyway, won't they?
And more importantly, we currently run upgradepath when packages are
*requested* for push, but the actual push might be some hours away, is that
right? So the result we produce is not really valid, in the meantime a lot
of things may change. That also speaks against sending some notifications.
But maybe I just got some stuff wrong. What do you think?
13 years, 7 months
Current state of ResultsDB
by Josef Skladanka
Hello gang!
Currently, there is a instance of ResultsDB core, running at <http://test1185.test.redhat.com/wsgi/resultsdb/xmlrpc>
The tested & working functionality is "storing the data into the DB" using beforementioned xmlrpc interface.
At the moment, "using resultsdb as storage" is implemented in the jskladan branch in git
<http://test1185.test.redhat.com/wsgi/resultsdb/xmlrpc>
Precisely these patches:
<http://git.fedorahosted.org/git/?p=autoqa.git;a=commit;h=46b4019b7608a794...>
<http://git.fedorahosted.org/git/?p=autoqa.git;a=commit;h=e1404be74b646ec5...>
As you can see, the only changes are in the AutoQATest base class (lib/python/test.py), and of course the API library
for ResultsDB is addded (lib/python/resuldsdb.py or lib/python/logging.py [in the first patch]).
The rest is simply taken care of automagically.
This patchset does not make use of the phases, as these would require changes into the sources of individual tests
and can be postponed until the basic functionality is in the master. But the functionality should be pretty much
at the same level, as the mailing list provides (although resultsdb is not _at the moment_ storing the "outputs" variable).
I have also created a wiki metadataa drafts:
Testplan:
https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Package_Update_Accep...
Testcases:
https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Rpmguard_Testcase_Me...
https://fedoraproject.org/wiki/User:Jskladan/Sandbox:Rpmlint_Testcase_Met...
...
As you can see, the main focus is on the Testplan part. The idea behind this wiki page is to provide
a simple, yet whole description of the testplan, so we can make a frontend, which will take it, and
visualize the 'progress' of the testplan for given envr (which seems to be the common denominator of
all the arguments given to a test).
To explain what does it all mean:
* "testcases" : _stripped_
- this has two means
1) sum up all the tests covered in the testplan
2) provide 'aliases' for further use in the metadata - so we can write "Initscripts" instead of the whole URL
* "testcase_classes" : ["mandatory", "introspection", "advisory"]
- this value specifies testcase 'classes' or 'types' - this has utterly semantic meaning, and these can be _anything_
what you find meaningfull. The idea behind this is to be able to somehow group the test in the frontend AND
to provide some means to bind the results of all the standalone tests into the "testplan result". See below.
* "mandatory" : {"testcases": ["Package sanity", "Repo sanity", "Conflicts", "Upgrade path"], "pass": ["PASSED", "WAIVED"], on_fail":"FAILED"}
- this says, that the ["Package sanity", "Repo sanity", "Conflicts", "Upgrade path"] tests will be considered "passed", if
the result stored in ResultsDB is in ["PASSED", "WAIVED"].
- If any of theses tests have a different result, the whole testplan is considered to be "failed" (on_fail":"FAILED")
* "introspection" : {"testcases": ["Rpmlint", "Initscripts"], "pass": ["PASSED", "WAIVED", "INFO"], "on_fail":"FAILED"}
- the same as the mandatory tests, but also satisfies with "INFO" as a result of the test.
* "advisory" : {"testcases": ["Rpmlint", "Rpmguard"], "pass": ["PASSED", "WAIVED", "INFO", "NEEDS_INSPECTION"], "on_fail":"NEEDS_INSPECTION"}
- if any of these testcases' result is out of the "pass" set, mark the testplan as "NEEDS_INSPECTION".
You noticed, that Rpmlint is both in the "introspecition" and "advisory" groups, this is based on the current state
of the testplan <https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_accept...> - I'm not really
sure how to base the "no errors" and "no warnings" into that, based solely on the result, but that's not the real issue for now.
This brought me to one more possible thing to think about: are the current "result states" (RUNNING, PASSED,
INFO, FAILED, ABORTED, CRASHED, WAIVED, NEEDS_INSPECTION) enough? AND - are we able to sort these according to the
'urgency' or 'importance' or 'relevance' or how to call it? Let's say we have two tests in a testplan,
one is ABORTED and the second is CRASHED - what would you like to see as 'overall result' Aborted or Crashed?
I'm not sure, if I was able to describe it clearly (I'm really hitting the limits of my descriptive skills here :-D),
so feel free to ask as many questions, as needed.
Joza
13 years, 7 months