3:51 p.m.
I recently wrote some code to generate detailed statistics of Fedora & EPEL updates
within bodhi. Eventually this will be auto-generated and exposed within bodhi itself, but
for now here are the initial metrics.
This report definitely conveys the shortcomings in our testing, however, it does show us
improving with each release. For Fedora 13, we implemented the No Frozen Rawhide process
with improved Critical Path policies, which were definitely a success. With these enhanced
procedures, along with the upcoming implementation of AutoQA and the new Package update
acceptance criteria (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
think we'll see these numbers drastically improve in the future.
You can find the code that generates these statistics here:
https://fedorahosted.org/bodhi/browser/bodhi/tools/metrics.py
https://fedorahosted.org/bodhi/browser/bodhi/tools/log_stats.py. If you have any ideas or
suggestions for different types of metrics to generate, or if you find any bugs in my
code, please let me know.
luke
=====================================================
Bodhi Statistics Report (Generated on June 8th, 2010)
=====================================================
Out of 17412 total updates, 2958 received feedback (16.99%)
Out of 1045 total unique karma submitters, the top 30 are:
* notting (424)
* mclasen (366)
* jkeating (321)
* adamwill (283)
* cwickert (161)
* rdieter (159)
* pbrobinson (141)
* kevin (141)
* cweyl (122)
* tomspur (119)
* mtasaka (110)
* xake (97)
* cschwangler (86)
* kwright (84)
* peter (83)
* hadess (80)
* michich (72)
* tagoh (69)
* pfrields (69)
* bpepple (69)
* iarnell (68)
* lkundrak (66)
* shinobi (65)
* sundaram (64)
* spot (62)
* pravins (62)
* markmc (62)
* thomasj (61)
* smooge (60)
* fab (59)
================================================================================
Fedora 13
================================================================================
* 3562 updates
* 3065 stable updates
* 427 testing updates
* 62 pending updates
* 8 obsolete updates
* 2371 bugfix updates (66.56%)
* 745 enhancement updates (20.92%)
* 89 security updates (2.50%)
* 357 newpackage updates (10.02%)
* 410 critical path updates (11.51%)
* 1155 updates received feedback (32.43%)
* 12120 +0 comments
* 2477 +1 comments
* 155 -1 comments
* 595 unique authenticated karma submitters
* 133 anonymous users gave feedback (1.57%)
* 2261 out of 3562 updates went through testing (63.48%)
* 1317 testing updates were pushed *without* karma (58.25%)
* 21 critical path updates pushed *without* karma
* Time spent in testing:
* mean = 11 days
* median = 9 days
* mode = 7 days
* 4 updates automatically unpushed due to karma (0.11%)
* 0 of which were critical path updates
* 231 updates automatically pushed due to karma (6.49%)
* 2 of which were critical path updates
* Time spent in testing of updates that were pushed by karma:
* mean = 11 days
* median = 7 days
* mode = 7 days
* Time spent in testing of updates that were unpushed by karma:
* mean = 9 days
* median = 5 days
* mode = 5 days
* 2445 packages updated (top 10 shown)
* selinux-policy: 13
* jd: 12
* openoffice.org: 12
* gdb: 12
* ibus-pinyin: 11
* nautilus: 10
* kernel: 10
* evolution: 9
* libfm: 9
* libmx: 9
================================================================================
Fedora 12
================================================================================
* 4844 updates
* 4291 stable updates
* 371 testing updates
* 113 pending updates
* 69 obsolete updates
* 2905 bugfix updates (59.97%)
* 1054 enhancement updates (21.76%)
* 201 security updates (4.15%)
* 684 newpackage updates (14.12%)
* 407 critical path updates (8.40%)
* 960 updates received feedback (19.82%)
* 16311 +0 comments
* 1899 +1 comments
* 554 -1 comments
* 758 unique authenticated karma submitters
* 576 anonymous users gave feedback (5.33%)
* 2873 out of 4844 updates went through testing (59.31%)
* 2138 testing updates were pushed *without* karma (74.42%)
* 188 critical path updates pushed *without* karma
* Time spent in testing:
* mean = 14 days
* median = 13 days
* mode = 17 days
* 12 updates automatically unpushed due to karma (0.25%)
* 4 of which were critical path updates
* 133 updates automatically pushed due to karma (2.75%)
* 13 of which were critical path updates
* Time spent in testing of updates that were pushed by karma:
* mean = 11 days
* median = 7 days
* mode = 7 days
* Time spent in testing of updates that were unpushed by karma:
* mean = 9 days
* median = 5 days
* mode = 5 days
* 2902 packages updated (top 10 shown)
* qbittorrent: 25
* gdb: 25
* selinux-policy: 22
* kernel: 15
* xorg-x11-server: 14
* ibus: 13
* jd: 13
* abrt: 11
* gvfs: 11
* gtk2: 11
================================================================================
Fedora 11
================================================================================
* 6987 updates
* 6381 stable updates
* 183 testing updates
* 99 pending updates
* 324 obsolete updates
* 3649 bugfix updates (52.23%)
* 1566 enhancement updates (22.41%)
* 350 security updates (5.01%)
* 1422 newpackage updates (20.35%)
* 383 critical path updates (5.48%)
* 729 updates received feedback (10.43%)
* 23427 +0 comments
* 1197 +1 comments
* 448 -1 comments
* 782 unique authenticated karma submitters
* 481 anonymous users gave feedback (3.58%)
* 4129 out of 6987 updates went through testing (59.10%)
* 3620 testing updates were pushed *without* karma (87.67%)
* 278 critical path updates pushed *without* karma
* Time spent in testing:
* mean = 15 days
* median = 14 days
* mode = 17 days
* 7 updates automatically unpushed due to karma (0.10%)
* 0 of which were critical path updates
* 64 updates automatically pushed due to karma (0.92%)
* 11 of which were critical path updates
* Time spent in testing of updates that were pushed by karma:
* mean = 11 days
* median = 7 days
* mode = 7 days
* Time spent in testing of updates that were unpushed by karma:
* mean = 9 days
* median = 5 days
* mode = 5 days
* 3787 packages updated (top 10 shown)
* libguestfs: 30
* jd: 24
* selinux-policy: 23
* kdebase-workspace: 19
* kernel: 18
* gdb: 16
* dovecot: 16
* qemu: 16
* kdebase-runtime: 16
* kdenetwork: 16
================================================================================
Fedora EPEL 5
================================================================================
* 1572 updates
* 1255 stable updates
* 198 testing updates
* 43 pending updates
* 76 obsolete updates
* 734 bugfix updates (46.69%)
* 236 enhancement updates (15.01%)
* 93 security updates (5.92%)
* 509 newpackage updates (32.38%)
* 20 critical path updates (1.27%)
* 103 updates received feedback (6.55%)
* 6076 +0 comments
* 156 +1 comments
* 19 -1 comments
* 243 unique authenticated karma submitters
* 41 anonymous users gave feedback (1.22%)
* 1176 out of 1572 updates went through testing (74.81%)
* 1092 testing updates were pushed *without* karma (92.86%)
* 19 critical path updates pushed *without* karma
* Time spent in testing:
* mean = 24 days
* median = 18 days
* mode = 16 days
* 0 updates automatically unpushed due to karma (0.00%)
* 0 of which were critical path updates
* 10 updates automatically pushed due to karma (0.64%)
* 0 of which were critical path updates
* Time spent in testing of updates that were pushed by karma:
* mean = 11 days
* median = 7 days
* mode = 7 days
* Time spent in testing of updates that were unpushed by karma:
* mean = 9 days
* median = 5 days
* mode = 5 days
* 1060 packages updated (top 10 shown)
* libguestfs: 26
* znc: 10
* vrq: 8
* cherokee: 8
* 389-ds-base: 8
* viewvc: 8
* 389-admin: 7
* pki-ca: 7
* wordpress-mu: 7
* Django: 7
================================================================================
Fedora EPEL 4
================================================================================
* 447 updates
* 359 stable updates
* 40 testing updates
* 11 pending updates
* 37 obsolete updates
* 222 bugfix updates (49.66%)
* 68 enhancement updates (15.21%)
* 40 security updates (8.95%)
* 117 newpackage updates (26.17%)
* 5 critical path updates (1.12%)
* 11 updates received feedback (2.46%)
* 1592 +0 comments
* 11 +1 comments
* 2 -1 comments
* 85 unique authenticated karma submitters
* 2 anonymous users gave feedback (0.24%)
* 320 out of 447 updates went through testing (71.59%)
* 311 testing updates were pushed *without* karma (97.19%)
* 5 critical path updates pushed *without* karma
* Time spent in testing:
* mean = 18 days
* median = 16 days
* mode = 16 days
* 0 updates automatically unpushed due to karma (0.00%)
* 0 of which were critical path updates
* 1 updates automatically pushed due to karma (0.22%)
* 0 of which were critical path updates
* Time spent in testing of updates that were pushed by karma:
* mean = 11 days
* median = 7 days
* mode = 7 days
* Time spent in testing of updates that were unpushed by karma:
* mean = 9 days
* median = 5 days
* mode = 5 days
* 313 packages updated (top 10 shown)
* cherokee: 8
* globus-common: 7
* R: 6
* voms: 6
* globus-gsi-proxy-ssl: 5
* globus-openssl-module: 5
* globus-gsi-proxy-core: 5
* bitlbee: 5
* flashrom: 5
* viewvc: 5
6:46 p.m.
Luke Macken wrote:
This report definitely conveys the shortcomings in our testing,
however,
it does show us improving with each release. For Fedora 13, we implemented
the No Frozen Rawhide process with improved Critical Path policies, which
were definitely a success. With these enhanced procedures, along with the
upcoming implementation of AutoQA and the new Package update acceptance
criteria
(https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
think we'll see these numbers drastically improve in the future.
Only because those numbers are taylored towards that very process (they
measure the exact same things that process is going to enforce) and do not
reflect the actual quality of the packages in any way.
You can make really anything a "success" by measuring the very symptoms of
the process and calling them a metric of "quality".
The reasons for which Bodhi karma (especially in its current incarnation) is
a completely broken indicator of quality have been pointed out in several
past threads.
Kevin Kofler
7:41 p.m.
On Tue, Jun 8, 2010 at 7:46 PM, Kevin Kofler <kevin.kofler(a)chello.at> wrote:
Luke Macken wrote:
> This report definitely conveys the shortcomings in our testing, however,
> it does show us improving with each release. For Fedora 13, we implemented
> the No Frozen Rawhide process with improved Critical Path policies, which
> were definitely a success. With these enhanced procedures, along with the
> upcoming implementation of AutoQA and the new Package update acceptance
> criteria
> (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
> think we'll see these numbers drastically improve in the future.
Only because those numbers are taylored towards that very process (they
measure the exact same things that process is going to enforce) and do not
reflect the actual quality of the packages in any way.
You can make really anything a "success" by measuring the very symptoms of
the process and calling them a metric of "quality".
The reasons for which Bodhi karma (especially in its current incarnation) is
a completely broken indicator of quality have been pointed out in several
past threads.
Kevin Kofler
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
I'll have to agree with Kevin. I can't how any of those numbers
represent the quality of anything.
8:48 p.m.
On Tue, Jun 8, 2010 at 6:41 PM, Brandon Lozza <brandon(a)pwnage.ca> wrote:
On Tue, Jun 8, 2010 at 7:46 PM, Kevin Kofler
<kevin.kofler(a)chello.at> wrote:
> Luke Macken wrote:
>> This report definitely conveys the shortcomings in our testing, however,
>> it does show us improving with each release. For Fedora 13, we implemented
>> the No Frozen Rawhide process with improved Critical Path policies, which
>> were definitely a success. With these enhanced procedures, along with the
>> upcoming implementation of AutoQA and the new Package update acceptance
>> criteria
>> (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
>> think we'll see these numbers drastically improve in the future.
>
> Only because those numbers are taylored towards that very process (they
> measure the exact same things that process is going to enforce) and do not
> reflect the actual quality of the packages in any way.
>
> You can make really anything a "success" by measuring the very symptoms of
> the process and calling them a metric of "quality".
>
> The reasons for which Bodhi karma (especially in its current incarnation) is
> a completely broken indicator of quality have been pointed out in several
> past threads.
I'll have to agree with Kevin. I can't how any of those numbers
represent the quality of anything.
Well the only person I see mentioning quality is Kevin. And for some
reason he is expecting it immediately when Luke said 3 things:
1) This was what he could initially see.
2) He thinks he sees improvement of what was measured, but he is
interested in what would be better indicators.
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
--
Stephen J Smoogen.
“The core skill of innovators is error recovery, not failure avoidance.”
Randy Nelson, President of Pixar University.
"We have a strategic plan. It's called doing things.""
— Herb Kelleher, founder Southwest Airlines
12:18 a.m.
Stephen John Smoogen wrote:
Well the only person I see mentioning quality is Kevin. And for some
reason he is expecting it immediately
You can't claim that there are "shortcomings in our testing" (the exact
words Luke used!) without a metric of quality. A shortcoming in testing
means less testing was done than needed. If the packages have good quality,
that means more testing is NOT needed, no matter what the actual amount of
testing was.
In addition, Luke is speculating about those numbers "improving" with
upcoming policy changes. But the word "improve" only makes sense for a
quantity which measures some kind of quality. Pure numbers without a
judgement of value can only "increase" or "decrease", not
"improve".
Kevin Kofler
12:28 a.m.
On 06/09/2010 10:48 AM, Kevin Kofler wrote:
Stephen John Smoogen wrote:
> Well the only person I see mentioning quality is Kevin. And for some
> reason he is expecting it immediately
>
You can't claim that there are "shortcomings in our testing" (the exact
words Luke used!) without a metric of quality. A shortcoming in testing
means less testing was done than needed. If the packages have good quality,
that means more testing is NOT needed, no matter what the actual amount of
testing was.
Packages without any testing is of unknown quality. If someone doesn't
test them before pushing, users are unknowingly playing that role.
Rahul
6:42 a.m.
On Wed, Jun 09, 2010 at 10:58:15AM +0530, Rahul Sundaram wrote:
On 06/09/2010 10:48 AM, Kevin Kofler wrote:
> Stephen John Smoogen wrote:
>
>> Well the only person I see mentioning quality is Kevin. And for some
>> reason he is expecting it immediately
>>
> You can't claim that there are "shortcomings in our testing" (the exact
> words Luke used!) without a metric of quality. A shortcoming in testing
> means less testing was done than needed. If the packages have good quality,
> that means more testing is NOT needed, no matter what the actual amount of
> testing was.
>
Packages without any testing is of unknown quality. If someone doesn't
test them before pushing, users are unknowingly playing that role.
That is true if you are making the assumption that the package maintainer did
no testing themselves. I would hope that isn't the common case.
josh
7:36 a.m.
On 06/09/2010 05:12 PM, Josh Boyer wrote:
That is true if you are making the assumption that the package
maintainer did
no testing themselves. I would hope that isn't the common case.
There are well known cases of that happening. Kevin has been public
about his position on that. Hopefully we can automatically catch and
prevent the obvious breakages soon.
Rahul
8:46 a.m.
On Wednesday 09 June 2010, Rahul Sundaram wrote:
There are well known cases of that happening. Kevin has been public
about his position on that. Hopefully we can automatically catch and
prevent the obvious breakages soon.
For the record, I did test (on one release) the latest bunch of updates I
pushed before pushing them out to stable. But sometimes a fix is very urgent
and trivial so I'll just take anyone's word that it works (or if the change is
really really trivial, e.g. a fixed Requires which can be verified to be
satisfied just by looking at the Rawhide report and noticing there's no broken
dependency, not even that) and issue it directly to stable (and I think that's
a good thing and the new process which will make this a PITA is going to
degrade our overall package quality by delaying urgent fixes). And for stuff
like the KDE updates, I know many people, including other KDE SIG members,
test them, so I usually don't spend my time testing those updates as well, the
testing feedback we get is largely sufficient.
Kevin Kofler
11:57 a.m.
On 06/09/2010 01:18 AM, Kevin Kofler wrote:
If the packages have good quality, that means more testing is NOT
needed, no matter what the actual amount of testing was.
Apart from the Bodhi issue, I disagree with the logic of your statement.
Quality doesn't exist (or at least is not provable) in an absence of
testing; as the philosopher DeNiro says in Ronin, "If there is any
doubt, there is no doubt".
On the contrary, there's the testing philosophy that says that if your
tests start showing less faults, you should _increase_ and diversify
testing, because you have simply exhausted previous test coverage profile.
12:53 a.m.
On Wed, 2010-06-09 at 01:46 +0200, Kevin Kofler wrote:
Luke Macken wrote:
> This report definitely conveys the shortcomings in our testing, however,
> it does show us improving with each release.
By 'shortcomings in our testing', I mean, 'shortcomings in the process
by which we currently use bodhi for "testing" updates'.
For Fedora 13, we implemented
> the No Frozen Rawhide process with improved Critical Path policies, which
> were definitely a success. With these enhanced procedures, along with the
> upcoming implementation of AutoQA and the new Package update acceptance
> criteria
> (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
> think we'll see these numbers drastically improve in the future.
Only because those numbers are taylored towards that very process (they
measure the exact same things that process is going to enforce) and do not
reflect the actual quality of the packages in any way.
You can make really anything a "success" by measuring the very symptoms of
the process and calling them a metric of "quality".
These metrics do not reflect the actual quality of our updates, the
policy or process behind them, or even the quality of testing that they
endured before being released. It is simply a measurement of how we
have been interacting with this specific piece of infrastructure.
By "success" I mean that I felt we were successful in drafting,
implementing, deploying, and utilizing the mentioned policies as
expected, and the results show increased community engagement.
Attempting to quantify the quality of these community interactions is
another story. As to whether or not the policies behind the processes
have helped or hindered the release quality overall, that has yet to be
determined.
The reasons for which Bodhi karma (especially in its current
incarnation) is
a completely broken indicator of quality have been pointed out in several
past threads.
I'm well aware, and I agree that the current bodhi karma implementation
does not convey an effective measurement of update quality. The problem
is not a lack of good ideas for improvement, but rather a lack of
developers to help improve.
luke
1:38 a.m.
Luke Macken wrote:
By "success" I mean that I felt we were successful in
drafting,
implementing, deploying, and utilizing the mentioned policies as
expected, and the results show increased community engagement.
This definition of "success" does not match mine nor the one you'll find in
a dictionary. So your terminology is misleading.
Kevin Kofler
1:54 a.m.
On Wed, 2010-06-09 at 08:38 +0200, Kevin Kofler wrote:
Luke Macken wrote:
> By "success" I mean that I felt we were successful in drafting,
> implementing, deploying, and utilizing the mentioned policies as
> expected, and the results show increased community engagement.
This definition of "success" does not match mine nor the one you'll find in
a dictionary. So your terminology is misleading.
Really, Kevin? We're digressing to a dictionary battle?
Fine, I'll play. First definition in the dictionary[0]: "an event that
accomplishes its intended purpose".
...which is exactly what I meant to being with.
luke
[0]: http://www.google.com/search?q=define%3Asuccess
2:10 a.m.
On 06/09/2010 08:54 AM, Luke Macken wrote:
On Wed, 2010-06-09 at 08:38 +0200, Kevin Kofler wrote:
> Luke Macken wrote:
>> By "success" I mean that I felt we were successful in drafting,
>> implementing, deploying, and utilizing the mentioned policies as
>> expected, and the results show increased community engagement.
>
> This definition of "success" does not match mine nor the one you'll
find in
> a dictionary. So your terminology is misleading.
Really, Kevin? We're digressing to a dictionary battle?
Fine, I'll play. First definition in the dictionary[0]: "an event that
accomplishes its intended purpose".
Exactly. Your definition differs from
Kevin's (and mine).
...which is exactly what I meant to being with.
To me, your definition of success is "compliance with *your* process".
Whether this process is suitable to improve package quality, whether the
technical system behind it is a good approach and whether your approach
actually improves package quality or is mere bureaucray is highly
questionable.
That said, all you demonstrated is your system not being entirely
broken, but I don't see any "success" related to QA in your statistic.
Ralf
11:48 a.m.
On Wed, 2010-06-09 at 09:10 +0200, Ralf Corsepius wrote:
On 06/09/2010 08:54 AM, Luke Macken wrote:
> On Wed, 2010-06-09 at 08:38 +0200, Kevin Kofler wrote:
>> Luke Macken wrote:
>>> By "success" I mean that I felt we were successful in drafting,
>>> implementing, deploying, and utilizing the mentioned policies as
>>> expected, and the results show increased community engagement.
>>
>> This definition of "success" does not match mine nor the one
you'll find in
>> a dictionary. So your terminology is misleading.
>
> Really, Kevin? We're digressing to a dictionary battle?
>
> Fine, I'll play. First definition in the dictionary[0]: "an event that
> accomplishes its intended purpose".
Exactly. Your definition differs from Kevin's (and mine).
Neither of you have mentioned "your" definition of the word
"success".
Care to enlighten us?
> ...which is exactly what I meant to being with.
To me, your definition of success is "compliance with *your* process".
If by "your process" you mean "the processes created by the Fedora
Community". I have had almost no say in the new updates criteria, nor
am I on any rubber-stamping committee to approve it. I am, however, one
of the *few* developers who actually works on bodhi, and I have a vested
interested in improving it for the greater good of the community. Now,
if the policies that are being approved do not actually benefit the
greater good of the community, we have bigger problems.
Whether this process is suitable to improve package quality, whether
the
technical system behind it is a good approach and whether your approach
actually improves package quality or is mere bureaucray is highly
questionable.
Yes, all of those are highly questionable, with regard to this "quality"
metric.
To improve a process we must first observe how it is currently being
utilized.
What comes out of bodhi is what the maintainers put into it.
Aside from that, we've been essentially been using it to "croud-source"
QA. As expected, this is far from perfect, but it's a start until we
have code in place that can perform rigorous and comprehensive testing.
That said, all you demonstrated is your system not being entirely
broken, but I don't see any "success" related to QA in your statistic.
The numbers show an increase in community interaction. More eyes are
looking at the updates and providing feedback. Considering we're
croud-sourcing QA, an increase in participation is called a success.
luke
9:59 p.m.
Luke Macken wrote:
Neither of you have mentioned "your" definition of the word
"success".
Care to enlighten us?
Success is the achievement of a worthwhile goal. If the original goal which
was set is worthless, "succeeding" at it is meaningless.
Now, if the policies that are being approved do not actually benefit
the
greater good of the community, we have bigger problems.
Yet that's exactly the problem we're having. :-(
We're being pushed towards more and more bureaucracy such as FORCED use of
updates-testing without drawing any lessons from our project history:
* Fedora Extras had no testing requirements, in fact it didn't even HAVE a
testing repository. All builds were pushed directly to the stable
repository. It worked great, to the point where Core merged with it.
* Fedora Legacy had stringent QA requirements very similar to the ones which
are about to be enforced now: updates could not move out of testing without
either a minimum amount of positive feedback or a timeout (which had to be
introduced because otherwise packages would never move out of testing).
Feedback was counted separately for each distro version, just as our new
policy will do it, which made it nearly impossible to get the required
positive feedback for some releases. Over time, the amount of required
positive feedback and the timeout had to be reduced several times because
the system was just not working. In the end, Fedora Legacy failed, because
it was impossible to deliver security updates in a timely manner with that
kind of QA requirements.
So why are we now going to use the Fedora Legacy model over the Fedora
Extras one? Does FESCo really want Fedora to fail?
Kevin Kofler
5:31 a.m.
On Thu, Jun 10, 2010 at 04:59:30AM +0200, Kevin Kofler wrote:
Luke Macken wrote:
> Neither of you have mentioned "your" definition of the word
"success".
> Care to enlighten us?
Success is the achievement of a worthwhile goal. If the original goal which
was set is worthless, "succeeding" at it is meaningless.
There are a number of people who disagree with the value of that goal, including
most of the past and current FESCo.
It is very clear you don't agree with the goal. That in and of itself does
not make it worthless. Luke's changes to bodhi are also not meaningless. He
is implementing towards the goal/process he was given and he's doing it rather
well.
You are devaluing the work of a Fedora contributor and being generally
poisonous. I suggest you stop this particular line of attack.
josh
10:10 a.m.
Josh Boyer wrote:
There are a number of people who disagree with the value of that
goal,
including most of the past and current FESCo.
You conveniently ignored the part of the mail where I pointed out WHY that
goal is broken. Why are we trying to emulate the failed Fedora Legacy
process rather than the successful Fedora Extras one? Why can't we learn
from our past?
Kevin Kofler
10:24 a.m.
On Thu, Jun 10, 2010 at 05:10:36PM +0200, Kevin Kofler wrote:
Josh Boyer wrote:
> There are a number of people who disagree with the value of that goal,
> including most of the past and current FESCo.
You conveniently ignored the part of the mail where I pointed out WHY that
goal is broken.
No, that was intentional. I was commenting directly on your abuse of someone
that was _told_ to implement towards a specific goal. Your questioning of that
goal, while personally annoying to me, is certainly acceptable. Your assertion
that someone's hard work and time on an implementation of that goal is
"worthless" and "meaningless" is not. It's hurtful, poisonous and
entirely
unwelcome.
josh
4:48 p.m.
* Kevin Kofler [10/06/2010 17:21] :
Why are we trying to emulate the failed Fedora Legacy
process rather than the successful Fedora Extras one? Why can't we learn
from our past?
I think you're rather quick to conclude that Extras succeded because it
allowed maintainers to push to stable directly and that Legacy failed for
the lack of same.
Others have given other reasons for Legacy's failure (these have ranged
to lack of infrastructure to lack of community involvement to conspiracy
theories involving Red Hat), so if stringent QA requirements really were
the cause for Legacy's failure, it's probably best to demonstrate this
before using it as an arguement in your assertions.
Emmanuel
7:28 p.m.
On Thu, 10 Jun 2010 04:59:30 +0200
Kevin Kofler <kevin.kofler(a)chello.at> wrote:
> Now, if the policies that are being approved do not actually
> benefit the greater good of the community, we have bigger problems.
Yet that's exactly the problem we're having. :-(
....snip...
So why are we now going to use the Fedora Legacy model over the
Fedora Extras one? Does FESCo really want Fedora to fail?
I think you are presenting a false dichotomy here.
There are many reasons why the current fedora is dissimilar to both
Fedora extras and Fedora Legacy. I think it's worth trying to give our
users a more stable updates stream. If it turns out that we don't get
this from the things we are trying to do, we will learn from it and try
something else.
I think we have had this discussion before, so I will leave it at that.
kevin
8:20 p.m.
On Tue, 2010-06-08 at 16:51 -0400, Luke Macken wrote:
================================================================================
Fedora 13
================================================================================
* 3562 updates
* 3065 stable updates
* 427 testing updates
* 62 pending updates
* 8 obsolete updates
Hey Luke,
are these the numbers of F13 updates since we branched, or since the F13
release ? The numbers certainly look like the former, but I think the
latter would be much more interesting.
Matthias
1:11 a.m.
On Tue, 2010-06-08 at 21:20 -0400, Matthias Clasen wrote:
On Tue, 2010-06-08 at 16:51 -0400, Luke Macken wrote:
> ================================================================================
> Fedora 13
> ================================================================================
>
> * 3562 updates
> * 3065 stable updates
> * 427 testing updates
> * 62 pending updates
> * 8 obsolete updates
Hey Luke,
are these the numbers of F13 updates since we branched, or since the F13
release ? The numbers certainly look like the former, but I think the
latter would be much more interesting.
These numbers are since we branched for F13.
luke
1:04 a.m.
On Tue, 2010-06-08 at 16:51 -0400, Luke Macken wrote:
================================================================================
Fedora 13
================================================================================
* 231 updates automatically pushed due to karma (6.49%)
* 2 of which were critical path updates
So I thought that this last metric would take into account critical path
updates that were approved under the new policy, but it did not. The
actual metric is for Fedora 13 is:
* 333 critical path updates approved
2:35 a.m.
On 06/08/2010 10:51 PM, Luke Macken wrote:
I recently wrote some code to generate detailed statistics of
Fedora& EPEL updates within bodhi. Eventually this will be auto-generated and exposed
within bodhi itself, but for now here are the initial metrics.
This report definitely conveys the shortcomings in our testing, however, it does show us
improving with each release. For Fedora 13, we implemented the No Frozen Rawhide process
with improved Critical Path policies, which were definitely a success. With these enhanced
procedures, along with the upcoming implementation of AutoQA and the new Package update
acceptance criteria (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
think we'll see these numbers drastically improve in the future.
I can't agree that this update policy is success (in any dictionary).
Since I'm forced to wait
two weeks for pushing into stable, I have more tickets about packages
that I've already fixed. Users want fixes immediately, they are not
interested in some processes. Many users don't even have FAS account and
they want fixes and updates now. Not in next release as offer enterprise
solution.
Other thing is that I'm forgetting packages in bodhi, because I believed
until yesterday that updates will be pushed after two weeks
automatically. Not sure whether policy changed or it's a bug [1].
Karma in bodhi updates doesn't speak about quality of updates at all. At
least my packages which has more dependencies are pushed without
testing, and less important are "tested" even testers don't know what
does it do [2]. For example here were testers happy because it was
updated without any warning message, but they don't know if it's working
or not.
I'm looking forward to qa project that could really improve quality -
like testing updates, which fixed bugs with bugzilla and testers really
reproduce them before and not after. Otherwise is bodhi good only for
checking update by yum without errors.
Best regards,
Marcela
[1] https://admin.fedoraproject.org/updates/perl-LDAP-0.40-1.fc13
[2] https://admin.fedoraproject.org/updates/perl-version-0.82-1.fc13
8:54 a.m.
Marcela Mašláňová wrote:
I can't agree that this update policy is success (in any
dictionary).
Since I'm forced to wait
two weeks for pushing into stable, I have more tickets about packages
that I've already fixed. Users want fixes immediately, they are not
interested in some processes. Many users don't even have FAS account and
they want fixes and updates now. Not in next release as offer enterprise
solution.
Exactly my point. But FESCo just won't listen. :-(
Kevin Kofler
8:59 a.m.
On Wed, 9 Jun 2010, Kevin Kofler wrote:
Marcela Mašláňová wrote:
> I can't agree that this update policy is success (in any dictionary).
> Since I'm forced to wait
> two weeks for pushing into stable, I have more tickets about packages
> that I've already fixed. Users want fixes immediately, they are not
> interested in some processes. Many users don't even have FAS account and
> they want fixes and updates now. Not in next release as offer enterprise
> solution.
Exactly my point. But FESCo just won't listen. :-(
FESCo listened. They did not agree with you.
-sv
11:03 a.m.
On Wed, 2010-06-09 at 09:35 +0200, Marcela Mašláňová wrote:
On 06/08/2010 10:51 PM, Luke Macken wrote:
> I recently wrote some code to generate detailed statistics of Fedora& EPEL
updates within bodhi. Eventually this will be auto-generated and exposed within bodhi
itself, but for now here are the initial metrics.
>
> This report definitely conveys the shortcomings in our testing, however, it does
show us improving with each release. For Fedora 13, we implemented the No Frozen Rawhide
process with improved Critical Path policies, which were definitely a success. With these
enhanced procedures, along with the upcoming implementation of AutoQA and the new Package
update acceptance criteria
(https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I think we'll see
these numbers drastically improve in the future.
>
I can't agree that this update policy is success (in any
dictionary).
If you can't agree that we, as a community, were successful in
accomplishing our intended purpose of implementing the No Frozen Rawhide
& Critical Path package processes that we drafted, then you're not
understanding the use of the word.
Again, the "update policy" that is currently rubber stamped does not
match what is enforced by the process. The "package update acceptance
criteria"[0] has been "approved", but not yet implemented. So, if you
feel like you have suggestions or some sort of constructive criticism to
offer, I recommend bringing it up with FESCo.
[0]: https://fedoraproject.org/wiki/Package_update_acceptance_criteria
Since I'm forced to wait two weeks for pushing into stable
For EPEL updates, yes, that is their policy. For Fedora, there is
nothing stopping you from pushing your non-critpath updates straight to
stable.
Other thing is that I'm forgetting packages in bodhi, because I
believed
until yesterday that updates will be pushed after two weeks
automatically. Not sure whether policy changed or it's a bug [1].
Bodhi has never done this, ever.
According to the new acceptance critera, updates will have to "spend
some minimum amount of time in updates-testing, currently one week".
Now, as to whether or not bodhi should auto-push after that week, that
I'm not quite sure.
Karma in bodhi updates doesn't speak about quality of updates at
all. At
least my packages which has more dependencies are pushed without
testing, and less important are "tested" even testers don't know what
does it do [2]. For example here were testers happy because it was
updated without any warning message, but they don't know if it's working
or not.
Right, karma does not equate quality. I don't think anyone ever claimed
that it did?
I'm looking forward to qa project that could really improve
quality -
like testing updates, which fixed bugs with bugzilla and testers really
reproduce them before and not after. Otherwise is bodhi good only for
checking update by yum without errors.
Yes, I'm looking forward to AutoQA as well. We all are.
The majority of karma is "this ran without exploding", which is far from
sufficient testing, but it's still valuable information.
luke
11:52 a.m.
On Wed, 09 Jun 2010 12:03:50 -0400, Luke wrote:
According to the new acceptance critera, updates will have to
"spend
some minimum amount of time in updates-testing, currently one week".
Now, as to whether or not bodhi should auto-push after that week, that
I'm not quite sure.
Rest assured that you are NOT expected to auto-push anything unless
packagers can opt-in or opt-out from such a process.
2:54 p.m.
On Wed, Jun 9, 2010 at 6:03 PM, Luke Macken <lmacken(a)redhat.com> wrote:
The majority of karma is "this ran without exploding",
which is far from
sufficient testing, but it's still valuable information.
Indeed. And to be honest, i can't agree with all that whining that
karma isn't useful. It means the user tried/used the software with his
use cases and in case of +1 it does work for him. What more can i
expect from a regular user/unpaid kind tester? I'm extremely thankful
that he uses the testing repo *and* report his experience back.
My thank you goes out to every tester. And a big thank you for
everyone who also spends the time to report it back via karma in
bodhi.
Thank you for the numbers you provided Luke.
--
LG Thomas
Dubium sapientiae initium
6:02 p.m.
On Wed, 2010-06-09 at 09:35 +0200, Marcela Mašláňová wrote:
that I've already fixed. Users want fixes immediately, they are
not
interested in some processes. Many users don't even have FAS account and
Users also want regressions not to happen (see how much belly-aching
there is over the nss multilib issue, or any of the famous historical
regressions - udev etc). These two things are not mutually compatible.
[1] https://admin.fedoraproject.org/updates/perl-LDAP-0.40-1.fc13
[2] https://admin.fedoraproject.org/updates/perl-version-0.82-1.fc13
perl-version is in the critpath, but perl-LDAP is not; there would be no
enforcement of update karma for perl-LDAP. There is no enforced waiting
period for updates at present, the only criterion that's ever been
enforced is that, during the pre-release phase, F13 updates required +1
from someone in rel-eng or qa and another +1 from anyone to be pushed.
This has not been the case since F13's release, as we're working on the
permanent process; in the meantime, it works as it always has
historically (you can push anything you want at any time).
Are you sure your updates didn't just get held up in the pre-release
freeze? There's nothing new about that, we've been freezing immediately
prior to release for years. But looking at the dates, I'm fairly sure
that at the time you did the perl-LDAP update, you could have pushed it
to stable without any problems, if you'd tried. Your perl-version update
got the required threshold of feedback exactly one week after it was
pushed to testing (submitted 2010-05-11, pushed 2010-05-12, second +1
received 2010-05-19); you could have submitted it to stable any time
after that second +1 was received.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
12:53 a.m.
On Thu, 2010-06-10 at 05:03 +0200, Kevin Kofler wrote:
Adam Williamson wrote:
> Users also want regressions not to happen
This is exactly why we need quick fixes, i.e. direct stable pushes: to be
able to push a fixed update IMMEDIATELY if somebody caught a regression.
We danced that tango last time, I'm not getting baited into it again.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
2:51 a.m.
On Tue, 8 Jun 2010 16:51:36 -0400 (EDT), Luke wrote:
=====================================================
Bodhi Statistics Report (Generated on June 8th, 2010)
=====================================================
Out of 17412 total updates, 2958 received feedback (16.99%)
Out of 1045 total unique karma submitters, the top 30 are:
Just for the record and some fairness, the top Fedora 11 testers
according to https://admin.fedoraproject.org/updates/metrics/?release=F11
alexlan
mschwendt
mrunge
dsd
dcottle
lmacken
mooninite
cwickert
from times when the fedora-easy-karma script wasn't available and
didn't make it possible to mass-vote on updates.
6:46 a.m.
On Wed, Jun 09, 2010 at 09:51:59AM +0200, Michael Schwendt wrote:
On Tue, 8 Jun 2010 16:51:36 -0400 (EDT), Luke wrote:
> =====================================================
> Bodhi Statistics Report (Generated on June 8th, 2010)
> =====================================================
>
> Out of 17412 total updates, 2958 received feedback (16.99%)
> Out of 1045 total unique karma submitters, the top 30 are:
Just for the record and some fairness, the top Fedora 11 testers
according to https://admin.fedoraproject.org/updates/metrics/?release=F11
alexlan
mschwendt
mrunge
dsd
dcottle
lmacken
mooninite
cwickert
from times when the fedora-easy-karma script wasn't available and
didn't make it possible to mass-vote on updates.
Thanks for this. I was wondering about exactly that.
josh
5:59 p.m.
On Wed, 2010-06-09 at 09:51 +0200, Michael Schwendt wrote:
from times when the fedora-easy-karma script wasn't available
and
didn't make it possible to mass-vote on updates.
easy-karma doesn't allow you to mass vote, you still have to vote on
each update individually. it simply streamlines the process somewhat, so
you don't spend most of the time poking through the Bodhi web interface.
just thought I'd note it...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
4:19 a.m.
On Tue, Jun 08, 2010 at 04:51:36PM -0400, Luke Macken wrote:
You can find the code that generates these statistics here:
https://fedorahosted.org/bodhi/browser/bodhi/tools/metrics.py
https://fedorahosted.org/bodhi/browser/bodhi/tools/log_stats.py. If you have any ideas or
suggestions for different types of metrics to generate, or if you find any bugs in my
code, please let me know.
* 595 unique authenticated karma submitters
I think it might be interesting to calculate this number for different
timeframes, e.g. last day, last week, last month, since it was released,
since F(N+1) was released. Maybe also the number of karma submitters
that submitted at least X comments for each week the release is
available. This would imho allow more precise to estimate how many
active testers are there.
Regards
Till
5:36 p.m.
On Tue, 2010-06-08 at 16:51 -0400, Luke Macken wrote:
I recently wrote some code to generate detailed statistics of Fedora
& EPEL updates within bodhi. Eventually this will be auto-generated and exposed within
bodhi itself, but for now here are the initial metrics.
This report definitely conveys the shortcomings in our testing, however, it does show us
improving with each release. For Fedora 13, we implemented the No Frozen Rawhide process
with improved Critical Path policies, which were definitely a success. With these enhanced
procedures, along with the upcoming implementation of AutoQA and the new Package update
acceptance criteria (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
think we'll see these numbers drastically improve in the future.
You can find the code that generates these statistics here:
https://fedorahosted.org/bodhi/browser/bodhi/tools/metrics.py
https://fedorahosted.org/bodhi/browser/bodhi/tools/log_stats.py. If you have any ideas or
suggestions for different types of metrics to generate, or if you find any bugs in my
code, please let me know.
Great stuff Luke, very interesting, and pleasing to see the improvements
in the f13 process in cold hard numbers - thanks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
5108
days inactive
5111
days old
37 comments
17 participants
participants (17)
-
Adam Williamson -
Brandon Lozza -
Emmanuel Seyman -
Josh Boyer -
Kevin Fenzi -
Kevin Kofler -
Luke Macken -
Marcela Mašláňová -
Matthias Clasen -
Michael Schwendt -
Przemek Klosowski -
Rahul Sundaram -
Ralf Corsepius -
Seth Vidal -
Stephen John Smoogen -
Thomas Janssen -
Till Maas