Build failed in Hudson: bodhi-py2.4 #43
by Luke Macken
See <http://ci.csh.rit.edu/job/bodhi-py2.4/43/changes>
Changes:
[Luke Macken] Update our admin_groups to include security_respons & proventesters
[Luke Macken] Store stderr from mash when it fails...
[Luke Macken] Only write out stderr if it is not None
------------------------------------------
Started by an SCM change
Checkout:workspace / <http://ci.csh.rit.edu/job/bodhi-py2.4/ws/> - hudson.remoting.LocalChannel@d5cdab
Last Built Revision: Revision 78195e8fec65f29801bbaca3aa8557f7e44f8a86 (origin/master)
Checkout:workspace / <http://ci.csh.rit.edu/job/bodhi-py2.4/ws/> - hudson.remoting.LocalChannel@d5cdab
GitAPI created
Env: _=/usr/bin/java
Env: BUILD_ID=2010-05-06_15-01-47
Env: BUILD_NUMBER=43
Env: BUILD_TAG=hudson-bodhi-py2.4-43
Env: BUILD_URL=http://ci.csh.rit.edu/job/bodhi-py2.4/43/
Env: CLASSPATH=
Env: EXECUTOR_NUMBER=1
Env: G_BROKEN_FILENAMES=1
Env: HISTSIZE=1000
Env: HOME=/home/hudson
Env: HOSTNAME=ci
Env: HUDSON_HOME=/home/hudson/.hudson
Env: HUDSON_URL=http://ci.csh.rit.edu/
Env: INPUTRC=/etc/inputrc
Env: JOB_NAME=bodhi-py2.4
Env: JOB_URL=http://ci.csh.rit.edu/job/bodhi-py2.4/
Env: LANG=en_US.UTF-8
Env: LD_LIBRARY_PATH=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/lib/i386/client:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/lib/i386:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/../lib/i386
Env: LESSOPEN=|/usr/bin/lesspipe.sh %s
Env: LOGNAME=hudson
Env: LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
Env: MAIL=/var/spool/mail/hudson
Env: NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat
Env: NODE_NAME=
Env: PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/hudson/bin
Env: PWD=/home/hudson
Env: SHELL=/bin/bash
Env: SHLVL=1
Env: TERM=screen
Env: USER=hudson
Env: WORKSPACE=<http://ci.csh.rit.edu/job/bodhi-py2.4/ws/>
Env: XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
Fetching changes from the remote Git repository
Fetching upstream changes from git://git.fedorahosted.org/git/bodhi
[workspace] $ git fetch -t git://git.fedorahosted.org/git/bodhi +refs/heads/*:refs/remotes/origin/*
[workspace] $ git ls-tree HEAD
[workspace] $ git tag -l master
[workspace] $ git rev-parse origin/master
Commencing build of Revision f097b382fb251860f5ec24c4b340b9620bdaec38 (origin/master)
GitAPI created
Env: _=/usr/bin/java
Env: BUILD_ID=2010-05-06_15-01-47
Env: BUILD_NUMBER=43
Env: BUILD_TAG=hudson-bodhi-py2.4-43
Env: BUILD_URL=http://ci.csh.rit.edu/job/bodhi-py2.4/43/
Env: CLASSPATH=
Env: EXECUTOR_NUMBER=1
Env: G_BROKEN_FILENAMES=1
Env: HISTSIZE=1000
Env: HOME=/home/hudson
Env: HOSTNAME=ci
Env: HUDSON_HOME=/home/hudson/.hudson
Env: HUDSON_URL=http://ci.csh.rit.edu/
Env: INPUTRC=/etc/inputrc
Env: JOB_NAME=bodhi-py2.4
Env: JOB_URL=http://ci.csh.rit.edu/job/bodhi-py2.4/
Env: LANG=en_US.UTF-8
Env: LD_LIBRARY_PATH=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/lib/i386/client:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/lib/i386:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/../lib/i386
Env: LESSOPEN=|/usr/bin/lesspipe.sh %s
Env: LOGNAME=hudson
Env: LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
Env: MAIL=/var/spool/mail/hudson
Env: NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat
Env: NODE_NAME=
Env: PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/hudson/bin
Env: PWD=/home/hudson
Env: SHELL=/bin/bash
Env: SHLVL=1
Env: TERM=screen
Env: USER=hudson
Env: WORKSPACE=<http://ci.csh.rit.edu/job/bodhi-py2.4/ws/>
Env: XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
Checking out Revision f097b382fb251860f5ec24c4b340b9620bdaec38 (origin/master)
[workspace] $ git checkout -f f097b382fb251860f5ec24c4b340b9620bdaec38
[workspace] $ git tag -a -f -m "Hudson Build #43" hudson-bodhi-py2.4-43
Recording changes in branch origin/master
[workspace] $ git log --pretty=format:%H 78195e8fec65f29801bbaca3aa8557f7e44f8a86..f097b382fb251860f5ec24c4b340b9620bdaec38
[workspace] $ git log -M --summary --pretty=raw -n 1 f097b382fb251860f5ec24c4b340b9620bdaec38
[workspace] $ git diff-tree -M -r f097b382fb251860f5ec24c4b340b9620bdaec38
[workspace] $ git log -M --summary --pretty=raw -n 1 7acb3166840815f1f7be8660bd4ad681dba0593c
[workspace] $ git diff-tree -M -r 7acb3166840815f1f7be8660bd4ad681dba0593c
[workspace] $ git log -M --summary --pretty=raw -n 1 8964eeb03966b029c5d3b0f1993a18d68a62e39b
[workspace] $ git diff-tree -M -r 8964eeb03966b029c5d3b0f1993a18d68a62e39b
[workspace] $ /bin/sh -xe /tmp/hudson3093495778394264606.sh
+ python setup.py egg_info
running egg_info
writing requirements to bodhi.egg-info/requires.txt
writing bodhi.egg-info/PKG-INFO
writing top-level names to bodhi.egg-info/top_level.txt
writing dependency_links to bodhi.egg-info/dependency_links.txt
writing entry points to bodhi.egg-info/entry_points.txt
reading manifest file 'bodhi.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'start-bodhi'
warning: no files found matching 'ChangeLog'
writing manifest file 'bodhi.egg-info/SOURCES.txt'
+ nosetests
...............F................FF..........................Exception in thread VisitManager:
Traceback (most recent call last):
File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
self.run()
File "/usr/lib/python2.4/site-packages/turbogears/visit/api.py", line 352, in run
self.update_queued_visits(queue)
File "/usr/lib/python2.4/site-packages/turbogears/visit/sovisit.py", line 86, in update_queued_visits
conn.query(conn.sqlrepr(u))
File "/usr/lib/python2.4/site-packages/sqlobject/dbconnection.py", line 846, in query
return self._dbConnection._query(self._connection, s)
File "/usr/lib/python2.4/site-packages/sqlobject/dbconnection.py", line 349, in _query
self._executeRetry(conn, conn.cursor(), s)
File "/usr/lib/python2.4/site-packages/sqlobject/sqlite/sqliteconnection.py", line 178, in _executeRetry
raise OperationalError(ErrorMessage(e))
OperationalError: no such table: visit
......................................................
======================================================================
FAIL: test_admin_push (bodhi.tests.test_controllers.TestControllers)
----------------------------------------------------------------------
Traceback (most recent call last):
File "<http://ci.csh.rit.edu/job/bodhi-py2.4/ws/bodhi/tests/test_controllers.py",> line 1111, in test_admin_push
assert PackageUpdate.select().count() == 1
AssertionError:
-------------------- >> begin captured stdout << ---------------------
/updates/save?builds=TurboGears-2.6.23.1-21.fc7¬es=Initial+release+of+new+package%21&request=stable&bugs=&type_=security&release=Fedora+7
--------------------- >> end captured stdout << ----------------------
======================================================================
FAIL: Ensure admins can submit critpath updates for pending releases to stable.
----------------------------------------------------------------------
Traceback (most recent call last):
File "<http://ci.csh.rit.edu/job/bodhi-py2.4/ws/bodhi/tests/test_controllers.py",> line 1685, in test_critpath_to_frozen_release_testing_admin_actions
assert len(update.comments) == 3, update.comments
AssertionError: [<Comment 1 timestamp='datetime.datetime...)' updateID=1 author=u'guest' karma=1 text=u'foobar' anonymous=False>, <Comment 2 timestamp='datetime.datetime...)' updateID=1 author=u'bob' karma=1 text=u'foobar' anonymous=False>]
-------------------- >> begin captured stdout << ---------------------
/updates/save?stable_karma=1&builds=kernel-2.6.31-1.fc7¬es=foobar&request=None&bugs=&unstable_karma=-1&type_=bugfix&release=Fedora+7
--------------------- >> end captured stdout << ----------------------
======================================================================
FAIL: Ensure releng/qa can push critpath updates to stable for pending releases
----------------------------------------------------------------------
Traceback (most recent call last):
File "<http://ci.csh.rit.edu/job/bodhi-py2.4/ws/bodhi/tests/test_controllers.py",> line 1560, in test_critpath_to_pending_release_num_approved_comments
assert update.request == 'stable'
AssertionError:
-------------------- >> begin captured stdout << ---------------------
/updates/save?stable_karma=1&builds=kernel-2.6.31-1.fc7¬es=foobar&request=None&bugs=&unstable_karma=-1&type_=bugfix&release=Fedora+7
--------------------- >> end captured stdout << ----------------------
----------------------------------------------------------------------
Ran 114 tests in 192.784s
FAILED (failures=3)
14 years
- 1
- 2
2 commits - bodhi/tools
by Luke Macken
bodhi/tools/metrics.py | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
New commits:
commit 98847c7dfd6a5cd3cb5d94d2bc26ce9da2e82cb5
Merge: 565cb9d... f097b38...
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed May 19 08:31:50 2010 -0400
Merge branch 'master' of git+ssh://git.fedorahosted.org/git/bodhi
commit 565cb9dabc860cb3f18758ce36ab0fc6ee5ca8b7
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed May 19 08:31:36 2010 -0400
Fix a bug in the metrics script
diff --git a/bodhi/tools/metrics.py b/bodhi/tools/metrics.py
index aa8e6dd..dfc5c39 100755
--- a/bodhi/tools/metrics.py
+++ b/bodhi/tools/metrics.py
@@ -29,8 +29,8 @@ def main():
updates = PackageUpdate.select(PackageUpdate.q.releaseID==release.id)
stats[release.name] = {
'num_updates': updates.count(),
- 'num_testing': 0,
- 'num_testing_without_karma': 0,
+ 'num_tested': 0,
+ 'num_tested_without_karma': 0,
'num_feedback': 0,
'num_anon_feedback': 0,
'num_critpath': 0,
@@ -97,9 +97,9 @@ def main():
if not feedback_done:
data['critpath_without_karma'].add(update)
if testingtime_done:
- data['num_testing'] += 1
+ data['num_tested'] += 1
if not feedback_done:
- data['num_testing_without_karma'] += 1
+ data['num_tested_without_karma'] += 1
data['deltas'].sort()
@@ -120,12 +120,12 @@ def main():
data['num_anon_feedback'], float(data['num_anon_feedback']) /
(data['num_anon_feedback'] + sum(data['karma'].values())) * 100)
print " * %d out of %d updates went through testing (%0.2f%%)" % (
- data['num_testing'], data['num_updates'],
- float(data['num_testing']) / data['num_updates'] * 100)
+ data['num_tested'], data['num_updates'],
+ float(data['num_tested']) / data['num_updates'] * 100)
print " * %d testing updates were pushed *without* karma (%0.2f%%)" %(
- data['num_testing_without_karma'],
- float(data['num_testing_without_karma']) /
- data['num_testing'] * 100)
+ data['num_tested_without_karma'],
+ float(data['num_tested_without_karma']) /
+ data['num_tested'] * 100)
print " * %d critical path updates pushed *without* karma" % (
len(data['critpath_without_karma']))
for update in data['critpath_without_karma']:
14 years
- 1
- 0
bodhi/masher.py
by Luke Macken
bodhi/masher.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
New commits:
commit f097b382fb251860f5ec24c4b340b9620bdaec38
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu May 6 15:43:47 2010 -0400
Only write out stderr if it is not None
diff --git a/bodhi/masher.py b/bodhi/masher.py
index 438a89b..9aa8f41 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -564,8 +564,9 @@ class MashTask(Thread):
% time.strftime("%y%m%d.%H%M"))
out = file(failed_output, 'w')
out.write(stdout)
- out.write('\n\nstderr:\n\n')
- out.write(stderr)
+ if stderr:
+ out.write('\n\nstderr:\n\n')
+ out.write(stderr)
out.close()
log.info("Wrote failed mash output to %s" % failed_output)
self.log = failed_output
14 years
- 1
- 0
bodhi/masher.py
by Luke Macken
bodhi/masher.py | 2 ++
1 file changed, 2 insertions(+)
New commits:
commit 7acb3166840815f1f7be8660bd4ad681dba0593c
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu May 6 15:36:24 2010 -0400
Store stderr from mash when it fails...
diff --git a/bodhi/masher.py b/bodhi/masher.py
index e61484b..438a89b 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -564,6 +564,8 @@ class MashTask(Thread):
% time.strftime("%y%m%d.%H%M"))
out = file(failed_output, 'w')
out.write(stdout)
+ out.write('\n\nstderr:\n\n')
+ out.write(stderr)
out.close()
log.info("Wrote failed mash output to %s" % failed_output)
self.log = failed_output
14 years
- 1
- 0
bodhi/config
by Luke Macken
bodhi/config/app.cfg | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
New commits:
commit 8964eeb03966b029c5d3b0f1993a18d68a62e39b
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu May 6 15:30:56 2010 -0400
Update our admin_groups to include security_respons & proventesters
diff --git a/bodhi/config/app.cfg b/bodhi/config/app.cfg
index 6726057..67bcdf5 100644
--- a/bodhi/config/app.cfg
+++ b/bodhi/config/app.cfg
@@ -74,7 +74,7 @@ release_team_address = None
security_team = None
# Superuser groups
-admin_groups = 'releng security_respons cvsadmin qa'
+admin_groups = 'security_respons proventesters'
# Where do we send update announcements to ?
# These variables should be named per: Release.prefix_id.lower()_announce_list
14 years
- 1
- 0
4 commits - bodhi/tools
by Luke Macken
bodhi/tools/log_stats.py | 101 +++++++++++++++++++++++++++++++
bodhi/tools/metrics.py | 152 ++++++++++++++++++++++++++++++++++++++++++-----
bodhi/tools/pickledb.py | 42 +++++++-----
3 files changed, 263 insertions(+), 32 deletions(-)
New commits:
commit 78195e8fec65f29801bbaca3aa8557f7e44f8a86
Author: Luke Macken <lmacken(a)redhat.com>
Date: Mon May 3 17:41:55 2010 -0400
Make our pickledb script skip duplicate updates & releases
diff --git a/bodhi/tools/pickledb.py b/bodhi/tools/pickledb.py
index 964fde3..cdc1dba 100755
--- a/bodhi/tools/pickledb.py
+++ b/bodhi/tools/pickledb.py
@@ -97,7 +97,10 @@ def load_db():
# {'updates': [], 'releases': []}
if isinstance(data, dict):
for release in data['releases']:
- Release(**release)
+ try:
+ Release.byName(release['name'])
+ except SQLObjectNotFound:
+ Release(**release)
data = data['updates']
progress = ProgressBar(maxValue=len(data))
@@ -124,20 +127,23 @@ def load_db():
if not u.has_key('date_modified'):
u['date_modified'] = None
- update = PackageUpdate(title=u['title'],
- date_submitted=u['date_submitted'],
- date_pushed=u['date_pushed'],
- date_modified=u['date_modified'],
- release=release,
- submitter=u['submitter'],
- updateid=u['updateid'],
- type=u['type'],
- status=u['status'],
- pushed=u['pushed'],
- notes=u['notes'],
- karma=u['karma'],
- request=request,
- approved=u['approved'])
+ try:
+ update = PackageUpdate.byTitle(u['title'])
+ except SQLObjectNotFound:
+ update = PackageUpdate(title=u['title'],
+ date_submitted=u['date_submitted'],
+ date_pushed=u['date_pushed'],
+ date_modified=u['date_modified'],
+ release=release,
+ submitter=u['submitter'],
+ updateid=u['updateid'],
+ type=u['type'],
+ status=u['status'],
+ pushed=u['pushed'],
+ notes=u['notes'],
+ karma=u['karma'],
+ request=request,
+ approved=u['approved'])
## Create Package and PackageBuild objects
for pkg, nvr in u['builds']:
commit bb59d772c55e2be3dd5e1a822fc38596e96ee22d
Author: Luke Macken <lmacken(a)redhat.com>
Date: Mon May 3 17:41:29 2010 -0400
Store the 'anonymous' comment property in our pickledb tool
diff --git a/bodhi/tools/pickledb.py b/bodhi/tools/pickledb.py
index bdac628..964fde3 100755
--- a/bodhi/tools/pickledb.py
+++ b/bodhi/tools/pickledb.py
@@ -74,7 +74,7 @@ def save_db():
data['pushed'] = update.pushed
data['notes'] = update.notes
data['request'] = update.request
- data['comments'] = [(c.timestamp, c.author, c.text, c.karma) for c in update.comments]
+ data['comments'] = [(c.timestamp, c.author, c.text, c.karma, c.anonymous) for c in update.comments]
if hasattr(update, 'approved'):
data['approved'] = update.approved
else:
@@ -167,9 +167,9 @@ def load_db():
except SQLObjectNotFound:
cve = CVE(cve_id=cve_id)
update.addCVE(cve)
- for timestamp, author, text, karma in u['comments']:
+ for timestamp, author, text, karma, anonymous in u['comments']:
comment = Comment(timestamp=timestamp, author=author, text=text,
- karma=karma, update=update)
+ karma=karma, update=update, anonymous=anonymous)
progress()
commit bd3daef805a45d8d81b2ccc07c42b1aa4ecc1bdf
Author: Luke Macken <lmacken(a)redhat.com>
Date: Mon May 3 17:39:52 2010 -0400
Add a metrics.py tool for generating more in-dept statistics
diff --git a/bodhi/tools/0day.py b/bodhi/tools/0day.py
old mode 100644
new mode 100755
diff --git a/bodhi/tools/metrics.py b/bodhi/tools/metrics.py
index d10e10b..aa8e6dd 100755
--- a/bodhi/tools/metrics.py
+++ b/bodhi/tools/metrics.py
@@ -1,30 +1,154 @@
-#!/usr/bin/python -tt
-
+#!/usr/bin/env -tt
"""
-A tool for spitting out basic update and bug metrics for each release
+A tool for generating statistics for each release.
+
+.. moduleauthor:: Luke Macken <lmacken(a)redhat.com>
"""
+from operator import itemgetter
from sqlobject import AND
+from datetime import timedelta
+from collections import defaultdict
from turbogears.database import PackageHub
-from bodhi.util import load_config
+from bodhi.util import load_config, header
from bodhi.model import PackageUpdate, Release
+statuses = ('stable', 'testing', 'pending', 'obsolete')
+types = ('bugfix', 'enhancement', 'security', 'newpackage')
+
def main():
load_config()
+ stats = {} # {release: {'stat': ...}}
+ feedback = 0 # total number of updates that received feedback
+ karma = {} # {username: # of karma submissions}
+ num_updates = PackageUpdate.select().count()
+
for release in Release.select():
- print release.long_name
- updates = PackageUpdate.select(
- AND(PackageUpdate.q.releaseID == release.id,
- PackageUpdate.q.status == 'stable'))
- num_updates = updates.count()
- print " * %d stable updates" % num_updates
- bugs = set()
- for update in updates:
+ print header(release.long_name)
+ updates = PackageUpdate.select(PackageUpdate.q.releaseID==release.id)
+ stats[release.name] = {
+ 'num_updates': updates.count(),
+ 'num_testing': 0,
+ 'num_testing_without_karma': 0,
+ 'num_feedback': 0,
+ 'num_anon_feedback': 0,
+ 'num_critpath': 0,
+ 'critpath_without_karma': set(),
+ 'bugs': set(),
+ 'karma': {},
+ 'deltas': [],
+ 'occurrences': {},
+ 'accumulative': timedelta(),
+ 'packages': defaultdict(int),
+ }
+ data = stats[release.name]
+
+ for status in statuses:
+ data['num_%s' % status] = PackageUpdate.select(AND(
+ PackageUpdate.q.releaseID==release.id,
+ PackageUpdate.q.status==status)).count()
+
+ for type in types:
+ data['num_%s' % type] = PackageUpdate.select(AND(
+ PackageUpdate.q.releaseID==release.id,
+ PackageUpdate.q.type==type)).count()
+
+ for update in release.updates:
+ for build in update.builds:
+ data['packages'][build.package] += 1
for bug in update.bugs:
- bugs.add(bug)
- print " * %d bugs" % len(bugs)
+ data['bugs'].add(bug.bz_id)
+
+ feedback_done = False
+ testingtime_done = False
+
+ for comment in update.comments:
+ if not feedback_done and (not comment.author.startswith('bodhi')
+ and comment.karma != 0 and '@' not in comment.author):
+ data['num_feedback'] += 1
+ feedback += 1
+ feedback_done = True
+ if not comment.author.startswith('bodhi'):
+ if comment.anonymous or '@' in comment.author:
+ if comment.karma != 0:
+ data['num_anon_feedback'] += 1
+ else:
+ if comment.author not in data['karma']:
+ data['karma'][comment.author] = 0
+ karma[comment.author] = 0
+ data['karma'][comment.author] += 1
+ karma[comment.author] += 1
+ if (not testingtime_done and
+ comment.text == 'This update has been pushed to testing'):
+ for othercomment in update.comments:
+ if othercomment.text == 'This update has been pushed to stable':
+ delta = othercomment.timestamp - comment.timestamp
+ data['deltas'].append(delta)
+ data['occurrences'][delta.days] = \
+ data['occurrences'].setdefault(
+ delta.days, 0) + 1
+ data['accumulative'] += delta
+ testingtime_done = True
+ break
+
+ if update.critpath:
+ data['num_critpath'] += 1
+ if not feedback_done:
+ data['critpath_without_karma'].add(update)
+ if testingtime_done:
+ data['num_testing'] += 1
+ if not feedback_done:
+ data['num_testing_without_karma'] += 1
+
+ data['deltas'].sort()
+
+ print " * %d updates" % data['num_updates']
+ for status in statuses:
+ print " * %d %s updates" % (data['num_%s' % status], status)
+ for type in types:
+ print " * %d %s updates (%0.2f%%)" % (data['num_%s' % type], type,
+ float(data['num_%s' % type]) / data['num_updates'] * 100)
+ print " * %d critical path updates (%0.2f%%)" % (data['num_critpath'],
+ float(data['num_critpath']) / data['num_updates'] * 100)
+ print " * %d updates received feedback (%0.2f%%)" % (
+ data['num_feedback'], (float(data['num_feedback']) /
+ data['num_updates'] * 100))
+ print " * %d unique authenticated karma submitters" % (
+ len(data['karma']))
+ print " * %d anonymous users gave feedback (%0.2f%%)" % (
+ data['num_anon_feedback'], float(data['num_anon_feedback']) /
+ (data['num_anon_feedback'] + sum(data['karma'].values())) * 100)
+ print " * %d out of %d updates went through testing (%0.2f%%)" % (
+ data['num_testing'], data['num_updates'],
+ float(data['num_testing']) / data['num_updates'] * 100)
+ print " * %d testing updates were pushed *without* karma (%0.2f%%)" %(
+ data['num_testing_without_karma'],
+ float(data['num_testing_without_karma']) /
+ data['num_testing'] * 100)
+ print " * %d critical path updates pushed *without* karma" % (
+ len(data['critpath_without_karma']))
+ for update in data['critpath_without_karma']:
+ print " * %s submitted by %s" % (update.title, update.submitter)
+ print " * Time spent in testing:"
+ print " * mean = %d days" % (data['accumulative'].days /
+ len(data['deltas']))
+ print " * median = %d days" % (
+ data['deltas'][len(data['deltas']) / 2].days)
+ print " * mode = %d days" % (
+ sorted(data['occurrences'].items(), key=itemgetter(1))[-1][0])
+ print " * %d packages updated" % (len(data['packages']))
+ for package in sorted(data['packages'].items(), key=itemgetter(1), reverse=True):
+ print " * %s: %d" % (package[0].name, package[1])
+ print
+ print
+ print "Out of %d total updates, %d received feedback (%0.2f%%)" % (
+ num_updates, feedback, (float(feedback) / num_updates * 100))
+ print "Out of %d total unique karma submitters, the top 30 are:" % (
+ len(karma))
+ for submitter in sorted(karma.iteritems(), key=itemgetter(1), reverse=True)[:30]:
+ print " * %s (%d)" % (submitter[0], submitter[1])
if __name__ == '__main__':
main()
commit e46ccc0366f2d263d3fda65a00c0aabc43d507e7
Author: Luke Macken <lmacken(a)redhat.com>
Date: Mon May 3 17:38:03 2010 -0400
Add a script that scrapes bodhi's mod_wsgi logs to calculate autokarma statistics.
In bodhi v2.0, we'll do better tracking of these transitions in the database,
and will not have to scrape logs.
diff --git a/bodhi/tools/log_stats.py b/bodhi/tools/log_stats.py
new file mode 100755
index 0000000..d8b3109
--- /dev/null
+++ b/bodhi/tools/log_stats.py
@@ -0,0 +1,101 @@
+#!/usr/bin/env python
+"""
+A script that scrapes bodhi's mod_wsgi logs to calculate autokarma statistics.
+
+In bodhi v2.0, we'll do better tracking of these transitions in the database,
+and will not have to scrape logs.
+"""
+
+import subprocess
+
+from datetime import timedelta
+from operator import itemgetter
+from sqlobject import SQLObjectNotFound
+from collections import defaultdict
+from turbogears.database import PackageHub
+
+from bodhi.util import load_config, header
+from bodhi.model import PackageUpdate, Release, PackageBuild
+
+def parse_output(out):
+ updates = defaultdict(set)
+ num_critpath = defaultdict(int)
+ debug = file('bodhi.debug', 'a')
+ deltas = []
+ occurrences = {}
+ accumulative = timedelta()
+
+ for line in out.split('\n'):
+ line = line.strip()
+ if '}' in line:
+ continue
+ if line:
+ debug.write(line + '\n')
+ title = line.split()[-1]
+ update = None
+ for build in title.split(','):
+ try:
+ update = PackageBuild.byNvr(build).updates[0]
+ break
+ except SQLObjectNotFound:
+ pass
+ #print "Cannot find update for %s" % build
+ if update:
+ if update.title not in updates[update.release.name]:
+ updates[update.release.name].add(update.title)
+ if update.critpath:
+ num_critpath[update.release.name] += 1
+ for comment in update.comments:
+ if comment.text == 'This update has been pushed to testing':
+ for othercomment in update.comments:
+ if othercomment.text == 'This update has been pushed to stable':
+ delta = othercomment.timestamp - comment.timestamp
+ deltas.append(delta)
+ occurrences[delta.days] = occurrences.setdefault(delta.days, 0) + 1
+ accumulative += delta
+ break
+ break
+ debug.close()
+ deltas.sort()
+ return updates, num_critpath, deltas, accumulative, occurrences
+
+
+def main():
+ unstable = subprocess.Popen('grep "\[Fedora Update\] \[unstable\]" bodhi.logs',
+ stdout=subprocess.PIPE, shell=True)
+ out, err = unstable.communicate()
+ (unstable_updates, unstable_critpath, unstable_deltas,
+ unstable_accum, unstable_occur) = parse_output(out)
+
+ stable = subprocess.Popen('grep "\[Fedora Update\] \[stablekarma\]" bodhi.logs',
+ stdout=subprocess.PIPE, shell=True)
+ out, err = stable.communicate()
+ (stable_updates, stable_critpath, stable_deltas,
+ stable_accum, stable_occur) = parse_output(out)
+
+ for release in Release.select():
+ print '\n' + header(release.long_name)
+ num_updates = PackageUpdate.select(
+ PackageUpdate.q.releaseID==release.id).count()
+ num_stable = len(stable_updates[release.name])
+ num_unstable = len(unstable_updates[release.name])
+ num_testing = len(unstable_deltas) + len(stable_deltas)
+ print " * %d updates automatically unpushed due to karma (%0.2f%%)" % (
+ num_unstable, float(num_unstable) / num_updates * 100)
+ print " * %d of which were critical path updates" % (
+ unstable_critpath[release.name])
+ print " * %d updates automatically pushed due to karma (%0.2f%%)" % (
+ num_stable, float(num_stable) / num_updates * 100)
+ print " * %d of which were critical path updates" % (
+ stable_critpath[release.name])
+
+ print " * Time spent in testing of updates that were pushed by karma:"
+ print " * mean = %d days" % (stable_accum.days / len(stable_deltas))
+ print " * median = %d days" % stable_deltas[len(stable_deltas)/2].days
+ print " * mode = %d days" % sorted(stable_occur.items(),
+ key=itemgetter(1))[-1][0]
+
+
+if __name__ == '__main__':
+ load_config()
+ main()
14 years
- 1
- 0
bodhi/config
by Luke Macken
bodhi/config/app.cfg | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
New commits:
commit 0072e8e85faf9435ffd41b079a53e64463b8a474
Author: Luke Macken <lmacken(a)redhat.com>
Date: Fri Apr 30 17:05:18 2010 -0400
Update our critpath list
diff --git a/bodhi/config/app.cfg b/bodhi/config/app.cfg
index 6a42e5d..6726057 100644
--- a/bodhi/config/app.cfg
+++ b/bodhi/config/app.cfg
@@ -152,8 +152,8 @@ tgcaptcha.key = 'Y`h`f&s}TZz'
## Critical Path Packages
##
## Hardcoded, until we can query the pkgdb for it.
-## http://kojipkgs.fedoraproject.org/mash/rawhide-20100217/logs/critpath.txt
-critpath = "ConsoleKit ConsoleKit-libs ConsoleKit-x11 DeviceKit-power GConf2 GConf2-gtk MAKEDEV ModemManager NetworkManager NetworkManager-glib ORBit2 acl alsa-lib anaconda anaconda-yum-plugins at-spi atk attr audit audit-libs authconfig authconfig-gtk avahi avahi-autoipd avahi-glib basesystem bash binutils bluecurve-cursor-theme bzip2 bzip2-libs c-ares ca-certificates cairo checkpolicy chkconfig cloog-ppl comps-extras constantine-backgrounds constantine-backgrounds-single control-center control-center-filesystem coreutils coreutils-libs cpio cpp cracklib cracklib-dicts cracklib-python createrepo cronie cronie-anacron crontabs cryptsetup-luks cryptsetup-luks-libs cups-libs curl cyrus-sasl cyrus-sasl-lib dash db4 db4-utils dbus dbus-glib dbus-libs dbus-python dbus-x11 deltarpm desktop-backgrounds-basic desktop-file-utils device-mapper device-mapper-event device-mapper-event-libs device-mapper-libs dhclient diffutils dmidecode dmraid dmraid-events dnsmasq dosfstools dracut e2fsprogs e2fsprogs-libs efibootmgr eggdbus elfutils elfutils-libelf elfutils-libs ethtool evolution-data-server expat fedora-gnome-theme fedora-icon-theme fedora-logos fedora-release file file-libs filesystem findutils fipscheck fipscheck-lib firstboot flac fontconfig freetype gamin gawk gcc gcc-c++ gdbm gdm genisoimage glib2 glibc glibc-common glibc-devel glibc-headers gmp gnome-desktop gnome-icon-theme gnome-keyring gnome-keyring-pam gnome-menus gnome-panel-libs gnome-python2 gnome-python2-canvas gnome-python2-gnome gnome-python2-gnomevfs gnome-session gnome-settings-daemon gnome-themes gnome-vfs2 gnupg2 gnutls gobject-introspection gpgme grep grub grubby gstreamer gstreamer-tools gtk2 gtk2-engines gzip hal hal-filesystem hal-info hal-libs hdparm hesiod hicolor-icon-theme hostname hwdata info initscripts iptables iptables-ipv6 iputils iscsi-initiator-utils iso-codes isomd5sum jasper-libs kbd kernel kernel-headers keyutils-libs koji kpartx krb5-libs less libICE libIDL libSM libX11 libX11-common libXScrnSaver libXau libXcomposite libXcursor libXdamage libXdmcp libXext libXfixes libXfont libXft libXi libXinerama libXmu libXrandr libXrender libXres libXt libXtst libXv libXvMC libXxf86misc libXxf86vm libacl libarchive libart_lgpl libasyncns libattr libblkid libbonobo libbonoboui libcanberra libcanberra-gtk2 libcap libcap-ng libcom_err libcroco libcurl libdaemon libdrm libedit libffi libfontenc libgail-gnome libgcc libgcrypt libglade2 libgnome libgnome-keyring libgnomecanvas libgnomekbd libgnomeui libgomp libgpg-error libgsf libgudev1 libgweather libical libidn libjpeg libmcpp libnl libnotify libogg libpcap libpciaccess libpng libproxy libproxy-bin libproxy-python librsvg2 libselinux libselinux-python libselinux-utils libsemanage libsepol libsndfile libsoup libss libssh2 libstdc++ libstdc++-devel libtasn1 libtdb libthai libtiff libtool-ltdl libudev libusb libuser libuser-python libutempter libuuid libvorbis libwnck libx86 libxcb libxkbfile libxklavier libxml2 libxml2-python libxslt linux-firmware livecd-tools logrotate lua lvm2 lvm2-libs m4 makebootfat mash mcpp mdadm metacity mingetty mobile-broadband-provider-info module-init-tools mpfr mtools mysql-libs ncurses ncurses-base ncurses-libs net-tools newt newt-python notification-daemon notification-daemon-engine-slider nspr nss nss-softokn nss-softokn-freebl nss-sysinit nss-util ntp ntpdate openldap openssh openssh-clients openssh-server openssl pam pango parted passwd patch pciutils-libs pcre perl perl-Crypt-PasswdMD5 perl-Digest-SHA1 perl-Module-Pluggable perl-Pod-Escapes perl-Pod-Simple perl-libs perl-version pinentry pixman pkgconfig plymouth plymouth-core-libs plymouth-gdm-hooks plymouth-scripts plymouth-utils pm-utils policycoreutils polkit polkit-desktop-policy polkit-gnome popt postfix ppl ppp procmail procps psmisc pth pulseaudio-gdm-hooks pulseaudio-libs pulseaudio-libs-glib2 pungi pyOpenSSL pycairo pygobject2 pygpgme pygtk2 pygtk2-libglade pykickstart pyparted python python-bugzilla python-cryptsetup python-decorator python-deltarpm python-ethtool python-imgcreate python-iniparse python-kid python-krbV python-libs python-meh python-nss python-pyblock python-pycurl python-slip python-urlgrabber pyxf86config radeontool rarian rarian-compat readline redhat-menus redhat-rpm-config repoview rootfiles rpm rpm-build rpm-libs rpm-python rsyslog sed selinux-policy selinux-policy-targeted sendmail setserial setup setuptool sgml-common sgpio shadow-utils shared-mime-info slang sound-theme-freedesktop sqlite squashfs-tools startup-notification sudo sysklogd syslinux system-config-date system-config-firewall-base system-config-keyboard system-config-users system-setup-keyboard sysvinit-tools tar tcp_wrappers-libs tigervnc-server tzdata udev unique unzip upstart usermode usermode-gtk ustr util-linux-ng vbetool vim-minimal which wpa_supplicant xcb-util xdg-utils xkeyboard-config xml-common xorg-x11-drivers xorg-x11-drv-acecad xorg-x11-drv-aiptek xorg-x11-drv-apm xorg-x11-drv-ast xorg-x11-drv-ati xorg-x11-drv-cirrus xorg-x11-drv-dummy xorg-x11-drv-elographics xorg-x11-drv-evdev xorg-x11-drv-fbdev xorg-x11-drv-fpit xorg-x11-drv-geode xorg-x11-drv-glint xorg-x11-drv-hyperpen xorg-x11-drv-i128 xorg-x11-drv-i740 xorg-x11-drv-intel xorg-x11-drv-keyboard xorg-x11-drv-mach64 xorg-x11-drv-mga xorg-x11-drv-mouse xorg-x11-drv-mutouch xorg-x11-drv-neomagic xorg-x11-drv-nouveau xorg-x11-drv-nv xorg-x11-drv-openchrome xorg-x11-drv-penmount xorg-x11-drv-r128 xorg-x11-drv-rendition xorg-x11-drv-s3virge xorg-x11-drv-savage xorg-x11-drv-siliconmotion xorg-x11-drv-sis xorg-x11-drv-sisusb xorg-x11-drv-synaptics xorg-x11-drv-tdfx xorg-x11-drv-trident xorg-x11-drv-v4l xorg-x11-drv-vesa xorg-x11-drv-vmmouse xorg-x11-drv-vmware xorg-x11-drv-void xorg-x11-drv-voodoo xorg-x11-drv-wacom xorg-x11-font-utils xorg-x11-fonts-misc xorg-x11-server-Xorg xorg-x11-server-common xorg-x11-server-utils xorg-x11-xauth xorg-x11-xinit xorg-x11-xkb-utils xz xz-libs yum yum-metadata-parser yum-utils zenity zlib"
+## http://kojipkgs.fedoraproject.org/mash/branched-20100429/logs/critpath.txt
+critpath = "ConsoleKit ConsoleKit-libs ConsoleKit-x11 GConf2 GConf2-gtk MAKEDEV ModemManager NetworkManager NetworkManager-glib ORBit2 acl alsa-lib anaconda anaconda-yum-plugins at-spi atk attr audit audit-libs authconfig authconfig-gtk avahi avahi-autoipd avahi-glib basesystem bash binutils bzip2 bzip2-libs c-ares ca-certificates cairo checkpolicy chkconfig cloog-ppl compat-db47 comps-extras control-center control-center-filesystem coreutils coreutils-libs cpio cpp cracklib cracklib-dicts cracklib-python createrepo cronie cronie-anacron crontabs cryptsetup-luks cryptsetup-luks-libs cups-libs curl cyrus-sasl cyrus-sasl-lib dash db4 db4-utils dbus dbus-glib dbus-libs dbus-python dbus-x11 deltarpm desktop-backgrounds-basic desktop-file-utils device-mapper device-mapper-event device-mapper-event-libs device-mapper-libs dhclient diffutils dmidecode dmraid dmraid-events dmz-cursor-themes dnsmasq dosfstools dracut e2fsprogs e2fsprogs-libs efibootmgr eggdbus elfutils elfutils-libelf elfutils-libs enchant evolution-data-server expat fedora-gnome-theme fedora-icon-theme fedora-logos fedora-release file file-libs filesystem findutils fipscheck fipscheck-lib firstboot flac fontconfig freetype gamin gawk gcc gcc-c++ gdbm gdm genisoimage glib2 glibc glibc-common glibc-devel glibc-headers gmp gnome-desktop gnome-icon-theme gnome-keyring gnome-keyring-pam gnome-menus gnome-panel-libs gnome-python2 gnome-python2-canvas gnome-python2-gnome gnome-python2-gnomevfs gnome-session gnome-settings-daemon gnome-themes gnome-vfs2 gnupg2 gnutls gobject-introspection goddard-backgrounds-gnome goddard-backgrounds-single gpgme grep grub grubby gstreamer gstreamer-tools gtk2 gtk2-engines gzip hal hal-filesystem hal-info hal-libs hdparm hesiod hicolor-icon-theme hostname hunspell hwdata info initscripts iptables iptables-ipv6 iputils iscsi-initiator-utils iso-codes isomd5sum jasper-libs kbd kernel kernel-headers keyutils-libs koji kpartx krb5-libs less libICE libIDL libSM libX11 libX11-common libXScrnSaver libXau libXcomposite libXcursor libXdamage libXdmcp libXext libXfixes libXfont libXft libXi libXinerama libXmu libXrandr libXrender libXres libXt libXtst libXv libXvMC libXxf86misc libXxf86vm libacl libarchive libart_lgpl libasyncns libattr libblkid libbonobo libbonoboui libcanberra libcanberra-gtk2 libcap libcap-ng libcom_err libcroco libcurl libdaemon libdrm libedit libffi libfontenc libgail-gnome libgcc libgcrypt libglade2 libgnome libgnome-keyring libgnomecanvas libgnomekbd libgnomeui libgomp libgpg-error libgsf libgudev1 libgweather libical libidn libjpeg libmcpp libnih libnl libnotify libogg libpcap libpciaccess libpng libproxy librsvg2 libselinux libselinux-python libselinux-utils libsemanage libsepol libsexy libsndfile libsoup libss libssh2 libstdc++ libstdc++-devel libtasn1 libtdb libthai libtiff libtool-ltdl libudev libusb libuser libuser-python libutempter libuuid libvorbis libwnck libx86 libxcb libxfce4util libxfcegui4 libxkbfile libxklavier libxml2 libxml2-python libxslt linux-firmware livecd-tools logrotate lua lvm2 lvm2-libs m4 makebootfat mash mcpp mdadm mesa-dri-drivers metacity mingetty mobile-broadband-provider-info module-init-tools mpfr mtools mysql-libs ncurses ncurses-base ncurses-libs net-tools newt newt-python notification-daemon notification-daemon-engine-slider nspr nss nss-softokn nss-softokn-freebl nss-sysinit nss-util ntp ntpdate openldap openssh openssh-clients openssh-server openssl pam pango parted passwd patch pciutils-libs pcre perl perl-Crypt-PasswdMD5 perl-Digest-SHA1 perl-Module-Pluggable perl-Pod-Escapes perl-Pod-Simple perl-libs perl-version pinentry pixman pkgconfig plymouth plymouth-core-libs plymouth-gdm-hooks plymouth-scripts plymouth-utils pm-utils policycoreutils polkit polkit-desktop-policy polkit-gnome popt postfix ppl ppp procmail procps psmisc pth pulseaudio-gdm-hooks pulseaudio-libs pulseaudio-libs-glib2 pungi pyOpenSSL pycairo pygobject2 pygpgme pygtk2 pygtk2-libglade pykickstart pyparted python python-bugzilla python-cryptsetup python-decorator python-deltarpm python-ethtool python-imgcreate python-iniparse python-kid python-krbV python-libs python-meh python-nss python-pyblock python-pycurl python-slip python-urlgrabber pyxf86config radeontool rarian rarian-compat readline redhat-menus redhat-rpm-config repoview rootfiles rpm rpm-build rpm-libs rpm-python rsyslog sed selinux-policy selinux-policy-targeted sendmail setserial setup setuptool sgml-common sgpio shadow-utils shared-mime-info slang sound-theme-freedesktop sqlite squashfs-tools startup-notification sudo sysklogd syslinux system-config-date system-config-firewall-base system-config-keyboard system-config-users system-setup-keyboard sysvinit-tools tar tcp_wrappers-libs tigervnc-server tzdata udev unique unzip upower upstart usermode usermode-gtk ustr util-linux-ng vbetool vim-minimal which wpa_supplicant xcb-util xdg-utils xfce4-notifyd xfconf xkeyboard-config xml-common xorg-x11-drivers xorg-x11-drv-acecad xorg-x11-drv-aiptek xorg-x11-drv-apm xorg-x11-drv-ast xorg-x11-drv-ati xorg-x11-drv-cirrus xorg-x11-drv-dummy xorg-x11-drv-elographics xorg-x11-drv-evdev xorg-x11-drv-fbdev xorg-x11-drv-fpit xorg-x11-drv-geode xorg-x11-drv-glint xorg-x11-drv-hyperpen xorg-x11-drv-i128 xorg-x11-drv-i740 xorg-x11-drv-intel xorg-x11-drv-keyboard xorg-x11-drv-mach64 xorg-x11-drv-mga xorg-x11-drv-mouse xorg-x11-drv-mutouch xorg-x11-drv-neomagic xorg-x11-drv-nouveau xorg-x11-drv-nv xorg-x11-drv-openchrome xorg-x11-drv-penmount xorg-x11-drv-r128 xorg-x11-drv-rendition xorg-x11-drv-s3virge xorg-x11-drv-savage xorg-x11-drv-siliconmotion xorg-x11-drv-sis xorg-x11-drv-sisusb xorg-x11-drv-synaptics xorg-x11-drv-tdfx xorg-x11-drv-trident xorg-x11-drv-v4l xorg-x11-drv-vesa xorg-x11-drv-vmmouse xorg-x11-drv-vmware xorg-x11-drv-void xorg-x11-drv-voodoo xorg-x11-drv-wacom xorg-x11-font-utils xorg-x11-fonts-misc xorg-x11-server-Xorg xorg-x11-server-common xorg-x11-server-utils xorg-x11-xauth xorg-x11-xinit xorg-x11-xkb-utils xz xz-libs yum yum-metadata-parser yum-utils zenity zlib"
# The number of admin approvals it takes to be able to push a critical path
# update to stable for a pending release.
14 years
- 1
- 0