Bodhi 0.5!
by Luke Macken
Hey everyone,
As many have already noticed, I performed a large bodhi upgrade recently. A
few weeks ago, during The Incident, I was forced to perform what was originally
going to be a week long bodhi migration and upgrade, overnight. During the
past two weeks I've pushed out 24 versions of bodhi to our infrastructure,
fixing various show-stoppers, and making sure that updates got out the door.
One of the most noticable changes is that bodhi is much more responsive.
Previously, bodhi was a single python process, running on a single server.
This single server was also responsible for composing the updates repositories,
and rawhide, among lots of other bodhi-related churn. This lead to much pain
and suffering for all.
The bodhi deployment has since changed. All bodhi requests are now load
balanced to a bunch of app servers, each running mod_wsgi with multiple bodhi
processes, each with multiple threads. All of the hard work is now done on an
isolated releng server. This separate bodhi "masher" is now responsible for
composing repositories, updating bugs, generating update notices, sending
emails, extended metadata generation, and calculating metrics. I also added
support for inter-bodhi communication, which allows our bodhi web frontends to
kick off push requests to our bodhi-masher instance.
Some of the new features in this release:
- A much more flexible karma automatism scheme. Stable/unstable karma
thresholds are now fully configurable
- Support for bug aliases
- A 'newpackage' update type
- Newer updates which obsolete older ones will now inherit their bugs and notes.
- A shiny new API in the fedora.client.bodhi module
- Lots of improved releng and security team support, making our lives little easier
- An improved `make update` template (be sure to update your Makefile.common)
- Some new bodhi-client features
- Creating updates for multiple releases using a single form
Note: This is not perfect yet. You can use the "New Update Form" to add any
number of builds for any number of releases, but bodhi will still create a
single update for each. This issue will be resolved in the next major bodhi
release, which will contain a full model redesign.
- updateinfo.xml generation takes about 20 seconds, instead of 20 minutes.
- A lot of metrics enhancements
- A ton of bug and usability fixes
Bodhi is far from being feature complete. Some new features in the pipeline:
- DeltaRPM generation
- Security issue (CVE) tracking and triaging
- Dependency closure verification, utilizing the power of rpmgrok.
- A complete remodeling from SQLObject to SQLAlchemy, which is almost complete,
will give us a lot more flexibility, speed, and power over our update model.
This will also allow for things such as having multi-builds for multiple
releases in a single update.
As always,
- File tickets here: https://fedorahosted.org/bodhi/newticket
- Help out here: https://fedorahosted.org/bodhi/report/1
- Subscribe to the bodhi mailing list here: https://fedorahosted.org/mailman/listinfo/bodhi
- You can always find bodhi here: http://bodhi.fedoraproject.org
Thank you all for your patience during the past few weeks,
luke
p.s. If you're having issues with the bodhi client, a fixed version will be
going out with the next batch of updates. For the impatient, you can pull
fixed versions from koji (also, make sure your Makefile.common is up to date):
koji download-build --arch=noarch python-fedora-0.3.5-1.fc10
koji download-build --arch=noarch bodhi-0.5.2-1.fc9
15 years, 8 months
bodhi/model.py
by Luke Macken
bodhi/model.py | 5 +++++
1 file changed, 5 insertions(+)
New commits:
commit ba2adb113bfc8441a26dc85b1b873559efb8b0d9
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 10 16:17:23 2008 -0400
Fix bug aliases (again)
diff --git a/bodhi/model.py b/bodhi/model.py
index 0e4dfbc..f2ac7e9 100644
--- a/bodhi/model.py
+++ b/bodhi/model.py
@@ -525,6 +525,11 @@ class PackageUpdate(SQLObject):
bug.destroySelf()
for bug in bugs:
try:
+ bug = int(bug)
+ except ValueError: # bug alias
+ bugzilla = Bugzilla.get_bz()
+ bug = bugzilla.getbug(bug).bug_id
+ try:
bz = Bugzilla.byBz_id(bug)
except SQLObjectNotFound:
if fetchdetails:
15 years, 8 months
9 commits - bodhi/admin.py bodhi/config bodhi/mail.py bodhi/masher.py bodhi/metrics.py bodhi/tools bodhi/util.py
by Luke Macken
bodhi/admin.py | 4 ++--
bodhi/config/app.cfg | 1 +
bodhi/mail.py | 8 ++++----
bodhi/masher.py | 37 +++++++++++++++++++++++++++----------
bodhi/metrics.py | 2 ++
bodhi/tools/tagcheck.py | 2 +-
bodhi/util.py | 9 ++++++++-
7 files changed, 45 insertions(+), 18 deletions(-)
New commits:
commit 9063fecc30e8bdbf7e384dc13947ff64a5c13b3e
Author: Luke Macken <lmacken(a)redhat.com>
Date: Tue Sep 9 13:11:40 2008 -0400
Properly update the lock this time
diff --git a/bodhi/masher.py b/bodhi/masher.py
index 3baf732..d62e995 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -263,7 +263,7 @@ class MashTask(Thread):
lock.close()
masher_state['composed_repos'] = self.composed_repos
lock = file(mash_lock, 'w')
- pickle.dump(masher_state)
+ pickle.dump(masher_state, lock)
lock.close()
def error_log(self, msg):
commit e37bfbf2e1c7b8980c29a9f39289cc19d285f998
Author: Luke Macken <lmacken(a)redhat.com>
Date: Tue Sep 9 12:16:20 2008 -0400
Update the MASHING state lock after the completion of every mash
diff --git a/bodhi/masher.py b/bodhi/masher.py
index 833c6f6..3baf732 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -251,6 +251,21 @@ class MashTask(Thread):
else:
log.error("Cannot find MashTask lock at %s" % mash_lock)
+ def _update_lock(self):
+ """
+ Update our masher state lockfile with any completed repos that we were
+ able to compose during this push
+ """
+ log.debug("Updating MASHING lock")
+ mash_lock = join(config.get('mashed_dir'), 'MASHING')
+ lock = file(mash_lock, 'r')
+ masher_state = pickle.load(lock)
+ lock.close()
+ masher_state['composed_repos'] = self.composed_repos
+ lock = file(mash_lock, 'w')
+ pickle.dump(masher_state)
+ lock.close()
+
def error_log(self, msg):
log.error(msg)
self.errors.append(msg)
@@ -535,6 +550,7 @@ class MashTask(Thread):
log.info("Wrote mash output to %s" % mash_output)
self.log = mash_output
self.composed_repos.append(mashdir)
+ self._update_lock()
self.mashing = False
log.debug("Mashed for %s seconds" % (time.time() - t0))
@@ -618,14 +634,7 @@ class MashTask(Thread):
# Update our masher state lockfile with any completed
# repos that we were able to compose during this push
log.debug("Mash unsuccessful, updating state lock")
- mash_lock = join(config.get('mashed_dir'), 'MASHING')
- lock = file(mash_lock, 'r')
- masher_state = pickle.load(lock)
- lock.close()
- masher_state['composed_repos'] = self.composed_repos
- lock = file(mash_lock, 'w')
- pickle.dump(masher_state)
- lock.close()
+ self._update_lock()
log.debug("MashTask done")
masher.done(self)
commit a2129d27e2242f7aef509a39485f9eaaf15efa79
Author: Luke Macken <lmacken(a)redhat.com>
Date: Tue Sep 9 01:37:06 2008 -0400
Avoid some circular deps with turbomail + setuptools
diff --git a/bodhi/mail.py b/bodhi/mail.py
index 5174500..3c19ad1 100644
--- a/bodhi/mail.py
+++ b/bodhi/mail.py
@@ -17,7 +17,6 @@ import logging
import turbomail
from textwrap import wrap
-from turbomail import MailNotEnabledException
from turbogears import config, identity
from bodhi.util import rpm_fileheader, to_unicode
@@ -365,6 +364,7 @@ def get_template(update, use_template=errata_template):
return templates
def send_mail(sender, to, subject, body):
+ from turbomail import MailNotEnabledException
message = turbomail.Message(sender, to, subject)
message.plain = body
try:
commit 2db1ac5a7fe263322a7b65645fedb338a5f25a88
Author: Luke Macken <lmacken(a)redhat.com>
Date: Tue Sep 9 01:34:04 2008 -0400
Update our tagcheck tool for F9
diff --git a/bodhi/tools/tagcheck.py b/bodhi/tools/tagcheck.py
index 11f6bd1..24d2bf6 100755
--- a/bodhi/tools/tagcheck.py
+++ b/bodhi/tools/tagcheck.py
@@ -16,7 +16,7 @@ def main():
load_config()
__connection__ = hub = PackageHub("bodhi")
koji = get_session()
- for tag in ('dist-fc7-updates-testing', 'dist-fc7-updates',
+ for tag in ('dist-f9-updates-testing', 'dist-f9-updates',
'dist-f8-updates-testing', 'dist-f8-updates'):
tagged = [build['nvr'] for build in koji.listTagged(tag)]
for nvr in tagged:
commit cbd1ccec36c6872f1bc615d4a82725adcadbd5a9
Author: Luke Macken <lmacken(a)redhat.com>
Date: Mon Sep 8 10:43:17 2008 -0400
Sort our metric data by month
diff --git a/bodhi/metrics.py b/bodhi/metrics.py
index a715acd..e2a7787 100644
--- a/bodhi/metrics.py
+++ b/bodhi/metrics.py
@@ -81,10 +81,12 @@ class AllMetric(Metric):
def done(self):
for update_type, data in self.timeline.items():
self.timeline[update_type] = data.items()
+ self.timeline[update_type].sort(key=lambda x: x[0])
# Append earlier months for newer years to the end of the graph
self.months = self.months.items()
self.months.sort(key=lambda x: x[0])
+
m = []
for num, month in self.months:
if num < self.earliest.month:
commit eab67e11aa7e06ce3f8d8f7eda1015fb92b95a14
Author: Luke Macken <lmacken(a)redhat.com>
Date: Fri Sep 5 19:39:48 2008 -0400
Fix a nasty unicode bug in our mail template generation
diff --git a/bodhi/mail.py b/bodhi/mail.py
index 50d5ee9..5174500 100644
--- a/bodhi/mail.py
+++ b/bodhi/mail.py
@@ -20,7 +20,7 @@ from textwrap import wrap
from turbomail import MailNotEnabledException
from turbogears import config, identity
-from bodhi.util import rpm_fileheader
+from bodhi.util import rpm_fileheader, to_unicode
from bodhi.exceptions import RPMNotFound
log = logging.getLogger(__name__)
@@ -345,7 +345,7 @@ def get_template(update, use_template=errata_template):
elif len(text) != 1:
oldtime = oldtime[0]
info['changelog'] = u"ChangeLog:\n\n%s%s" % \
- (unicode(build.get_changelog(oldtime)), line)
+ (to_unicode(build.get_changelog(oldtime)), line)
except RPMNotFound:
log.error("Cannot find 'latest' RPM for generating ChangeLog: %s" %
lastpkg)
@@ -359,7 +359,7 @@ def get_template(update, use_template=errata_template):
# We can't trust the strings we get from RPM
log.debug("UnicodeDecodeError! Will try again after decoding")
for (key, value) in info.items():
- if value: info[key] = value.decode('utf8', 'replace')
+ if value: info[key] = to_unicode(value)
templates.append((info['subject'], use_template % info))
return templates
commit a82c46a589103a83f18b301c508b001b4f24ee53
Author: Luke Macken <lmacken(a)redhat.com>
Date: Fri Sep 5 19:26:18 2008 -0400
Add a to_unicode method to bodhi.util
diff --git a/bodhi/util.py b/bodhi/util.py
index 4706c14..1e23cdf 100644
--- a/bodhi/util.py
+++ b/bodhi/util.py
@@ -309,7 +309,6 @@ class Singleton(object):
def __new__(cls, *args, **kw):
if not '_instance' in cls.__dict__:
- log.debug("Creating new singleton instance!")
cls._instance = object.__new__(cls)
return cls._instance
@@ -415,3 +414,11 @@ def sanity_check_repodata(myurl):
if errorstrings:
raise RepodataException(','.join(errorstrings))
+
+
+def to_unicode(obj, encoding='utf-8'):
+ if isinstance(obj, basestring):
+ if not isinstance(obj, unicode):
+ obj = unicode(obj, encoding, 'replace')
+ return obj
+
commit 6f32efa9db9d6902cf2fe1a0a61a27a3901a5174
Author: Luke Macken <lmacken(a)redhat.com>
Date: Fri Sep 5 18:04:33 2008 -0400
Fix bodhi's wait_for_sync to watch the newkey repomd if necessary
diff --git a/bodhi/config/app.cfg b/bodhi/config/app.cfg
index d4903eb..b108de3 100644
--- a/bodhi/config/app.cfg
+++ b/bodhi/config/app.cfg
@@ -44,6 +44,7 @@ comps_cvs = ':pserver:anonymous@cvs.fedora.redhat.com:/cvs/pkgs'
##
file_url = 'http://download.fedoraproject.org/pub/fedora/linux/updates'
master_repomd = 'http://download.fedora.redhat.com/pub/fedora/linux/updates/%d/i386/repoda...'
+master_newkey_repomd = 'http://download.fedora.redhat.com/pub/fedora/linux/updates/%d/i386.newkey...'
## The base url of this application (should not include server.webpath)
base_address = 'http://localhost:8084'
diff --git a/bodhi/masher.py b/bodhi/masher.py
index 6934a75..833c6f6 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -682,15 +682,23 @@ class MashTask(Thread):
self.updates.add(update)
mashdir = config.get('mashed_dir')
repo = "%s-updates" % release.name.lower()
+ master_repomd = config.get('master_repomd')
repomd = join(mashdir, repo, 'i386', 'repodata', 'repomd.xml')
if not exists(repomd):
log.error("Cannot find local repomd: %s" % repomd)
- return
+ # Hack, for new gpg key
+ repomd = join(mashdir, repo, 'i386.newkey', 'repodata', 'repomd.xml')
+ if not exists(repomd):
+ log.error("Cannot find local repomd: %s" % repomd)
+ return
+ else:
+ log.info("Found repomd at %s" % repomd)
+ master_repomd = config.get('master_newkey_repomd')
checksum = sha.new(file(repomd).read()).hexdigest()
while True:
sleep(600)
try:
- masterrepomd = urllib2.urlopen(config.get('master_repomd') % release.get_version())
+ masterrepomd = urllib2.urlopen(master_repomd % release.get_version())
except urllib2.URLError, e:
log.error("Error fetching repomd.xml: %s" % str(e))
continue
commit db9ef5ec8b9bdce1980858b11b962d956062b98b
Author: Luke Macken <lmacken(a)redhat.com>
Date: Fri Sep 5 00:03:18 2008 -0400
Update our AdminController to reflect the new mash state lock format
diff --git a/bodhi/admin.py b/bodhi/admin.py
index 3da5e0b..4d9baa5 100644
--- a/bodhi/admin.py
+++ b/bodhi/admin.py
@@ -159,10 +159,10 @@ class AdminController(Controller, SecureResource):
mash_lock = join(mashed_dir, 'MASHING')
if exists(mash_lock):
mash_lock = file(mash_lock)
- mashing_updates = pickle.load(mash_lock)
+ mash_state = pickle.load(mash_lock)
mash_lock.close()
mash_data['mashing'] = masher.mashing
- mash_data['updates'] = mashing_updates
+ mash_data['updates'] = mash_state['updates']
return mash_data
def _masher_request(self, method, **kwargs):
15 years, 8 months
3 commits - bodhi/admin.py bodhi/masher.py
by Luke Macken
bodhi/admin.py | 4 ++--
bodhi/masher.py | 13 +++++++++++++
2 files changed, 15 insertions(+), 2 deletions(-)
New commits:
commit fe03f84b5575ec1d2c77c1c3f18d221680824363
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 23:55:19 2008 -0400
Update our masher state lockfile with any completed repos if a mash fails
diff --git a/bodhi/masher.py b/bodhi/masher.py
index 795a59f..6934a75 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -614,6 +614,18 @@ class MashTask(Thread):
if self.success:
log.debug("Success! Unlocking repo")
self._unlock()
+ else:
+ # Update our masher state lockfile with any completed
+ # repos that we were able to compose during this push
+ log.debug("Mash unsuccessful, updating state lock")
+ mash_lock = join(config.get('mashed_dir'), 'MASHING')
+ lock = file(mash_lock, 'r')
+ masher_state = pickle.load(lock)
+ lock.close()
+ masher_state['composed_repos'] = self.composed_repos
+ lock = file(mash_lock, 'w')
+ pickle.dump(masher_state)
+ lock.close()
log.debug("MashTask done")
masher.done(self)
commit 4df99dd8f2a196aa3c36927d61c1bf6e8bbb21e3
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 23:55:00 2008 -0400
Add completed repos to our composed_repos list
diff --git a/bodhi/masher.py b/bodhi/masher.py
index 06eb15c..795a59f 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -534,6 +534,7 @@ class MashTask(Thread):
out.close()
log.info("Wrote mash output to %s" % mash_output)
self.log = mash_output
+ self.composed_repos.append(mashdir)
self.mashing = False
log.debug("Mashed for %s seconds" % (time.time() - t0))
commit 7c1becb6f45fc682c3ddc6fee4ac562585ebade7
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 23:54:37 2008 -0400
See if we can fix this status message
diff --git a/bodhi/admin.py b/bodhi/admin.py
index 7995281..3da5e0b 100644
--- a/bodhi/admin.py
+++ b/bodhi/admin.py
@@ -134,8 +134,8 @@ class AdminController(Controller, SecureResource):
# If we're not The Masher, then proxy this request to it
if config.get('masher'):
data = self._masher_request('/admin/mash', updates=updates, resume=resume) or {}
- flash_log('Push request %s' % data.get('success') and 'succeeded'
- or 'failed')
+ flash_log('Push request %s' % (data.get('success') and 'succeeded'
+ or 'failed'))
raise redirect('/admin/masher')
from bodhi.masher import masher
15 years, 8 months
bodhi/masher.py
by Luke Macken
bodhi/masher.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
New commits:
commit e5f3031d3c8441d469709b33af16c3166b83c72e
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 23:36:09 2008 -0400
s/completed_repos/composed_repos/g
diff --git a/bodhi/masher.py b/bodhi/masher.py
index b82888a..06eb15c 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -240,7 +240,7 @@ class MashTask(Thread):
lock = file(mash_lock, 'w')
pickle.dump({
'updates': [update.title for update in self.updates],
- 'completed_repos': self.composed_repos,
+ 'composed_repos': self.composed_repos,
}, lock)
lock.close()
@@ -498,7 +498,7 @@ class MashTask(Thread):
self.update_comps()
# {'f9-updates': '/mnt/koji/mash/updates/f9-updates-080905.0057',}
finished_repos = dict([('-'.join(basename(repo).split('-')[:-1]), repo)
- for repo in self.completed_repos])
+ for repo in self.composed_repos])
for repo in self.repos:
# Skip mashing this repo if we successfully mashed it previously
if repo in finished_repos:
15 years, 8 months
bodhi/masher.py
by Luke Macken
bodhi/masher.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
New commits:
commit cc578b908bfd125dd7adcb8fc63171a8e1d64ae0
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 23:14:42 2008 -0400
s/add/append/
diff --git a/bodhi/masher.py b/bodhi/masher.py
index 76ff9c8..b82888a 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -227,7 +227,7 @@ class MashTask(Thread):
except SQLObjectNotFound:
log.warning("Cannot find %s" % up)
for repo in masher_state['composed_repos']:
- self.composed_repos.add(repo)
+ self.composed_repos.append(repo)
else:
log.error('Unknown masher lock format: %s' % masher_state)
raise MashTaskException
15 years, 8 months
3 commits - bodhi/admin.py bodhi/masher.py
by Luke Macken
bodhi/admin.py | 5 ++---
bodhi/masher.py | 54 ++++++++++++++++++++++++++++++++++++++++++++----------
2 files changed, 46 insertions(+), 13 deletions(-)
New commits:
commit fb19c3be45cee5dfbec6b1cee1686b0e51c7c4ac
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 23:12:11 2008 -0400
Update our masher state lock format so we can skip previously mashed repos when resuming a push.
diff --git a/bodhi/masher.py b/bodhi/masher.py
index 09be1ce..76ff9c8 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -26,7 +26,7 @@ import cPickle as pickle
from sqlobject import SQLObjectNotFound
from threading import Thread, Lock
from turbogears import config
-from os.path import exists, join, islink, isdir, dirname
+from os.path import exists, join, islink, isdir, dirname, basename
from time import sleep
from bodhi import buildsys, mail
@@ -183,11 +183,13 @@ class MashTask(Thread):
self.genmd = False
self.resume = resume
self.testing_digest = {}
+ self.composed_repos = [] # previously mashed repos from our MASHING lock
self._lock()
self._find_repos()
def _lock(self):
- """ Write out what updates we are pushing to our MASHING lock """
+ """ Write out what updates we are pushing and any successfully mashed
+ repositories to our MASHING lock """
mashed_dir = config.get('mashed_dir')
mash_stage = config.get('mashed_stage_dir')
mash_lock = join(mashed_dir, 'MASHING')
@@ -201,14 +203,34 @@ class MashTask(Thread):
if self.resume:
log.debug("Resuming previous push!")
lock = file(mash_lock, 'r')
- ups = pickle.load(lock)
+ masher_state = pickle.load(lock)
lock.close()
- for up in ups:
- try:
- up = PackageUpdate.byTitle(up)
- self.updates.add(up)
- except SQLObjectNotFound:
- log.warning("Cannot find %s" % up)
+
+ # For backwards compatability, we need to make sure we handle
+ # masher state that is just a list of updates, as well as a
+ # dictionary of updates and successfully mashed repos
+ if isinstance(masher_state, list):
+ for up in masher_state:
+ try:
+ up = PackageUpdate.byTitle(up)
+ self.updates.add(up)
+ except SQLObjectNotFound:
+ log.warning("Cannot find %s" % up)
+
+ # { 'updates' : [PackageUpdate.title,],
+ # 'repos' : ['/path_to_completed_repo',] }
+ elif isinstance(masher_state, dict):
+ for up in masher_state['updates']:
+ try:
+ up = PackageUpdate.byTitle(up)
+ self.updates.add(up)
+ except SQLObjectNotFound:
+ log.warning("Cannot find %s" % up)
+ for repo in masher_state['composed_repos']:
+ self.composed_repos.add(repo)
+ else:
+ log.error('Unknown masher lock format: %s' % masher_state)
+ raise MashTaskException
else:
log.error("Previous mash not complete! Either resume the last "
"push, or remove %s" % mash_lock)
@@ -216,7 +238,10 @@ class MashTask(Thread):
else:
log.debug("Creating lock for updates push: %s" % mash_lock)
lock = file(mash_lock, 'w')
- pickle.dump([update.title for update in self.updates], lock)
+ pickle.dump({
+ 'updates': [update.title for update in self.updates],
+ 'completed_repos': self.composed_repos,
+ }, lock)
lock.close()
def _unlock(self):
@@ -471,7 +496,16 @@ class MashTask(Thread):
t0 = time.time()
self.mashing = True
self.update_comps()
+ # {'f9-updates': '/mnt/koji/mash/updates/f9-updates-080905.0057',}
+ finished_repos = dict([('-'.join(basename(repo).split('-')[:-1]), repo)
+ for repo in self.completed_repos])
for repo in self.repos:
+ # Skip mashing this repo if we successfully mashed it previously
+ if repo in finished_repos:
+ log.info('Skipping previously mashed repo %s' % repo)
+ self.mashed_repos[repo] = finished_repos[repo]
+ continue
+
mashdir = join(config.get('mashed_dir'), repo + '-' + \
time.strftime("%y%m%d.%H%M"))
self.mashed_repos[repo] = mashdir
commit 4ed489a0a0522fc1e49d5246d110eafa25aecab5
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 03:47:25 2008 -0400
Pass the resume argument to our masher request
diff --git a/bodhi/admin.py b/bodhi/admin.py
index 843cd7b..7995281 100644
--- a/bodhi/admin.py
+++ b/bodhi/admin.py
@@ -133,7 +133,7 @@ class AdminController(Controller, SecureResource):
# If we're not The Masher, then proxy this request to it
if config.get('masher'):
- data = self._masher_request('/admin/mash', updates=updates) or {}
+ data = self._masher_request('/admin/mash', updates=updates, resume=resume) or {}
flash_log('Push request %s' % data.get('success') and 'succeeded'
or 'failed')
raise redirect('/admin/masher')
commit 598212ff661204c9b86c962aa5c5aac6923baa76
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 03:29:44 2008 -0400
Ugh.
diff --git a/bodhi/admin.py b/bodhi/admin.py
index 2a6d843..843cd7b 100644
--- a/bodhi/admin.py
+++ b/bodhi/admin.py
@@ -148,8 +148,7 @@ class AdminController(Controller, SecureResource):
def _current_mash(self):
""" Get the update list for the current mash """
- data = self._masher_request('/admin/current_mash')
- return data.get('mash')
+ return self._masher_request('/admin/current_mash')
@expose(allow_json=True)
def current_mash(self):
15 years, 8 months
4 commits - bodhi/admin.py
by Luke Macken
bodhi/admin.py | 24 ++++++++++--------------
1 file changed, 10 insertions(+), 14 deletions(-)
New commits:
commit 9ad166fda29d7d82cb38fc8535bb8af39a8a52ad
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 03:21:41 2008 -0400
Properly return our mash data from AdminController._current_mash
diff --git a/bodhi/admin.py b/bodhi/admin.py
index 598ded0..2a6d843 100644
--- a/bodhi/admin.py
+++ b/bodhi/admin.py
@@ -149,7 +149,7 @@ class AdminController(Controller, SecureResource):
def _current_mash(self):
""" Get the update list for the current mash """
data = self._masher_request('/admin/current_mash')
- return dict(mash=data.get('mash'))
+ return data.get('mash')
@expose(allow_json=True)
def current_mash(self):
commit ec161cca35eaedccf602dae6bce3aeeb3a1ef6e5
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 03:12:28 2008 -0400
Yay typos
diff --git a/bodhi/admin.py b/bodhi/admin.py
index 0173d2b..598ded0 100644
--- a/bodhi/admin.py
+++ b/bodhi/admin.py
@@ -155,7 +155,7 @@ class AdminController(Controller, SecureResource):
def current_mash(self):
""" Return details about the mash in process """
from bodhi.masher import masher
- mash_data = {'mashing': , 'updates': []}
+ mash_data = {'mashing': False, 'updates': []}
mashed_dir = config.get('mashed_dir')
mash_lock = join(mashed_dir, 'MASHING')
if exists(mash_lock):
commit 59765039c3a419afcc53ed763eb6d93ee9fcb6cf
Author: Luke Macken <lmacken(a)redhat.com>
Date: Thu Sep 4 02:56:42 2008 -0400
Hopefully fix our current_mash call
diff --git a/bodhi/admin.py b/bodhi/admin.py
index c189ab1..0173d2b 100644
--- a/bodhi/admin.py
+++ b/bodhi/admin.py
@@ -133,7 +133,7 @@ class AdminController(Controller, SecureResource):
# If we're not The Masher, then proxy this request to it
if config.get('masher'):
- data = self._masher_request('/admin/mash', updates=updates)
+ data = self._masher_request('/admin/mash', updates=updates) or {}
flash_log('Push request %s' % data.get('success') and 'succeeded'
or 'failed')
raise redirect('/admin/masher')
@@ -146,21 +146,16 @@ class AdminController(Controller, SecureResource):
flash("Updates queued for mashing")
raise redirect('/admin/masher')
- @expose(allow_json=True)
- def current_mash(self):
+ def _current_mash(self):
""" Get the update list for the current mash """
- mash_data = None
- if config.get('masher'):
- data = self._masher_request('/admin/current_mash')
- mash_data = data.get('mash')
- else:
- mash_data = self._current_mash()
- return dict(mash=mash_data)
+ data = self._masher_request('/admin/current_mash')
+ return dict(mash=data.get('mash'))
- def _current_mash(self):
+ @expose(allow_json=True)
+ def current_mash(self):
""" Return details about the mash in process """
from bodhi.masher import masher
- mash_data = {'mashing': False, 'updates': []}
+ mash_data = {'mashing': , 'updates': []}
mashed_dir = config.get('mashed_dir')
mash_lock = join(mashed_dir, 'MASHING')
if exists(mash_lock):
commit 9b3feef35ac9e458ddd737d009f5e8f02d7627f2
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 21:38:44 2008 -0400
Fix some of our lame json handling in our masher
diff --git a/bodhi/admin.py b/bodhi/admin.py
index a02e808..c189ab1 100644
--- a/bodhi/admin.py
+++ b/bodhi/admin.py
@@ -125,10 +125,11 @@ class AdminController(Controller, SecureResource):
handling all of the update requests, composing fresh repositories,
generating and sending update notices, closing bugs, etc.
"""
- if request_format() == 'json':
- updates = simplejson.loads(updates.replace("u'", "\"").replace("'", "\""))
if not isinstance(updates, list):
- updates = [updates]
+ if isinstance(updates, basestring):
+ updates = simplejson.loads(updates.replace("u'", "\"").replace("'", "\""))
+ else:
+ updates = [updates]
# If we're not The Masher, then proxy this request to it
if config.get('masher'):
15 years, 8 months
2 commits - bodhi/config bodhi/masher.py bodhi.spec
by Luke Macken
bodhi.spec | 2 +-
bodhi/config/mash.conf | 4 ++--
bodhi/masher.py | 6 +++---
3 files changed, 6 insertions(+), 6 deletions(-)
New commits:
commit 02451f312252430b3c68ae1e0c947d2885036c0d
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 20:57:35 2008 -0400
Update our mash config for production
diff --git a/bodhi.spec b/bodhi.spec
index a10fd1e..34aca07 100644
--- a/bodhi.spec
+++ b/bodhi.spec
@@ -3,7 +3,7 @@
Name: bodhi
Version: 0.5.2
-Release: 10%{?dist}
+Release: 11%{?dist}
Summary: A modular framework that facilitates publishing software updates
Group: Applications/Internet
License: GPLv2+
diff --git a/bodhi/config/mash.conf b/bodhi/config/mash.conf
index e8a4d44..2123043 100644
--- a/bodhi/config/mash.conf
+++ b/bodhi/config/mash.conf
@@ -1,8 +1,8 @@
[defaults]
-symlink = True
+symlink = False
configdir = /etc/bodhi/
buildhost = http://koji.fedoraproject.org/kojihub
repodir = /mnt/koji
fork = True
use_sqlite = True
-strict_keys = False
+strict_keys = True
commit 937535596b25eb7a6a95de8ddd2f5f2a5c57f5e4
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 20:49:21 2008 -0400
Raise a MashTaskException if any of our repo sanity checks fail.
diff --git a/bodhi/masher.py b/bodhi/masher.py
index dd89470..09be1ce 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -418,21 +418,21 @@ class MashTask(Thread):
for arch in config.get('arches').split():
if arch not in arches and arch not in [arch.split('.')[0] for arch in arches]:
self.error_log("Cannot find arch %s in %s" % (arch, newrepo))
- return
+ raise MashTaskException
# sanity check our repodata
try:
sanity_check_repodata(join(newrepo, arch, 'repodata'))
except Exception, e:
self.error_log("Repodata sanity check failed!\n%s" % str(e))
- return
+ raise MashTaskException
# make sure that mash didn't symlink our packages
for pkg in os.listdir(join(newrepo, arches[0])):
if pkg.endswith('.rpm'):
if islink(join(newrepo, arches[0], pkg)):
self.error_log("Mashed repository full of symlinks!")
- return
+ raise MashTaskException
break
# move the new repo to our mash stage
15 years, 8 months
10 commits - bodhi/admin.py bodhi.egg-info/entry_points.txt bodhi/jobs.py bodhi/masher.py bodhi.spec bodhi/util.py setup.py
by Luke Macken
bodhi.egg-info/entry_points.txt | 3 +++
bodhi.spec | 9 ++++++++-
bodhi/admin.py | 18 +++++++++++-------
bodhi/jobs.py | 2 +-
bodhi/masher.py | 18 ++++++++++++++----
bodhi/util.py | 1 +
setup.py | 3 +++
7 files changed, 41 insertions(+), 13 deletions(-)
New commits:
commit 27ee1006ff1efea72efe24b6580edec706d6ff45
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 20:28:55 2008 -0400
* Wed Sep 03 2008 Luke Macken <lmacken(a)redhat.com> - 0.5.2-2
- Add the masher deps to BuildRequires, since it now resides
on the turbogears.extensions entry point and will be
imported by pkg_resources at build time.
diff --git a/bodhi.spec b/bodhi.spec
index 9c7f850..a10fd1e 100644
--- a/bodhi.spec
+++ b/bodhi.spec
@@ -3,7 +3,7 @@
Name: bodhi
Version: 0.5.2
-Release: 1%{?dist}
+Release: 10%{?dist}
Summary: A modular framework that facilitates publishing software updates
Group: Applications/Internet
License: GPLv2+
@@ -13,7 +13,9 @@ Source0: bodhi-%{version}.tar.bz2
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
BuildArch: noarch
+BuildRequires: python-setuptools
BuildRequires: python-setuptools-devel
+BuildRequires: python-fedora python-TurboMail TurboGears yum
BuildRequires: python-devel
BuildRequires: TurboGears
@@ -108,6 +110,11 @@ rm -rf bodhi/tests bodhi/tools/test-bodhi.py
%changelog
+* Wed Sep 03 2008 Luke Macken <lmacken(a)redhat.com> - 0.5.2-2
+- Add the masher deps to BuildRequires, since it now resides
+ on the turbogears.extensions entry point and will be
+ imported by pkg_resources at build time.
+
* Wed Sep 03 2008 Luke Macken <lmacken(a)redhat.com> - 0.5.2-1
- Latest upstream bugfix release
commit 6cc12910d633cdadd0a63d534fd472487e066a8e
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 20:27:14 2008 -0400
Debugging statements are fun
diff --git a/bodhi/masher.py b/bodhi/masher.py
index be2589b..dd89470 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -725,6 +725,7 @@ class MashTask(Thread):
def start_extension():
global masher
+ log.info("Starting masher extension")
masher = Masher()
def shutdown_extension():
diff --git a/bodhi/util.py b/bodhi/util.py
index d618c46..4706c14 100644
--- a/bodhi/util.py
+++ b/bodhi/util.py
@@ -309,6 +309,7 @@ class Singleton(object):
def __new__(cls, *args, **kw):
if not '_instance' in cls.__dict__:
+ log.debug("Creating new singleton instance!")
cls._instance = object.__new__(cls)
return cls._instance
commit 81edec36bddbd7917b7b7e10a6e495c7174cad4e
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 20:26:26 2008 -0400
Delay our metrics refresh a bit
diff --git a/bodhi/jobs.py b/bodhi/jobs.py
index 8501df4..ae37db3 100644
--- a/bodhi/jobs.py
+++ b/bodhi/jobs.py
@@ -178,5 +178,5 @@ def schedule():
log.debug("Scheduling refresh_metrics job")
scheduler.add_interval_task(action=refresh_metrics,
taskname='Refresh our metrics',
- initialdelay=0,
+ initialdelay=1200,
interval=172800)
commit bc5b92295376f53baf608b7085c5db94c0d10ddd
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 20:26:14 2008 -0400
Make the Masher a real turbogears.extension
diff --git a/bodhi.egg-info/entry_points.txt b/bodhi.egg-info/entry_points.txt
index be5abad..a92b426 100644
--- a/bodhi.egg-info/entry_points.txt
+++ b/bodhi.egg-info/entry_points.txt
@@ -1,3 +1,6 @@
+[turbogears.extensions]
+masher = bodhi.masher
+
[console_scripts]
bodhi-rmrelease = bodhi.tools.rmrelease:main
start-bodhi = bodhi.commands:start
diff --git a/setup.py b/setup.py
index 03cb807..f9f40a5 100644
--- a/setup.py
+++ b/setup.py
@@ -174,5 +174,8 @@ setup(
'bodhi-devinit = bodhi.tools.dev_init:main',
'bodhi-rmrelease = bodhi.tools.rmrelease:main',
),
+ 'turbogears.extensions': (
+ 'masher = bodhi.masher'
+ ),
}
)
commit a9bb756f273a5b326cf35f6c736ef3f9d5ac53ba
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 20:23:56 2008 -0400
Fix masher interaction from our AdminController
diff --git a/bodhi/admin.py b/bodhi/admin.py
index ee9a90f..a02e808 100644
--- a/bodhi/admin.py
+++ b/bodhi/admin.py
@@ -30,7 +30,6 @@ from fedora.tg.util import request_format
from fedora.client.proxyclient import ProxyClient
from bodhi.util import flash_log
-from bodhi.masher import Masher
from bodhi.model import Release, PackageUpdate
@@ -55,16 +54,18 @@ class AdminController(Controller, SecureResource):
data = {'masher_str': 'Unable to contact the masher','tags': []}
return dict(masher_str=data['masher_str'], tags=data['tags'])
else:
+ from bodhi.masher import masher
tags = []
for release in Release.select():
tags.append('%s-updates' % release.dist_tag.split('-')[1])
tags.append('%s-updates-testing' % release.dist_tag.split('-')[1])
- return dict(masher_str=str(Masher()), tags=tags)
+ return dict(masher_str=str(masher), tags=tags)
@expose(template='bodhi.templates.text')
def lastlog(self):
""" Return the last mash log """
- (logfile, data) = Masher().lastlog()
+ from bodhi.masher import masher
+ (logfile, data) = masher.lastlog()
return dict(title=logfile, text=data)
@expose(allow_json=True)
@@ -80,7 +81,8 @@ class AdminController(Controller, SecureResource):
flash_log("Mash request %s" %
data.get('success', 'failed'))
else:
- Masher().mash_tags([tag])
+ from bodhi.masher import masher
+ masher.mash_tags([tag])
flash_log("Mashing tag: %s" % tag)
raise redirect('/admin/masher')
@@ -135,8 +137,9 @@ class AdminController(Controller, SecureResource):
or 'failed')
raise redirect('/admin/masher')
- Masher().queue([PackageUpdate.byTitle(title) for title in updates],
- resume=resume)
+ from bodhi.masher import masher
+ masher.queue([PackageUpdate.byTitle(title) for title in updates],
+ resume=resume)
if request_format() == 'json':
return dict(success=True)
flash("Updates queued for mashing")
@@ -155,6 +158,7 @@ class AdminController(Controller, SecureResource):
def _current_mash(self):
""" Return details about the mash in process """
+ from bodhi.masher import masher
mash_data = {'mashing': False, 'updates': []}
mashed_dir = config.get('mashed_dir')
mash_lock = join(mashed_dir, 'MASHING')
@@ -162,7 +166,7 @@ class AdminController(Controller, SecureResource):
mash_lock = file(mash_lock)
mashing_updates = pickle.load(mash_lock)
mash_lock.close()
- mash_data['mashing'] = Masher().mashing
+ mash_data['mashing'] = masher.mashing
mash_data['updates'] = mashing_updates
return mash_data
commit f4d772f4f86f616936d64675a2a9cab8c32dcb7b
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 18:38:12 2008 -0400
Make our Masher class *not* be a Singleton, since we keep one instance per AdminController now.
diff --git a/bodhi/masher.py b/bodhi/masher.py
index a05c6db..be2589b 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -30,7 +30,7 @@ from os.path import exists, join, islink, isdir, dirname
from time import sleep
from bodhi import buildsys, mail
-from bodhi.util import synchronized, sanity_check_repodata, Singleton
+from bodhi.util import synchronized, sanity_check_repodata
from bodhi.model import PackageUpdate
from bodhi.metadata import ExtendedMetadata
from bodhi.exceptions import MashTaskException
@@ -55,7 +55,7 @@ def get_mash_conf():
log.error("No mash configuration found!")
return None
-class Masher(Singleton):
+class Masher(object):
""" The Masher.
This is an extension that is in charge of queueing and dispatching update
commit bfc7e7df10bb5f4f949826f1a4f42753aee8f9f6
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 18:12:20 2008 -0400
Add another newkey hack to fix our repodata caching
diff --git a/bodhi/masher.py b/bodhi/masher.py
index f7307fc..a05c6db 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -668,6 +668,10 @@ class MashTask(Thread):
olduinfo = join(config.get('mashed_dir'), '%s.repodata' % repo,
'i386', 'updateinfo.xml.gz')
olduinfo = exists(olduinfo) and olduinfo or None
+ if not olduinfo:
+ olduinfo = join(config.get('mashed_dir'), '%s.repodata' % repo,
+ 'i386.newkey', 'updateinfo.xml.gz')
+ olduinfo = exists(olduinfo) and olduinfo or None
repo = join(mashdir, repo)
log.debug("Generating updateinfo.xml.gz for %s" % repo)
uinfo = ExtendedMetadata(repo, olduinfo)
commit 9c0704adb429b9a6eb4d8208d101070fca323076
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 17:57:19 2008 -0400
Make the sanity checks work with our normal tree, and the .newkey trees
diff --git a/bodhi/masher.py b/bodhi/masher.py
index aa91df8..f7307fc 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -416,7 +416,7 @@ class MashTask(Thread):
# make sure the new repository has our arches
for arch in config.get('arches').split():
- if arch not in arches + ['%s.newkey' % arch]:
+ if arch not in arches and arch not in [arch.split('.')[0] for arch in arches]:
self.error_log("Cannot find arch %s in %s" % (arch, newrepo))
return
@@ -572,11 +572,15 @@ class MashTask(Thread):
log.error("Exception thrown in MashTask %d" % self.id)
self.error_log(str(e))
log.exception(str(e))
- except MashTaskException:
+ except MashTaskException, e:
+ log.error("MashTaskException thrown! %s" % str(e))
self.success = False
if self.success:
+ log.debug("Success! Unlocking repo")
self._unlock()
+
+ log.debug("MashTask done")
masher.done(self)
def add_to_digest(self,update):
@@ -599,6 +603,7 @@ class MashTask(Thread):
Send digest mail to mailing lists
'''
for prefix, content in self.testing_digest.items():
+ log.debug("Sending digest for updates-testing %s" % prefix)
maildata = u'The following builds have been pushed to %s updates-testing\n\n' % prefix
# get a list af all nvr's
updlist = content.keys()
commit abf681e95b11ce10c9abcbde77e7979821000e35
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 17:09:28 2008 -0400
Fix another lame typo
diff --git a/bodhi/masher.py b/bodhi/masher.py
index d06e046..aa91df8 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -416,7 +416,7 @@ class MashTask(Thread):
# make sure the new repository has our arches
for arch in config.get('arches').split():
- if arch not in arches + '%s.newkey' % arch:
+ if arch not in arches + ['%s.newkey' % arch]:
self.error_log("Cannot find arch %s in %s" % (arch, newrepo))
return
commit be54111756f9415ec5337ab5bbac52366a784c4d
Author: Luke Macken <lmacken(a)redhat.com>
Date: Wed Sep 3 14:47:43 2008 -0400
Fix some of our masher sanity checking to work with our newkey hack
diff --git a/bodhi/masher.py b/bodhi/masher.py
index 3874837..d06e046 100644
--- a/bodhi/masher.py
+++ b/bodhi/masher.py
@@ -416,7 +416,7 @@ class MashTask(Thread):
# make sure the new repository has our arches
for arch in config.get('arches').split():
- if arch not in arches:
+ if arch not in arches + '%s.newkey' % arch:
self.error_log("Cannot find arch %s in %s" % (arch, newrepo))
return
15 years, 8 months