[copr] master: fix the host_list issue in the dispatcher for terminating instances, too :( (f6508e3)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : master
>---------------------------------------------------------------
commit f6508e3a3120f496ef8234bb12b27cee069e9df9
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Wed Jan 16 09:06:46 2013 -0500
fix the host_list issue in the dispatcher for terminating instances, too :(
>---------------------------------------------------------------
backend/dispatcher.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/backend/dispatcher.py b/backend/dispatcher.py
index 4b828b1..5fa6aa0 100644
--- a/backend/dispatcher.py
+++ b/backend/dispatcher.py
@@ -154,7 +154,7 @@ class Worker(multiprocessing.Process):
stats = callbacks.AggregateStats()
playbook_cb = SilentPlaybookCallbacks(verbose=False)
runner_cb = callbacks.DefaultRunnerCallbacks()
- play = ansible.playbook.PlayBook(host_list=[ip], stats=stats, playbook=self.opts.terminate_playbook,
+ play = ansible.playbook.PlayBook(host_list=ip +',', stats=stats, playbook=self.opts.terminate_playbook,
callbacks=playbook_cb, runner_callbacks=runner_cb,
remote_user='root')
11 years, 5 months
[copr] master: ansible 0.9 changes the host_list option from taking an actual list to a comma separated list - grumble (1ce0561)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : master
>---------------------------------------------------------------
commit 1ce0561e2586735ae416dc0187a990395ec1de91
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Wed Jan 16 02:00:07 2013 -0500
ansible 0.9 changes the host_list option from taking an actual list to
a comma separated list - grumble
>---------------------------------------------------------------
backend/mockremote.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/backend/mockremote.py b/backend/mockremote.py
index 8c81580..2e1e19f 100755
--- a/backend/mockremote.py
+++ b/backend/mockremote.py
@@ -96,7 +96,7 @@ def get_ans_results(results, hostname):
def _create_ans_conn(hostname, username, timeout):
ans_conn = ansible.runner.Runner(remote_user=username,
- host_list=[hostname], pattern=hostname, forks=1,
+ host_list=hostname + ',', pattern=hostname, forks=1,
timeout=timeout)
return ans_conn
@@ -363,7 +363,8 @@ class Builder(object):
raise BuilderError('%s could not be resolved' % self.hostname)
# connect as user
- ans = ansible.runner.Runner(host_list=[self.hostname], pattern='*',
+
+ ans = ansible.runner.Runner(host_list=self.hostname + ',', pattern='*',
remote_user=self.username, forks=1, timeout=20)
ans.module_name = "shell"
ans.module_args = str("/bin/rpm -q mock rsync")
11 years, 5 months
Arbitrary chroots set for copr
by Bohuslav Kabrda
Hi all,
as you have probably figured out of the bunch of commits, I have implemented selecting arbitrary chroots set for copr. That means that you can now have a copr with e.g. fedora-rawhide-x86_64 and fedora-18-i386. I've tested, but not that much, so if anyone can try it out and tell me about any bugs, I'd be glad :)
Thanks.
--
Regards,
Bohuslav "Slavek" Kabrda.
11 years, 5 months
[copr] master: the hob grab event log is just too noisy - removing (09e5152)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : master
>---------------------------------------------------------------
commit 09e5152ba07635722c5f615705d1a9d453e6b6ea
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Tue Jan 15 16:29:01 2013 -0500
the hob grab event log is just too noisy - removing
>---------------------------------------------------------------
copr-be.py | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
diff --git a/copr-be.py b/copr-be.py
index e76e54f..dfcd4c0 100644
--- a/copr-be.py
+++ b/copr-be.py
@@ -38,7 +38,6 @@ class CoprJobGrab(multiprocessing.Process):
self.events.put({'when':time.time(), 'who':'jobgrab', 'what':what})
def fetch_jobs(self):
- self.event('fetching jobs')
try:
r = requests.get('%s/waiting_builds/' % self.opts.frontend_url) # auth stuff here? maybe/maybenot
except requests.RequestException, e:
11 years, 5 months
[copr] master: typos or thinkos pychecker unveiled for me in mockremote fixed (bc8ad33)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : master
>---------------------------------------------------------------
commit bc8ad33549ef654c2b049d733ec56e80ae431200
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Tue Jan 15 16:15:01 2013 -0500
typos or thinkos pychecker unveiled for me in mockremote fixed
>---------------------------------------------------------------
backend/mockremote.py | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/backend/mockremote.py b/backend/mockremote.py
index 45af993..8c81580 100755
--- a/backend/mockremote.py
+++ b/backend/mockremote.py
@@ -203,7 +203,7 @@ class CliLogCallBack(DefaultCallBack):
try:
open(self.logfn, 'a').write(str(now) + ':' + msg + '\n')
except (IOError, OSError), e:
- print >>sys.stderr, 'Could not write to logfile %s - %s' % (self.lf, str(e))
+ print >>sys.stderr, 'Could not write to logfile %s - %s' % (self.logfn, str(e))
if not self.quiet:
print msg
@@ -299,7 +299,7 @@ class Builder(object):
#print ' Running %s on %s' % (buildcmd, hostname)
# run the mockchain command async
# this runs it sync - FIXME
- self.callback.log('executing: %r' % buildcmd)
+ self.mockremote.callback.log('executing: %r' % buildcmd)
self.conn.module_name="shell"
self.conn.module_args = str(buildcmd)
results = self.conn.run()
11 years, 5 months
[copr] master: seperate out what pkg is being logged from mockchain log (b2f5c8f)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : master
>---------------------------------------------------------------
commit b2f5c8f101a85700dae15fab111948790528d5e8
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Tue Jan 15 16:07:29 2013 -0500
seperate out what pkg is being logged from mockchain log
>---------------------------------------------------------------
backend/mockremote.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/backend/mockremote.py b/backend/mockremote.py
index 2d13a57..45af993 100755
--- a/backend/mockremote.py
+++ b/backend/mockremote.py
@@ -487,7 +487,7 @@ class MockRemote(object):
if not os.path.exists(self.destdir + '/' + self.chroot):
os.makedirs(self.destdir + '/' + self.chroot)
r_log = open(self.destdir + '/' + self.chroot + '/mockchain.log', 'a')
- r_log.write('%s\n' % pkg)
+ r_log.write('\n\n%s\n\n' % pkg)
r_log.write(b_out)
if b_err:
r_log.write('\nstderr\n')
11 years, 5 months
[copr] master: - return status to main log for builds - output the complete mockchain command that will be run since it seems that required repos are not being included (ebe3000)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : master
>---------------------------------------------------------------
commit ebe3000ddce455ab295a3180b5e9e150167c8cd5
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Tue Jan 15 15:48:29 2013 -0500
- return status to main log for builds
- output the complete mockchain command that will be run since it seems that
required repos are not being included
>---------------------------------------------------------------
backend/dispatcher.py | 2 +-
backend/mockremote.py | 1 +
2 files changed, 2 insertions(+), 1 deletions(-)
diff --git a/backend/dispatcher.py b/backend/dispatcher.py
index 5a73a08..4b828b1 100644
--- a/backend/dispatcher.py
+++ b/backend/dispatcher.py
@@ -317,7 +317,7 @@ class Worker(multiprocessing.Process):
job.status = status
self.return_results(job)
self.callback.log('worker finished build: %s' % ip)
- self.event('build end: user:%s copr:%s build:%s ip:%s pid:%s' % (job.user_name, job.copr_name, job.build_id, ip, self.pid))
+ self.event('build end: user:%s copr:%s build:%s ip:%s pid:%s status:%s' % (job.user_name, job.copr_name, job.build_id, ip, self.pid, job.status))
# clean up the instance
if self.create:
self.terminate_instance(ip)
diff --git a/backend/mockremote.py b/backend/mockremote.py
index 7017a51..2d13a57 100755
--- a/backend/mockremote.py
+++ b/backend/mockremote.py
@@ -299,6 +299,7 @@ class Builder(object):
#print ' Running %s on %s' % (buildcmd, hostname)
# run the mockchain command async
# this runs it sync - FIXME
+ self.callback.log('executing: %r' % buildcmd)
self.conn.module_name="shell"
self.conn.module_args = str(buildcmd)
results = self.conn.run()
11 years, 5 months
[copr] master: minor changes to log output (0f3d301)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : master
>---------------------------------------------------------------
commit 0f3d30104a3dca947ef15cf00fbe0d6e6f3dc3c8
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Tue Jan 15 14:02:33 2013 -0500
minor changes to log output
>---------------------------------------------------------------
backend/dispatcher.py | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/backend/dispatcher.py b/backend/dispatcher.py
index 6762ac5..5a73a08 100644
--- a/backend/dispatcher.py
+++ b/backend/dispatcher.py
@@ -270,10 +270,10 @@ class Worker(multiprocessing.Process):
job.started_on = time.time()
self.mark_started(job)
- self.event('job start: user:%s copr:%s build:%s ip:%s pid:%s' % (job.user_name, job.copr_name, job.build_id, ip, self.pid))
+ self.event('build start: user:%s copr:%s build:%s ip:%s pid:%s' % (job.user_name, job.copr_name, job.build_id, ip, self.pid))
for chroot in job.chroots:
- self.event('job chroot start: chroot:%s user:%s copr:%s build:%s ip:%s pid:%s' % (chroot, job.user_name, job.copr_name, job.build_id, ip, self.pid))
+ self.event('chroot start: chroot:%s user:%s copr:%s build:%s ip:%s pid:%s' % (chroot, job.user_name, job.copr_name, job.build_id, ip, self.pid))
chroot_destdir = job.destdir + '/' + chroot
# setup our target dir locally
if not os.path.exists(chroot_destdir):
@@ -317,7 +317,7 @@ class Worker(multiprocessing.Process):
job.status = status
self.return_results(job)
self.callback.log('worker finished build: %s' % ip)
- self.event('job end: user:%s copr:%s build:%s ip:%s pid:%s' % (job.user_name, job.copr_name, job.build_id, ip, self.pid))
+ self.event('build end: user:%s copr:%s build:%s ip:%s pid:%s' % (job.user_name, job.copr_name, job.build_id, ip, self.pid))
# clean up the instance
if self.create:
self.terminate_instance(ip)
11 years, 5 months
[copr] master: make the event log have a bit more detail and be simpler to follow (c74ba30)
by skvidal＠fedorahosted.org
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : master
>---------------------------------------------------------------
commit c74ba308810fc05666fca8444446e24ec1a6e312
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Tue Jan 15 12:27:23 2013 -0500
make the event log have a bit more detail and be simpler to follow
>---------------------------------------------------------------
backend/dispatcher.py | 11 +++++++----
copr-be.py | 4 ++--
2 files changed, 9 insertions(+), 6 deletions(-)
diff --git a/backend/dispatcher.py b/backend/dispatcher.py
index 1485285..6762ac5 100644
--- a/backend/dispatcher.py
+++ b/backend/dispatcher.py
@@ -178,6 +178,8 @@ class Worker(multiprocessing.Process):
jobdata.repos.append(jobdata.results)
jobdata.copr_id = build['copr']['id']
jobdata.user_id = build['user_id']
+ jobdata.user_name = build['copr']['owner']['name']
+ jobdata.copr_name = build['copr']['name']
return jobdata
# maybe we move this to the callback?
@@ -206,7 +208,7 @@ class Worker(multiprocessing.Process):
# maybe we move this to the callback?
def mark_started(self, job):
- self.event('job start: user:%s copr:%s build:%s ip:%s pid:%s' % (job.user_id, job.copr_id, job.build_id, self.ip, self.pid))
+
build = {'id':job.build_id,
'started_on': job.started_on,
'results': job.results,
@@ -218,7 +220,6 @@ class Worker(multiprocessing.Process):
# maybe we move this to the callback?
def return_results(self, job):
- self.event('job end: user:%s copr:%s build:%s ip:%s pid:%s' % (job.user_id, job.copr_id, job.build_id, self.ip, self.pid))
self.callback.log('%s status %s. Took %s seconds' % (job.build_id, job.status, job.ended_on - job.started_on))
build = {'id':job.build_id,
@@ -269,8 +270,10 @@ class Worker(multiprocessing.Process):
job.started_on = time.time()
self.mark_started(job)
- for chroot in job.chroots:
+ self.event('job start: user:%s copr:%s build:%s ip:%s pid:%s' % (job.user_name, job.copr_name, job.build_id, ip, self.pid))
+ for chroot in job.chroots:
+ self.event('job chroot start: chroot:%s user:%s copr:%s build:%s ip:%s pid:%s' % (chroot, job.user_name, job.copr_name, job.build_id, ip, self.pid))
chroot_destdir = job.destdir + '/' + chroot
# setup our target dir locally
if not os.path.exists(chroot_destdir):
@@ -309,12 +312,12 @@ class Worker(multiprocessing.Process):
if mr.failed:
status = 0
self.callback.log('Finished build: id=%r builder=%r timeout=%r destdir=%r chroot=%r repos=%r' % (job.build_id, ip, job.timeout, job.destdir, chroot, str(job.repos)))
-
job.ended_on = time.time()
job.status = status
self.return_results(job)
self.callback.log('worker finished build: %s' % ip)
+ self.event('job end: user:%s copr:%s build:%s ip:%s pid:%s' % (job.user_name, job.copr_name, job.build_id, ip, self.pid))
# clean up the instance
if self.create:
self.terminate_instance(ip)
diff --git a/copr-be.py b/copr-be.py
index 78256a7..e76e54f 100644
--- a/copr-be.py
+++ b/copr-be.py
@@ -35,7 +35,7 @@ class CoprJobGrab(multiprocessing.Process):
self.added_jobs = []
def event(self, what):
- self.events.put({'when':time.time(), 'who':'job', 'what':what})
+ self.events.put({'when':time.time(), 'who':'jobgrab', 'what':what})
def fetch_jobs(self):
self.event('fetching jobs')
@@ -99,7 +99,7 @@ class CoprLog(multiprocessing.Process):
def log(self, event):
when = time.strftime('%F %T', time.gmtime(event['when']))
- msg = '%s : %s %s' % (when, event['who'], event['what'].strip())
+ msg = '%s : %s: %s' % (when, event['who'], event['what'].strip())
try:
open(self.logfile, 'a').write(msg + '\n')
11 years, 5 months
[copr] master: - sleep a bit between fetchjob actions - actually log stuff :) (7437f45)
by Seth Vidal
Repository : http://git.fedorahosted.org/cgit/copr.git
On branch : master
>---------------------------------------------------------------
commit 7437f45c069593a1ad7c3b1995e41e0301e6be29
Author: Seth Vidal <skvidal(a)fedoraproject.org>
Date: Tue Jan 15 12:14:27 2013 -0500
- sleep a bit between fetchjob actions
- actually log stuff :)
>---------------------------------------------------------------
copr-be.py | 8 +++++---
1 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/copr-be.py b/copr-be.py
index 881c30d..78256a7 100644
--- a/copr-be.py
+++ b/copr-be.py
@@ -73,6 +73,8 @@ class CoprJobGrab(multiprocessing.Process):
self.jobs.put(f)
self.added_jobs.append(n)
self.event('adding to work queue id %s' % n)
+ time.sleep(self.opts.sleeptime)
+
class CoprLog(multiprocessing.Process):
"""log mechanism where items from the events queue get recorded"""
@@ -109,9 +111,9 @@ class CoprLog(multiprocessing.Process):
def run(self):
abort = False
while not abort:
- for e in self.events.get():
- if 'when' in e and 'who' in e and 'what' in e:
- self.log(e)
+ e = self.events.get()
+ if 'when' in e and 'who' in e and 'what' in e:
+ self.log(e)
class CoprBackend(object):
"""core process - starts/stops/initializes workers"""
11 years, 5 months
