setting up a koji stage env
by Mike McLean
There are a number of ways one might use to set up a stage instance of
koji that is based on a production instance. This is one that I have
used on my koji systems.
The two major bits of data that koji contains are:
1) the koji db
2) /mnt/koji
With the db it is pretty easy to make a duplicate db based on a
snapshot. The filesystem can be trickier. If you're lucky, your
filesystem may support writable snapshots, in which case, awesome.
Otherwise, you might like this approach.
Koji has supported the notion of split storage since 1.7.0. There's a
brief summary of how that works here:
https://lists.fedoraproject.org/pipermail/buildsys/2012-May/003892.html
In this staging approach, we mount the production volume READ-ONLY on
the staging systems, and make /mnt/koji/vol/prod on the _stage_ fs point
to that RO mount.
When we restore the database from the production snapshot, we update the
builds to point to the prod volume (seen the attached example script).
The stage koji instance will then know where to find them.
Because the builds have changed locations, the old repodata becomes
invalid, so all the repos need to be regenerated. The attached script
just marks them expired (which should trigger the kojira running in
stage to regen them).
The attached example sql script is based on one that I have used before.
It will need to be adapted to your particular case.
8 years, 10 months
Any chance for koji 1.10?
by Pat Riehecky
I realize we are all excited about koji 2.0, but there are a few minor
fixes in the current koji tree I'd love to see on my build servers.
Any chance for a koji 1.10 release?
Pat
--
Pat Riehecky
Scientific Linux developer
Fermi National Accelerator Laboratory
www.fnal.gov
www.scientificlinux.org
8 years, 10 months
[PATCH] fix more #pylint issues
by Mike McLean
---
hub/kojixmlrpc.py | 2 +-
koji/__init__.py | 4 ++--
koji/daemon.py | 2 +-
koji/server.py | 2 +-
4 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/hub/kojixmlrpc.py b/hub/kojixmlrpc.py
index 0e51012..efb99a6 100644
--- a/hub/kojixmlrpc.py
+++ b/hub/kojixmlrpc.py
@@ -192,7 +192,7 @@ class HandlerAccess(object):
return self.__reg.get(__name)(*args, **kwargs)
def get(self, name):
- return self.__Reg.get(name)
+ return self.__reg.get(name)
class ModXMLRPCRequestHandler(object):
diff --git a/koji/__init__.py b/koji/__init__.py
index c3a0990..e38a208 100644
--- a/koji/__init__.py
+++ b/koji/__init__.py
@@ -2037,10 +2037,10 @@ class ClientSession(object):
chk_opts['verify'] = 'adler32'
result = self._callMethod('checkUpload', (path, name), chk_opts)
if int(result['size']) != ofs:
- raise koji.GenericError, "Uploaded file is wrong length: %s/%s, %s != %s" \
+ raise GenericError, "Uploaded file is wrong length: %s/%s, %s != %s" \
% (path, name, result['sumlength'], ofs)
if problems and result['hexdigest'] != full_chksum.hexdigest():
- raise koji.GenericError, "Uploaded file has wrong checksum: %s/%s, %s != %s" \
+ raise GenericError, "Uploaded file has wrong checksum: %s/%s, %s != %s" \
% (path, name, result['hexdigest'], full_chksum.hexdigest())
self.logger.debug("Fast upload: %s complete. %i bytes in %.1f seconds", localfile, size, t2)
diff --git a/koji/daemon.py b/koji/daemon.py
index d76355b..0570faa 100644
--- a/koji/daemon.py
+++ b/koji/daemon.py
@@ -266,7 +266,7 @@ class SCM(object):
# check for validity: params should be empty, query may be empty, everything else should be populated
if params :
- raise koji.GenericError, 'Unable to parse SCM URL: %s . Param element %s should be empty.' % (self.url,param)
+ raise koji.GenericError, 'Unable to parse SCM URL: %s . Params element %s should be empty.' % (self.url, params)
if not scheme :
raise koji.GenericError, 'Unable to parse SCM URL: %s . Could not find the scheme element.' % self.url
if not netloc :
diff --git a/koji/server.py b/koji/server.py
index 7cc8be8..52f13f5 100644
--- a/koji/server.py
+++ b/koji/server.py
@@ -158,7 +158,7 @@ class WSGIWrapper(object):
for chunk in result:
if chunk and not self.set_headers:
raise RuntimeError, "write() called before start_response()"
- write(data)
+ write(chunk)
if not req.bytes_sent:
#application sent nothing back
req.set_content_length(0)
--
1.9.3
8 years, 10 months
[PATCH] handle symlinks in BuildMavenTask._zip_dir()
by Mike McLean
---
builder/kojid | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/builder/kojid b/builder/kojid
index dcd0840..ad2cf1f 100755
--- a/builder/kojid
+++ b/builder/kojid
@@ -1281,7 +1281,17 @@ class BuildMavenTask(BaseBuildTask):
dirnames.remove(skip)
for filename in filenames:
filepath = os.path.join(dirpath, filename)
- zfo.write(filepath, filepath[roottrim:])
+ if os.path.islink(filepath):
+ content = os.readlink(filepath)
+ st = os.lstat(filepath)
+ mtime = time.localtime(st.st_mtime)
+ info = zipfile.ZipInfo(filepath[roottrim:])
+ info.external_attr |= 0120000 << 16L # symlink file type
+ info.compress_type = zipfile.ZIP_STORED
+ info.date_time = mtime[:6]
+ zfo.writestr(info, content)
+ else:
+ zfo.write(filepath, filepath[roottrim:])
zfo.close()
def checkHost(self, hostdata):
--
1.9.3
8 years, 10 months
[PATCH] Avoid errors logging anonymous retries (ticket 317)
by Mike McLean
see: https://fedorahosted.org/koji/ticket/317
---
koji/__init__.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/koji/__init__.py b/koji/__init__.py
index 4ca9c08..36772bb 100644
--- a/koji/__init__.py
+++ b/koji/__init__.py
@@ -1946,7 +1946,7 @@ class ClientSession(object):
if self.logger.isEnabledFor(logging.DEBUG):
tb_str = ''.join(traceback.format_exception(*sys.exc_info()))
self.logger.debug(tb_str)
- self.logger.info("Try #%d for call %s (%s) failed: %s", tries, self.callnum, name, e)
+ self.logger.info("Try #%s for call %s (%s) failed: %s", tries, self.callnum, name, e)
if tries > 1:
# first retry is immediate, after that we honor retry_interval
time.sleep(interval)
--
1.9.3
8 years, 10 months
Koji 2.0 planning
by Mike McLean
It's been eight years since koji 1.0. In that time Koji has grown a lot,
but always incrementally and with great care to avoid breaking
compatibility. Over the years, we've found numerous things that we
wanted to add or change, but that we dismissed as too big, too
complicated, or too invasive.
Bumping the major release number gives use the freedom to shake things
up a bit. Koji 2.0 is about making major changes, otherwise it would
just be koji 1.13.
Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is
sufficiently stable. Some features (e.g. content generators) will also
appear in 1.x.
The list below is probably not complete, it is ambitious, and it is
certainly open to discussion. If you are a Koji user, or otherwise
invested in Koji, then you are encouraged to join in. I expect there
will be a lot to say, so I've created a new mailing list devoted
specifically to Koji development.
https://lists.fedorahosted.org/mailman/listinfo/koji-devel
Also, apologies for the tense summary listing. I can certain expound on
any of these as necessary. I hope this suffices to start some conversation.
= High level goals =
• better documentation
• more community involvement
• refactor/modernize code
• more modular design
∘ content generators
∘ broader, better plugin framework
• better support for different types of build processes
• better support for for different types build output
• make hard-wired restrictions more configurable
• easier to deploy
• better qa process
• better release process
= Highlights/Major changes =
• python3 support
∘ the bulk of the code will target python 2.6 + python-six
∘ we'll create a basic client lib for older systems (e.g rhel5
clients/builders)
• drop xmlrpc in favor a json based rpc
• build namespaces
∘ allow for use cases that require multiple builds of the same NVR
(or NVRA)
• refactor task scheduling
• extend content generator support
∘ content generators will land in 1.x fairly soon, but in 2.0 they
will be more integral
∘ refactor kojid to use content generator calls
∘ (possibly) tighter integration in the db
• unify handling of rpms with other build types in the db
∘ e.g. unify rpminfo and archiveinfo tables
• support different ways of building (possibly via content generators)
• utilize jsonb fields in postgres
• modular auth
∘ make it easier to add new auth mechanisms
∘ support openid auth
• improve plugins
∘ make the plugin framework cleaner and more pythonic
∘ support plugins in cli and web ui
• improve/update web ui
∘ drop cheetah templates in favor of python-jinja2
∘ more parity with cli
∘ history browsing
∘ landing page search or history
∘ support plugins
• change how tag pkglists (and blocking) work
• refactor package ownership
• refactor uploads
• more flexible gc
= Yet more changes =
• store all task requests by named args
∘ (for ease of inspection)
• get rid of tagBuild tasks
• drop odd event refererences in favor of timestamps
• streamlined cli options
• marker files for many things on disk
• more history data
• drop modpython support
• policy code
∘ more robust syntax
‣ test negation
‣ OR
‣ parentheses
‣ quoted strings
∘ multiple result policies (non-terminal actions)
∘ all-matches policy (needed for scheduler?)
∘ break action (breaks out of nesting)
∘ stop action (halts processing of an all-matches policy)
8 years, 10 months
