On Thu, Jan 26, 2012 at 02:57:03PM -0500, Luke Macken wrote:
During the meeting today, Seth asked me to write up details on the
interdependencies of the services that we're currently running on
community01.dev.
Indexer Cron Job
- Runs nightly, hits Koji, downloads RPMs using yum from the master
mirror.
- Creates a xapian database, and swaps out the old with the new
Git repo updater cron job
- Queries koji for a list of builds since $LAST_RUN
- Does a git pull of all updated package repos
Fedora Packages (
https://community.dev.fedoraproject.org/packages)
- Queries the local xapain db for search queries and package info
- Queries bodhi, koji, pkgdb, bugzilla, via python-{fedora,bugzilla}
- Uses fedpkg to clones git repos
Fedora Tagger (
https://community.dev.fedoraproject.org/tagger)
- Queries the local xapain db for package info
- Uses postgres on db02.stg currently.
So as far as production deployment goes, we could potentially have a
separate box just for our indexer & git repos cron jobs, then we could
rsync them over to our multiple app frontends. What do you guys think?
A bunch of us talked about this on IRC few updates:
* xapian may be usable remotely -- if so we could treat as just a different
kind of database server.
* The git checkouts (and xapian if it turns out it can't be used remotely)
would be good candidates to start trying out glusterfs/hekafs/other cloud
filesystems on.
* Packages is also using yum repodata for provides-requires and filelists.
That's also a good candidate to store on glusterfs.
* If cloud filesystems are a bust, having a bapp-style box and then rsyncing
is a possibility due to the relatively small size of the data so far.
-Toshio