On Thu, Jan 26, 2012 at 02:57:03PM -0500, Luke Macken wrote:

During the meeting today, Seth asked me to write up details on the interdependencies of the services that we're currently running on community01.dev.

Indexer Cron Job
    - Runs nightly, hits Koji, downloads RPMs using yum from the master
      mirror.
    - Creates a xapian database, and swaps out the old with the new

Git repo updater cron job
    - Queries koji for a list of builds since $LAST_RUN
    - Does a git pull of all updated package repos

Fedora Packages (https://community.dev.fedoraproject.org/packages)
    - Queries the local xapain db for search queries and package info
    - Queries bodhi, koji, pkgdb, bugzilla, via python-{fedora,bugzilla}
    - Uses fedpkg to clones git repos

Fedora Tagger (https://community.dev.fedoraproject.org/tagger)
    - Queries the local xapain db for package info
    - Uses postgres on db02.stg currently.

So as far as production deployment goes, we could potentially have a separate box just for our indexer & git repos cron jobs, then we could rsync them over to our multiple app frontends. What do you guys think?

A bunch of us talked about this on IRC few updates:

* xapian may be usable remotely -- if so we could treat as just a different kind of database server. * The git checkouts (and xapian if it turns out it can't be used remotely) would be good candidates to start trying out glusterfs/hekafs/other cloud filesystems on. * Packages is also using yum repodata for provides-requires and filelists. That's also a good candidate to store on glusterfs. * If cloud filesystems are a bust, having a bapp-style box and then rsyncing is a possibility due to the relatively small size of the data so far.

-Toshio