On Tue, 22 Jul 2014 12:12:36 -0600 Kevin Fenzi kevin@scrye.com wrote:

On Tue, 22 Jul 2014 10:56:12 -0600 Tim Flink tflink@redhat.com wrote:

...

Now that Taskotron staging is up and running, I'm not sure what's generally required for services before moving to production.

I'm still working out a few kinks in the initial production systems but I think that the only big thing left is the proxy settings, which I'm leaving alone for the moment.

Is there any review process or checklist for moving things from stg to prod?

http://infrastructure.fedoraproject.org/infra/docs/requestforresources.txt

Thanks, I don't think that I would have looked in that doc.

I'd like monitoring added. What should we monitor to make sure it's working as expected?

It depends on how complicated we want to get, I suppose. The most basic things would be to check to make sure resultsdb, resultsdb_frontend and buildbot are responding to http but there are json apis for resultsdb and buildbot which would give more details. I'd like to monitor free disk space if that's not already done by default.

Are most other services monitored with a GET to a url to see if they're up?

Also, do we need any SOP's written up? Anything special in stopping/starting things? updates order or anything?

Yeah, the triggered jobs need to be buffered during downtime for playback after everything comes up. I've always used my local system for that but I'm not sure what the best choice is for infra.

Also, I need to make sure we have backups going on the db server at least. Anything else that needs backing up?

/home/buildmaster/master on the taskotron server (taskotron01.qa for production) should be backed up as that's where all the logs are stored. Everything else that needs to be backed up is in the databases.

Tim