Lessons learned: Initial check/step list for updates
by Stephen John Smoogen
0. Plan time in infrastructure(a)lists.fedoraprojects.org
1. Open ticket on infrastructure for downtime.
Updates will occur during day
Reboots will occur during evening
2. Send email to devel-announce, announce, infrastructure
3. Update servers during working hours and work out issues in ticket.
** releng updates the following boxes:
cvs01, pkgs01, nfs01, bnfs01, bxen*,
x86-*, ppc*, koji*, db03, xb-01,
compose-*, sign-vault01
4. Change DNS to turn off proxy on bodhost01 (or similar external
proxy server).
5. Reboot bodhost01
6. Confirm proxy is working on bodhost/fix issues.
7. Change proxy dns to only go to bodhost01
8. Turn off nagios for servers.
9. Turn off nagios-external for services.
10. Reboot order counts
11. releng deals with the boxes listed above unless told otherwise.
12. reboots with database servers first
xen15: db02
xen12: db01
13. reboot PHX2 boxes
xen03:
xen04:
xen06:
xen07:
xen09:
xen10:
xen11:
xen13:
backup01:
14. reboot Outside boxes (can be in parallel to PHX2)
cnode01:
cnode02:
cnode03:
ibiblio01:
internetx01:
osuosl01:
people01:
serverbeach1:
serverbeach2:
serverbeach3:
serverbeach4:
serverbeach5:
telia1:
tummy1:
15. reboot bastion.fedoraproject.org
log into bastion1 from outside system
log into bastion2 from outside world
log into xen05 from bastion01
bastion01:
sudo su /usr/sbin/puppetd --disable
sudo su /sbin/service openvpn start
bastion02
sudo su /sbin/service openvpn start
xen05
sudo /sbin/shutdown -r now
once xen05/bastion2 server is back up, we can
bastion01:
sudo su /sbin/service openvpn stop
sudo su /usr/sbin/puppetd --enable
16. reboot puppet01
log into bastion2 from outside world
ssh xen14
sudo /sbin/shutdown -r now
17. re-enable DNS for proxy servers
test proxy servers from puppet01
edit dns in git puppet
make ns1
18. re-enable nagios on internal/external
19. Setup transifex agent on app servers: app01 app02 app03 app04 app07
sudo -u transifex /var/lib/transifex/ssh-add.sh -f
20. Log and report problems to list.
21. Close ticket.
--
Stephen J Smoogen.
“The core skill of innovators is error recovery, not failure avoidance.”
Randy Nelson, President of Pixar University.
"We have a strategic plan. It's called doing things.""
— Herb Kelleher, founder Southwest Airlines
13 years, 7 months
speeding up the group/dump query in FAS
by Matt Domsch
Currently this query takes several minutes. HAProxy (or something in the proxy series) times out the request, and returns a 500 to the client.
https://admin.fedoraproject.org/accounts/group/dump/
Any chance that could be optimized? FWIW, I use this query in my ftbfs script to convert package owner names to email addresses, so I can send the FTBFS report to the owner's emails directly (on bcc).
Thanks,
Matt
--
Matt Domsch
Technology Strategist
Dell | Office of the CTO
13 years, 7 months
[PATCH/RFC] func: Simplify minion.conf creation
by Todd Zullinger
Puppet conveniently makes most variables from Puppet.settings available
via Puppet[:variable]. Use this to generate minion.conf via a standard
ERB template.
---
This change is the sort of thing that really begs for a second set of
eyes, as it would cause too much trouble if it didn't work as
intended.
A quick way to test that the template does the right thing is to grab
http://fedorapeople.org/~tmz/tmp/func-minion-template-test.pp and run
it through puppet, e.g. sudo puppet /tmp/func-minion-template-test.pp.
I noticed this while poking the func setup to see what settings were
needed to make func use puppet's certificates with the new code from
Seth in func's git tree. (I also noticed that the overlord.conf isn't
puppet managed, and that's the one I was more curious to see, easily
found on puppet01. ;)
modules/func/files/make-minion-conf.sh | 24 ------------------------
modules/func/manifests/init.pp | 24 ++++++++++++------------
modules/func/templates/minion.conf.erb | 13 +++++++++++++
3 files changed, 25 insertions(+), 36 deletions(-)
delete mode 100755 modules/func/files/make-minion-conf.sh
create mode 100644 modules/func/templates/minion.conf.erb
diff --git a/modules/func/files/make-minion-conf.sh b/modules/func/files/make-minion-conf.sh
deleted file mode 100755
index ea50c95..0000000
--- a/modules/func/files/make-minion-conf.sh
+++ /dev/null
@@ -1,24 +0,0 @@
-#!/bin/bash
-
-
-certfile=`ruby -e "require 'puppet'; Puppet.parse_config; puts Puppet.settings['hostcert']"`
-keyfile=`ruby -e "require 'puppet'; Puppet.parse_config; puts Puppet.settings['hostprivkey']"`
-cacert=`ruby -e "require 'puppet'; Puppet.parse_config; puts Puppet.settings['localcacert']"`
-
-
-cat <<EOM>/etc/func/minion.conf
-# configuration for minions
-
-[main]
-log_level = DEBUG
-acl_dir = /etc/func/minion-acl.d
-
-listen_addr =
-listen_port = 51234
-use_certmaster = 0
-
-cert_file = $certfile
-key_file = $keyfile
-ca_file = $cacert
-
-EOM
diff --git a/modules/func/manifests/init.pp b/modules/func/manifests/init.pp
index 50d6ed8..b9d39fe 100644
--- a/modules/func/manifests/init.pp
+++ b/modules/func/manifests/init.pp
@@ -23,23 +23,23 @@ class func::funcd {
}
}
+ File {
+ require => Package['func'],
+ }
+
file { '/etc/logrotate.d/func_rotate':
source => 'puppet:///func/func_rotate',
- require => Package['func']
- }
- file { '/usr/local/bin/make-minion-conf.sh':
- source => 'puppet:///func/make-minion-conf.sh',
- require => Package['func'],
- mode => 0750,
- notify => Exec['makeminionconf']
}
- exec { 'makeminionconf':
- require => File['/usr/local/bin/make-minion-conf.sh'],
- command => '/bin/bash /usr/local/bin/make-minion-conf.sh',
- refreshonly => true,
- notify => Service['funcd']
+
+ file { '/etc/func/minion.conf':
+ content => template('func/minion.conf.erb'),
+ notify => Service['funcd'],
}
+ # This can be removed once all systems are updated
+ file { '/usr/local/bin/make-minion-conf.sh':
+ ensure => absent,
+ }
}
class func::certmaster {
diff --git a/modules/func/templates/minion.conf.erb b/modules/func/templates/minion.conf.erb
new file mode 100644
index 0000000..33395ca
--- /dev/null
+++ b/modules/func/templates/minion.conf.erb
@@ -0,0 +1,13 @@
+# configuration for minions
+
+[main]
+log_level = DEBUG
+acl_dir = /etc/func/minion-acl.d
+
+listen_addr =
+listen_port = 51234
+use_certmaster = 0
+
+cert_file = <%= Puppet[:hostcert] %>
+key_file = <%= Puppet[:hostprivkey] %>
+ca_file = <%= Puppet[:localcacert] %>
--
1.7.2.2
--
Todd OpenPGP -> KeyID: 0xBEAF0CE3 | URL: www.pobox.com/~tmz/pgp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
History teaches that grave threats to liberty often come in times of
urgency, when constitutional rights seem too extravagant to endure.
-- Justice Thurgood Marshall, 1989
13 years, 7 months
