Hi all,
I'd like to push out a production bodhi upgrade, along with some ansible changes that will change the permissions of how the masher is run.
The latest bodhi code contains a number of fairly important bug fixes, and a couple of minor enhancements:
https://github.com/fedora-infra/bodhi/compare/2.1.7...develop
The backend masher changes that are in ansible will hopefully provide a huge speedup in terms of how long it takes to mash the updates repositories. The changes for this involve running the fedmsg-hub as the apache user, and changing the ownership of /mnt/koji/mash/{atomic,updates} accordingly.
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=94608f0... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=9889ac7... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=afbfecb... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=5b25234...
I tested the bodhi & db upgrades in staging, and performed limited testing of the masher due to various constraints.
Thanks,
luke
+1 from me On Mar 24, 2016 12:22, "Luke Macken" lmacken@redhat.com wrote:
Hi all,
I'd like to push out a production bodhi upgrade, along with some ansible changes that will change the permissions of how the masher is run.
The latest bodhi code contains a number of fairly important bug fixes, and a couple of minor enhancements:
https://github.com/fedora-infra/bodhi/compare/2.1.7...develop
The backend masher changes that are in ansible will hopefully provide a huge speedup in terms of how long it takes to mash the updates repositories. The changes for this involve running the fedmsg-hub as the apache user, and changing the ownership of /mnt/koji/mash/{atomic,updates} accordingly.
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=94608f0...
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=9889ac7...
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=afbfecb...
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=5b25234...
I tested the bodhi & db upgrades in staging, and performed limited testing of the masher due to various constraints.
Thanks,
luke
infrastructure mailing list infrastructure@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraprojec...
This upgrade has been deployed.
We'll be starting an updates push shortly to see what kind of performance gain we've achieved.
Thanks! luke
On Thu, Mar 24, 2016 at 12:25:36PM -0600, Stephen John Smoogen wrote:
+1 from me
On Mar 24, 2016 12:22, "Luke Macken" lmacken@redhat.com wrote:
Hi all, I'd like to push out a production bodhi upgrade, along with some ansible changes that will change the permissions of how the masher is run. The latest bodhi code contains a number of fairly important bug fixes, and a couple of minor enhancements: https://github.com/fedora-infra/bodhi/compare/2.1.7...develop The backend masher changes that are in ansible will hopefully provide a huge speedup in terms of how long it takes to mash the updates repositories. The changes for this involve running the fedmsg-hub as the apache user, and changing the ownership of /mnt/koji/mash/{atomic,updates} accordingly. https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id= 94608f070d2f61bf518ab27cec2e2ef0cc29f964 https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id= 9889ac7446a8de92019cc3b5f39893b1d91a20a7 https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id= afbfecbbd765c184b155f4e713442fc954dffa84 https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id= 5b252344eeacdf9b05cd58feb227dee73d2c1112 I tested the bodhi & db upgrades in staging, and performed limited testing of the masher due to various constraints. Thanks, luke _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/ infrastructure@lists.fedoraproject.org
infrastructure mailing list infrastructure@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraprojec...
On Thu, 24 Mar 2016 14:10:39 -0600 Luke Macken lmacken@redhat.com wrote:
This upgrade has been deployed.
We'll be starting an updates push shortly to see what kind of performance gain we've achieved.
Thanks! luke
We need to fix a nagios check related to this:
diff --git a/roles/nagios/client/templates/check_fedmsg_masher_proc.cfg.j2 b/roles/nagios/client/templates/check_fedmsg index b6ec46a..b6ad466 100644 --- a/roles/nagios/client/templates/check_fedmsg_masher_proc.cfg.j2 +++ b/roles/nagios/client/templates/check_fedmsg_masher_proc.cfg.j2 @@ -1 +1 @@ -command[check_fedmsg_masher_proc]={{ libdir }}/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u masher +command[check_fedmsg_masher_proc]={{ libdir }}/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u apache diff --git a/roles/nagios/server/files/nrpe.cfg b/roles/nagios/server/files/nrpe.cfg index 04dd746..752bca5 100644 --- a/roles/nagios/server/files/nrpe.cfg +++ b/roles/nagios/server/files/nrpe.cfg @@ -234,7 +234,7 @@ command[check_fedmsg_hub_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C ' command[check_fedmsg_gateway_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-gateway' -u fedmsg command[check_fedmsg_irc_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-irc' -u fedmsg command[check_fedmsg_tweet_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-tweet' -u fedmsg -command[check_fedmsg_masher_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u masher +command[check_fedmsg_masher_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u apache command[check_supybot_fedmsg_plugin]=/usr/lib64/nagios/plugins/check_supybot_plugin -t fedmsg command[check_haproxy_conns]=/usr/lib64/nagios/plugins/check_haproxy_conns.py command[check_redis_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'redis-server' -u redis
Needs to run over noc and bodhi-backends at least and we might just run it over all nagios_client to make sure they are up to date too.
kevin
+1
On 24 March 2016 at 14:39, Kevin Fenzi kevin@scrye.com wrote:
On Thu, 24 Mar 2016 14:10:39 -0600 Luke Macken lmacken@redhat.com wrote:
This upgrade has been deployed.
We'll be starting an updates push shortly to see what kind of performance gain we've achieved.
Thanks! luke
We need to fix a nagios check related to this:
diff --git a/roles/nagios/client/templates/check_fedmsg_masher_proc.cfg.j2 b/roles/nagios/client/templates/check_fedmsg index b6ec46a..b6ad466 100644 --- a/roles/nagios/client/templates/check_fedmsg_masher_proc.cfg.j2 +++ b/roles/nagios/client/templates/check_fedmsg_masher_proc.cfg.j2 @@ -1 +1 @@ -command[check_fedmsg_masher_proc]={{ libdir }}/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u masher +command[check_fedmsg_masher_proc]={{ libdir }}/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u apache diff --git a/roles/nagios/server/files/nrpe.cfg b/roles/nagios/server/files/nrpe.cfg index 04dd746..752bca5 100644 --- a/roles/nagios/server/files/nrpe.cfg +++ b/roles/nagios/server/files/nrpe.cfg @@ -234,7 +234,7 @@ command[check_fedmsg_hub_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C ' command[check_fedmsg_gateway_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-gateway' -u fedmsg command[check_fedmsg_irc_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-irc' -u fedmsg command[check_fedmsg_tweet_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-tweet' -u fedmsg -command[check_fedmsg_masher_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u masher +command[check_fedmsg_masher_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u apache command[check_supybot_fedmsg_plugin]=/usr/lib64/nagios/plugins/check_supybot_plugin -t fedmsg command[check_haproxy_conns]=/usr/lib64/nagios/plugins/check_haproxy_conns.py command[check_redis_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'redis-server' -u redis
Needs to run over noc and bodhi-backends at least and we might just run it over all nagios_client to make sure they are up to date too.
kevin
infrastructure mailing list infrastructure@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraprojec...
On Thu, Mar 24, 2016 at 02:39:25PM -0600, Kevin Fenzi wrote:
We need to fix a nagios check related to this:
diff --git a/roles/nagios/client/templates/check_fedmsg_masher_proc.cfg.j2 b/roles/nagios/client/templates/check_fedmsg index b6ec46a..b6ad466 100644 --- a/roles/nagios/client/templates/check_fedmsg_masher_proc.cfg.j2 +++ b/roles/nagios/client/templates/check_fedmsg_masher_proc.cfg.j2 @@ -1 +1 @@ -command[check_fedmsg_masher_proc]={{ libdir }}/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u masher +command[check_fedmsg_masher_proc]={{ libdir }}/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u apache diff --git a/roles/nagios/server/files/nrpe.cfg b/roles/nagios/server/files/nrpe.cfg index 04dd746..752bca5 100644 --- a/roles/nagios/server/files/nrpe.cfg +++ b/roles/nagios/server/files/nrpe.cfg @@ -234,7 +234,7 @@ command[check_fedmsg_hub_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C ' command[check_fedmsg_gateway_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-gateway' -u fedmsg command[check_fedmsg_irc_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-irc' -u fedmsg command[check_fedmsg_tweet_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-tweet' -u fedmsg -command[check_fedmsg_masher_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u masher +command[check_fedmsg_masher_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'fedmsg-hub' -u apache command[check_supybot_fedmsg_plugin]=/usr/lib64/nagios/plugins/check_supybot_plugin -t fedmsg command[check_haproxy_conns]=/usr/lib64/nagios/plugins/check_haproxy_conns.py command[check_redis_proc]=/usr/lib64/nagios/plugins/check_procs -c 1:1 -C 'redis-server' -u redis
Needs to run over noc and bodhi-backends at least and we might just run it over all nagios_client to make sure they are up to date too.
+1
Thanks,
luke
On Thu, Mar 24, 2016 at 12:22:04PM -0600, Luke Macken wrote:
Hi all,
I'd like to push out a production bodhi upgrade, along with some ansible changes that will change the permissions of how the masher is run.
The latest bodhi code contains a number of fairly important bug fixes, and a couple of minor enhancements:
https://github.com/fedora-infra/bodhi/compare/2.1.7...develop
The backend masher changes that are in ansible will hopefully provide a huge speedup in terms of how long it takes to mash the updates repositories. The changes for this involve running the fedmsg-hub as the apache user, and changing the ownership of /mnt/koji/mash/{atomic,updates} accordingly.
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=94608f0... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=9889ac7... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=afbfecb... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=5b25234...
I tested the bodhi & db upgrades in staging, and performed limited testing of the masher due to various constraints.
How easy would it be to revert if needed? And will you be available tomorrow and this week-end if something goes south? (I seem to remember tomorrow is a day off).
Pierre
On Thu, Mar 24, 2016 at 07:32:42PM +0100, Pierre-Yves Chibon wrote:
On Thu, Mar 24, 2016 at 12:22:04PM -0600, Luke Macken wrote:
Hi all,
I'd like to push out a production bodhi upgrade, along with some ansible changes that will change the permissions of how the masher is run.
The latest bodhi code contains a number of fairly important bug fixes, and a couple of minor enhancements:
https://github.com/fedora-infra/bodhi/compare/2.1.7...develop
The backend masher changes that are in ansible will hopefully provide a huge speedup in terms of how long it takes to mash the updates repositories. The changes for this involve running the fedmsg-hub as the apache user, and changing the ownership of /mnt/koji/mash/{atomic,updates} accordingly.
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=94608f0... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=9889ac7... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=afbfecb... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=5b25234...
I tested the bodhi & db upgrades in staging, and performed limited testing of the masher due to various constraints.
How easy would it be to revert if needed? And will you be available tomorrow and this week-end if something goes south? (I seem to remember tomorrow is a day off).
Reverting will be fairly easy, and yes I will be available tomorrow and this weekend.
I'll have a production db snapshot ready just in-case the alembic migration fails. I've tested it in stg with the prod snapshots many times already.
If the masher aspect fails, it'll be simple to re-chown things back to the way they were and revert the ansible changes.
luke
On Thu, Mar 24, 2016 at 01:07:30PM -0600, Luke Macken wrote:
On Thu, Mar 24, 2016 at 07:32:42PM +0100, Pierre-Yves Chibon wrote:
On Thu, Mar 24, 2016 at 12:22:04PM -0600, Luke Macken wrote:
Hi all,
I'd like to push out a production bodhi upgrade, along with some ansible changes that will change the permissions of how the masher is run.
The latest bodhi code contains a number of fairly important bug fixes, and a couple of minor enhancements:
https://github.com/fedora-infra/bodhi/compare/2.1.7...develop
The backend masher changes that are in ansible will hopefully provide a huge speedup in terms of how long it takes to mash the updates repositories. The changes for this involve running the fedmsg-hub as the apache user, and changing the ownership of /mnt/koji/mash/{atomic,updates} accordingly.
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=94608f0... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=9889ac7... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=afbfecb... https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=5b25234...
I tested the bodhi & db upgrades in staging, and performed limited testing of the masher due to various constraints.
How easy would it be to revert if needed? And will you be available tomorrow and this week-end if something goes south? (I seem to remember tomorrow is a day off).
Reverting will be fairly easy, and yes I will be available tomorrow and this weekend.
I'll have a production db snapshot ready just in-case the alembic migration fails. I've tested it in stg with the prod snapshots many times already.
If the masher aspect fails, it'll be simple to re-chown things back to the way they were and revert the ansible changes.
+1 for me as well then, thanks for answering :)
Pierre
infrastructure@lists.fedoraproject.org
-
Dennis Gilmore -
Kevin Fenzi -
Luke Macken -
Pierre-Yves Chibon -
Stephen John Smoogen