After talking in #fedora-noc, we would like to make phx2 app servers talk to other app servers via proxy01, rather than (potentially) going out to a server across the country for the benefit of load balancing. The issue this solves is that because apps hosted on admin.fp.o talk to each other using the same admin.fp.o roundrobin that users use, when an application hits a random proxy and that proxy has gone down for some reason, the application would block its thread waiting for a response. Eventually this would happen on enough applications that we would get a flood of alerts, and ultimately cause downtime. By using (only) proxy01, it means re-adding a single point of failure, but it seems likely that the case of "proxy01 is down so everything else is down" would mean that phx2 was having a network issue anyway, meaning we'd be in the same position (i.e., app servers unreachable). It seems unlikely that proxy01 will just die at random, and if it does then that is a whole new issue that we should address. So yes, the single point of failure is bad, but it seems marginally better than what we have been seeing lately. I would like +1's to push this to puppet, which effectively reverts 1bac8c9a and 23ceebd5. [codeblock@lockbox01 puppet]$ git show commit f0445cffd64d3db980a4b689517fd4b95f6e7686 Author: Ricky Elrod <codeblock(a)lockbox01.phx2.fedoraproject.org&gt; Date: Fri Dec 6 17:55:38 2013 +0000 Make phx2 boxen use proxy01 for admin.fp.o again diff --git a/manifests/services/phx.pp b/manifests/services/phx.pp index 72c24be..317bfc2 100644 --- a/manifests/services/phx.pp +++ b/manifests/services/phx.pp @@ -11,11 +11,10 @@ class phx { } case $environment { 'production' : { -# host { 'admin.fedoraproject.org': - #ip => '10.5.126.52', - #ip => '66.35.62.166', -# ensure => absent, -# } + host { 'admin.fedoraproject.org': + ip => '10.5.126.52', + ensure => present, + } host { 'cvs.fedoraproject.org': ip => '10.5.125.151', host_aliases => ['cvs']