Freeze break request: Switch back to bastion02 for now
by Kevin Fenzi
We've continued to have issues with bastion03 and this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=725332
It's been requiring a reboot every day or two, resulting in 5-10min of
downtime and about 90 pages. ;(
I hate changes on fridays and more so during a freeze, but I think we
need to switch back to bastion02 for now to avoid this issue until we
can get a fix.
So, I would like to:
- commit the following patch.
- puppet update nameservers to get the new info.
- puppet update bastion02/03 to get openvpn running on 02 and stopped
on 03
- Make sure everything reconnects.
Unfortunately this will result in a small outage, but no worse than the
bastion03 ones have been. If we don't want to do it now, I can wait
until the next time bastion03 freaks out and just change it then, since
it should be all prepped below:
diff --git a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp b/manifests/nodes/bastion02.phx2.fe
index 4018ec9..1a0ee7c 100644
--- a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
+++ b/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
@@ -1,6 +1,5 @@
node bastion02{
- # Moving openvpn over to bastion03
- $enable_openvpn = false
+ $enable_openvpn = true
include phx
$syncFasAliases = true
include gateway
diff --git a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp b/manifests/nodes/bastion03.phx2.fe
index 8c5fca9..b7b0f32 100644
--- a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
+++ b/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
@@ -3,7 +3,7 @@ node bastion03{
# comment out the line below when bastion02 is down or going to be down.
# Under normal situations, only one bastion host should be running openvpn
# or we'll end up with a split-brain problem in the network
- #$enable_openvpn = false
+ $enable_openvpn = false
include phx
$syncFasAliases = true
include gateway
diff --git a/modules/bind/files/master/fedoraproject.org b/modules/bind/files/master/fedoraproject.o
index 5b72f2d..f3dc836 100644
--- a/modules/bind/files/master/fedoraproject.org
+++ b/modules/bind/files/master/fedoraproject.org
@@ -1,6 +1,6 @@
$TTL 3600
@ IN SOA ns01.fedoraproject.org. hostmaster.fedoraproject.org. (
- 2011091301 ; Serial
+ 2011091601 ; Serial
1H ; refresh
10M ; retry
4W ; expire
@@ -85,7 +85,7 @@ autoqa-stg IN A 209.132.181.9
; need rhit to fix.
bastion01 IN A 209.132.181.2
bastion02 IN A 209.132.181.3
-bastion IN A 209.132.181.2
+bastion IN A 209.132.181.3
bastion-comm01 IN A 209.132.181.13
backup02 IN A 152.19.134.140
blogs IN CNAME wildcard
diff --git a/modules/bind/files/master/phx2.fedoraproject.org b/modules/bind/files/master/phx2.fedor
index b8caea3..7c9eed7 100644
--- a/modules/bind/files/master/phx2.fedoraproject.org
+++ b/modules/bind/files/master/phx2.fedoraproject.org
@@ -28,7 +28,7 @@ bapp01 IN A 10.5.126.38
bapp1 IN CNAME bapp01
bapp02 IN A 10.5.126.39
bapp2 IN CNAME bapp02
-bastion IN CNAME bastion03
+bastion IN CNAME bastion02
;bastion01 IN A 10.5.126.13
;bastion1 IN CNAME bastion01
bastion02 IN A 10.5.126.11
12 years, 7 months
Plan for tomorrow's Fedora Infrastructure meeting (2011-09-15)
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow
2011-09-15 at 1900 UTC in #fedora-meeting on the freenode network.
Suggested topics (suggested by whom):
* New folks introductions and Apprentice tasks.
* Password/Ssh-key/Cert reset flag day discussion.
* Bastion outages/openvpn discussion.
* Upcoming Tasks/Items (nirik)
2011-09-13 - 27: Beta change freeze
2011-09-27: Fedora 16 Beta
2011-10-01 mail fi-apprentice folks.
2011-10-09 Remove inactive fi-apprentice people.
2011-10-18 - 2011-11-01: Final change freeze
2011-11-01: Fedora 16 release.
* Meeting tagged tickets:
https://fedorahosted.org/fedora-infrastructure/report/10
NOTE: I have cleaned out the ones that have been sitting around with
this tag. Anytime anyone would like to discuss a ticket, just add the
meeting keyword and we will get to it in this section.
* Request for Resources progress report
- askfedora
- paste
Submit your agenda items, as tickets in the trac instance and send a
note replying to this thread.
More info here:
https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings
Thanks
kevin
12 years, 7 months
RHEL 6 packages information
by Roberto Sassu
Hi all
i'm trying to obtain the update type information
about released RHEL 6 packages. Unfortunately,
this information is not available in the Fedora
database. There is a place where i can get it?
Thanks in advance for replies
Regards
Roberto Sassu
12 years, 7 months
bastion03 outage this afternoon
by Kevin Fenzi
We had an outage of bastion03 this afternoon.
Because it's the vpn hub, lots of services were affected.
It looks like it's running into
https://bugzilla.redhat.com/show_bug.cgi?id=725332
So, if this re-occurs, we should either:
a) switch it to use e1000 instead of virtio network devices.
or
b) switch back to bastion02 (which is still there and rhel5).
So, keep an eye out for outages of services where you see a lot of ping
loss or not answer at all from bastion.fedoraproject.org.
For the record, the outage started at 22:45 and ran until 22:55 or so.
kevin
12 years, 7 months
[Change Request] Fix checkout and pull ordering.
by Ricky Zhou
This change is needed for the beta website and releases.txt changes.
---
modules/fedora-web/files/syncStatic.sh | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/modules/fedora-web/files/syncStatic.sh b/modules/fedora-web/files/syncStatic.sh
index ffbf781..5cf0644 100644
--- a/modules/fedora-web/files/syncStatic.sh
+++ b/modules/fedora-web/files/syncStatic.sh
@@ -51,8 +51,8 @@ cd /srv/web/fedora-web
/usr/bin/git reset -q --hard || exit 1
#/usr/bin/git checkout -q master || exit 1
-/usr/bin/git pull -q --ff-only || exit 1
/usr/bin/git checkout -q f16-alpha || exit 1
+/usr/bin/git pull -q --ff-only || exit 1
build fedoraproject.org
build spins.fedoraproject.org
--
1.7.2.1
12 years, 7 months
todo: ask everyone what their todo is for the week
by Kevin Fenzi
Greetings.
I thought I would try a little something out this week.
For folks working in infrastructure, what are you working on this week?
Note that I am not looking for some detailed todo list, just after
having fought your monday and read your emails and looked at tickets
and bugs and requests, what kind of things are you going to try and
work on this week? On friday, if you looked and finished X, you would
feel happy about the week. whats X?
Here's mine:
- Document bastion machines and their oddities in infa-docs.
- Going to try and fix syslog from non vpn machines.
- Get our Beta tickets filed and the freeze setup tomorrow (announce
it, etc)
- Look at updating CSI
- Work on askbot in stg some more.
- Continue new machine (rhel6) plans.
- Typical putting out fires, fixing things as they break, fixing things
that send me email.
anyone else have any goals for the week?
kevin
12 years, 7 months
Proposal for action: SSH Key, User Cert and Password Flag Day
by Seth Vidal
Given recent events in the linux-y world I think it might do us a
service to impose an ssh-key, user cert and password enforced change
flag day.
The idea would be everyone would be required to change their passwords,
ssh keys and any user certs they have before being allowed to do
anything else on our systems.
Anyone failing to change them would be locked out after a specific
date.
In particular I would like to make sure that ssh keys get changed - so
much so that I would want to keep a copy of the existing ssh keys and
verify that the new one does not match the old one before allowing it to
be used.
I'd like to discuss the efficacy and timing of this. If anyone has
perspective that is helpful, please share it.
I think this should be done soon, personally.
-sv
12 years, 7 months
Introduction
by Magyar Zoltán
Hi All!
My name is Zoltán Magyar. Basically I live in Hungary (but for now I'm in
Ireland until 1st of March, 2012).
I'm using fedora since 4 or 5 years now, and I pretty like it so far. I
guess it's time for me to take part in helping this OS to get even better.
I have some knowledge in software development in general, I'm currently
learning Java, Python, have used SQL a bit. I'm mostly looking for some
programming tasks, but I'm also interested in networking and cluster and HA
stuff.
I work as a full-time software developer at Ericsson, mainly doing function
test related stuff.
Hard to tell the time I could contribute, but I hope I can free up at least
3-4 hours a week!
I'm happy to listen to any pointers about where/how to start contributing!
:-)
Cheers,
Zoli
12 years, 7 months
new hosts plans and thoughts
by Kevin Fenzi
As some of you may have noted, I've been building up some new hosts of
late. I thought I would give an overview here and plans moving forward
and provide a list left to do for those folks in sysadmin-main who wish
to help out. :)
We have 4 old xen boxes that are going out of service at the end of
next month, so we need to get all guests off them before then. Those
are: xen03, xen05, xen09, xen15.
We have new machines: virthost05-10 and bvirthost04
On the rel-eng side:
* I have setup a releng03 to replace releng01 on bvirthost04
* I have setup a releng04 to replace releng02 on bvirthost01
* I have setup a kojipkgs02 to replace kojipkgs02 on bvirthost04
I need to get them finished up and access to the netapp mounts, then we
can look at migrating over to them. Once we migrate, bxen03 will be
empty and can be re-installed as bvirthost05. We might look then at
moving things around a bit to take advantage of bvirthost05.
The only outage here would be kojipkgs, and we could make that pretty
short/fast.
On the staging front:
* We need to make a db02.stg thats rhel6. Soon all our db's will be
rhel6, and we want staging to match up. I think we should use
virthost10 for this.
* We need to migrate fas01.stg and pkgs01.stg to virthost10. These are
already rhel6. I suppose we could re-install them, but migrating
seems reasonable to me.
* We may want to migrate some or all the stg on virthost13. virthost13
is now out of warentee. We might not care too much since it's only
stg there. Thoughts?
On the production front:
* I have made a db01 on virthost04. This will replace db02. I'd like to
look at doing this wed next week. (2011-09-07). This WILL require an
outage, since db02 has fas on it along with many other things. Should
we do this outage in the evening? Other thoughts on time?
* I have made a bastion03 on virthost04. This will replace bastion02.
We will need to swap IP's for it to work with the external IP in
phx2. I figure we can do this at the same time as the db outage
above.
* Machines that still need building/setting up for you sysadmin-main
folks who want to help out:
- bastion04 to replace bastion01. Should be on a different virthost
from bastion03.
- proxy08 to replace proxy1 (rhel5).
- value03 and value04 (on different virthosts) to replace value01/02.
- app02 and app04 are on those xen boxes, so we need to move them. We
can't do RHEL6 yet however. So, we could wait a bit more in hopes of
being able to rebuild them as RHEL6, or just migrate them, or rebuild
them as RHEL5.
Once this is all done that leaves us with just xen04 as a xen box in
phx2. We can then look at moving stuff off it and repaving it as
virthost11.
kevin
12 years, 7 months
infrastructure
by rexchou
Hello Does fedora on the C language project?
12 years, 7 months
