infrastructure January 2011

infrastructure@lists.fedoraproject.org

35 participants
19 discussions

Lost access to retrace01 machine
by Michal Toman 05 Feb '11

05 Feb '11

Hi, after changing my (mtoman) FAS password and SSH key, I'm not able to SSH retrace01.fedoraproject.org anymore (pubkey, password). Login to fedorapeople and fedorahosted works correctly. Could you look at the problem and possibly renew my access? Thanks, Michal

3 3

Fwd: Permission denied (publickey)
by Tomas Lestach 04 Feb '11

04 Feb '11

Could anybody check the problem, please? It's been 8 weeks since I reported the issue. Regards, Tomas -- Tomas Lestach RHN Satellite Engineering, Red Hat -------- Original Message -------- Subject: Permission denied (publickey) Date: Thu, 02 Dec 2010 11:15:41 +0100 From: Tomas Lestach <tlestach(a)redhat.com> To: infrastructure(a)lists.fedoraproject.org Hello, my fedoraproject.org account is: tlestach My Roles are: * Signed CLA Group (user) * Fedora CLA Group (user) * Spacewalk Git Commit Group (administrator) When I try to upload a tar.gz to fedorahosted.org:spacewalk, I get: $ scp /tmp/tito/spacewalk-backend-1.3.20.tar.gz fedorahosted.org:spacewalk Permission denied (publickey). lost connection According to https://fedorahosted.org/web/faq, I shall have just a fedoraproject account and a CLA, what I have. I'm sure, my public key is correct, because I'm a member of one of the Git Commit Groups and I have no trouble with committing, where the public key get's verified. Is there something I miss? Regards, Tomas -- Tomas Lestach RHN Satellite Engineering, Red Hat

3 4

RE: New Fedora Mirror in Vietnam
by Matt Domsch 04 Feb '11

04 Feb '11

We would very much welcome you as a mirror. Please see http://fedoraproject.org/wiki/Infrastructure/Mirroring, and create an account for yourself in the Fedora Account System, and register your mirror in MirrorManager, our database, at https://admin.fedoraproject.org/mirrormanager . You can enter your own mirror there, and update your information as needed. Within 2 hours of adding your info to the database, you will start seeing traffic. If you have any questions, please let us know. Thanks, Matt Fedora Mirror Wrangler -- Matt Domsch Technology Strategist Dell | Office of the CTO -----Original Message----- From: mirror-admin-bounces(a)fedoraproject.org [mailto:mirror-admin-bounces@fedoraproject.org] On Behalf Of Gmail - Pham Duc Hoang Sent: Tuesday, December 21, 2010 9:11 PM To: mirror-admin(a)fedoraproject.org; fedora-infrastructure-list(a)redhat.com Subject: New Fedora Mirror in Vietnam Dear , We launched a new mirror for the project fedoraproject.org. You can add a mirror to the overall list. Below the information about mirror: * My Company : DIGIPOWER Co.,ltd * URL of mirror: http://mirrors.digipower.vn/fedora/releases * Country where the mirror is located: VietNam * Contact email address: support(a)digipower.vn , duchoang(a)digipower.vn * Update frequency: 4 times a day * Rsync repository used: mirrors.digipower.vn::fedora * Approximate bandwidth: 100Mbps * IP : 112.78.6.8 . * Service : HTTP - FTP - RSYNC * Available content : Fedora Linux 13 - 14 * Architectures : All Best Regards , _______________________________________________ Mirror-admin mailing list Mirror-admin(a)fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/mirror-admin

2 2

Support for BFO?
by Ray Van Dolson 02 Feb '11

02 Feb '11

Where is the proper place to seek support for BFO (boot.fp.org)? I didn't really see a component for it in Bugzilla, nor were there any contact links on the website or in the FAQ. I know it spawned out of Infrastructure, so also checked Trac -- that may be the right place but thought I'd ask here first. I have an issue with my NIC being supported. Probably can resolve upstream, but who here would need to be pinged to implement any changes? Thanks, Ray

5 6

Re: Security incident on Fedora infrastructure on 23 Jan 2011
by Jose Manimala 28 Jan '11

28 Jan '11

Awesome work and happy that nothing bad has happened. One question is should a password length and secure password creation check be enforced on the FAS system. Like regular expression checks and stuff. I know this is asking a lot, the current implementation allows me to have a simple password if I remember(need to check) been long. And password expiry? :) On Tue, Jan 25, 2011 at 1:14 PM, Jared K. Smith <jsmith(a)fedoraproject.org> wrote: > Summary: Fedora infrastructure intrusion but no impact on product integrity > > On January 22, 2011 a Fedora contributor received an email from the Fedora > Accounts System indicating that his account details had been changed. He > contacted the Fedora Infrastructure Team indicating that he had received > the email, but had not made changes to his FAS account. The Infrastructure > Team immediately began investigating, and confirmed that the account had > indeed been compromised. > > At this time, the Infrastructure Team has evidence that indicates the account > credentials were compromised externally, and that the Fedora Infrastructure was > not subject to any code vulnerability or exploit. > > The account in question was not a member of any sysadmin or Release Engineering > groups. The following is a complete list of privileges on the account: > * SSH to fedorapeople.org (user permissions are very limited on this machine). > * Push access to packages in the Fedora SCM. > * Ability to perform builds and make updates to Fedora packages. > > The Infrastructure Team took the following actions after being > notified of the issue: > 1. Lock down access to the compromised account > 2. Take filesystem snapshots of all systems the account had access to > (pkgs.fedoraproject.org, fedorapeople.org) > 3. Audit SSH, FAS, Git, and Koji logs from the time of compromise to the > present > Here, we found that the attacker did: > * Change the account's SSH key in FAS > * Login to fedorapeople.org > The attacker did not: > * Push any changes to the Fedora SCM or access pkgs.fedoraproject.org in > any way > * Generate a koji cert or perform any builds > * Push any package updates > > Based on the results of our investigation so far, we do not believe that any > Fedora packages or other Fedora contributor accounts were affected by this > compromise. > > While the user in question had the ability to commit to Fedora SCM, the > Infrastructure Team does not believe that the compromised account was used to > do this, or cause any builds or updates in the Fedora build system. The > Infrastructure Team believes that Fedora users are in no way threatened by this > security breach and we have found no evidence that the compromise extended > beyond this single account. > > As always, Fedora packagers are recommended to regularly review commits to > their packages and report any suspicious activity that they notice. > > Fedora contributors are strongly encouraged to choose a strong FAS password. > Contributors should *NOT* use their FAS password on any other websites or > user accounts. If you receive an email from FAS notifying you of changes to > your account that you did not make, please contact the Fedora Infrastructure > team immediately via admin(a)fedoraproject.org. > > We are still performing a more in-depth investigation and security audit and we > will post again if there are any material changes to our understanding. > > -- > Jared Smith > Fedora Project Leader > -- > announce mailing list > announce(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/announce > -- Cheers Jose http://josemanimala.eu.org/blog Ph: +64221033100

9 8

rpm -Va report
by Seth Vidal 28 Jan '11

28 Jan '11

As we talked about in the infrastructure meeting (and beyond) yesterday. I did a full rpm -Va run on every host we maintain. the results are on puppet1 in /var/tmp/global-rpm-va some of them are icko -sv

1 0

Meeting reminder
by Derek Carter 27 Jan '11

27 Jan '11

Yeah, it's last minute. We have a pretty full meeting item list on the trac: https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assi… What other things do we need to talk about? -- Derek aka goozbach

1 0

Introductions
by Bassford John 25 Jan '11

25 Jan '11

Hello Fedora-Infrastructure I`d like to introduction myself, my name is John Bassford (irc jbass29503 on irc.freenode.net#fedora-admin) and was looking to learn the in/outs of how the fedora infrstucture group works and possible start to assist with small tasks/projects to assist the fedora project. I have been a sysadmin for about 13 years now, working both linux and windows sides. So please look at the info below which is what was recommended from the fedora-wiki, and don`t be in shock. Name: John Bassford Time Zone/Country: Eastern/US BasicSkills/Experiences: SystemAdministrator/Engineer, bash,perl, ldap,rpm packaging, nagios, dns admin, fimilar with puppet Developerskills: not really, know php, and a little java, but not python whyJoin: to give back to opensource community, learn new technology and ways, in general to boarden my horizons Looking to do: observe and get a feel of the infrastructure group and possible help with these two open tickets to start with, I think they are small and something good for someone new to the group to start with. (tickets 1152 and 1084 if there really is still work to be done for them, as these active tickets are rather old) an maybe the Nagios upgrade. (2275) HowMuchTime: couple hours a day and/or more on a weekend. I have some hardware to do simulated testing with, a dell server running vmware esx (freeware) which is what I just installed fedora14 on, but somewhat limited in disk space abou 100GB total local. Thanks JB

1 0

Methods for reducing spam
by Mike McGrath 23 Jan '11

23 Jan '11

We publish a lot of emails for various reasons, I thought I'd pass this along, it's worth a read: http://techblog.tilllate.com/2008/07/20/ten-methods-to-obfuscate-e-mail-add… -Mike

2 1

OUTAGE: PHX2 Network outage - 2011-01-25 02:00 UTC
by Stephen John Smoogen 20 Jan '11

20 Jan '11

There will be an outage starting at 2011-01-25 02:00 UTC, which will last approximately 3 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2011-01-25 02:00 UTC' Reason for outage: We are moving from older netapp to newer one due to hardware limitations and aging hardware. Due to change in disk architectures and such, we will be having a total outage of Fedora services. Affected Services: BFO - http://boot.fedoraproject.org/ Bodhi - https://admin.fedoraproject.org/updates/ Buildsystem - http://koji.fedoraproject.org/ CVS / Source Control Docs - http://docs.fedoraproject.org/ Email system Fedora Account System - https://admin.fedoraproject.org/accounts/ Fedora Community - https://admin.fedoraproject.org/community/ Fedora Hosted - https://fedorahosted.org/ Fedora People - http://fedorapeople.org/ Fedora Talk - http://talk.fedoraproject.org/ Main Website - http://fedoraproject.org/ Mirror List - https://mirrors.fedoraproject.org/ Mirror Manager - https://admin.fedoraproject.org/mirrormanager/ Package Database - https://admin.fedoraproject.org/pkgdb/ Smolt - http://smolts.org/ Spins - http://spins.fedoraproject.org/ Start - http://start.fedoraproject.org/ Torrent - http://torrent.fedoraproject.org/ Translation Services - http://translate.fedoraproject.org/ Wiki - http://fedoraproject.org/wiki/ Unaffected Services: DNS - ns1.fedoraproject.org, ns2.fedoraproject.org Contact Information: Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage. Ticket: https://fedorahosted.org/fedora-infrastructure/ticket/2571 -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren

1 0

Meeting Re: Eucalyptus hosting
by Garrett Holmstrom 18 Jan '11

18 Jan '11

Hi, At the infrastructure meeting yesterday Seth brought up the topic of cloud hosting for Fedora services. With EC2 and Rackspace being closed platforms, I suggested Eucalyptus as an alternative. I wasn't able to find any hosting providers in the US that use it, but Rich Wolski, the CTO of Eucalyptus Systems, expressed an interest in chatting with you infrastructure folks about whether or not we at Eucalyptus can just host it ourselves, depending on Fedora's needs and such. Would there be a good time and place to discuss this? The weekly IRC meeting? (For Rich's benefit: the weekly IRC meeting is at noon PST on Thursdays in #fedora-meeting on irc.freenode.net.) Another IRC meeting in case the weekly one would be too pressed for time? A conference call? Might an email thread be better? -- Garrett Holmstrom

3 3

Wiki extension request
by Adam Williamson 17 Jan '11

17 Jan '11

Hi, everyone. I'd like to ask about adding an extension to the Fedora wiki instance. Many of you may have noticed the package-specific test case project I've been working on lately - https://fedorahosted.org/fedora-qa/ticket/154 . Part of the plan for this project is to allow tools such as Bodhi to use mediawiki categories to figure out certain sets of test cases. One important one is 'all critical path test cases for package foo'. However, this requires the ability to do multi-category search in the wiki - find all pages that are members of two (or more) different categories. We've identified a couple of extensions which can potentially achieve this: http://www.mediawiki.org/wiki/Extension:Multi-Category_Search http://www.mediawiki.org/wiki/Extension:DynamicPageList_%28Wikimedia%29 I'd like to ask that we look at including one of these (or someone tell me that we already do :>). There are a couple of concerns that I don't have sufficient expertise to address. There would need to be some kind of, I guess, API access to the searches, so that an external tool could do the search - both extensions seem rather geared to doing the search entirely within the Wiki, for instance, including the results of the search in another page. I don't know how to tell which, if either, extension is most suited to this. The other concern is recursion - does either extension handle this case: page is a member of categories A and Z, but not directly of category B category A is a member of category B if searching for 'all pages that are in category B and Z', does the page show up? For our purposes I'd like that the be the case. If anyone can help with this it'd be greatly appreciated! Thanks. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net

2 2

Querying FAS details through IRC
by Misha Shnurapet 13 Jan '11

13 Jan '11

Hello. It is possible to query contributors' information on IRC with a command that fetches data from FAS. While it's OK to provide extensive information on users through the web interface (which is now only possible for the community members, AFAIK, as it requires to log in), I think it might come reasonable not to provide email addresses on such queries on IRC. -- Best regards, Misha Shnurapet, Fedora Project Contributor http://fedoraproject.org/wiki/User:Shnurapet shnurapet AT fedoraproject.org, GPG: 00217306

6 8

Outage notes
by Seth Vidal 12 Jan '11

12 Jan '11

Hi Everyone, I took some notes while we were rebooting boxes I wanted to share them with everyone for future outages. Ordering of the bounces: 1. xen14: puppet is on there and if that is back up first we have a place to stand for pushing out any changes (dns changes for example via puppet) - xen14 takes about 4 minutes to restart/POST 2. xen15: bastion01, db02 are on there - same 4 minute restart window once this is up you'll want to logout of bastion02 and into bastion01 so you have a firm place to do the xen05 reboots from which will take out bastion02. 3. edit dns on puppet to remove proxy01 from the wildcard/roundrobin and push that to the ns* servers and verify. 4. xen05: bastion02 (openvpn), proxy01 - 4-5 minutes for this machine to restart. once xen05 is completely up log back in and verify the vpn is back online 5. edit dns remove all the other proxy hosts and put proxy01 back in. Push and verify 6. virthost01 - I had to halt each of the kvms from a login - virsh shutdown didn't work. - 4 minute restart time on the hw. Note: make sure virthost01 is completely up - especially fas03. since taking down virthost02 next will take out fas02- you want to make sure you don't leave fas01 all by itself. 7. virthost02 - fas02 was not setup to autostart - that's now fixed. 8. virthost13 - uneventful 9. xen03 - spin01 spewed lots of umount issues - those are from the spin creation paths - they can be safely ignored - fas01.stg was running on xen03 according to the logs but there's no definition for it on the system - so not sure what the story is there. - neither of the other staging hosts were set to autostart 10. xen04: we apparently have a number of hosts w/only one dns record internally and they point to ns03 only. B/c when ns03 went away - lots of things got VERY VERY SLOW trying to resolve names. This is on my list to address. You must wait for xen04 to be completely up and ns03 running before you can take down xen07. Otherwise we'll be w/o dns internally to phx. 11. xen07: iscsi disks didn't come up right away - this kept ns04 from coming up immediately - needed to run /etc/init.d/iscsi start and they showed up. 12. xen09: uneventful 13. xen10: log01 needed an fsck b/c of the time since last mount - this took a long time. 14. xen11: secondary1 needed an fsck. also a 5-6 minute hw reboot time. 15. xen12: db1->db01 naming change kept it from coming up at boot b/c of the 'auto' symlink to db1. db01 had to fsck 16. cnode01 - 6-7 minute reboot time - nothing was set to autostart in xen - this is now fixed - autoqa01 and dhcp02 are set to autostart 17. db03: fsck took FOREVER to complete and this takes a lot of things done - for the future move db03 reboot higher up the stack, just in case. This machine's restart/POST time is REALLY high like 7-10 minutes. The console for it is less than forthcoming, too. 18. backup01: uneventful At this point internal was back online - except for the build xen systems and servers. External hosts: 19. - bodhost01: 5-6 minute machine reboot time - people01 - uneventful. - ibiblio01 - 5-7 minute machine reboot time. uneventful 20. - internetx01: uneventful - osuosl01: uneventful 21. - sb2 - must wait for ibiblio01 to be up b/c of not having any external name servers - sb3 - uneventful - sb4 - hosted1 listed more 'maxmem' in its config that sb4 had available - so that had to be edited down. Not sure how that EVER started - sb5 - uneventful 22. telia01 - proxy5 did not restart on its own - unknown as to WHY yet - but it did start manually. - retrace01 was not set to autostart tummy1 - uneventful Now all the proxy* rebooting is over so we can: 23. edit dns: put the other proxy hosts in the wildcard/RR - push and verify Build boxes: - bxen03 had koji2 listed in its set of hosts - but it wasn't running. This led to some confusion as to how to start the hosts on bxen03 b/c of insufficient memory for all guests. Eventually I realized bxen04 is where koji02 was running and that the left over guest file was never cleaned up on bxen03. Things to think about post-outage: - check all the raid arrays for lost disks - we saw this a couple of times - it's not pleasant. - check for downed vpns and/or broken resolution - we need to get a firm handle on why this is a hassle so often. Overall things to think about for the future: 1. dumping a complete virsh list - including how much memory is actually being used per vm per server before we start reboots 2. checking what disks need fscks because of mounted time and doing those earlier or separately. 3. verifying that all running vms are: a. intended to be running b. have a config file c. are set to autostart 4. verifying that all NOT running vms are: a. intended to be off b. are NOT set to autostart thoughts welcome. -sv

4 6

Outage: Fedora related services 2010-01-11 1900 UTC -> 2300 UTC.
by Stephen John Smoogen 11 Jan '11

11 Jan '11

Outage: Rolling update outages - 2010-01-11 19:00 UTC There will be an outage starting at 2010-01-11 19:00 UTC, which will last approximately 4 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2010-01-11 19:00 UTC' Reason for outage: Kernel and glibc updates require servers to be rebooted. Outages will be occurring on our virtual servers requiring all services to be down at some point. However due to rolling nature of outages no service should be affected long or deeply. Affected Services: BFO - http://boot.fedoraproject.org/ Bodhi - https://admin.fedoraproject.org/updates/ Buildsystem - http://koji.fedoraproject.org/ CVS / Source Control DNS - ns1.fedoraproject.org, ns2.fedoraproject.org Docs - http://docs.fedoraproject.org/ Email system Fedora Account System - https://admin.fedoraproject.org/accounts/ Fedora Community - https://admin.fedoraproject.org/community/ Fedora Hosted - https://fedorahosted.org/ Fedora People - http://fedorapeople.org/ Fedora Talk - http://talk.fedoraproject.org/ Main Website - http://fedoraproject.org/ Mirror List - https://mirrors.fedoraproject.org/ Mirror Manager - https://admin.fedoraproject.org/mirrormanager/ Package Database - https://admin.fedoraproject.org/pkgdb/ Smolt - http://smolts.org/ Spins - http://spins.fedoraproject.org/ Start - http://start.fedoraproject.org/ Torrent - http://torrent.fedoraproject.org/ Translation Services - http://translate.fedoraproject.org/ Wiki - http://fedoraproject.org/wiki/ Unaffected Services: Ticket Link: https://fedorahosted.org/fedora-infrastructure/ticket/2554 Contact Information: Please join #fedora-admin in irc.freenode.net or respond to this email to track the status of this outage. -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren

1 1

Turbogears downgraded on fas servers
by Toshio Kuratomi 10 Jan '11

10 Jan '11

I've re-downgraded TurboGears to 1.0.9 on the fas servers. Another incompatibility was discovered i nthe TG-.1.1.x series. EPEL6 is going out of beta soon so on Monday I'll need to decide whether to revert TG in EPEL6 to TG-1.0.x or work on fixing fas further. -Toshio

7 13

REMINDER: publictest box motd [this means you :)]
by Stephen John Smoogen 05 Jan '11

05 Jan '11

Ok when testing a project it is important to update the /etc/motd file with the following information: [fas username] - [date started] [date updated] project title - project summary directories/areas looked at. Example: Project Owner: smooge - Started: 2010-12-31 Updated: 2010-01-30 gnusong - a project to digitize the reading of all source code to a dance beat. /usr/local/bin, /srv/web/songs 1) Systems that DO NOT have updated motd's will be wiped, reinstalled to a fresh state and turned off until an RFR is needed for it. Systems whose last update was over 90 days will also be subject to wiping/reinstall/turn off. 2) publictest systems are NOT backed up. they are test/scratch systems that are meant for eventual integration with some sort of development/staging infrastructure. 3) due to communal nature of systems, people need to work together on them to make sure they are not overwriting and beating each other up. -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren

5 5

delta rpm compose debugging change
by Kevin Fenzi 03 Jan '11

03 Jan '11

Greetings. Of late openoffice.org and a few other packages have been not generating delta rpms in pushes. ;( They error with: Error genDeltaRPM for chunkd: exitcode was 256 - Reported Error: bad cpio archive But looking later at both rpms, things seem fine. ;( I've talked with jdieter and we would like to add in more debugging: <jdieter> nirik: We're still having the "bad cpio archive" with openoffice.org updates <jdieter> I have a patched version of deltarpm available at: http://www.lesloueizeh.com/jdieter/deltarpm-3.6-0.1.20101230git.fc14.src.rpm <jdieter> If run with -vv, it will report old rpm, new rpm, and tell us what the actual first six bytes of what's actually in the rpm <jdieter> vs the 070701 that it's supposed to be <jdieter> I don't have an el6 system, so it will have to be rebuilt for releng2 <jdieter> /usr/lib/python2.4/site-packages/createrepo/deltarpms.py needs to modified as well <jdieter> The line with "makedeltarpm -v" needs to be changed to "makedeltarpm -vv" epel5 scratch build at: http://koji.fedoraproject.org/koji/taskinfo?taskID=2694315 (releng02 is rhel5). If I can get enough +'s I'd be happy to put this in place whenever. Otherwise we can wait until next week after the freeze. Thoughts? Comments? Other ideas? kevin

3 4

Back
by Stephen John Smoogen 01 Jan '11

01 Jan '11

I am back and starting to catch up with emails. I ended up with almost 0 internet access last week so thanks to those who covered and watched ibiblio go up and down ( I was drawing talking with Crush the Turtle at the time :)). -- Stephen J Smoogen. "The core skill of innovators is error recovery, not failure avoidance." Randy Nelson, President of Pixar University. "Let us be kind, one to another, for most of us are fighting a hard battle." -- Ian MacLaren

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure January 2011