============================================ #fedora-meeting: Infrastructure (2014-05-01) ============================================ Meeting started by nirik at 18:00:03 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2014-05-01/infrastructure... . Meeting summary --------------- * greetings starfighters (nirik, 18:00:03) * New folks introductions and Apprentice tasks (nirik, 18:02:13) * LINK: http://fedoraproject.com/easyfix (webpigeon, 18:07:12) * Applications status / discussion (nirik, 18:08:02) * some work on a flask re-write of mirrormanager ongoing (nirik, 18:14:14) * ui work on pkgdb2 ongoing (nirik, 18:14:24) * hyperkitty came up in the news a few times this week in slashdot and lwn, pointing to our stg instance. (nirik, 18:14:48) * LINK: https://fedorahosted.org/fedora-infrastructure/ticket/4044 (threebean, 18:17:15) * LINK: http://threebean.org/blog/fedmsg-collectd-ng/ (threebean, 18:17:19) * Sysadmin status / discussion (nirik, 18:18:47) * storage move had soe issues, but hopefully we have worked them out now. (nirik, 18:21:14) * new bvirthosts are on-line (nirik, 18:21:20) * LINK: https://admin.fedoraproject.org/nagios/ is our main nagios (nirik, 18:33:10) * Upcoming Tasks/Items (nirik, 18:35:09) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:35:09) * LINK: https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014 (threebean, 18:35:54) * nirik will be out saturday to next thursday. (nirik, 18:36:06) * Open Floor (nirik, 18:40:09) Meeting ended at 18:56:22 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (108) * threebean (23) * henderbj (15) * pingou (15) * bwood09 (13) * smooge (13) * mattdm (9) * zodbot_ (7) * ootbro (6) * danrimal (5) * Daredel (4) * danofsatx-work (3) * nj0y (3) * relrod (2) * webpigeon (2) * janeznemanic (1) * mpduty (1) * lmacken (1) * ghostalker (1) * danofsatx|kvirc (1) * oddshocks (1) * dgilmore (1) * mdomsch (0) * puiterwijk (0) * abadger1999 (0) -- 18:00:03 <nirik> #startmeeting Infrastructure (2014-05-01) 18:00:03 <zodbot_> Meeting started Thu May 1 18:00:03 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:03 <zodbot_> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:03 <nirik> #meetingname infrastructure 18:00:03 <nirik> #topic greetings starfighters 18:00:03 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:03 <zodbot_> The meeting name has been set to 'infrastructure' 18:00:03 <zodbot_> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:10 * relrod waves 18:00:26 * pingou 18:00:31 * webpigeon waves 18:00:48 <janeznemanic> hi 18:01:04 * lmacken 18:01:33 <danofsatx-work> I'm here, but if Konversation wigs out on me again 18:01:47 <danofsatx|kvirc> I'll be here instead 18:02:00 <nirik> :) 18:02:06 <nirik> ok, lets go ahead and get started.... 18:02:08 * threebean is here 18:02:13 <nirik> #topic New folks introductions and Apprentice tasks 18:02:27 <danrimal> hi, i am new here 18:02:34 <nirik> any new folks want to do a quick one line introduction of themselves? or apprentices with questions or comments? 18:02:46 <danrimal> yes, sure 18:02:48 <danrimal> I am sysadmin, my job is high load and high availability application as web, mail and databases services as well as bgp and ospf networking 18:03:00 <ootbro> I'll also jump in as one of the newbies 18:03:02 <danrimal> and i am interested in sysadmin things, if available 18:03:31 <ootbro> I've been using Linux for years and now want to contribute here. I think the testing FIG is my best starting point, with an eye toward sysadmin-main (eventually) 18:03:38 <nirik> danrimal: welcome! sure thing... see me in #fedora-admin after the meeting and I can get you setup in our apprentice program... 18:03:52 <nirik> ootbro: welcome again. ;) 18:03:59 <ootbro> thanks 18:04:22 <nj0y> As I introduced myself in the mailing, i'm a sysadmin/engineer from switzerland and very interested getting in charge here. And i search some fig to join. I think also fig testing or fig web is a good place for me. 18:05:00 <nirik> nj0y: welcome also. ;) 18:05:16 * ghostalker is here 18:05:17 <nj0y> thanks, glad i'm here. 18:05:30 <nirik> always good to have new folks around... do chime in with questions or comments anytime... 18:05:31 <mpduty> .fasinfo mohanprakash 18:05:32 <zodbot_> mpduty: User: mohanprakash, Name: Mohan Prakash, email: mpduty(a)gmail.com, Creation: 2013-12-27, IRC Nick: mpduty, Timezone: Asia/Kolkata, Locale: en, GPG key ID: 0xAF620142, Status: active 18:05:36 <zodbot_> mpduty: Unapproved Groups: l10n-editor l10n-commits marketing 18:05:39 <zodbot_> mpduty: Approved Groups: fi-apprentice cvsl10n cla_done cla_fpca 18:06:26 <nirik> I can assist anyone after the meeting over in #fedora-admin who wants to join our apprentice group or would like to be pointed at easyfix tickets, etc. :) 18:06:31 <nirik> Welcome again everyone! 18:06:52 <ootbro> many thanks.... I could use the help in getting started 18:07:01 <nj0y> me too. 18:07:08 <danrimal> ok, thanks 18:07:12 <webpigeon> http://fedoraproject.com/easyfix 18:07:23 * mattdm is lurking 18:07:35 * bwood09 is here 18:07:46 <nirik> see also http://fedoraproject.org/wiki/Infrastructure/GettingStarted and https://fedoraproject.org/wiki/Infrastructure_Apprentice 18:08:02 <nirik> #topic Applications status / discussion 18:08:14 <nirik> any application side news from the previous week or upcoming? 18:08:27 * pingou has done a good chunk of work on mirrormanager2 this week 18:08:42 <pingou> and over the last two days I am on the re-design of some the page of pkgdb2 18:08:48 * Daredel is late 18:08:48 <pingou> including http://209.132.184.188/package/R-DBI/ 18:08:53 <nirik> pingou: this is the flask re-write of it? or is it tg2? or ? 18:08:57 <pingou> nirik: flask 18:09:10 <pingou> but I only worked on the UI 18:09:14 <nirik> have you contacted mdomsch any on it? (I know he's not been around) 18:09:24 <threebean> pingou: that looks *much* nicer. 18:09:43 <relrod> yeah, that looks nice :) 18:09:56 * nirik is waiting for load. ;) 18:10:08 <nirik> I bet a bunch of us clicked it at the same time. 18:10:35 <danofsatx-work> i waited a bit, came up fine for me ;) 18:10:48 <nirik> or... it could be my pesky wireless. ;( 18:10:53 <danofsatx-work> which is amazing, considering what I've been fighting with locally...... 18:11:33 <nirik> pingou: how is 'package administrator' determined? 18:11:41 <pingou> threebean: designed by mizmo, I can't beat that :) 18:11:47 * oddshocks here late, in lecture as usual 18:11:53 <pingou> nirik: Contacts are the POC, Admins are the users with approveacls 18:12:56 <nirik> pingou: ok, so anyone with any approveacls? 18:13:20 <pingou> yes 18:13:36 <pingou> nirik: or pending approveacls (then there is a (?) icon next to them) 18:13:51 <nirik> ok, cool. 18:14:14 <nirik> #info some work on a flask re-write of mirrormanager ongoing 18:14:24 <nirik> #info ui work on pkgdb2 ongoing 18:14:48 <nirik> #info hyperkitty came up in the news a few times this week in slashdot and lwn, pointing to our stg instance. 18:15:24 <threebean> are we any closer to cutting another list over? 18:15:41 <threebean> erm, by that I mean changing some existing lists from mailman2 to mailman3 18:15:54 <nirik> I sent abompard some issues and he was going to fix them up... then we were going to see where we were. 18:16:05 <nirik> hopefully soon tho. 18:16:09 <threebean> cool. newly queued stuff.. 18:16:11 * threebean nods 18:16:12 <nirik> I'd be happy to move the infra list. 18:16:20 <threebean> yeah, agreed 18:16:36 <nirik> there may also be some fixes from this recent press on it... 18:17:10 <threebean> unrelated, janeznemanic and I have been working on some fedmsg monitoring stuff and made some progress this week 18:17:15 <threebean> https://fedorahosted.org/fedora-infrastructure/ticket/4044 18:17:17 <nirik> excellent. 18:17:19 <threebean> http://threebean.org/blog/fedmsg-collectd-ng/ 18:17:34 <threebean> collectd is in place and fun. nagios checks coming soon. 18:17:56 * bwood09 starts reading the entirety of threebean.org 18:18:17 <nirik> any other application type news? or shall we move on to sysadmin? 18:18:47 <nirik> #topic Sysadmin status / discussion 18:19:02 <nirik> smooge got some of our new build virthosts up and running yesterday. 18:19:21 <smooge> yay 18:19:22 <nirik> Tuesday night we moved our backend storage from one netapp to another less loaded one... 18:19:28 <smooge> hahah 18:19:29 <nirik> but we have had some issues since then. ;( 18:19:34 <smooge> boo 18:19:58 <nirik> It's looking a lot like those issues are related to some virthosts having an emulated realtek network card instead of virtio. 18:20:07 <bwood09> nirik, will those new virthosts need to be added to nagios and the such? 18:20:09 <nirik> something in the move caused them to start dropping packets like mad 18:20:16 <nirik> bwood09: they will indeed. ;) 18:20:42 <nirik> I can file a ticket on them after the meeting. 18:20:45 <nirik> or smooge can 18:20:47 <bwood09> I'm going to go through today and tomorrow and take care of the nagios stuff, so if you drop a ticket for them in easyfix-- yeah 18:20:49 <bwood09> lol 18:20:51 <nirik> or really anyone can. ;) 18:21:14 <nirik> #info storage move had soe issues, but hopefully we have worked them out now. 18:21:20 <nirik> #info new bvirthosts are on-line 18:21:57 * smooge opens an easyfix ticket that someone can open an easyfix ticket to add monitoring for several hosts 18:22:11 <henderbj> Hello all... i am late... already read previous messages 18:22:19 <bwood09> Also, not sure if this is the place to do this, but I want to get on with the sysadmin-hosted group 18:22:26 * pingou gtg 18:22:33 <nirik> welcome henderbj 18:22:36 <smooge> bye pingou 18:22:38 <nirik> bye pingou 18:23:46 <nirik> bwood09: what sorts of things do you want to work on there? any tickets in specific? or just adding new projects and such? 18:24:27 <nirik> we had some plans in there we could look at again and see if you might want to work on them... 18:24:30 <bwood09> I'm going to look at the tickets today and see if there's anything I want to tackle. Recently, most of my experience has been git, svn, hg, and bzr so I figure I'd be a good fit 18:24:52 <nirik> sure thing. Let me (or any other hosted sponsor know) and we can see about helping you along. 18:24:58 <bwood09> Alrighty 18:25:10 <nirik> on nagios... we had a lot more alerts this last week I fear... 18:25:21 <nirik> 273 I see since last thursday. 18:25:45 <threebean> oo 18:25:47 <bwood09> What's the norm for those? 18:25:47 <nirik> the vast majority of which I think were related in one way or another to the storage move. 18:25:58 <threebean> this is a fun new routine. :p 18:26:01 <dgilmore> damn storage 18:26:32 * nirik looks back at the previous weeks 18:27:18 <nirik> 77 the week before 18:27:34 <bwood09> oh wow 18:28:18 <nirik> I'd like to reduce them as much as we can... I fear it will be impossible to make them 0 without making them not alert when theres problems users will notice. 18:28:35 <henderbj> well, that's normal... a lot of alarms when someone touches anything! 18:29:13 <nirik> well, most of the 'normal' ones are network related. We have a very wide network... so if our monitoring host can't reach some datacenter, it alerts. 18:30:07 <nirik> some of the ones this last week were also from a datacenter where we started to see packet loss... they were being hit by a DDOS. 18:30:12 <smooge> or where we aren't losing pings but they are taking close to a second to travel around the world 18:30:29 <nirik> anyhow, if anyone wants to dig thru nagios logs and propose changes that would be lovely. ;) 18:30:53 <henderbj> i will be testing nagios on my own testing machine 18:31:04 <nirik> we may be able to tune the network related ones down some, but not too far. 18:31:21 <henderbj> When get into it, i will pick something about nagios to help 18:31:42 <ootbro> question..... is there a way in nagios to not try a set of hosts if a "core" host is unreachable due to a network outage? 18:31:44 <nirik> henderbj: sounds great. Feel free to ask in #fedora-noc or #fedora-admin if you have any questions about our setup 18:31:54 <nirik> ootbro: yeah, it has dependencies... 18:32:01 <henderbj> Tnx, nirik, sure 18:32:09 <nirik> I think they should be in pretty good shape now, I revamped them all a while back 18:32:32 <nirik> so if say virthost01 is down, it will only alert about that, not the vm's running on it also 18:32:42 <nirik> or a router is down, etc. 18:33:10 <nirik> https://admin.fedoraproject.org/nagios/ is our main nagios 18:33:22 <nirik> and https://admin.fedoraproject.org/nagios-external/ is a smaller one we have at a secondary datacenter 18:33:43 <nirik> anyone should be able to login with their fedora account login/pass 18:34:29 <nirik> ok, any other sysadmin related stuff? 18:35:09 <nirik> #topic Upcoming Tasks/Items 18:35:09 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:35:13 <threebean> good stuff 18:35:20 <nirik> anything upcoming anyone would like to schedule or note? 18:35:34 * pingou has none 18:35:44 <threebean> heh, kinda like a broken record... but we have the bodhi2 FAD upcoming in June 18:35:47 <nirik> I'd like to note that I will be GONE from saturday until thursday (back late wed night) 18:35:48 <smooge> just more hardware to install 18:35:54 <threebean> https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014 18:36:02 <threebean> nothing new to note.. just reminding that its happening. 18:36:06 <nirik> #info nirik will be out saturday to next thursday. 18:36:11 <pingou> oh, during the meeting I pushed the change the 'Manage ACL' page: see http://209.132.184.188/package/guake/acl/commit/ (replace guake by a package you own) 18:36:22 <nirik> if you need me for anything before then, please find me today/tomorrow. ;) 18:36:43 <threebean> nirik: if you have specific things you need taken care of while you're gone, feel free to tell us either here or offline. 18:36:43 <smooge> during that time threebean will technically be in charge but in an undisclosed bunker. I will be available as Alexander Haig 18:37:16 * threebean promotes smooge 18:37:27 <nirik> can do. ;) I will have cell saturday and wed, but won't even have that the rest of the time. Hurray wilderness! :) 18:37:29 <pingou> I'm out from Sunday to Saturday next week 18:37:45 <bwood09> I'm probably going to be out for the same ^ 18:37:54 <bwood09> Supposed to be going to Georgia 18:38:00 <pingou> I'll likely check on emails once in a while, but I'll try to stay away from irc :) 18:38:06 <nirik> popular vacation week. ;) 18:38:35 <smooge> nirik, threebean with you and pingou gone.. should we go to warm slush for changes? 18:39:12 <nirik> well, I'd say to be carefull sure... dunno if we need anything formal 18:39:16 <smooge> eg changes need at least a IRC +1 from someone else who can review it before commit/push 18:39:25 <nirik> since I won't have phone, I don't care... can't bother me. ;) 18:39:45 <threebean> you'll come back and we'll have a chef setup in place 18:39:53 <nirik> :) 18:40:02 * smooge goes to find his contacts in the Smoke Jumpers to see if they can fix that 18:40:06 <nirik> anyhow... 18:40:09 <nirik> #topic Open Floor 18:40:20 <nirik> anyone have anything for open floor? questions? comments? 18:40:41 <mattdm> nirik yeah I have one 18:40:43 <ootbro> I was able to get into nagios with my regular FP id 18:40:51 <mattdm> just filed https://fedorahosted.org/fedora-infrastructure/ticket/4350 18:40:51 <henderbj> i have one... a moment please 18:41:13 <henderbj> can apprentice members ssh to lockbox01? 18:41:17 <mattdm> colin walters requests a slightly-less ad hoc place to do ostree experimetnation fedora 18:41:39 <Daredel> hi, i get late for the New folks introductions and Apprentice tasks, i'm new and really exited about contributing to the community 18:41:46 <nirik> mattdm: hum, ok, I already promised walters one of our old virthosts once we move a new one in... is that for this same thing or something different? 18:42:04 <mattdm> nirik I *think* this is the same thing? maybe he is just getting antsy? :) 18:42:15 <nirik> henderbj: absolutely. See the ssh access link off the apprentice page. ;) 18:42:31 <nirik> Daredel: welcome! are you interested in sysadmin or application devel or both? 18:42:32 * mattdm did not know about that. or forgot if i did 18:43:03 <nirik> mattdm: ok. We have been backloged by heartbleed, then virthosts getting shipped the wrong place, then storage hell, etc. We are getting there tho. 18:43:11 <Daredel> i think both, but most of all devel 18:43:21 <mattdm> nirik ok I will relay that. 18:43:34 <nirik> smooge: did we decide what 2 old virthosts we were going to save? one for ostree the other for cloud lockbox? 18:44:01 <nirik> Daredel: great. See me after the meeting in #fedora-admin and I can help set you up with the apprentice group... #fedora-apps can help with application devel stuff. :) 18:44:13 <henderbj> I read it before.. but from bastion01 i get: Permission denied (publickey). 18:44:14 <Daredel> ok thanks :D 18:44:42 <bwood09> henderbj, how are you authenticating? And did you upload your public key to FAS? 18:44:54 <nirik> henderbj: can assist you after the meeting in #fedora-admin, but you should be doing 'ssh lockbox01.phx2.fedoraproject.org' from your home machine, it should use bastion01 as a proxy... 18:45:02 <smooge> nirik, I have not yet. I keep doing so and then forgetting which 2 I saved and start over 18:45:19 <nirik> smooge: yeah, we should see if we can hurry on one for ostree stuff. 18:45:43 <nirik> mattdm: we will try and hurry it along. 18:45:51 <mattdm> nirik thanks. :) 18:45:58 <mattdm> nirik is the previous ticket https://fedorahosted.org/fedora-infrastructure/ticket/4200 ? 18:46:09 <henderbj> I created the ~/.ssh/config file, then ssh to bastion01 , and from there, i did: ssh lockbox01.phx2.fedoraproject.org, and get as response: Permission denied (publickey). 18:46:16 <nirik> mattdm: could be yeah 18:47:04 <nirik> henderbj: you can't do it that way.. ;) bastion doesn't (and shouldn't) have your config and keys on it... you should run the 'ssh lockbox01.phx2.fedoraproject.org' from your home machine. The config takes care of the proxying part. 18:47:17 <henderbj> ok... i will trying to connect after the meeting 18:47:50 <nirik> we will get it working. :) 18:47:51 <smooge> mattdm, we are having to do a lot of yak shaving to get these boxes available. it may be mid may 18:49:07 <nirik> anyhow, we will get there as soon as we can. 18:49:23 <nirik> smooge: lets both go over them and come up with a pair... 18:50:26 <nirik> ok, anything else? or shall we call it a meeting? 18:51:05 <henderbj> well, about easyfix tickets 18:51:36 <nirik> sure, shoot... 18:51:37 <henderbj> are those easyfix tickets from 2011-2012 really need any work done? 18:51:50 <nirik> if they are still open, yes. 18:52:17 <nirik> they may have been things that weren't urgent enough for someone else to do... 18:52:41 <threebean> henderbj: if you have one or two in particular in mind, drop a link to them in channel 18:52:56 <nirik> if they don't need anything anymore, we can close them. ;) 18:53:06 <threebean> otherwise, I can only guess... 18:53:49 <henderbj> i reviewed this one: https://fedorahosted.org/fedora-infrastructure/ticket/3617 18:54:50 <threebean> yeah, I'm pretty sure that one still needs work 18:54:50 <henderbj> After my "quick" review, i didn0t find anything to do... i left it because it was too old ;) 18:54:51 <nirik> yeah, probibly needs the current output added, but I can do that if you want to work on it. ;) 18:56:00 <nirik> ok, lets all move over to #fedora-admin, #fedora-noc and #fedora-apps... 18:56:04 <henderbj> Ok... if any question i will post it on the ticket to get going to close it 18:56:15 <nirik> thanks for coming everyone. And welcome again to all the new folks. ;) 18:56:19 <nirik> henderbj: sounds great. 18:56:22 <nirik> #endmeeting