============================================ #fedora-meeting: Infrastructure (2014-04-03) ============================================ Meeting started by nirik at 18:00:08 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2014-04-03/infrastructure... . Meeting summary --------------- * welcome y'all (nirik, 18:00:08) * New folks introductions and Apprentice tasks. (nirik, 18:02:52) * all easyfix trac tickets in fedora-infrastructure trac have been reset to new and milestone corrected. (nirik, 18:09:22) * Applications status / discussion (nirik, 18:12:52) * apps now has deep linking support: https://apps.fedoraproject.org/#ForPackagers (nirik, 18:15:04) * jenkins-fedmsg plugin is now working. Just needs deployment work/access. (nirik, 18:15:29) * LINK: http://threebean.org/blog/pkgdb-acls-speedup/ (threebean, 18:15:50) * fedmsg triggered genacls is now live. (nirik, 18:16:15) * LINK: https://fedorahosted.org/fedora-infrastructure/ticket/3925 (threebean, 18:39:40) * ACTION: nirik to file a ticket for fasClient changes (nirik, 18:39:46) * fasClient could likely be made to run MUCH less often without too much work (nirik, 18:40:00) * Sysadmin status / discussion (nirik, 18:40:43) * easyfix tickets have been updated and cleaned up some. (nirik, 18:41:54) * updates to http://infrastructure.fedoraproject.org/infra/docs/ always welcome (nirik, 18:48:52) * Upcoming Tasks/Items (nirik, 18:51:06) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:51:07) * nirik at centos dojo in denver next week (nirik, 18:51:55) * bodhi2/taskotron fad planning ongoing, in june in denver (nirik, 18:52:15) * flock talk submissions closes very very very soon. (nirik, 18:53:09) * Open Floor (nirik, 18:54:15) Meeting ended at 18:59:27 UTC. Action Items ------------ * nirik to file a ticket for fasClient changes Action Items, by person ----------------------- * nirik * nirik to file a ticket for fasClient changes * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (97) * threebean (35) * dgilmore (21) * willo (14) * smooge (10) * SmootherFrOgZ (9) * jmosco (7) * danofsatx-work (6) * emedentsii_ (5) * zodbot (4) * relrod (4) * axil42 (3) * webpigeon (3) * tflink (2) * janeznemanic (2) * abadger1999 (1) * relrod_ (1) * puiterwijk (0) * lmacken (0) * mdomsch (0) * pingou (0) -- 18:00:08 <nirik> #startmeeting Infrastructure (2014-04-03) 18:00:08 <zodbot> Meeting started Thu Apr 3 18:00:08 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:08 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:08 <nirik> #meetingname infrastructure 18:00:08 <nirik> #topic welcome y'all 18:00:08 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:08 <zodbot> The meeting name has been set to 'infrastructure' 18:00:08 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:16 <dgilmore> hola 18:00:25 * relrod_ here 18:00:47 * danofsatx-work is waiting for the official "roll call" call..... 18:01:06 <smooge> here 18:01:34 * threebean is here 18:02:29 * tflink is here 18:02:43 <nirik> ok, lets go ahead and get started. ;) 18:02:52 <nirik> #topic New folks introductions and Apprentice tasks. 18:03:00 <nirik> any new folks like to introduce themselves? 18:03:06 <nirik> Or apprentices with questions or comments? 18:03:10 <nirik> emedentsii_: ^ 18:03:46 <danofsatx-work> last week, smooge mentioned maybe setting up some 'easyfix' ticket in regards to documentation. And thoughts on that? 18:03:46 <jmosco> I am new, hello everyone 18:03:49 <emedentsii_> hi 18:04:26 <nirik> welcome jmosco and emedentsii_. Can you each give a short intro about yourselves? and are you more interested in sysadmin or application development 18:04:27 <emedentsii_> I'm new too. jmosco , go ahead=) 18:04:48 * willo is here, sorry too many things open. laptop was hung for a good couple of mins 18:05:03 <jmosco> I would say around 60/40 18:05:06 <jmosco> im trying to learn more dev 18:05:27 <jmosco> I have been working in a devops style role now with a focus on automation 18:05:55 <jmosco> ive been in infra now for around 5 years 18:06:33 <jmosco> :), and very happy to start working with the fedora team 18:06:33 <nirik> awesome. welcome. See me after the meeting if you want to be added to our apprentice program... also #fedora-noc (for sysadmin) and #fedora-apps (for application devel) are good channels to ask questions in. 18:06:52 <jmosco> thank you 18:08:21 <nirik> cool. Welcome again. I talked with emedentsii_ some earlier also 18:08:53 <nirik> any other new folks ? 18:09:22 <nirik> #info all easyfix trac tickets in fedora-infrastructure trac have been reset to new and milestone corrected. 18:09:34 <nirik> So, if you are looking for an easyfix, do take another look today... 18:09:40 <emedentsii_> My name is Eugene. I have 2 year expirience in Automation QA, writing tests using selenium WebDriver and python. Also i involved in DevOps process as a member of our contin integration team. I would like to develop application for fedora, also i think i can be usefull for some infrastructure tasks. 18:10:30 <threebean> oo, selenium and continuous integration? that gives me an idea.. 18:10:32 <nirik> emedentsii_: great. :) You may also be interested in our qa devel stuff... see tflink on that 18:10:55 <nirik> s/see/talk to/ 18:10:59 <nirik> threebean: oh? 18:11:08 <threebean> emedentsii_: real quick -> we have a project called 'rube' that is supposed to test our staging infrastructure with selenium https://github.com/fedora-infra/rube/ 18:11:35 <threebean> emedentsii_: right now, we just run it by hand every now and then.. but it would be neat to run it periodically via some automatic means. 18:11:44 <emedentsii_> ok, great. i will check it 18:12:09 <nirik> yeah. :) it could definitely be expanded to cover more too. 18:12:43 <nirik> cool. 18:12:52 <nirik> #topic Applications status / discussion 18:13:00 <nirik> Any application status or news this week? 18:13:12 <nirik> pingou is off at a conference without much coverage. 18:13:27 <threebean> one small thing.. our little landing page at https://apps.fedoraproject.org got some new behavior this week. 18:13:58 <threebean> it now has 'deep-linking' where you can click on an item, share the link, and it should navigate to the same position in the graph 18:14:02 <threebean> like this: https://apps.fedoraproject.org/#ForPackagers 18:14:06 <nirik> oh nice. 18:14:12 <relrod> I just got the jenkins-fedmsg plugin working locally (with threebean's help :D) - Need to talk to threebean later about how to deploy it, because our jenkins master node (a cloud node) will need to be able to access fedmsg somehow...but it is working fine locally, so that is good. 18:14:29 <threebean> relrod: that's awesome :P 18:14:49 <nirik> cool. 18:15:04 <nirik> #info apps now has deep linking support: https://apps.fedoraproject.org/#ForPackagers 18:15:29 <nirik> #info jenkins-fedmsg plugin is now working. Just needs deployment work/access. 18:15:45 <threebean> oh, and we got janeznemanic's genacls + fedmsg updater deployed, too. 18:15:50 <threebean> http://threebean.org/blog/pkgdb-acls-speedup/ 18:16:04 <janeznemanic> not just mine 18:16:09 <threebean> :D 18:16:10 <janeznemanic> yours too 18:16:15 <nirik> #info fedmsg triggered genacls is now live. 18:16:35 <threebean> it is a neat proof-of-concept for replacing other cronjobs we have. 18:16:54 <threebean> fasclient, for example, syncs shell credentials from the FAS webapp every hour. 18:16:58 <nirik> yeah. ;) its nice to only run things when they need to run and not midlessly every hour or whatever 18:18:09 <relrod> I see charul just made some commits to https://github.com/charulagrl/mw-Datagrepper - would like to work with her on getting that deployed soon 18:18:20 <nirik> we talked about a lot of ways to make fasClient better... not sure if we are doing all of them or have picked some or what. :) We should figure out the actual roadmap 18:19:01 * tflink wonders if it would be worth looking into making taskotron's trigger more generic 18:19:23 <willo> question though about fedmsg generated events during outages 18:19:26 <threebean> relrod: she actually had a puppet patch to deploy it to production.. it just needs to be reworked to be tested in staging, I think. very soon :) 18:19:48 <relrod> threebean: awesome! 18:20:36 <willo> what happens when the gitolite side of things that's consuming the fedmsg is not running and a pkgdb event triggers a message to be produced and place on the bus 18:20:49 <threebean> yeah, it will just miss it. 18:20:52 * relrod remembers the day he deployed mw-FedoraBadges, and haproxy saving his butt a few times... fun day that was. :P 18:20:53 <willo> how does gitolite know the event occured in the past 18:21:15 <willo> ah, ok, so cron job might stay as the fall back 18:21:16 <threebean> heh, it won't. as things stand now, it would either need to be run by hand.. or the next acl change would trigger both to be picked up. 18:21:47 <threebean> willo: hm, so an idea was talked about but never implemented.. 18:22:05 * abadger1999 arrives from the fpc meeting 18:22:12 <threebean> .. to have fedmsg consumers leave some sort of file around indicating the time that they were last running. 18:22:18 <nirik> I wonder tho, perhaps we could have fedmsg emit a 'I am shutting down now' message when a machine is stopped... 18:22:32 <nirik> and a start and then we could see between them? 18:22:42 <threebean> and when they start up, to look for that file and then query the history for all messages (of whatever type) since then. 18:22:46 <willo> yeah, was thinking about something like that last week as well 18:22:52 * threebean nods 18:23:07 <nirik> in practice I don't think this is a big deal right now, but would be nice to think about... 18:23:31 <willo> so, I guess network outages are not a common occurance. :) 18:23:43 <threebean> yeah, I'm for it. It just needs some sustained attention from some people to implement/test/deploy. :p 18:24:07 * SmootherFrOgZ here 18:24:20 <nirik> willo: not too often... and most of the fedmsg emitting things are all in the same datacenter 18:25:19 <willo> nirik: ok, yeah cause I was wondering about fedmsg across the vpn's that might drop ocassionally 18:25:19 * smooge wonders if we could cram this in the journal and have systemd-fedmsg 18:25:32 <nirik> :) 18:25:59 <willo> damn internet, why can't it be more reliable. :) 18:26:07 <nirik> indeed. 18:26:16 <dgilmore> smooge: ewwwwwwwww 18:26:19 <dgilmore> ewwww 18:27:08 <nirik> abadger1999 / SmootherFrOgZ: so do you recall the exact plans we had for fasClient? I know we talked about push mode, running from ansible, etc... but did push mode get implemented? 18:27:09 <smooge> I think if we do this by fed22 we could have use everyone's unused cycles for bitcoin mining and profit 18:27:55 * threebean puts smooge in charge 18:28:08 * dgilmore takes threebean and smooge out back 18:28:13 <nirik> fedmsg-bot: bitcoin.mine.start -- smooges server started mining bitcoins. 18:28:14 <SmootherFrOgZ> nirik: I've a branch with a 1rst-step implementation where fasClient listen to fedmsg to do stuff. 18:28:36 <nirik> SmootherFrOgZ: awesome. ;) This is 2.0? 18:28:50 <willo> :) 18:28:53 <nirik> so wait, fasClient runs all the time and listens? or ? 18:29:53 <SmootherFrOgZ> nirik: for now, yes. but we could work from this implementation to tell ansible do fire specific fasClient. 18:30:08 <dgilmore> id rather push via ansible 18:30:28 <dgilmore> not make something extra listen on every box 18:31:09 <nirik> yeah, I think... thing on lockbox01 listens for fedmsgs related to accounts. When they come in, it looks to see if they matter for any. If they do, it fires off to run fasClient on those machines. I think we could implement this today. 18:31:25 <dgilmore> nirik: yep 18:31:47 <nirik> but that would be specific to our setup, might be nice for a more generic thing in fas upstream... but not sure how much where. 18:32:15 <dgilmore> nirik: would need some kind of policy engine 18:32:26 <dgilmore> that has a config with all teh amppings 18:32:36 <dgilmore> I wish i could spell 18:33:10 <dgilmore> nirik: I think it could be easily done today 18:33:22 <dgilmore> and would be generic enough to go upstream 18:33:35 <threebean> .. and, the mappings/policy. what is that? 18:33:43 <nirik> so wait... the only things we care about are: ssh key changed and user added/deleted right? 18:33:47 <threebean> a mapping of "changes to these fas groups mean push changes to these machines?" 18:33:57 <dgilmore> threebean: today its spread through puppet and ansible 18:34:21 <dgilmore> which fas groups land on which boxes 18:34:26 <nirik> user added/deleted from group we care about that is. 18:34:35 * threebean nods 18:34:49 <dgilmore> passwords or ssh keys changed 18:34:52 <nirik> so, really we could do a first cut of this that is dirt simple: 18:34:54 <dgilmore> users added or removed 18:35:08 <nirik> watch for ssh key, user added/removed from group. Run fasClient on all machines. 18:35:17 <nirik> because that doesn't happen that often at all anyhow. 18:35:22 <nirik> it would cut down runs vastly 18:35:31 <SmootherFrOgZ> nirik: yep. my changes handle what you want. you could look at it as sample 18:35:37 <nirik> dgilmore: don't need passwords, fas doesnt store them anymore 18:35:40 <dgilmore> nirik: and passwords? 18:35:47 <nirik> well, fas does 18:35:56 <dgilmore> nirik: how are they getting on the boxes for sudo? 18:35:56 <nirik> but fasClient doesn't put them in the nss db 18:36:15 <threebean> boxes ask fas via pam_url, correct? 18:36:17 <nirik> sudo uses 2fa 18:36:19 <nirik> yeah 18:36:27 <nirik> except for some small number of weird hosts. 18:36:31 <SmootherFrOgZ> nirik: fasClient does 18:36:33 <SmootherFrOgZ> actually 18:36:44 <nirik> I thought we disabled that? 18:37:28 <SmootherFrOgZ> hm...is this upstream changes or just config updates? 18:37:47 <nirik> can't recall, but when we moved to 2fa we saw no need to expose passwords on all hosts. 18:37:57 <nirik> I can double check that out of meeting 18:38:03 * SmootherFrOgZ nods 18:38:20 <dgilmore> nirik: i know that the secondary sigul boxes dont do 2 facter 18:38:32 <dgilmore> and one of the compose boxes doesnt 18:38:36 <nirik> dgilmore: yeah, there's about 4 or 5 weird hosts that don't 18:38:48 <nirik> but we should move them to it too. ;) 18:39:08 <dgilmore> anyway 18:39:22 <nirik> right, sorry... I will file a ticket on the fasClient thing and we can see what we can get going 18:39:34 <nirik> any other application news? 18:39:40 <threebean> https://fedorahosted.org/fedora-infrastructure/ticket/3925 18:39:46 <nirik> #action nirik to file a ticket for fasClient changes 18:40:00 <nirik> #info fasClient could likely be made to run MUCH less often without too much work 18:40:11 <nirik> ha too fast. ;) 18:40:14 <nirik> thanks threebean 18:40:29 <webpigeon> Evening 18:40:43 <nirik> #topic Sysadmin status / discussion 18:40:49 <nirik> morning webpigeon. ;) 18:40:56 <nirik> ok, on the sysadmin side of things... 18:41:08 <nirik> we have new machines racked, we just need networking on them and can get them installed. 18:41:15 <nirik> smooge: is there a ticket on networking those new boxes? 18:41:54 <nirik> #info easyfix tickets have been updated and cleaned up some. 18:41:55 * axil42 sneaks in late 18:42:09 <nirik> we did a mass reboot on tuesday. It went pretty well. 18:42:19 <SmootherFrOgZ> threebean: for the record: https://github.com/fedora-infra/fas/blob/feature/new_fasClient/client/fas... 18:42:20 <axil42> nirik, thanks for the cleanup ;) 18:44:02 <nirik> no problem 18:44:38 <nirik> I still need to make some sundries servers and look at the ansible wiki migration diffs. 18:44:53 <nirik> I did also file a few new easyfix tickets 18:45:02 * webpigeon looks 18:45:17 <smooge> I need to put in some documentation tickets.. 18:46:37 <nirik> smooge: cool. Docs on what? 18:47:10 <danofsatx-work> From what I see, a lot of the how-tos on the Fedora docs page haven't been updated since 14 to 16. 18:47:12 <smooge> well I figured we needed to go over documentation we have already.. and see if it actually meaans anything 18:47:27 <smooge> well that is different documentation 18:47:40 <nirik> danofsatx-work: docs.fedoraproject.org? yeah, thats the fedora docs group that does those. 18:47:40 <danofsatx-work> oh, ok...wrong docs ;) 18:48:32 <nirik> we have these: http://infrastructure.fedoraproject.org/infra/docs/ 18:48:38 * danofsatx-work was befuddled 18:48:38 <threebean> SmootherFrOgZ: cool :) 18:48:39 <nirik> updates to which are always welcome. 18:48:52 <nirik> #info updates to http://infrastructure.fedoraproject.org/infra/docs/ always welcome 18:48:53 <smooge> I am talking about 'how are we set up?' 'what do we expect people to be able to do on people/pkgs/hosted', 'what commands should we do in git', 'why is the sky blue and vegemite black', 'who was on the grassy knoll' 18:49:08 <axil42> smooge, +1 18:49:10 * nirik makes a 'jfk.txt' doc 18:49:24 <willo> :) 18:49:36 <smooge> then take the csi docs and see if we still 'follow' them, 18:49:55 <nirik> yeah, csi also needs love. Just not had time from my side. 18:51:06 <nirik> #topic Upcoming Tasks/Items 18:51:07 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:51:14 <nirik> anything upcoming folks would like to note or schedule? 18:51:40 <nirik> looks like I will be giving an ansible thing at the centos dojo in denver next week. ;) Need to write that up... 18:51:53 <threebean> nirik: cool :) 18:51:55 <nirik> #info nirik at centos dojo in denver next week 18:52:15 <nirik> #info bodhi2/taskotron fad planning ongoing, in june in denver 18:52:26 <threebean> the Bodhi2 FAD is still coming up in June.. 18:52:47 <threebean> I'm still trying to figure out some of the budget stuff but I haven't had luck getting in touch with rsuehle this week. 18:52:48 <nirik> I might be away first week of may... still havent decided my plans there yet, but will let folks know when I do. 18:53:09 <nirik> #info flock talk submissions closes very very very soon. 18:54:15 <nirik> #topic Open Floor 18:54:21 <nirik> any items for open floor? questions, comments/ 18:55:34 <smooge> not from me 18:55:46 <smooge> nirik, ah man now I want to go to the denver dojo 18:55:55 <threebean> random item -> 18:56:01 <danofsatx-work> all's quiet on the southern front 18:56:07 <nirik> smooge: come up. :) It's only a 5-6 hour drive... 18:56:13 <threebean> github users, I packaged up a cli tool: https://bugzilla.redhat.com/show_bug.cgi?id=1083344 18:56:42 <threebean> it is fun to use.. EOM 18:56:42 <willo> threebean: cool 18:57:13 <nirik> nice 18:57:24 <webpigeon> shiny :) 18:57:41 <nirik> ok, if nothing else, will close out the meeting in a minute here. 18:58:17 <willo> a lot less typing to clone github repo, sweet 18:58:30 <threebean> willo: yeah, it can interact with pull requests and stuff too 18:58:39 <emedentsii_> Thank you! 18:58:53 <willo> nice 18:59:25 <nirik> ok, thanks for coming everyone. :) Lets go continue over in #fedora-admin, #fedora-apps and #fedora-noc. See you all there. 18:59:27 <nirik> #endmeeting