Summary/Minutes for today's Fedora Infrastructure meeting (2015-02-26)

============================================ #fedora-meeting: Infrastructure (2015-02-26) ============================================ Meeting started by nirik at 18:00:05 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2015-02-26/infrastructure... . Meeting summary --------------- * aloha (nirik, 18:00:06) * New folks introductions and Apprentice tasks (nirik, 18:02:25) * Applications status / discussion (nirik, 18:06:10) * new nuancier deployed (pingou, 18:06:35) * new anitya deployed (pingou, 18:06:40) * fedora notifications and the-new-hotness had releases just before freeze. (nirik, 18:11:46) * Sysadmin status / discussion (nirik, 18:15:30) * nagios/alerts recap (nirik, 18:22:27) * LINK: http://ur1.ca/jr7j4 (nirik, 18:22:27) * Upcoming Tasks/Items (nirik, 18:26:03) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:26:03) * Meeting process (nirik, 18:27:00) * LINK: https://sprint.ly/blog/kanban/ ? (intrk, 18:32:27) * will try adding a shared doc and see if it makes our meetings more helpfull. (nirik, 18:35:11) * ACTION: nirik to make doc and mail list. (nirik, 18:35:21) * Learn about an app: squid / kojipkgs (nirik, 18:35:45) * Open Floor (nirik, 18:47:22) * everyone go test progit. :) http://209.132.184.222 (nirik, 18:52:40) Meeting ended at 18:54:12 UTC. Action Items ------------ * nirik to make doc and mail list. Action Items, by person ----------------------- * nirik * nirik to make doc and mail list. * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (116) * pingou (28) * dgilmore (10) * threebean (9) * intrk (7) * tflink (7) * smooge (6) * kushalk124 (6) * zodbot (4) * kushal (2) * lmacken (2) * sborza (1) * puiterwijk (1) * danofsatx (1) * oddshocks (1) * danielbruno (1) * relrod (1) * janeznemanic (1) * abadger1999 (0) * mdomsch (0) -- 18:00:05 <nirik> #startmeeting Infrastructure (2015-02-26) 18:00:05 <zodbot> Meeting started Thu Feb 26 18:00:05 2015 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:06 <nirik> #meetingname infrastructure 18:00:06 <zodbot> The meeting name has been set to 'infrastructure' 18:00:06 <nirik> #topic aloha 18:00:06 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:06 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:22 <smooge> morning 18:00:35 <puiterwijk> hi. I'm here, but also in another meeting 18:00:36 * danielbruno is here 18:00:38 <janeznemanic> hi 18:00:38 * sborza waves hello 18:00:39 * lmacken 18:00:52 * pingou here 18:00:57 <kushal> I am here 18:01:02 <kushal> though sleeping I guess 18:01:16 <nirik> :) 18:01:37 * relrod here 18:01:48 * tflink is here 18:02:16 <nirik> morning everyone. ;) 18:02:21 <threebean> hey hey 18:02:25 <nirik> #topic New folks introductions and Apprentice tasks 18:02:32 <nirik> any new folks like to introduce themselves? 18:02:38 <nirik> or apprentices with questions or comments? 18:03:36 <kushalk124> Hey, I have started with some packaging, and looking for a reviewer and sponsor for my first package. 18:03:47 <kushalk124> And I would like to be added to the apprentice group again :) 18:03:55 <nirik> kushalk124: cool. have a link? sure, we can add you... 18:04:04 <kushalk124> Also, I would like to contribute to some python apps. 18:04:24 <kushalk124> Hey nirik, https://bugzilla.redhat.com/show_bug.cgi?id=1194576 18:04:29 <nirik> excellent. 18:04:38 <kushalk124> :) 18:04:54 <danofsatx> good day, folks. 18:05:02 <nirik> kushalk124: see me in #fedora-admin after the meeting and I'll get you added... 18:05:09 <nirik> morning danofsatx 18:05:17 <kushalk124> nirik, sure 18:05:34 <dgilmore> hey 18:05:40 <nirik> morning dgilmore 18:06:10 <nirik> #topic Applications status / discussion 18:06:20 <nirik> any applications side news this week? or upcoming? 18:06:35 <pingou> #info new nuancier deployed 18:06:40 <pingou> #info new anitya deployed 18:06:56 <nirik> isn't there a nuancier election going on now? 18:06:57 <pingou> (with a nice bunch of fixes and new features :)) 18:07:02 <pingou> nirik: yes it is :(: 18:07:04 <pingou> :( 18:07:42 <nirik> pingou: ok. 18:07:56 <pingou> I wasn't happy about it, but I learned about the new election when it was already started 18:08:21 <pingou> while there was 6 months work piled for the release :/ so we tested it carefully with gnokii before pushing to prod 18:08:26 <nirik> as long as the update didn't break them... 18:08:32 <pingou> (and so far, no reports so looks like it went ok :)) 18:09:04 <nirik> ok, good. 18:09:43 <nirik> ok, anything else? I know since we are in freeze things are a bit quiet. ;) 18:09:56 <threebean> yeah, a new fmn release went out right before freeze 18:09:57 <intrk> did I miss the new comers' introduction? 18:10:09 <threebean> contained changes that made the format of some emails nicer (mostly). 18:10:11 <nirik> intrk: yeah... but feel free to jump in now with one. :) 18:10:19 <threebean> an update of the-new-hotness went out too with similar cosmetic changes. 18:11:46 <nirik> #info fedora notifications and the-new-hotness had releases just before freeze. 18:12:41 <intrk> okay, well... Hi! I'm a fairly young engineer, 25yo. Started at the Helpdesk, went through AV/Security support tech job and now I'm a Network Security technical support engineer. Got my RHCSA last year but since I work mostly on firewalls at the moment I want to have some more experience with Linux and thought this might be the place. 18:12:54 <nirik> intrk: cool. welcome. ;) 18:13:17 <nirik> I guess from that background you are more interested in sysadmin type things? or are you also interested in application devel stuff? 18:13:53 <intrk> Both to be honest, however, I haven't dabbled in programming too much 18:14:31 <intrk> I think I can read basic code and have passed an introductory course for Python and would love to get my feet with that also 18:14:32 <nirik> cool. :) We have both.... 18:15:03 <nirik> see me in #fedora-admin after the meeting and I can get you added to our apprentice group... and see #fedora-apps for the application side of things. 18:15:09 <nirik> and again welcome. ;) 18:15:30 <nirik> #topic Sysadmin status / discussion 18:15:40 <intrk> :) superb, thank you 18:15:50 <nirik> lets see... I wish I could say it was a quiet week with the freeze and all, but it's not really been. ;) 18:16:13 <nirik> we have a possibly ongoing issue with out squid proxy. (I'll explain that more in a bit here) 18:16:53 <nirik> I did manage to clean up some more playbooks monday... our ansible check/diff output is getting lower. 18:17:06 <nirik> smooge reinstalled osuosl02 (one less machine in puppet) 18:17:13 <pingou> cool 18:17:19 <pingou> how many left? 18:17:27 <nirik> (after I live migrated guests off it.) 18:17:38 <nirik> For the record, virsh migrate is pretty awesome. ;) 18:17:59 <smooge> I reinstalled sb10 but it crashed out and we are in a email trail of dead letters on getting it back 18:18:07 <nirik> 25 hosts left in puppet. 18:18:24 <pingou> nirik: any big one remaining in there? 18:19:05 <nirik> pingou: most of the ones left will be tricky. I sent a list to the list. 18:19:16 <pingou> oh :( 18:20:22 <nirik> I also looked into torrent software. There's no good news there really. Opentracker (what we use now) has some new commits, but nothing that seems very useful. 18:21:17 <nirik> I did have the thought to see if we could switch from offering people torrent files to offering them magnet: uris... in that you can list hosts you can download from, so we could list our download.fedoraproject.org, etc. 18:22:05 <nirik> I'll look at proposing something around that sometime. ;) 18:22:08 <nirik> lets see... 18:22:27 <nirik> #topic nagios/alerts recap 18:22:27 <nirik> http://ur1.ca/jr7j4 18:22:28 <pingou> googling how to make magnet uri isn't really helpful :/ 18:22:49 <nirik> pingou: http://en.wikipedia.org/wiki/Magnet_uri 18:23:06 <pingou> better than google :D 18:23:17 <nirik> so basically that would allow us to not seed. We would just track and get some people to download the images from https first and seed. 18:23:57 <nirik> oh, another thought I had was to make another torrent server so we could have both listed in the magnet link and can reboot one anytime we need to without outage... 18:24:55 <nirik> anyhow, nagios has been somewhat less noisy... 18:25:14 <nirik> I see proxy07 is high on the list, but I don't think any of those caused actual notifications 18:26:03 <nirik> #topic Upcoming Tasks/Items 18:26:03 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:26:12 <nirik> anything upcoming anyone would like to schedule or note? 18:26:56 <nirik> ok, I have 2 items now... 18:27:00 <nirik> #topic Meeting process 18:27:28 <nirik> so, we have had this same meeting process for a long time, I read something the other day (which I cannot for the life of me find now) that had a interesting process... 18:27:37 <nirik> basically the way it worked was: 18:27:51 <nirik> * Open a shared document (for us this would be on gobby or something) 18:28:25 <nirik> * Over the next week people add things to it. There's a 'status/info' section, there's a 'discuss' section and a 'todo/actions' 18:28:44 <nirik> * next meeting people read that and we don't talk about status/info stuff... unless it needs discussion. 18:29:07 <nirik> then we discuss the things on the list to discuss, and then make todos/actions which get added to the next shared doc 18:29:35 <nirik> So, just a thought... I might send this to the list first and get more feedback before we change anything. 18:29:54 <nirik> but I wanted to toss it out here too. ;) 18:30:09 <threebean> interesting 18:30:26 <threebean> yeah, the meetings take on a kind of pre-fab feel since the agenda is the same each week. 18:30:46 <intrk> We do something similar at my current job with a share OneNote document. It works out fairly well. 18:30:48 <nirik> yeah, and we all forget what we worked on really. 18:30:53 <threebean> yeah :P 18:31:03 <nirik> if we add stuff as we do it/think of it... it makes it easier. 18:31:10 <lmacken> I like it. The zodbot output is nice, but having a rw doc could be more collaborative if people have a week to poke at it 18:31:13 <tflink> interestingly enough, we've been talking about something similar for qadevel meetings 18:31:31 <nirik> tflink: cool. did you read something similar to this process recently? 18:31:42 <nirik> it's driving me crazy that I cannot find the blog post or whatever I read. ;( 18:31:52 <tflink> nirik: no, just trying to figure out how to make the meetings go faster 18:32:01 * tflink wasn't the person who suggested the shared document 18:32:27 <intrk> https://sprint.ly/blog/kanban/ ? 18:32:29 * nirik wonders if it could help blocker meetings too. 18:32:47 <tflink> we decided to try suggesting prepared #infos first because it's less complex 18:32:48 <nirik> nope, that wasn't it. :( oh well. 18:33:02 <tflink> nirik: we've wanted to support async blocker debate for a long time 18:33:04 * oddshocks pops in 18:33:12 <nirik> so, should we try this for next week? or discuss on list first and try week after? 18:33:58 * tflink is fine with either but suspects that there may not be much feedback from a list discussion 18:33:59 <nirik> I think having status/info stuff as #info in the doc would be nice... could cut and paste it into the meeting 18:34:25 <nirik> tflink: true. Most people on irc meetings are... in the irc meeting. ;) 18:34:50 <nirik> ok, so let me do this: after the meeting I will make a doc, and send info about it to the list... 18:34:57 <nirik> then we can see what we can do with it next week. ;) 18:35:00 <threebean> cool :) 18:35:11 <nirik> #info will try adding a shared doc and see if it makes our meetings more helpfull. 18:35:21 <nirik> #action nirik to make doc and mail list. 18:35:33 <nirik> So, the other thing I wanted to go over was: 18:35:45 <nirik> #topic Learn about an app: squid / kojipkgs 18:35:50 <threebean> oooo 18:36:01 <nirik> So, we have currently having some issues with squid. So, I thought I would describe how it's setup 18:36:13 <nirik> and how we might do it better or replace it or find out how to fix it. 18:36:42 <nirik> so, we have a machine that is 'kojipkgs01' internally, and kojipkgs.fedoraproject.org externally. 18:36:51 <nirik> It runs apache/httpd on port 8080 locally only. 18:36:59 <nirik> It has a nfs mount to the koji packages store. 18:37:13 <nirik> it runs squid listening on port 80 and 443. 18:37:36 <nirik> squid has it setup that the httpd on port 8080 is a 'parent' cache, so it queries it for the actual files 18:37:56 <nirik> then squid caches in memory or disk for requests. 18:38:07 <nirik> This worked without problem for a long time. 18:38:31 <nirik> Then recently, I nuked the old rhel6 version and replaced it with a shiny rhel7 one. 18:38:52 <nirik> I gave the new one tons and tons of memory with the idea it could memory cache most things. 18:39:08 <nirik> But we are running into an issue where after a few days/week it just gets really really slow. 18:39:18 <nirik> restarting it 'fixes' it 18:39:27 <pingou> takes to long to search in its cache? 18:39:33 <nirik> I adjusted the memory cache down today to see if that would help. 18:39:51 <nirik> it's not clear. There's 0 errors in squid logs or messages or dmesg on the host. 18:40:16 <nirik> squid is also running in it's default mode of 1 process. 18:40:29 <nirik> squid has a smp mode, but I cannot seem to get it to function at all. 18:41:14 <nirik> Some ideas: make a kojipgks02 setup the same way and make kojipkgs a round robin to both... that way at least when we restart builds don't die. 18:41:25 <nirik> Replace with varnish? but not sure how that would work. 18:41:33 <dgilmore> nirik: maybe we could use active/active 18:41:35 <nirik> Replace with memcached? but again not sure how that would work 18:41:49 <dgilmore> replace with mod_cache 18:42:08 <nirik> I think I looked at that and couldn't get it working, but I could try again. 18:42:15 <dgilmore> replace with something else 18:42:45 <smooge> 3) Run the RHEL-6 version on the RHEL-7 system and see if it is a kernel/architecture problem? 18:43:14 <dgilmore> run a Fedora 22 version 18:43:23 <dgilmore> or fedora 21 18:43:27 <nirik> I can't get smp mode working here on f21. ;) 18:43:31 <smooge> Or see if we built a RHEL-6 version with equivalent memory if it still happened 18:43:39 <dgilmore> maybe its a bug in rhel 7's squid that is fixed in fedora 18:43:52 <nirik> I still have the old rhel6 disk, but that is really not something I want to go back to. 18:44:06 <nirik> I can see if there's any differences. 18:44:35 <nirik> there's a big version difference. 18:45:16 <nirik> squid-3.3.8-12.el7_0.x86_64 vs squid-3.4.9-3.fc22.x86_64 18:45:48 <dgilmore> nirik: maybe a quick test is rebuild fedora's squid for rhel7 18:45:48 <nirik> ok, some things to try. ;) 18:46:06 <nirik> sure, although I hate to have to maintain it... for security updates forever 18:46:34 <dgilmore> indeed 18:46:47 <dgilmore> or we run Fedora server 18:47:16 <nirik> perhaps. 18:47:22 <nirik> #topic Open Floor 18:47:30 <nirik> anyone have any items for open floor? 18:47:36 <nirik> questions, comments, ideas? 18:47:51 <pingou> oh 18:48:06 <pingou> I was wondering if there would some more people interested in testing progit 18:48:27 <pingou> afaics it starts behaving nicely 18:48:50 <pingou> UI could be improved, but functionality is there 18:49:41 <nirik> cool. Perhaps a call to list? 18:49:44 <smooge> how would people test it? 18:49:51 <pingou> at one point I'll start looking at packaging it and writing some doc on how to deploy it 18:50:04 <pingou> smooge: for the moment the easiest is to use the dev instance: http://209.132.184.222 18:50:17 <pingou> make a project, fork one, check the pull-requests, issues 18:50:24 <pingou> see what needs improvement 18:50:49 <pingou> and flood me with requests and bug report :] 18:51:19 <smooge> okie dokie 18:52:18 <nirik> pingou: or perhaps devel list at this point? 18:52:24 <pingou> nirik: sure thing 18:52:29 <nirik> cool. 18:52:33 <pingou> thanks :) 18:52:40 <nirik> #info everyone go test progit. :) http://209.132.184.222 18:52:49 <nirik> ok, if nothing else will close out in a minute. 18:52:52 <pingou> I should blog about it as well :) 18:54:03 <nirik> yes. :) 18:54:09 <nirik> ok, thanks for coming everyone! 18:54:12 <nirik> #endmeeting