============================================ #fedora-meeting: Infrastructure (2011-08-11) ============================================

Meeting started by nirik at 19:00:00 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2011-08-11/infrastructure.20...

Meeting summary --------------- * Robot Roll Call (nirik, 19:00:01)

* New folks introductions and apprentice tasks/feedback (nirik, 19:01:06)

* F16 Alpha Freeze reminder and tickets (nirik, 19:05:47)

* Upcoming Tasks/Items (nirik, 19:07:33) * LINK: https://fedorahosted.org/fedora-infrastructure/report/14 (nirik, 19:10:31)

* Open Floor (nirik, 19:30:26) * ACTION: skvidal to curse at apache error logs (skvidal, 20:07:35)

Meeting ended at 20:11:58 UTC.

Action Items ------------ * skvidal to curse at apache error logs

Action Items, by person ----------------------- * skvidal * skvidal to curse at apache error logs * **UNASSIGNED** * (none)

People Present (lines said) --------------------------- * nirik (126) * skvidal (125) * smooge (39) * abadger1999 (24) * dgilmore (11) * LoKoMurdoK (7) * athmane (6) * pingou (6) * zodbot (5) * CodeBlock (4) * lmacken (3) * ricky (0) * codeblock (0) -- 19:00:00 <nirik> #startmeeting Infrastructure (2011-08-11) 19:00:00 <zodbot> Meeting started Thu Aug 11 19:00:00 2011 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:01 <nirik> #meetingname infrastructure 19:00:01 <zodbot> The meeting name has been set to 'infrastructure' 19:00:01 <nirik> #topic Robot Roll Call 19:00:01 <nirik> #chair smooge skvidal codeblock ricky nirik abadger1999 19:00:01 <zodbot> Current chairs: abadger1999 codeblock nirik ricky skvidal smooge 19:00:16 <CodeBlock> here 19:00:22 * dgilmore is here 19:00:23 <smooge> Crowbot here 19:00:32 * athmane is here 19:00:42 <abadger1999> hola 19:00:52 <nirik> welcome everyone. 19:01:06 <nirik> #topic New folks introductions and apprentice tasks/feedback 19:01:28 <nirik> Any new folks like to introduce themselves? or any apprentices like to talk about tickets or other issues? 19:01:58 * pingou around 19:02:13 <nirik> I did clean out the apprentice group the other day. A few more folks let me know they should be around more at some point... at which time we can re-add them. 19:02:43 <nirik> Does everyone think the apprentice program is helpfull? 19:02:49 * skvidal is here 19:03:00 <nirik> The feedback I have gotten is that it is, but I'm interested in other thoughts too. 19:03:03 <nirik> morning skvidal 19:03:16 <athmane> nirik: I was an apprentice :) 19:03:19 <skvidal> feed back on what? (sorry for being late) 19:03:24 <nirik> Does everyone think the apprentice program is helpfull? 19:03:34 <nirik> athmane: did you find the apprentice group useful? 19:04:10 <athmane> nirik: sure, having access to puppet is very useful 19:04:50 <nirik> yeah. I think it might help some, but we still have the issue of getting people over the 'where do I contribute' thing. ;) Oh well, we can keep trying and adjust as we come up with more ideas. 19:05:18 * nirik will move on in a sec then. 19:05:47 <nirik> #topic F16 Alpha Freeze reminder and tickets 19:06:01 <nirik> reminder that we are still pre-release frozen for f16 alpha. 19:06:04 <LoKoMurdoK> I'm here 19:06:12 <nirik> and since it's slipped a week, we are frozen for an extra week. 19:06:16 <nirik> morning LoKoMurdoK. 19:06:27 <LoKoMurdoK> hi nirik 19:07:06 <nirik> looks like we have the tickets all assigned (thanks smooge!) 19:07:33 <nirik> #topic Upcoming Tasks/Items 19:07:42 <smooge> I will be working on them this week so we are ready for Alpha 19:07:45 <nirik> Any upcoming tasks or items folks would like to talk about? 19:07:54 <nirik> thanks smooge 19:08:21 <nirik> I'll shout out a few tasks that anyone can work on (especially during a freeze): 19:08:27 <CodeBlock> smooge: let me know if I can help at all, finally starting to have a bit more time again, seems like a good place to get back into the game 19:08:43 <nirik> .infra 2906 19:09:22 <nirik> .ticket 2906 19:09:26 <zodbot> nirik: #2906 (Migrate SOP documents to infra-docs git repo) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2906 19:10:26 <nirik> and any of the easyfix ones: 19:10:31 <nirik> https://fedorahosted.org/fedora-infrastructure/report/14 19:10:36 <LoKoMurdoK> is my ticket 19:10:59 <nirik> LoKoMurdoK: yeah, you mentioned you wanted to work on that. ;) If we can assist or help you get started, please let us know. 19:11:32 <nirik> we can discuss in #fedora-admin after meeting too. 19:11:54 <nirik> I'll also note that our new machines have been racked in phx2... so we will be installing new machines hopefully before too long. 19:11:58 <LoKoMurdoK> the 2 weeks had problems in the office, very busy 19:12:10 <nirik> Then we will look at migrating some old instances to new instances on those new machines. 19:12:15 <nirik> LoKoMurdoK: no problems. 19:12:17 <LoKoMurdoK> but I'm free 19:12:49 <LoKoMurdoK> I can now migrate the infra-docs 19:13:03 <LoKoMurdoK> ok 19:13:05 <nirik> abadger1999: is everything looking ok for rhel6 app servers? (this would be a post freeze item) 19:13:09 <nirik> LoKoMurdoK: great. 19:13:29 <abadger1999> nirik: I think that fedoracommunity is still a problem 19:13:40 <abadger1999> nirik: but everything else looks okay. 19:13:54 <abadger1999> lmacken: Is that accurate? Or do you have fixes for that? 19:14:01 <abadger1999> (fedoracommunity on rhel6) 19:14:12 * nirik recalls someone working on it the other day, not sure if for rhel6 specifically tho . 19:14:38 <abadger1999> yeah, comphappy was working on fedoracommunity... I think it was general bugfixing, though. 19:14:44 <nirik> ok. 19:14:54 <abadger1999> at least, the questions he asked me were general bugfixing :-) 19:15:22 <abadger1999> nirik: I'd love for us to migrate to rhel6... means half the work for apps when testing. 19:15:30 <abadger1999> (vs mixed rhel5 and rhel6) 19:15:37 <nirik> I posted proposed post freeze changes to log02 as well to the list. If everyone could review and tell me if they have any problems/reservations with those changes that would be good. 19:15:45 <nirik> abadger1999: yeah, agreed. 19:16:33 * nirik looks over his list for any other upcoming things to discuss. 19:17:14 <nirik> Oh, we have 3 items (that I know of) in Request for resource process: ask, paste and nitrate. All are moving forward as they can... 19:17:37 * skvidal has something for openfloor when we have time 19:17:43 <skvidal> s/when/if/ 19:18:05 <nirik> ok, should get there soonish. ;) 19:18:21 <lmacken> abadger1999: I'm working on getting a new moksha & fcomm release prepped (including comphappy's patches), then I'll be able to test it on RHEL6. 19:18:36 <nirik> lmacken: cool. ;) 19:18:55 <abadger1999> lmacken: k any time estimate? Think we'll be able to deploy rhel6 app servers after alpha freeze? 19:19:17 * nirik notes the current freeze ends the 23rd unless we slip again. 19:19:35 <nirik> then we have 3 weeks until beta freeze. 19:20:32 <lmacken> abadger1999: I'll try and shoot for after the alpha freeze 19:20:57 <nirik> I can try and have a rhel6 app server instance prepped... 19:21:03 <abadger1999> lmacken: Excellent :-) 19:21:35 <nirik> well, we have app02.stg for testing, but I mean for adding into prod mix. ;) 19:21:50 <abadger1999> <nod> 19:22:02 <abadger1999> For converting all app servers... are we waiting on more hardware? 19:22:29 <nirik> waiting on re-installing hosts as rhel6 and freeze to be over. 19:22:45 <nirik> we have new hardware racked now, we will be setting that up in the next few weeks. 19:23:08 <abadger1999> Cool. So we control our destiny ;-) 19:23:24 <nirik> yeah, sadly we don't control 48 hour days. :) 19:23:36 <skvidal> yum install tardis 19:23:39 <skvidal> there we go 19:23:52 <pingou> "just yum it"© 19:24:01 <nirik> I also posted a plan for hosted... more feedback on that would be great. It sounds like no one minded my short term plans at least. 19:24:43 <smooge> are we still looking at a complete stg purge and build? 19:24:54 <smooge> I will look at the hosted after the meeting. 19:25:14 <nirik> yeah, staging is still needing dealing with. 19:25:29 <smooge> by the way which fedoracommunity were we talking about above? 19:25:35 <nirik> I'm not sure the best way forward on it... perhaps I will try and send out a plan on that too for people to poke holes in. 19:25:59 <nirik> smooge: this one: https://admin.fedoraproject.org/community/ 19:26:07 <smooge> ok just wanted to confirm 19:26:08 <smooge> thanks 19:26:49 <nirik> ok, anything else upcoming? or shall we move to open floor? 19:27:13 <smooge> retrace build 19:27:21 <nirik> oh yeah, thats another new machine... 19:27:24 <smooge> that was my only focus after I dug out of email hell 19:27:44 <nirik> that would be a nice one to get going. I assume we just install it, and then hand it off to retrace folks? 19:28:23 <smooge> yeah.. I am just going to have it set up minimally so it can be updated/rebuilt quickly and then have them tell us what sections to back up. 19:28:32 <nirik> once they are ready to cut over we change dns and recommision our retrace01... 19:28:43 <smooge> after that its getting it a public ip address/port and poof they can actually run Fedora 15/16 dumps 19:29:01 <nirik> smooge: you might ask them if they want config management on it. I still need to setup a qa bcfg2 instance, but it could go in that if they wanted. 19:30:20 <nirik> ok, on to open floor... 19:30:26 <nirik> #topic Open Floor 19:30:33 <skvidal> I've been working on the epylog reports some more 19:30:49 <skvidal> for those people receiving them - I'd like to know what you find useful and not useful 19:31:00 <skvidal> and what things you'd like to see change 19:31:40 * skvidal hears the echoes 19:31:53 <nirik> skvidal: so, one question... who should we allow to be in sysadmin-logs? apprentice folks? or sysadmin group people only? or just based on our judgement... 19:31:59 <skvidal> <shrug> 19:32:09 <skvidal> I don't think it is any different than who can login to log02, is it? 19:32:15 <nirik> right. 19:32:19 <skvidal> so 19:32:45 <nirik> so, if I change still so apprentice can look at logs, they should probibly be allowed in sysadmin-logs. Or for that matter we could just change log02 to allow sysadmin-logs 19:33:03 <skvidal> indeed 19:33:13 <athmane> so logs are only readable by root, so you need sudo there 19:33:14 <nirik> anyhow, side topic. 19:33:21 <athmane> s/so/some/ 19:33:26 <skvidal> athmane: no 19:33:29 <nirik> athmane: nope. 19:33:34 <skvidal> athmane: our central log server allows log reading 19:33:37 <skvidal> on its merged logs 19:33:50 <nirik> and the non merged ones are currently open to 'sysadmin' 19:33:52 <smooge> skvidal, I find them useful. I need a page so I can see old logs 19:34:02 <nirik> but only main and noc can login there. 19:34:07 <skvidal> smooge: so you want an index of some kind? 19:34:37 <smooge> yeah. because I see a report.. then I have to go searching through email for the link to the previous one etc 19:34:41 * nirik finds the logs very useful. We have fixed a number of issues and noted problems or things we should investigate from them. It's very nice to be proactive sometimes. 19:34:51 <skvidal> what are folks' thoughts on auth for log access via the webserver... I'll admit to not LOVING the idea of adding the mod_auth_pg to those boxes 19:35:43 <skvidal> is there a way for us to require the auth from admin.fp.o BEFORE it redirects to log02 for the actual content? 19:35:44 <smooge> well how do we use it to get to other parts like nagios, collectd and such 19:35:58 <dgilmore> skvidal: i dont like it 19:36:02 <dgilmore> but thats just me 19:36:06 <skvidal> dgilmore: which part? 19:36:12 <skvidal> dgilmore: the log report in general? 19:36:37 <skvidal> dgilmore: or the mod_auth_pg thing? 19:36:38 <nirik> collectd is all open I think... 19:36:45 <nirik> but nagios uses the auth_pg thing. 19:36:55 <CodeBlock> indeed 19:36:57 <skvidal> nirik: and it is installed on the noc server isn't it? 19:37:04 <CodeBlock> skvidal: yep 19:37:05 <dgilmore> skvidal: mod_auth_pg 19:37:08 <skvidal> nirik: not at thr admin.fp. level 19:37:14 <athmane> apache and haproxy status are open too 19:37:15 <dgilmore> skvidal: and accessing logs via browser 19:37:18 <nirik> skvidal: right. 19:37:32 <skvidal> dgilmore: well we're not alking about accessing the logs by browser - just the log report 19:37:40 <nirik> we could do ssh tunnels, but thats a bit ugly. ;) 19:38:06 <dgilmore> skvidal: can we do it using lynx on log02? 19:38:27 <skvidal> dgilmore: of course you can - but I think lynx isn't installed but elinks is 19:38:36 <dgilmore> skvidal: that works 19:38:40 <dgilmore> lets just do that 19:38:44 <skvidal> the log report is just prettier in a real web browser 19:38:55 <skvidal> and it is much more convenient to access it from your mail that way 19:39:50 <skvidal> brb 19:40:37 <nirik> so what are the issuess with mod_auth_pg? just more deps? 19:40:59 <skvidal> nirik: more deps - and more tightly coupled to the db server 19:41:01 <skvidal> so... 19:41:07 <skvidal> if we need to look at the logs/log reports 19:41:07 <nirik> yeah, if db is down... right. 19:41:11 <skvidal> and the db server is down 19:41:12 <skvidal> right 19:41:24 <nirik> there is that OAUTH plugin... but it's not packaged... 19:41:26 <skvidal> if there was a sane way to do mod_auth_pam against nss_db 19:41:30 <skvidal> I'd be all over it 19:41:47 <skvidal> this is one of those cases where one-off service passwords would be wonderful 19:42:41 <dgilmore> skvidal: yubikey auth? 19:42:50 <nirik> huh... as an alternative... how about htaccess with a shared password. You get that when you join sysadmin-log? another password and could be grabbed around... 19:43:00 <nirik> dgilmore: still requires the db I think... 19:43:22 <dgilmore> nirik: it requires a db 19:43:39 <dgilmore> in that you need to yubikey validation server up and running 19:43:43 <nirik> yeah. 19:44:52 <nirik> anyhow, I guess lets ponder on it more and try and come up with the least anoying alternative? ;) 19:46:35 <smooge> I figure an .htaccess pushed from puppet is not ok? 19:47:35 <nirik> smooge: well, we could put the pass in private... the issues would be: password could get leaked by someone and no easy way to account who was reading them (except possibly by IP). 19:47:45 <skvidal> smooge: and we'd have to keep updatinging it 19:47:52 <skvidal> nirik: oh - nm 19:47:53 <skvidal> sorry 19:47:55 <skvidal> I just caught up 19:47:56 <skvidal> shared pw 19:48:00 <skvidal> not individual ones 19:48:12 <skvidal> hahah, I know what we could do 19:48:21 <skvidal> I could have the epylog mail send out the password ;) 19:48:23 <nirik> yeah, I was talking shared. 19:48:32 * skvidal kids - that's no better than how we are now 19:48:35 <smooge> oh I figured it would just be like the password files. puppet runs on lockbox, gets the passwords out of the db, builds an htaccess and pushes that out 19:48:52 <nirik> I suppose it could be a per user pass in there too. 19:49:10 <smooge> actually lockbox wouldn't do it.. log02 would 19:49:11 <skvidal> smooge: oh - so you mean puppet generates an htpasswd file with all sysadmin-log members in it? 19:49:12 <smooge> duh 19:49:22 <smooge> skvidal, yeah. 19:49:23 <skvidal> hmm 19:49:26 <smooge> or a python script 19:49:34 <skvidal> can aapache deal with the passwd crypt 19:49:35 <smooge> that is cron run or something 19:49:38 <skvidal> we have in the nssdb files? 19:49:54 <skvidal> smooge: I like it, in principal... 19:50:03 <nirik> skvidal: possibly. 19:50:25 <smooge> reality will probably intrude and say no 19:50:29 <skvidal> nirik: we run this risk of having these files more available so people could compromise them 19:50:31 <nirik> dump crypted pass from nssdb -> htaccess file -> bobs your uncle. 19:50:36 <nirik> ? 19:50:44 <skvidal> nirik:well the files have to be readable by apache 19:50:52 <skvidal> which makes them slightly more vulnerable than shadow nss_db 19:50:56 <nirik> true. 19:51:08 * skvidal hmms 19:51:23 <nirik> but anyone can get crypted pass from fas 19:51:26 <nirik> or wait. 19:51:29 <nirik> no they can't. 19:52:03 <skvidal> no, they can't 19:52:15 <skvidal> getent won't do what you want 19:52:23 <skvidal> it's not like nis 19:52:38 <skvidal> and you'd need the fas host password to grab it from fas 19:52:44 <nirik> I was thinking of someone remotely running fasClient, but it doesn't allow non priv accounts that info 19:53:19 <smooge> hey how about we add a field into fas that keeps that password? 19:53:31 <smooge> it should be all of what 5 minutes of abadger1999's time :) 19:53:47 <skvidal> smooge: umm...... 19:53:57 <smooge> anyway.. I think I have derailed this 19:54:02 <skvidal> smooge: if we were going to do it - I'd say we add an arbitrary service password field 19:54:16 <skvidal> and then make it so you can pull down that service password 19:54:22 <smooge> well yes. that was I meant 19:54:23 <skvidal> and that will not be a 5m project 19:54:27 * nirik wonders if there's a clever way to have apache auth with ssh keys. ;) 19:54:30 <smooge> I figured that also 19:54:36 <skvidal> and to be fair to abadger1999 I doubt he has the 5m to spare 19:54:53 <abadger1999> hehe 19:55:10 <skvidal> abadger1999: if I'm wrong about that then GO GET TO WORK! 19:55:10 <skvidal> :) 19:55:59 <skvidal> okay 19:56:04 <nirik> anyhow, I think we keep thinking... 19:56:08 <skvidal> now 19:56:12 <skvidal> about the log reports 19:56:16 <skvidal> is there any CONTENT in the log report 19:56:20 <skvidal> that folks like or don't like 19:56:23 <skvidal> or would like to see 19:56:44 <skvidal> I have been thinking about how we could make apache logs happen and it is just not frelling obvious to me 19:56:53 <skvidal> it will involve, I suspect, a lot of work 19:57:20 <nirik> yeah, on that, could we switch them to syslog? or is that too much work/load/network problem? 19:57:38 * nirik also wondered about getting audit setup to log to a central place. 19:57:41 <abadger1999> skvidal: Okay, there's currently a config table in fas that you can access via json. 19:57:42 <skvidal> nirik: given our web log volume I'm worried how it would beat up log02 19:57:49 <abadger1999> That allows you to keep arbitrary data in fas. 19:57:57 <skvidal> abadger1999: orly? 19:58:01 <abadger1999> asterisk for instance was kept there. 19:58:10 <skvidal> abadger1999: huh 19:58:24 <nirik> skvidal: yeah, would need some testing. 19:58:28 <abadger1999> Let me see if I can find that. 19:58:38 <smooge> nirik, too much load when I tried apache logs a year or so ago 19:58:48 <nirik> I've also wondered about having a log03 somewhere. backs up log02 read-only in case something happens to it. 19:58:56 <nirik> smooge: ok. 19:59:03 <abadger1999> skvidal: https://fedorahosted.org/releases/p/y/python-fedora/doc/existing.html#fedora... 19:59:08 <abadger1999> and get_config_like() 19:59:13 <smooge> skvidal, I would go with a seperate epylog report for apache logs.. 19:59:15 <abadger1999> get_configs_like() 19:59:25 <skvidal> smooge: oh of course 19:59:32 <skvidal> smooge: I wouldn't want it mixed with the other logs atall 19:59:40 <skvidal> smooge: but I mean just handling the data at all 20:00:13 * smooge is confused then. I thought epylog dealt with it per domain 20:00:20 <smooge> or did I misremember 20:00:26 <nirik> epylog works on a merged log of all hosts. 20:00:44 <nirik> we could merge the apache logs too and run on them, but I bet it's going to be frigging gigantic. 20:00:49 <skvidal> smooge: epylog doesn't deal with apache logs at all right now 20:00:53 <skvidal> smooge: it has no module to handle them 20:01:00 <smooge> ah I thought for apache it did something different. I must be thinking of splunk or something 20:01:02 <skvidal> nirik: no betting about it 20:01:04 <nirik> yeah, so something else operating on them might be good. 20:01:17 <skvidal> the other issue is this 20:01:26 <skvidal> epylog is more about reporting and pointing up issues 20:01:26 <nirik> we do have awstats, but it's just hits and such, not errors. 20:01:35 <skvidal> apache logs often are about stat reporting 20:01:39 <skvidal> less about error reporting 20:01:44 <pingou> nirik: 404 errors are in iirc 20:01:45 <skvidal> and I _think_ we want to know more about errors 20:01:45 * nirik nods. 20:01:52 <nirik> however, there are errors. 20:02:04 <nirik> which currently we ignore. ;) 20:02:09 <smooge> apache errorlogs are an abomination 20:02:10 <skvidal> that's where I would like to be 20:02:13 <skvidal> to be fair 20:02:15 <skvidal> I would love it 20:02:17 <nirik> pingou: yeah, but not tracebacks and such. 20:02:23 <skvidal> if we could get just apache error logsand app traces 20:02:25 <skvidal> via syslog 20:02:32 <skvidal> which should, in theory, be much less 20:02:39 <smooge> oooooh 20:02:42 <smooge> hmmm 20:03:10 <skvidal> now we could modify our apache configs 20:03:21 <nirik> sadly, I think it's the bulk of apache logs. 20:03:23 <skvidal> to have the error logs be spat out via syslog to a local# facility 20:03:31 <skvidal> nirik: you think most of our apache logs are errors? 20:03:37 <skvidal> nirik: that feels..... bad 20:03:45 <nirik> -rw-r--r-- 1 root root 2320563738 Aug 11 20:03 error_log 20:03:57 <skvidal> how OLD is that/ 20:04:02 <skvidal> head -1 error_log 20:04:24 <nirik> [Thu Aug 11 04:02:22 2011] [error] /usr/sbin/pkgdb.wsgi:19: DeprecationWarning: fedora.tg.util is deprecated. Switch to one of these instead: TG1 apps: fedora.tg.tg1utils TG2 apps: fedora.tg.tg2utils. This file will disappear in 0.4 20:04:41 <nirik> 1 day 20:04:42 <skvidal> wait 20:04:44 <skvidal> noway 20:04:49 <skvidal> what the hell? 20:04:59 <nirik> way. ;) 20:05:00 <skvidal> it's 2.2GB in ONE DAY?! 20:05:03 <nirik> app01 20:05:17 <nirik> I think all our monitoring is showing up in there. 20:05:28 <nirik> [Thu Aug 11 20:05:17 2011] [error] - - "GET /pkgdb/collections/ HTTP/1.0" 200 22395 "" "" 20:05:43 <skvidal> wow 20:05:47 <nirik> or perhaps pkgdb is just doing everything to error. ;) 20:05:49 <skvidal> like 5 logs a second 20:05:53 <skvidal> christ on a crutch 20:05:54 <smooge> wow that has gotten bad 20:06:05 <smooge> I don't think it was that bad last fall. 20:06:08 <nirik> right. So, we need to clean this up some before we can even look at syslog. 20:06:15 <skvidal> nirik: no kidding 20:06:20 <skvidal> [Thu Aug 11 20:05:51 2011] [error] - - "GET /voting/ HTTP/1.0" 200 14378 "" "" 20:06:23 <skvidal> that's our monitoring 20:06:30 <skvidal> also 20:06:38 <skvidal> why is a 200 from that an 'error'? 20:06:45 <nirik> no idea. 20:07:01 <smooge> well it could be buried in a line below that 20:07:15 <skvidal> okay 20:07:17 <skvidal> you know what 20:07:24 <skvidal> I'll take a good long look at that in the coming week 20:07:30 <nirik> skvidal: thanks. 20:07:35 <skvidal> #action skvidal to curse at apache error logs 20:07:41 <nirik> any other items? or shall we close out the meeting? 20:07:55 <smooge> not me I think I broke things enough 20:08:01 <skvidal> wow 20:08:02 <skvidal> I'm wrong 20:08:07 <skvidal> 101 error log lines per second 20:08:23 <skvidal> on app01 20:08:24 <abadger1999> That's something we need lmacken to look into -- it's the curse of the TG1 logging system that we've never figured out completely. 20:08:24 <skvidal> ONLY 20:08:29 <nirik> yeah, app01 alerted the other night because / was more than 85% full. ;( 20:08:51 <skvidal> I'm going to see if I can do the user outlier log report today and tomorrow 20:08:55 <skvidal> and then get to the apache logs 20:08:57 <skvidal> and be angry 20:09:21 <pingou> sed -i -e "s|200|d" error_log ? :d 20:09:28 <abadger1999> We either seem to not get tracebacks in the logs or way too much information or both. 20:10:20 <nirik> ok, thanks for coming everyone! 20:10:24 <lmacken> abadger1999: yeah, our TG1 logging setup is inconsistent, and awful. 20:10:37 <pingou> abadger1999: todo: check the tg2 version 20:10:52 * nirik waits a min more since we have more info. ;) 20:11:20 <nirik> pingou: is voting changed to tg2? or not in prod yet? 20:11:34 <pingou> nirik: afaik not yet 20:11:42 <nirik> ok. 20:11:54 <nirik> ok, will close out then.... 20:11:58 <nirik> #endmeeting