3:12 p.m.
============================================
#fedora-meeting: Infrastructure (2011-08-11)
============================================
Meeting started by nirik at 19:00:00 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-meeting/2011-08-11/infrastructure...
Meeting summary
---------------
* Robot Roll Call (nirik, 19:00:01)
* New folks introductions and apprentice tasks/feedback (nirik,
19:01:06)
* F16 Alpha Freeze reminder and tickets (nirik, 19:05:47)
* Upcoming Tasks/Items (nirik, 19:07:33)
* LINK: https://fedorahosted.org/fedora-infrastructure/report/14
(nirik, 19:10:31)
* Open Floor (nirik, 19:30:26)
* ACTION: skvidal to curse at apache error logs (skvidal, 20:07:35)
Meeting ended at 20:11:58 UTC.
Action Items
------------
* skvidal to curse at apache error logs
Action Items, by person
-----------------------
* skvidal
* skvidal to curse at apache error logs
* **UNASSIGNED**
* (none)
People Present (lines said)
---------------------------
* nirik (126)
* skvidal (125)
* smooge (39)
* abadger1999 (24)
* dgilmore (11)
* LoKoMurdoK (7)
* athmane (6)
* pingou (6)
* zodbot (5)
* CodeBlock (4)
* lmacken (3)
* ricky (0)
* codeblock (0)
--
19:00:00 <nirik> #startmeeting Infrastructure (2011-08-11)
19:00:00 <zodbot> Meeting started Thu Aug 11 19:00:00 2011 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <nirik> #topic Robot Roll Call
19:00:01 <nirik> #chair smooge skvidal codeblock ricky nirik abadger1999
19:00:01 <zodbot> Current chairs: abadger1999 codeblock nirik ricky skvidal smooge
19:00:16 <CodeBlock> here
19:00:22 * dgilmore is here
19:00:23 <smooge> Crowbot here
19:00:32 * athmane is here
19:00:42 <abadger1999> hola
19:00:52 <nirik> welcome everyone.
19:01:06 <nirik> #topic New folks introductions and apprentice tasks/feedback
19:01:28 <nirik> Any new folks like to introduce themselves? or any apprentices like
to talk about tickets or other issues?
19:01:58 * pingou around
19:02:13 <nirik> I did clean out the apprentice group the other day. A few more
folks let me know they should be around more at some point... at which time we can re-add
them.
19:02:43 <nirik> Does everyone think the apprentice program is helpfull?
19:02:49 * skvidal is here
19:03:00 <nirik> The feedback I have gotten is that it is, but I'm interested in
other thoughts too.
19:03:03 <nirik> morning skvidal
19:03:16 <athmane> nirik: I was an apprentice :)
19:03:19 <skvidal> feed back on what? (sorry for being late)
19:03:24 <nirik> Does everyone think the apprentice program is helpfull?
19:03:34 <nirik> athmane: did you find the apprentice group useful?
19:04:10 <athmane> nirik: sure, having access to puppet is very useful
19:04:50 <nirik> yeah. I think it might help some, but we still have the issue of
getting people over the 'where do I contribute' thing. ;) Oh well, we can keep
trying and adjust as we come up with more ideas.
19:05:18 * nirik will move on in a sec then.
19:05:47 <nirik> #topic F16 Alpha Freeze reminder and tickets
19:06:01 <nirik> reminder that we are still pre-release frozen for f16 alpha.
19:06:04 <LoKoMurdoK> I'm here
19:06:12 <nirik> and since it's slipped a week, we are frozen for an extra
week.
19:06:16 <nirik> morning LoKoMurdoK.
19:06:27 <LoKoMurdoK> hi nirik
19:07:06 <nirik> looks like we have the tickets all assigned (thanks smooge!)
19:07:33 <nirik> #topic Upcoming Tasks/Items
19:07:42 <smooge> I will be working on them this week so we are ready for Alpha
19:07:45 <nirik> Any upcoming tasks or items folks would like to talk about?
19:07:54 <nirik> thanks smooge
19:08:21 <nirik> I'll shout out a few tasks that anyone can work on (especially
during a freeze):
19:08:27 <CodeBlock> smooge: let me know if I can help at all, finally starting to
have a bit more time again, seems like a good place to get back into the game
19:08:43 <nirik> .infra 2906
19:09:22 <nirik> .ticket 2906
19:09:26 <zodbot> nirik: #2906 (Migrate SOP documents to infra-docs git repo) -
Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2906
19:10:26 <nirik> and any of the easyfix ones:
19:10:31 <nirik> https://fedorahosted.org/fedora-infrastructure/report/14
19:10:36 <LoKoMurdoK> is my ticket
19:10:59 <nirik> LoKoMurdoK: yeah, you mentioned you wanted to work on that. ;) If
we can assist or help you get started, please let us know.
19:11:32 <nirik> we can discuss in #fedora-admin after meeting too.
19:11:54 <nirik> I'll also note that our new machines have been racked in
phx2... so we will be installing new machines hopefully before too long.
19:11:58 <LoKoMurdoK> the 2 weeks had problems in the office, very busy
19:12:10 <nirik> Then we will look at migrating some old instances to new instances
on those new machines.
19:12:15 <nirik> LoKoMurdoK: no problems.
19:12:17 <LoKoMurdoK> but I'm free
19:12:49 <LoKoMurdoK> I can now migrate the infra-docs
19:13:03 <LoKoMurdoK> ok
19:13:05 <nirik> abadger1999: is everything looking ok for rhel6 app servers? (this
would be a post freeze item)
19:13:09 <nirik> LoKoMurdoK: great.
19:13:29 <abadger1999> nirik: I think that fedoracommunity is still a problem
19:13:40 <abadger1999> nirik: but everything else looks okay.
19:13:54 <abadger1999> lmacken: Is that accurate? Or do you have fixes for that?
19:14:01 <abadger1999> (fedoracommunity on rhel6)
19:14:12 * nirik recalls someone working on it the other day, not sure if for rhel6
specifically tho .
19:14:38 <abadger1999> yeah, comphappy was working on fedoracommunity... I think it
was general bugfixing, though.
19:14:44 <nirik> ok.
19:14:54 <abadger1999> at least, the questions he asked me were general bugfixing
:-)
19:15:22 <abadger1999> nirik: I'd love for us to migrate to rhel6... means half
the work for apps when testing.
19:15:30 <abadger1999> (vs mixed rhel5 and rhel6)
19:15:37 <nirik> I posted proposed post freeze changes to log02 as well to the list.
If everyone could review and tell me if they have any problems/reservations with those
changes that would be good.
19:15:45 <nirik> abadger1999: yeah, agreed.
19:16:33 * nirik looks over his list for any other upcoming things to discuss.
19:17:14 <nirik> Oh, we have 3 items (that I know of) in Request for resource
process: ask, paste and nitrate. All are moving forward as they can...
19:17:37 * skvidal has something for openfloor when we have time
19:17:43 <skvidal> s/when/if/
19:18:05 <nirik> ok, should get there soonish. ;)
19:18:21 <lmacken> abadger1999: I'm working on getting a new moksha & fcomm
release prepped (including comphappy's patches), then I'll be able to test it on
RHEL6.
19:18:36 <nirik> lmacken: cool. ;)
19:18:55 <abadger1999> lmacken: k any time estimate? Think we'll be able to
deploy rhel6 app servers after alpha freeze?
19:19:17 * nirik notes the current freeze ends the 23rd unless we slip again.
19:19:35 <nirik> then we have 3 weeks until beta freeze.
19:20:32 <lmacken> abadger1999: I'll try and shoot for after the alpha freeze
19:20:57 <nirik> I can try and have a rhel6 app server instance prepped...
19:21:03 <abadger1999> lmacken: Excellent :-)
19:21:35 <nirik> well, we have app02.stg for testing, but I mean for adding into
prod mix. ;)
19:21:50 <abadger1999> <nod>
19:22:02 <abadger1999> For converting all app servers... are we waiting on more
hardware?
19:22:29 <nirik> waiting on re-installing hosts as rhel6 and freeze to be over.
19:22:45 <nirik> we have new hardware racked now, we will be setting that up in the
next few weeks.
19:23:08 <abadger1999> Cool. So we control our destiny ;-)
19:23:24 <nirik> yeah, sadly we don't control 48 hour days. :)
19:23:36 <skvidal> yum install tardis
19:23:39 <skvidal> there we go
19:23:52 <pingou> "just yum it"©
19:24:01 <nirik> I also posted a plan for hosted... more feedback on that would be
great. It sounds like no one minded my short term plans at least.
19:24:43 <smooge> are we still looking at a complete stg purge and build?
19:24:54 <smooge> I will look at the hosted after the meeting.
19:25:14 <nirik> yeah, staging is still needing dealing with.
19:25:29 <smooge> by the way which fedoracommunity were we talking about above?
19:25:35 <nirik> I'm not sure the best way forward on it... perhaps I will try
and send out a plan on that too for people to poke holes in.
19:25:59 <nirik> smooge: this one: https://admin.fedoraproject.org/community/
19:26:07 <smooge> ok just wanted to confirm
19:26:08 <smooge> thanks
19:26:49 <nirik> ok, anything else upcoming? or shall we move to open floor?
19:27:13 <smooge> retrace build
19:27:21 <nirik> oh yeah, thats another new machine...
19:27:24 <smooge> that was my only focus after I dug out of email hell
19:27:44 <nirik> that would be a nice one to get going. I assume we just install it,
and then hand it off to retrace folks?
19:28:23 <smooge> yeah.. I am just going to have it set up minimally so it can be
updated/rebuilt quickly and then have them tell us what sections to back up.
19:28:32 <nirik> once they are ready to cut over we change dns and recommision our
retrace01...
19:28:43 <smooge> after that its getting it a public ip address/port and poof they
can actually run Fedora 15/16 dumps
19:29:01 <nirik> smooge: you might ask them if they want config management on it. I
still need to setup a qa bcfg2 instance, but it could go in that if they wanted.
19:30:20 <nirik> ok, on to open floor...
19:30:26 <nirik> #topic Open Floor
19:30:33 <skvidal> I've been working on the epylog reports some more
19:30:49 <skvidal> for those people receiving them - I'd like to know what you
find useful and not useful
19:31:00 <skvidal> and what things you'd like to see change
19:31:40 * skvidal hears the echoes
19:31:53 <nirik> skvidal: so, one question... who should we allow to be in
sysadmin-logs? apprentice folks? or sysadmin group people only? or just based on our
judgement...
19:31:59 <skvidal> <shrug>
19:32:09 <skvidal> I don't think it is any different than who can login to
log02, is it?
19:32:15 <nirik> right.
19:32:19 <skvidal> so
19:32:45 <nirik> so, if I change still so apprentice can look at logs, they should
probibly be allowed in sysadmin-logs. Or for that matter we could just change log02 to
allow sysadmin-logs
19:33:03 <skvidal> indeed
19:33:13 <athmane> so logs are only readable by root, so you need sudo there
19:33:14 <nirik> anyhow, side topic.
19:33:21 <athmane> s/so/some/
19:33:26 <skvidal> athmane: no
19:33:29 <nirik> athmane: nope.
19:33:34 <skvidal> athmane: our central log server allows log reading
19:33:37 <skvidal> on its merged logs
19:33:50 <nirik> and the non merged ones are currently open to 'sysadmin'
19:33:52 <smooge> skvidal, I find them useful. I need a page so I can see old logs
19:34:02 <nirik> but only main and noc can login there.
19:34:07 <skvidal> smooge: so you want an index of some kind?
19:34:37 <smooge> yeah. because I see a report.. then I have to go searching through
email for the link to the previous one etc
19:34:41 * nirik finds the logs very useful. We have fixed a number of issues and noted
problems or things we should investigate from them. It's very nice to be proactive
sometimes.
19:34:51 <skvidal> what are folks' thoughts on auth for log access via the
webserver... I'll admit to not LOVING the idea of adding the mod_auth_pg to those
boxes
19:35:43 <skvidal> is there a way for us to require the auth from admin.fp.o BEFORE
it redirects to log02 for the actual content?
19:35:44 <smooge> well how do we use it to get to other parts like nagios, collectd
and such
19:35:58 <dgilmore> skvidal: i dont like it
19:36:02 <dgilmore> but thats just me
19:36:06 <skvidal> dgilmore: which part?
19:36:12 <skvidal> dgilmore: the log report in general?
19:36:37 <skvidal> dgilmore: or the mod_auth_pg thing?
19:36:38 <nirik> collectd is all open I think...
19:36:45 <nirik> but nagios uses the auth_pg thing.
19:36:55 <CodeBlock> indeed
19:36:57 <skvidal> nirik: and it is installed on the noc server isn't it?
19:37:04 <CodeBlock> skvidal: yep
19:37:05 <dgilmore> skvidal: mod_auth_pg
19:37:08 <skvidal> nirik: not at thr admin.fp. level
19:37:14 <athmane> apache and haproxy status are open too
19:37:15 <dgilmore> skvidal: and accessing logs via browser
19:37:18 <nirik> skvidal: right.
19:37:32 <skvidal> dgilmore: well we're not alking about accessing the logs by
browser - just the log report
19:37:40 <nirik> we could do ssh tunnels, but thats a bit ugly. ;)
19:38:06 <dgilmore> skvidal: can we do it using lynx on log02?
19:38:27 <skvidal> dgilmore: of course you can - but I think lynx isn't
installed but elinks is
19:38:36 <dgilmore> skvidal: that works
19:38:40 <dgilmore> lets just do that
19:38:44 <skvidal> the log report is just prettier in a real web browser
19:38:55 <skvidal> and it is much more convenient to access it from your mail that
way
19:39:50 <skvidal> brb
19:40:37 <nirik> so what are the issuess with mod_auth_pg? just more deps?
19:40:59 <skvidal> nirik: more deps - and more tightly coupled to the db server
19:41:01 <skvidal> so...
19:41:07 <skvidal> if we need to look at the logs/log reports
19:41:07 <nirik> yeah, if db is down... right.
19:41:11 <skvidal> and the db server is down
19:41:12 <skvidal> right
19:41:24 <nirik> there is that OAUTH plugin... but it's not packaged...
19:41:26 <skvidal> if there was a sane way to do mod_auth_pam against nss_db
19:41:30 <skvidal> I'd be all over it
19:41:47 <skvidal> this is one of those cases where one-off service passwords would
be wonderful
19:42:41 <dgilmore> skvidal: yubikey auth?
19:42:50 <nirik> huh... as an alternative... how about htaccess with a shared
password. You get that when you join sysadmin-log? another password and could be grabbed
around...
19:43:00 <nirik> dgilmore: still requires the db I think...
19:43:22 <dgilmore> nirik: it requires a db
19:43:39 <dgilmore> in that you need to yubikey validation server up and running
19:43:43 <nirik> yeah.
19:44:52 <nirik> anyhow, I guess lets ponder on it more and try and come up with the
least anoying alternative? ;)
19:46:35 <smooge> I figure an .htaccess pushed from puppet is not ok?
19:47:35 <nirik> smooge: well, we could put the pass in private... the issues would
be: password could get leaked by someone and no easy way to account who was reading them
(except possibly by IP).
19:47:45 <skvidal> smooge: and we'd have to keep updatinging it
19:47:52 <skvidal> nirik: oh - nm
19:47:53 <skvidal> sorry
19:47:55 <skvidal> I just caught up
19:47:56 <skvidal> shared pw
19:48:00 <skvidal> not individual ones
19:48:12 <skvidal> hahah, I know what we could do
19:48:21 <skvidal> I could have the epylog mail send out the password ;)
19:48:23 <nirik> yeah, I was talking shared.
19:48:32 * skvidal kids - that's no better than how we are now
19:48:35 <smooge> oh I figured it would just be like the password files. puppet runs
on lockbox, gets the passwords out of the db, builds an htaccess and pushes that out
19:48:52 <nirik> I suppose it could be a per user pass in there too.
19:49:10 <smooge> actually lockbox wouldn't do it.. log02 would
19:49:11 <skvidal> smooge: oh - so you mean puppet generates an htpasswd file with
all sysadmin-log members in it?
19:49:12 <smooge> duh
19:49:22 <smooge> skvidal, yeah.
19:49:23 <skvidal> hmm
19:49:26 <smooge> or a python script
19:49:34 <skvidal> can aapache deal with the passwd crypt
19:49:35 <smooge> that is cron run or something
19:49:38 <skvidal> we have in the nssdb files?
19:49:54 <skvidal> smooge: I like it, in principal...
19:50:03 <nirik> skvidal: possibly.
19:50:25 <smooge> reality will probably intrude and say no
19:50:29 <skvidal> nirik: we run this risk of having these files more available so
people could compromise them
19:50:31 <nirik> dump crypted pass from nssdb -> htaccess file -> bobs your
uncle.
19:50:36 <nirik> ?
19:50:44 <skvidal> nirik:well the files have to be readable by apache
19:50:52 <skvidal> which makes them slightly more vulnerable than shadow nss_db
19:50:56 <nirik> true.
19:51:08 * skvidal hmms
19:51:23 <nirik> but anyone can get crypted pass from fas
19:51:26 <nirik> or wait.
19:51:29 <nirik> no they can't.
19:52:03 <skvidal> no, they can't
19:52:15 <skvidal> getent won't do what you want
19:52:23 <skvidal> it's not like nis
19:52:38 <skvidal> and you'd need the fas host password to grab it from fas
19:52:44 <nirik> I was thinking of someone remotely running fasClient, but it
doesn't allow non priv accounts that info
19:53:19 <smooge> hey how about we add a field into fas that keeps that password?
19:53:31 <smooge> it should be all of what 5 minutes of abadger1999's time :)
19:53:47 <skvidal> smooge: umm......
19:53:57 <smooge> anyway.. I think I have derailed this
19:54:02 <skvidal> smooge: if we were going to do it - I'd say we add an
arbitrary service password field
19:54:16 <skvidal> and then make it so you can pull down that service password
19:54:22 <smooge> well yes. that was I meant
19:54:23 <skvidal> and that will not be a 5m project
19:54:27 * nirik wonders if there's a clever way to have apache auth with ssh keys.
;)
19:54:30 <smooge> I figured that also
19:54:36 <skvidal> and to be fair to abadger1999 I doubt he has the 5m to spare
19:54:53 <abadger1999> hehe
19:55:10 <skvidal> abadger1999: if I'm wrong about that then GO GET TO WORK!
19:55:10 <skvidal> :)
19:55:59 <skvidal> okay
19:56:04 <nirik> anyhow, I think we keep thinking...
19:56:08 <skvidal> now
19:56:12 <skvidal> about the log reports
19:56:16 <skvidal> is there any CONTENT in the log report
19:56:20 <skvidal> that folks like or don't like
19:56:23 <skvidal> or would like to see
19:56:44 <skvidal> I have been thinking about how we could make apache logs happen
and it is just not frelling obvious to me
19:56:53 <skvidal> it will involve, I suspect, a lot of work
19:57:20 <nirik> yeah, on that, could we switch them to syslog? or is that too much
work/load/network problem?
19:57:38 * nirik also wondered about getting audit setup to log to a central place.
19:57:41 <abadger1999> skvidal: Okay, there's currently a config table in fas
that you can access via json.
19:57:42 <skvidal> nirik: given our web log volume I'm worried how it would beat
up log02
19:57:49 <abadger1999> That allows you to keep arbitrary data in fas.
19:57:57 <skvidal> abadger1999: orly?
19:58:01 <abadger1999> asterisk for instance was kept there.
19:58:10 <skvidal> abadger1999: huh
19:58:24 <nirik> skvidal: yeah, would need some testing.
19:58:28 <abadger1999> Let me see if I can find that.
19:58:38 <smooge> nirik, too much load when I tried apache logs a year or so ago
19:58:48 <nirik> I've also wondered about having a log03 somewhere. backs up
log02 read-only in case something happens to it.
19:58:56 <nirik> smooge: ok.
19:59:03 <abadger1999> skvidal:
https://fedorahosted.org/releases/p/y/python-fedora/doc/existing.html#fed...
19:59:08 <abadger1999> and get_config_like()
19:59:13 <smooge> skvidal, I would go with a seperate epylog report for apache
logs..
19:59:15 <abadger1999> get_configs_like()
19:59:25 <skvidal> smooge: oh of course
19:59:32 <skvidal> smooge: I wouldn't want it mixed with the other logs atall
19:59:40 <skvidal> smooge: but I mean just handling the data at all
20:00:13 * smooge is confused then. I thought epylog dealt with it per domain
20:00:20 <smooge> or did I misremember
20:00:26 <nirik> epylog works on a merged log of all hosts.
20:00:44 <nirik> we could merge the apache logs too and run on them, but I bet
it's going to be frigging gigantic.
20:00:49 <skvidal> smooge: epylog doesn't deal with apache logs at all right
now
20:00:53 <skvidal> smooge: it has no module to handle them
20:01:00 <smooge> ah I thought for apache it did something different. I must be
thinking of splunk or something
20:01:02 <skvidal> nirik: no betting about it
20:01:04 <nirik> yeah, so something else operating on them might be good.
20:01:17 <skvidal> the other issue is this
20:01:26 <skvidal> epylog is more about reporting and pointing up issues
20:01:26 <nirik> we do have awstats, but it's just hits and such, not errors.
20:01:35 <skvidal> apache logs often are about stat reporting
20:01:39 <skvidal> less about error reporting
20:01:44 <pingou> nirik: 404 errors are in iirc
20:01:45 <skvidal> and I _think_ we want to know more about errors
20:01:45 * nirik nods.
20:01:52 <nirik> however, there are errors.
20:02:04 <nirik> which currently we ignore. ;)
20:02:09 <smooge> apache errorlogs are an abomination
20:02:10 <skvidal> that's where I would like to be
20:02:13 <skvidal> to be fair
20:02:15 <skvidal> I would love it
20:02:17 <nirik> pingou: yeah, but not tracebacks and such.
20:02:23 <skvidal> if we could get just apache error logsand app traces
20:02:25 <skvidal> via syslog
20:02:32 <skvidal> which should, in theory, be much less
20:02:39 <smooge> oooooh
20:02:42 <smooge> hmmm
20:03:10 <skvidal> now we could modify our apache configs
20:03:21 <nirik> sadly, I think it's the bulk of apache logs.
20:03:23 <skvidal> to have the error logs be spat out via syslog to a local#
facility
20:03:31 <skvidal> nirik: you think most of our apache logs are errors?
20:03:37 <skvidal> nirik: that feels..... bad
20:03:45 <nirik> -rw-r--r-- 1 root root 2320563738 Aug 11 20:03 error_log
20:03:57 <skvidal> how OLD is that/
20:04:02 <skvidal> head -1 error_log
20:04:24 <nirik> [Thu Aug 11 04:02:22 2011] [error] /usr/sbin/pkgdb.wsgi:19:
DeprecationWarning: fedora.tg.util is deprecated. Switch to one of these instead: TG1
apps: fedora.tg.tg1utils TG2 apps: fedora.tg.tg2utils. This file will disappear in 0.4
20:04:41 <nirik> 1 day
20:04:42 <skvidal> wait
20:04:44 <skvidal> noway
20:04:49 <skvidal> what the hell?
20:04:59 <nirik> way. ;)
20:05:00 <skvidal> it's 2.2GB in ONE DAY?!
20:05:03 <nirik> app01
20:05:17 <nirik> I think all our monitoring is showing up in there.
20:05:28 <nirik> [Thu Aug 11 20:05:17 2011] [error] - - "GET
/pkgdb/collections/ HTTP/1.0" 200 22395 "" ""
20:05:43 <skvidal> wow
20:05:47 <nirik> or perhaps pkgdb is just doing everything to error. ;)
20:05:49 <skvidal> like 5 logs a second
20:05:53 <skvidal> christ on a crutch
20:05:54 <smooge> wow that has gotten bad
20:06:05 <smooge> I don't think it was that bad last fall.
20:06:08 <nirik> right. So, we need to clean this up some before we can even look at
syslog.
20:06:15 <skvidal> nirik: no kidding
20:06:20 <skvidal> [Thu Aug 11 20:05:51 2011] [error] - - "GET /voting/
HTTP/1.0" 200 14378 "" ""
20:06:23 <skvidal> that's our monitoring
20:06:30 <skvidal> also
20:06:38 <skvidal> why is a 200 from that an 'error'?
20:06:45 <nirik> no idea.
20:07:01 <smooge> well it could be buried in a line below that
20:07:15 <skvidal> okay
20:07:17 <skvidal> you know what
20:07:24 <skvidal> I'll take a good long look at that in the coming week
20:07:30 <nirik> skvidal: thanks.
20:07:35 <skvidal> #action skvidal to curse at apache error logs
20:07:41 <nirik> any other items? or shall we close out the meeting?
20:07:55 <smooge> not me I think I broke things enough
20:08:01 <skvidal> wow
20:08:02 <skvidal> I'm wrong
20:08:07 <skvidal> 101 error log lines per second
20:08:23 <skvidal> on app01
20:08:24 <abadger1999> That's something we need lmacken to look into -- it's
the curse of the TG1 logging system that we've never figured out completely.
20:08:24 <skvidal> ONLY
20:08:29 <nirik> yeah, app01 alerted the other night because / was more than 85%
full. ;(
20:08:51 <skvidal> I'm going to see if I can do the user outlier log report
today and tomorrow
20:08:55 <skvidal> and then get to the apache logs
20:08:57 <skvidal> and be angry
20:09:21 <pingou> sed -i -e "s|200|d" error_log ? :d
20:09:28 <abadger1999> We either seem to not get tracebacks in the logs or way too
much information or both.
20:10:20 <nirik> ok, thanks for coming everyone!
20:10:24 <lmacken> abadger1999: yeah, our TG1 logging setup is inconsistent, and
awful.
20:10:37 <pingou> abadger1999: todo: check the tg2 version
20:10:52 * nirik waits a min more since we have more info. ;)
20:11:20 <nirik> pingou: is voting changed to tg2? or not in prod yet?
20:11:34 <pingou> nirik: afaik not yet
20:11:42 <nirik> ok.
20:11:54 <nirik> ok, will close out then....
20:11:58 <nirik> #endmeeting