So, the recent fas vulnerability made us realize that we were not collecting and saving httpd logs from staging machines.
I've since added:
app01.stg app02.stg proxy01.stg
to have their httpd logs pulled over to log02 and kept.
This however got me thinking. Since we are moving to a model where each app has it's own server, should we widen the servers we pull httpd logs from? For example, ask01/02? fedocal? blockerbugs?
Or should we figure out a better way to collect and store the httpd logs.
Thoughts?
kevin
On Thu, May 09, 2013 at 02:17:35PM -0600, Kevin Fenzi wrote:
So, the recent fas vulnerability made us realize that we were not collecting and saving httpd logs from staging machines.
I've since added:
app01.stg app02.stg proxy01.stg
Did you leave out fas01.stg on purpose?
to have their httpd logs pulled over to log02 and kept.
This however got me thinking. Since we are moving to a model where each app has it's own server, should we widen the servers we pull httpd logs from? For example, ask01/02? fedocal? blockerbugs?
I think so. If we're looking to analyze logs for tracebacks, errors, etc, then having consolidated logging makes it much easier than logging into our redundant servers for that service.
Or should we figure out a better way to collect and store the httpd logs.
Thoughts?
I don't think we have any ideas for how to do something better immediately... So we probably should just add all the other hosts for now.
-Toshio
On Fri, 10 May 2013 11:15:56 -0700 Toshio Kuratomi a.badger@gmail.com wrote:
On Thu, May 09, 2013 at 02:17:35PM -0600, Kevin Fenzi wrote:
So, the recent fas vulnerability made us realize that we were not collecting and saving httpd logs from staging machines.
I've since added:
app01.stg app02.stg proxy01.stg
Did you leave out fas01.stg on purpose?
Nope. Forgot it. ;)
to have their httpd logs pulled over to log02 and kept.
This however got me thinking. Since we are moving to a model where each app has it's own server, should we widen the servers we pull httpd logs from? For example, ask01/02? fedocal? blockerbugs?
I think so. If we're looking to analyze logs for tracebacks, errors, etc, then having consolidated logging makes it much easier than logging into our redundant servers for that service.
Right. ok.
Would someone like to look at adding those other servers? Otherwise I will do it monday.
kevin
On Thu, 9 May 2013 14:17:35 -0600 Kevin Fenzi kevin@scrye.com wrote:
So, the recent fas vulnerability made us realize that we were not collecting and saving httpd logs from staging machines.
I've since added:
app01.stg app02.stg proxy01.stg
to have their httpd logs pulled over to log02 and kept.
This however got me thinking. Since we are moving to a model where each app has it's own server, should we widen the servers we pull httpd logs from? For example, ask01/02? fedocal? blockerbugs?
Or should we figure out a better way to collect and store the httpd logs.
The ways I know of to collect/store httpd logs are: 1. rsyncing after the fact 2. redir out to logger to dump to syslog 3. other syslog-redirection trick 4. direct-to-mysql log writes.
I'm sure there are lots of variations on 3 using non-syslog to replicate the logs.
the disadvantage of 1 is that we don't get the logs from 'just now' if something goes wrong. That's where we are now. The second issue is that we have to constantly update that list of hosts/files to replicate those logs.
the disad of 2 and 3 is that http logs can kick the crap out of syslog in short order. it may, however, be worth trying it with our system to see how much damage the httpd logs do.
the disad of 4 is the dep on a db server (and the disads from 1)
thoughts on trying to log http to rsyslog/log02?
-sv
On Mon, 13 May 2013 12:10:15 -0400 seth vidal skvidal@fedoraproject.org wrote:
The ways I know of to collect/store httpd logs are:
- rsyncing after the fact
- redir out to logger to dump to syslog
- other syslog-redirection trick
- direct-to-mysql log writes.
I'm sure there are lots of variations on 3 using non-syslog to replicate the logs.
the disadvantage of 1 is that we don't get the logs from 'just now' if something goes wrong. That's where we are now. The second issue is that we have to constantly update that list of hosts/files to replicate those logs.
Yeah.
the disad of 2 and 3 is that http logs can kick the crap out of syslog in short order. it may, however, be worth trying it with our system to see how much damage the httpd logs do.
the disad of 4 is the dep on a db server (and the disads from 1)
thoughts on trying to log http to rsyslog/log02?
well, we do have the fas servers sending error_log messages there, I'd really be worried that it wouldn't be able to keep up. Especially since some of our web apps are really really noisy. ;(
I guess we could try adding them one by one and see when/if it breaks, but it might be better to revisit after we clean up apps logging some?
kevin
On Mon, May 13, 2013 at 10:25:50AM -0600, Kevin Fenzi wrote:
On Mon, 13 May 2013 12:10:15 -0400 seth vidal skvidal@fedoraproject.org wrote:
The ways I know of to collect/store httpd logs are:
- rsyncing after the fact
- redir out to logger to dump to syslog
- other syslog-redirection trick
- direct-to-mysql log writes.
I'm sure there are lots of variations on 3 using non-syslog to replicate the logs.
the disadvantage of 1 is that we don't get the logs from 'just now' if something goes wrong. That's where we are now. The second issue is that we have to constantly update that list of hosts/files to replicate those logs.
Yeah.
the disad of 2 and 3 is that http logs can kick the crap out of syslog in short order. it may, however, be worth trying it with our system to see how much damage the httpd logs do.
the disad of 4 is the dep on a db server (and the disads from 1)
thoughts on trying to log http to rsyslog/log02?
well, we do have the fas servers sending error_log messages there, I'd really be worried that it wouldn't be able to keep up. Especially since some of our web apps are really really noisy. ;(
If you want to see if it breaks real quickly -- fas's access_log is probably the quickest growing log (although that may change once fas_openid is prevalent).
-Toshio
On Mon, 13 May 2013 09:38:59 -0700 Toshio Kuratomi a.badger@gmail.com wrote:
If you want to see if it breaks real quickly -- fas's access_log is probably the quickest growing log (although that may change once fas_openid is prevalent).
I'd guess we get a lot more hits to the mirrorlist, for example, than fas, wouldn't we?
-sv
On Mon, May 13, 2013 at 01:02:10PM -0400, seth vidal wrote:
On Mon, 13 May 2013 09:38:59 -0700 Toshio Kuratomi a.badger@gmail.com wrote:
If you want to see if it breaks real quickly -- fas's access_log is probably the quickest growing log (although that may change once fas_openid is prevalent).
I'd guess we get a lot more hits to the mirrorlist, for example, than fas, wouldn't we?
You're right -- I was unconsciously thinking in terms of the heavyweight web apps but mirrorlist trumps all of those.
-Toshio
infrastructure@lists.fedoraproject.org
-
Kevin Fenzi -
Seth Vidal -
Toshio Kuratomi