On Fri, Mar 18, 2011 at 9:04 AM, seth vidal <skvidal(a)fedoraproject.org&gt; wrote: > Hi folks, > some thoughts have been slowly coalescing in my head about how we're > managing our boxes/services and I have some suggestions I've passed by > various folks but I wanted to check them out with everyone: > > > 1. puppetd sucks..... memory. Right now we have puppetd running on every > box and it wakes up every half hour and runs itself. This is fine but in > the time where it is not doing anything it just eats memory for no good > reason. I'd like to suggest we move to a cron-driven model instead of > puppetd. I'd write a simple cron job that runs every half hour to run > puppetd, if a lock file is not found. Pretty straightforward, of > course. I'd be happy to help get this going. I've set up puppet a few times in this fashion now and it's pretty easy to do.

2. monitoring if puppetd has run properly: > two things we want to know about puppet runs: > a. when they last happened per-box > b. if they fell over in a horrible way. It might be overkill, but puppet dashboard is pretty nice. It's a web interface, kind of like a nagios for puppet, telling you exactly the things you want to know above. Plus, it has some pretty graphs :) It runs on cron jobs too. I've set it up once about a year ago, pretty nice. I'm sure it's improved some since then. Again, I'd be happy to help set this up.

On Fri, 2011-03-18 at 09:16 -0600, Clint Savage wrote: > On Fri, Mar 18, 2011 at 9:04 AM, seth vidal <skvidal(a)fedoraproject.org&gt; wrote: > > Hi folks, > >  some thoughts have been slowly coalescing in my head about how we're > > managing our boxes/services and I have some suggestions I've passed by > > various folks but I wanted to check them out with everyone: > > > > > > 1. puppetd sucks..... memory. Right now we have puppetd running on every > > box and it wakes up every half hour and runs itself. This is fine but in > > the time where it is not doing anything it just eats memory for no good > > reason. I'd like to suggest we move to a cron-driven model instead of > > puppetd. I'd write a simple cron job that runs every half hour to run > > puppetd, if a lock file is not found. Pretty straightforward, of > > course. > > I'd be happy to help get this going.  I've set up puppet a few times > in this fashion now and it's pretty easy to do. I have a script now ($RANDOM is my friend) - I was just going to drop it into a path and put a crontab into /etc/cron.d/ is there more to it than that?

> 2. monitoring if puppetd has run properly: > >   two things we want to know about puppet runs: > >   a. when they last happened per-box > >   b. if they fell over in a horrible way. > > It might be overkill, but puppet dashboard is pretty nice.  It's a web > interface, kind of like a nagios for puppet, telling you exactly the > things you want to know above.  Plus, it has some pretty graphs :)  It > runs on cron jobs too.  I've set it up once about a year ago, pretty > nice.  I'm sure it's improved some since then.  Again, I'd be happy to > help set this up. feels like overkill. I'm not partial to web interfaces, personally, b/c to get them running and accessible on puppet.fp.o requires a fair bit of hoop jumping and reverse proxying. -sv

Hi folks, some thoughts have been slowly coalescing in my head about how we're managing our boxes/services and I have some suggestions I've passed by various folks but I wanted to check them out with everyone: 1. puppetd sucks..... memory. Right now we have puppetd running on every box and it wakes up every half hour and runs itself. This is fine but in the time where it is not doing anything it just eats memory for no good reason. I'd like to suggest we move to a cron-driven model instead of puppetd. I'd write a simple cron job that runs every half hour to run puppetd, if a lock file is not found. Pretty straightforward, of course.

2. monitoring if puppetd has run properly: two things we want to know about puppet runs: a. when they last happened per-box b. if they fell over in a horrible way. (a) can be known by looking at the $nodename.yaml file which lives on the puppetmaster. I've written a script to check if that file is older than 1 hour and report the nodename if it is. (b) can be done via the cron job - ie: taking error output from the puppet run and mailing to people until we fix it! :)

3. sign** boxes. problems here: a. These boxes are falling out of date, repeatedly, b/c they aren't in our normal updating path. b. these boxes don't email out to the same locations as the other boxes c. these boxes don't get faspassword updates properly d. these boxes don't get config changes normally via puppet (a) I'd like to suggest that they be put into a normal updating path and/or we setup a nag mail to tell us about them (b) obviously, fix their mail configs (c) fasclient is failing b/c of a missing token b/c, most likely, of (d) I'm open to suggestions on those but it is a bit annoying b/c while I understand their 'sensitivity' I think our way of treating them is making the problem WORSE not better.

On Fri, Mar 18, 2011 at 11:04:32AM -0400, seth vidal wrote: > Hi folks, > some thoughts have been slowly coalescing in my head about how we're > managing our boxes/services and I have some suggestions I've passed by > various folks but I wanted to check them out with everyone: > > > 1. puppetd sucks..... memory. Right now we have puppetd running on every > box and it wakes up every half hour and runs itself. This is fine but in > the time where it is not doing anything it just eats memory for no good > reason. I'd like to suggest we move to a cron-driven model instead of > puppetd. I'd write a simple cron job that runs every half hour to run > puppetd, if a lock file is not found. Pretty straightforward, of > course. > +1 Might need to update kickstarts and/or the SOP pages: http://fedoraproject.org/wiki/Kickstart_Infrastructure_SOP http://fedoraproject.org/wiki/Puppet_Infrastructure_SOP > 2. monitoring if puppetd has run properly: > two things we want to know about puppet runs: > a. when they last happened per-box > b. if they fell over in a horrible way. > > (a) can be known by looking at the $nodename.yaml file which lives > on the puppetmaster. I've written a script to check if that file is > older than 1 hour and report the nodename if it is. > (b) can be done via the cron job - ie: taking error output from the > puppet run and mailing to people until we fix it! :) > +1 > 3. sign** boxes. problems here: > a. These boxes are falling out of date, repeatedly, b/c they aren't > in our normal updating path. > b. these boxes don't email out to the same locations as the other > boxes > c. these boxes don't get faspassword updates properly > d. these boxes don't get config changes normally via puppet > > (a) I'd like to suggest that they be put into a normal updating path > and/or we setup a nag mail to tell us about them > (b) obviously, fix their mail configs > (c) fasclient is failing b/c of a missing token b/c, most likely, of > (d) > > I'm open to suggestions on those but it is a bit annoying b/c while I > understand their 'sensitivity' I think our way of treating them is > making the problem WORSE not better. > I agree with your assessment. I guess we need to tell releng our concerns and figure out what needs to be done For a: perhaps have releng okay us/a specific subset of sysadmins to run updates along with all the other updates. -Toshio _______________________________________________ infrastructure mailing list infrastructure(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hi folks, some thoughts have been slowly coalescing in my head about how we're managing our boxes/services and I have some suggestions I've passed by various folks but I wanted to check them out with everyone: 1. puppetd sucks..... memory. Right now we have puppetd running on every box and it wakes up every half hour and runs itself. This is fine but in the time where it is not doing anything it just eats memory for no good reason. I'd like to suggest we move to a cron-driven model instead of puppetd. I'd write a simple cron job that runs every half hour to run puppetd, if a lock file is not found. Pretty straightforward, of course.

2. monitoring if puppetd has run properly: two things we want to know about puppet runs: a. when they last happened per-box b. if they fell over in a horrible way. (a) can be known by looking at the $nodename.yaml file which lives on the puppetmaster. I've written a script to check if that file is older than 1 hour and report the nodename if it is. (b) can be done via the cron job - ie: taking error output from the puppet run and mailing to people until we fix it! :)