On Thu, 1 Jan 2009, Kostas Georgiou wrote:
On Thu, Jan 01, 2009 at 01:17:38AM -0600, Mike McGrath wrote:
On Thu, 1 Jan 2009, James Antill wrote:
On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote:
Lets pool some knowledge together because at this point, I'm missing something.
I've been doing all measurements with sar as bonnie, etc, causes builds to timeout.
Problem: We're seeing slower then normal disk IO. At least I think we are. This is a PERC5/E and MD1000 array.
When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get around 4-6MBytes/s
This _might_ not be "IO" in a normal sense, -a to cp means:
file data + file inode + ACLs + selinux + xattrs [+ file capabilities]
...esp. given that you aren't getting large IOWait times, you might want to strace -T the cp and do some perl/whatever on the result to see what is eating up the time.
Even with non cp type things (like a bacula backup) it just doesn't seem as fast as I would expect it to be. I've never actually done trending at this level / scale on a filesystem / drive before. So I really don't have a good baseline except that it just seems slow to me.
Other then the much faster direct block access and the large file reads, I don't have much else to go on that makes me think its slow.
Do writes show the same pattern? If you use selinux/ACLs/xattrs the default inode size of 128 can cause slowdowns (#205161 for example).
One reason I'm trying to ramp this up now is because the koji share is still under 50% utilized. If it turns out to be something in the filesystem, its not too late for us to shrink the main filesystem, create the new, copy, and grow the new.
Can you run blktrace+seekwatcher (both in EPEL) to get an idea on what is going on? An iostat -x -k /dev/sde 1 output will also be helpfull.
I'll take a look at those two applications as well, here's the iostat:
Linux 2.6.18-92.1.18.el5xen (xen2.fedora.phx.redhat.com) 01/01/2009
avg-cpu: %user %nice %system %iowait %steal %idle 0.55 0.01 1.35 0.10 6.28 91.71
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sde 1389.22 95.13 161.74 270.46 6693.75 1670.16 38.70 1.09 2.51 1.48 64.04
-Mike