On Wed, 31 Dec 2008, Mike McGrath wrote:
Lets pool some knowledge together because at this point, I'm
missing
something.
I've been doing all measurements with sar as bonnie, etc, causes builds to
timeout.
Problem: We're seeing slower then normal disk IO. At least I think we
are. This is a PERC5/E and MD1000 array.
When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get
around 4-6MBytes/s
When I do a cp of a large file "cp /mnt/koji/out /tmp/" I get
30-40MBytes/s.
Then I "dd if=/dev/sde of=/dev/null" I get around 60-70 MBytes/s read.
If I "cat /dev/sde > /dev/null" I get between 225-300MBytes/s read.
The above tests are pretty consistent. /dev/sde is a raid5 array,
hardware raid.
So my question here is, wtf? I've been working to do a backup which I
would think would either cause network utilization to max out, or disk io
to max out. I'm not seeing either. Sar says the disks are 100% utilized
but I can cause major increases in actual disk reads and writes by just
running additional commands. Also, if the disks were 100% utilized I'd
expect we would see lots more iowait. We're not though, iowait on the box
is only %0.06 today.
So, long story short, we're seeing much better performance when just
reading or writing lots of data (though dd is many times slower then cat).
But with our real-world traffic, we're just seeing crappy crappy IO.
Thoughts, theories or opinions? Some of the sysadmin noc guys have access
to run diagnostic commands, if you want more info about a setting, let me
know.
I should also mention there's lots going on with this box, for example its
hardware raid, lvm and I've got xen running on it (though the tests above
were not in a xen guest).
Also for the curious:
dumpe2fs 1.39 (29-May-2006)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype
needs_recovery sparse_super large_file
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 1342177280
Block count: 2684354560
Reserved block count: 134217728
Free blocks: 1407579323
Free inodes: 1336866363
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 384
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16384
Inode blocks per group: 512
Filesystem created: Thu Jan 17 14:52:03 2008
Last mount time: Fri Dec 5 18:51:44 2008
Last write time: Fri Dec 5 18:51:44 2008
Mount count: 17
Maximum mount count: 24
Last checked: Sat May 24 03:14:41 2008
Check interval: 15552000 (6 months)
Next check after: Thu Nov 20 03:14:41 2008
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: tea
Directory Hash Seed: 1b6393b1-472c-4005-ae87-9603eea9f45b
Journal backup: inode blocks
Journal size: 128M