tuning fdatasync for kvm?
by Bill McGonigle
Hi, all,
The subject line is a bit of a guess. I'm running the preview release
on an updated F12, with a Vista guest (no virtio drivers yet) and after
an initial OS install the software updates have been running for a bit
more than 12 hours, and the system's hard drives have been thrashing
heartily during it.
I've got a data-journaled ext4 on luks on raid-1. That's clearly not a
write-optimized stack, but performance has been pretty good in the past
with KVM & Vista and fine for normal system operations. If I check out
iotop while it's thrashing it's all in [jbd2/dm-2-8] and [kdmflush].
Looking for the blocked tasks (below), I think I see kvm is
fdatasync'ing, on what I'd presume is a very frequent basis. I noticed
that fsync was replaced with fdatasync not too long ago, but that should
have helped performance, not clobbered it (I think...). Ideally I could
tell kvm to fdatasync every 5 seconds or something like that and get
batched writes. I've tried switching schedules to cfq, deadline, and
noop, with no big difference.
Does this seem sensible or am I totally off-base here? I'm rebuilding
this virtual machine after a virtio driver install went south, so I'd
like to at least have a usable solution (but not libeatmydata) without
them. I also run OS's with no virtio driver support at times.
Thanks,
-Bill
-----
qemu-system-x86-0.12.3-6.fc12.x86_64
kernel-2.6.32.11-99.fc12.x86_64
-----
SysRq : Show Blocked State
task PC stack pid father
kdmflush D ffff8802218d4880 0 1818 2 0x00000080
ffff8802243a9d40 0000000000000046 ffff8802243a9ca0 ffffffffa006b33d
ffff8802228d1a70 ffff8802228d1a70 ffff8802243a9fd8 ffff8802243a9fd8
ffff880223c21b38 000000000000f980 0000000000015740 ffff880223c21b38
Call Trace:
[<ffffffffa006b33d>] ? raid1_unplug+0x29/0x2d [raid1]
[<ffffffff8107c359>] ? ktime_get_ts+0x85/0x8e
[<ffffffff81454c1d>] io_schedule+0x43/0x5d
[<ffffffff81378b9e>] dm_wait_for_completion+0xf7/0x129
[<ffffffff81050c59>] ? default_wake_function+0x0/0x14
[<ffffffff81379824>] dm_flush+0x59/0x5e
[<ffffffff813798ea>] dm_wq_work+0xc1/0x173
[<ffffffff8107027c>] worker_thread+0x1a9/0x237
[<ffffffff81379829>] ? dm_wq_work+0x0/0x173
[<ffffffff8107488b>] ? autoremove_wake_function+0x0/0x39
[<ffffffff810700d3>] ? worker_thread+0x0/0x237
[<ffffffff8107459e>] kthread+0x7f/0x87
[<ffffffff81012d6a>] child_rip+0xa/0x20
[<ffffffff8107451f>] ? kthread+0x0/0x87
[<ffffffff81012d60>] ? child_rip+0x0/0x20
jbd2/dm-2-8 D 0000000000000002 0 1915 2 0x00000080
ffff88022586fbe0 0000000000000046 ffff8802218d4800 ffff880222eee880
ffff88022586fba0 ffffffff8137a489 ffff88022586ffd8 ffff88022586ffd8
ffff880222c71b38 000000000000f980 0000000000015740 ffff880222c71b38
Call Trace:
[<ffffffff8137a489>] ? dm_table_unplug_all+0x58/0xc0
[<ffffffff8107c359>] ? ktime_get_ts+0x85/0x8e
[<ffffffff8114132a>] ? sync_buffer+0x0/0x44
[<ffffffff8114132a>] ? sync_buffer+0x0/0x44
[<ffffffff81454c1d>] io_schedule+0x43/0x5d
[<ffffffff8114136a>] sync_buffer+0x40/0x44
[<ffffffff81455170>] __wait_on_bit+0x48/0x7b
[<ffffffff81455211>] out_of_line_wait_on_bit+0x6e/0x79
[<ffffffff8114132a>] ? sync_buffer+0x0/0x44
[<ffffffff810748c4>] ? wake_bit_function+0x0/0x33
[<ffffffff8114128d>] __wait_on_buffer+0x24/0x26
[<ffffffff811c6cc9>] wait_on_buffer+0x3d/0x41
[<ffffffff811c79d2>] jbd2_journal_commit_transaction+0xb5c/0x1102
[<ffffffff81048024>] ? pick_next_task_fair+0xdb/0xec
[<ffffffff810653cd>] ? try_to_del_timer_sync+0x73/0x81
[<ffffffff811cdf31>] kjournald2+0xc6/0x203
[<ffffffff8107488b>] ? autoremove_wake_function+0x0/0x39
[<ffffffff811cde6b>] ? kjournald2+0x0/0x203
[<ffffffff8107459e>] kthread+0x7f/0x87
[<ffffffff81012d6a>] child_rip+0xa/0x20
[<ffffffff8107451f>] ? kthread+0x0/0x87
[<ffffffff81012d60>] ? child_rip+0x0/0x20
qemu-kvm D ffff8801d1a8fdc0 0 17177 1 0x00000080
ffff8801d1a8fd78 0000000000000086 0000000000000002 0000000000000046
ffff8801d1a8fcd8 ffffffff81045a01 ffff8801d1a8ffd8 ffff8801d1a8ffd8
ffff8802143ac9f8 000000000000f980 0000000000015740 ffff8802143ac9f8
Call Trace:
[<ffffffff81045a01>] ? task_rq_unlock+0x11/0x13
[<ffffffff811cd8b3>] jbd2_log_wait_commit+0xc6/0x119
[<ffffffff8107488b>] ? autoremove_wake_function+0x0/0x39
[<ffffffff811c5b98>] jbd2_journal_stop+0x205/0x235
[<ffffffff811c6b08>] jbd2_journal_force_commit+0x28/0x2c
[<ffffffff811a911c>] ext4_force_commit+0x27/0x2d
[<ffffffff8118fd57>] ext4_sync_file+0x16f/0x174
[<ffffffff8113e610>] vfs_fsync_range+0x82/0xb2
[<ffffffff8113e6aa>] vfs_fsync+0x1d/0x1f
[<ffffffff8113e6e0>] do_fsync+0x34/0x4b
[<ffffffff8113e70a>] sys_fdatasync+0x13/0x17
[<ffffffff81011d32>] system_call_fastpath+0x16/0x1b
--
Bill McGonigle, Owner
BFC Computing, LLC
http://bfccomputing.com/
Telephone: +1.603.448.4440
Email, IM, VOIP: bill(a)bfccomputing.com
VCard: http://bfccomputing.com/vcard/bill.vcf
Social networks: bill_mcgonigle/bill.mcgonigle
14 years
Fedora Virtualization Test Day
by Justin Forbes
A big thank you to everyone who participated in the official Fedora
Virtualization Test Day yesterday. For those of you with other
obligations, we can still use your help. The wiki page still exists at
http://fedoraproject.org/wiki/Test_Day:2010-04-08_Virtualization and the
live CD images will remain for a week. It is not too late to test and
give feedback.
There are also a number of bugs found as a result of the test day, which
need attention. If you would rather tackle bugs than test, we are okay
with that too (really, we are!).
Thanks,
Justin
14 years, 1 month
Fedora Virtualization Test Day tomorrow April 8th
by Justin Forbes
ve another Fedora Test Day for virtualization coming up this Thursday,
April 8.
We're making good progress fleshing out a wiki with test cases here:
http://fedoraproject.org/wiki/Test_Day:2010-04-08_Virtualization
Please do come along and help out with the testing, 'cause we've got
plenty to test! :-)
If you think you can make it on Thursday, pick a test area (or areas)
and put your name where it says "add your name here".
Prepare now by making sure your F13 test hosts are up to date. We should
have updated live ISOs available for test day as well for those of you who
don't have F13 installed.
See you Thursday on #fedora-test-day!
Thanks,
Justin
14 years, 1 month
Windows XP KVM annoyance
by Tom Horsley
I posted this on the test list since it is ff13, but perhaps
there might be more insight on the virt list:
I just tried running my Windows XP KVM on fedora 13
(I installed the KVM under fedora 12). It seemed to work
well, but I'm using the redhat virtio drivers for
network and disk, and Windows insisted on re-installing
the drivers (claiming "new hardware") the first time
I booted under fedora 13 libvirtd.
Strangely, when I booted the same KVM after going back
to fedora 12 on the host, it didn't insist on
re-installing the drivers yet again.
None of this triggered any Windows activation nonsense
(which is why it is only annoying).
I guess I should poke around in the windows hardware
manager for the details of what changed, but it would
sure be nice if things wouldn't change at all :-).
14 years, 1 month
Re: [fedora-virt] Nagios plugin for monitoring libvirt: multiple servers not possible?
by Richard W.M. Jones
On Thu, Apr 01, 2010 at 06:40:27PM +0200, Eveline wrote:
> If you have better suggestion, I'd be glad to hear them. :)
My best idea was to encode the server name into the hostname, eg:
host1:vm1
host1:vm2
host2:vm1
and in check_virt split the name into the two parts, one of which gets
passed in the virsh URI, the other gets used as the $HOSTNAME.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://et.redhat.com/~rjones/libguestfs/
See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html
14 years, 1 month
Re: [fedora-virt] Nagios plugin for monitoring libvirt: multiple servers not possible?
by Richard W.M. Jones
On Thu, Apr 01, 2010 at 06:11:14PM +0200, Eveline wrote:
> On Thu, Apr 01, 2010 at 12:23:03 +0200, Andrea Modesto Rossi wrote:
> >
> > On Gio, 1 Aprile 2010 9:49 am, Richard W.M. Jones wrote:
> > >
> > > You could perhaps try something with libvirt remote support:
> > > http://libvirt.org/remote.html
> >
> > I use ZABBIX to monitor my KVM cluster, both qemu processes and libvirtd
> > service. Simply, on zabbiz client:
> >
> > cat /etc/zabbix/zabbix_agent.conf
> >
> > .....
> > UserParameter=proc.num[libvirtd],ps axu | grep "libvirtd" | awk '{print
> > $11}' | grep "libvirtd" | wc -l
> > UserParameter=proc.num[qemu-kvm],ps axu | grep "qemu-kvm" | awk '{print
> > $11}' | grep "qemu-kvm" | wc -l
>
> Hmm, we do also have zabbix, so this might indeed be a solution. As far as
> I know zabbix is mostly used for statistics in our setup, and nagios for
> problem reports.
>
> Thanks!
>
> The remote support for libvirt seems like what virt-manager uses to
> connect to all KVM servers for me. I recoqnize the qemu:// URLS from the
> virt-manager config.
>
> But how to combine that with the nagios-virt plugin, I really don't have a
> clue. I was thinking myself to create a nagios plugin that runs the plugin
> on the remote servers over ssh. A bit like virt-manager does (it asks for
> ssh keys/passwords).
Basically you need to edit the check_virt script that gets installed.
Locate the line which runs virsh:
status=$(sudo /usr/bin/virsh domstate "$HOSTNAME" | head -1)
and modify the command so it passes a URI to virsh, eg:
... /usr/bin/virsh -c 'xen+ssh://host/' ...
(That will only handle a single remote host; handling multiple remote
hosts is a good deal more complicated than that).
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/
14 years, 1 month
