8 a.m.
Hello,
posting here as it could have occurred in Fedora too, during crossing
versions of components...
hope anyone could drive me some hints..
Also because I test qemu/kvm ibvirt in Fedora too...
I have 3 nodes with CentOS, two with 6.1 version + some updates (but below 6.2)
and one with 6.2.
I have a vm on 6.1 hypervisor and I'm able to live migrate it to the
6.2 host, but then I am not able to migrate from 6.2 to either one of
the 6.1....
relevant components:
6.1 hosts
qemu-kvm-0.12.1.2-2.160.el6_1.8.x86_64
libvirt-0.8.7-18.el6_1.1.x86_64
kernel-2.6.32-131.17.1.el6.x86_64
6.2 hosts
qemu-kvm-0.12.1.2-2.209.el6_2.1.x86_64
libvirt-0.9.4-23.el6_2.1.x86_64
kernel-2.6.32-220.2.1.el6.x86_64
guest is named dacsmaster and it is rh el 5.4
the migration from 6.2 to 6.1 fails with
# clusvcadm -M vm:dacsmaster -m intrarhev2
Trying to migrate vm:dacsmaster to intrarhev2...Failed; service
running on original owner
/var/log/messages information:
on source hypervisor:
Jan 12 15:45:17 rhev1 rgmanager[9267]: Migrating vm:dacsmaster to intrarhev2
Jan 12 15:45:18 rhev1 rgmanager[22193]: [vm] Migrate dacsmaster to
intrarhev2 failed:
Jan 12 15:45:18 rhev1 rgmanager[22215]: [vm] error: internal error
missing hostuuid element in migration data
Jan 12 15:45:18 rhev1 rgmanager[9267]: migrate on vm "dacsmaster"
returned 150 (unspecified)
Jan 12 15:45:18 rhev1 rgmanager[9267]: Migration of vm:dacsmaster to
intrarhev2 failed; return code 150
on target hypervisor:
Jan 12 15:45:18 rhev2 kernel: device vnet4 entered promiscuous mode
Jan 12 15:45:18 rhev2 kernel: brvlan65: topology change detected, propagating
Jan 12 15:45:18 rhev2 kernel: brvlan65: port 3(vnet4) entering forwarding state
Jan 12 15:45:18 rhev2 libvirtd: 15:45:18.113: 31182: warning :
qemudStartVMDaemon:3336 : Executing /usr/libexec/qemu-kvm
Jan 12 15:45:18 rhev2 libvirtd: 15:45:18.119: 31182: warning :
qemudStartVMDaemon:3346 : Executing done /usr/libexec/qemu-kvm
Jan 12 15:45:18 rhev2 qemu-kvm: Could not find keytab file:
/etc/qemu/krb5.tab: No such file or directory
Jan 12 15:45:18 rhev2 libvirtd: 15:45:18.332: 8787: error :
virCgroupRemoveRecursively:679 : Unable to remove
/cgroup/cpu/libvirt/qemu/dacsmaster/ (16)
Jan 12 15:45:18 rhev2 libvirtd: 15:45:18.332: 8787: error :
virCgroupRemoveRecursively:679 : Unable to remove
/cgroup/cpuacct/libvirt/qemu/dacsmaster/ (16)
Jan 12 15:45:18 rhev2 libvirtd: 15:45:18.332: 8787: error :
virCgroupRemoveRecursively:679 : Unable to remove
/cgroup/cpuset/libvirt/qemu/dacsmaster/ (16)
Jan 12 15:45:18 rhev2 libvirtd: 15:45:18.332: 8787: error :
virCgroupRemoveRecursively:679 : Unable to remove
/cgroup/memory/libvirt/qemu/dacsmaster/ (16)
Jan 12 15:45:18 rhev2 libvirtd: 15:45:18.332: 8787: error :
virCgroupRemoveRecursively:679 : Unable to remove
/cgroup/devices/libvirt/qemu/dacsmaster/ (16)
Jan 12 15:45:18 rhev2 libvirtd: 15:45:18.332: 8787: error :
virCgroupRemoveRecursively:679 : Unable to remove
/cgroup/freezer/libvirt/qemu/dacsmaster/ (16)
Jan 12 15:45:18 rhev2 libvirtd: 15:45:18.332: 8787: error :
virCgroupRemoveRecursively:679 : Unable to remove
/cgroup/blkio/libvirt/qemu/dacsmaster/ (16)
Jan 12 15:45:18 rhev2 kernel: brvlan65: port 3(vnet4) entering disabled state
Jan 12 15:45:18 rhev2 kernel: device vnet4 left promiscuous mode
Jan 12 15:45:18 rhev2 kernel: brvlan65: port 3(vnet4) entering disabled state
powering off the guest (through "disable" action of the related rhcs
service) and powering it on on 6.1 host, I'm able to live migrate it
either to the other 6.1 host and to the 6.2 one...
/etc/libvirt/libvirtd.conf doesn't contain anything for the host_uuid
part as I find in messages of source 6.2 hypervisor.....
and dmidecode command succeeds on all the three systems.
they are identical servers from an hw point of view and for example
output on one of them gives:
# dmidecode -s system-uuid
34353439-3036-435A-4A38-303330393332
Some additional notes:
If I freeze the vm related cluster service and on source 6.2 host I
manually run the virsh command that took place in 6.1 successfully, it
fails (as expected). This let me keep away considering the different
release versions of cluster related components between 6.1 and 6.2,
concentrating on different releaases of kernel/libvirt/qemu-kvm (at
least I
think so...)
intrarhevN is the host name on custer lan that is also used for migration
(the migration fails also using the service network, the only
difference being it asks a password as
on service network there is no ssh host equivalence)
# virsh migrate --live dacsmaster qemu+ssh://intrarhev2/system tcp:intrarhev2
error: internal error missing hostuuid element in migration data
It seems I'm able to connect through virsh from rhev1 (6.2) to rhev2
(6.1) as I can run
rhev1 prompt-- # virsh
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh # connect qemu+ssh://intrarhev2/system
virsh # list
and this command shows me the domains currently running on rhev2 as expected
So I presume there is something related to migration itself and
possibly different needed parameters in 6.2 shipped version of libvirt
(0.9.4 vs 0.8.7).
Any hint on these and/or possible debug options?
The error seems quite obscure, to me at least
Gianluca
8:36 a.m.
On Sat, Jan 14, 2012 at 3:00 PM, Gianluca Cecchi wrote:
Hello,
posting here as it could have occurred in Fedora too, during crossing
versions of components...
hope anyone could drive me some hints..
[snip]
# virsh migrate --live dacsmaster qemu+ssh://intrarhev2/system
tcp:intrarhev2
error: internal error missing hostuuid element in migration data
It seems I'm able to connect through virsh from rhev1 (6.2) to rhev2
(6.1) as I can run
rhev1 prompt-- # virsh
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh # connect qemu+ssh://intrarhev2/system
virsh # list
and this command shows me the domains currently running on rhev2 as expected
So I presume there is something related to migration itself and
possibly different needed parameters in 6.2 shipped version of libvirt
(0.9.4 vs 0.8.7).
possibly the answer is inside this thread, from a view of source
changes contained inside:
http://web.archiveorange.com/archive/v/8XiUW68vUOENyUOzpfSt
at line
[OP] Daniel P. Berrange Thu 12 May 2011 06:12:33 PM CEST
"
The migration protocol has support for a 'cookie' parameter which
is an opaque array of bytes as far as libvirt is concerned. Drivers
may use this for passing around arbitrary extra data they might
need during migration. The QEMU driver needs to do a few things:
- Pass hostname/uuid to allow strict protection against localhost
migration attempts
- Pass SPICE/VNC server port from the target back to the source to
allow seamless relocation of client sessions
- Pass lock driver state from source to destination
This patch introduces the basic glue for handling cookies
but only includes the host/guest UUID & name.
"
any way/option to bypass the parameter in virsh command?
Or any way to easily understand where in libvirt was added for rhel 6?
Thanks
Gianluca
4:18 a.m.
Just to confirm that upgrading another node to rh el 6.2, I'm able to
migrate 6.2 to 6.2 hosted vm.
Summarizing, starting from environment like this:
6.1 hosts
qemu-kvm-0.12.1.2-2.160.el6_1.8.x86_64
libvirt-0.8.7-18.el6_1.1.x86_64
kernel-2.6.32-131.17.1.el6.x86_64
6.2 hosts
qemu-kvm-0.12.1.2-2.209.el6_2.1.x86_64
libvirt-0.9.4-23.el6_2.1.x86_64
kernel-2.6.32-220.2.1.el6.x86_64
symbol "--->" corresponds to live migration
6.1 --> 6.1 ok
6.1 --> 6.2 ok
6.2 --> 6.2 ok
but
6.2 --> 6.1 ko
Gianluca
9:32 a.m.
On 01/18/2012 03:18 AM, Gianluca Cecchi wrote:
symbol "--->" corresponds to live migration
6.1 --> 6.1 ok
6.1 --> 6.2 ok
6.2 --> 6.2 ok
Yes, those are all supported migration patterns.
but
6.2 --> 6.1 ko
This is expected - there is no guarantee of downgrade compatibility,
only upgrade compatibility. The qemu on 6.2 has different machine
models for 6.1 vs. 6.2 hosting, so that it can continue to run a guest
migrated in from a 6.1 host. But if a guest is started from scratch on
6.2 rather than migrated in from 6.1, then chances are quite high that
the state of the machine will be using 6.2 features that are not
understood by 6.1, and thus you can't migrate to a downgraded qemu.
--
Eric Blake eblake(a)redhat.com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
9:51 a.m.
On Wed, Jan 18, 2012 at 4:32 PM, Eric Blake wrote:
On 01/18/2012 03:18 AM, Gianluca Cecchi wrote:
> symbol "--->" corresponds to live migration
>
> 6.1 --> 6.1 ok
> 6.1 --> 6.2 ok
> 6.2 --> 6.2 ok
Yes, those are all supported migration patterns.
>
> but
>
> 6.2 --> 6.1 ko
This is expected - there is no guarantee of downgrade compatibility,
only upgrade compatibility. The qemu on 6.2 has different machine
models for 6.1 vs. 6.2 hosting, so that it can continue to run a guest
migrated in from a 6.1 host. But if a guest is started from scratch on
6.2 rather than migrated in from 6.1, then chances are quite high that
the state of the machine will be using 6.2 features that are not
understood by 6.1, and thus you can't migrate to a downgraded qemu.
--
yes, the general concept is ok.
My concern was about guests that were born on 6.1 and after being
migrated to 6.2 were not able to come back again to 6.1
(in live migration way)
Possibly to cover a scenario where you migrate one host, then collapse
there again some machines, but due to problems (performance, hangs or
so on..) you are forced to come back.
You cannot do this live but giving downtime inside all the migrated guests.
And with 3 nodes as it is my case you have a little more flexibility
because you pass from a stage where you still have two 6.1 hosts.
In case of 2-nodes clusters you have to migrate all of the guests on
the upgraded host and then if something goes wrong you have to give
downtime for all the vms.
Probably a sort of compatible switch for pre-6.2 guests would have been good.
And note that if I shut down the guest on 6.2 and then power it on
again on 6.1 it boots without problem.
The guests' xml didn't change: it is still dated April 2011.
The limiting point is concerning live migration where there is this
hostuuid thing in 6.2 and not in 6.1...
Gianluca
8:13 a.m.
On Wed, Jan 18, 2012 at 04:51:53PM +0100, Gianluca Cecchi wrote:
On Wed, Jan 18, 2012 at 4:32 PM, Eric Blake wrote:
> On 01/18/2012 03:18 AM, Gianluca Cecchi wrote:
>> symbol "--->" corresponds to live migration
>>
>> 6.1 --> 6.1 ok
>> 6.1 --> 6.2 ok
>> 6.2 --> 6.2 ok
>
> Yes, those are all supported migration patterns.
>
>>
>> but
>>
>> 6.2 --> 6.1 ko
>
> This is expected - there is no guarantee of downgrade compatibility,
> only upgrade compatibility. The qemu on 6.2 has different machine
> models for 6.1 vs. 6.2 hosting, so that it can continue to run a guest
> migrated in from a 6.1 host. But if a guest is started from scratch on
> 6.2 rather than migrated in from 6.1, then chances are quite high that
> the state of the machine will be using 6.2 features that are not
> understood by 6.1, and thus you can't migrate to a downgraded qemu.
>
> --
yes, the general concept is ok.
My concern was about guests that were born on 6.1 and after being
migrated to 6.2 were not able to come back again to 6.1
(in live migration way)
Possibly to cover a scenario where you migrate one host, then collapse
there again some machines, but due to problems (performance, hangs or
so on..) you are forced to come back.
You cannot do this live but giving downtime inside all the migrated guests.
And with 3 nodes as it is my case you have a little more flexibility
because you pass from a stage where you still have two 6.1 hosts.
In case of 2-nodes clusters you have to migrate all of the guests on
the upgraded host and then if something goes wrong you have to give
downtime for all the vms.
Probably a sort of compatible switch for pre-6.2 guests would have been good.
And note that if I shut down the guest on 6.2 and then power it on
again on 6.1 it boots without problem.
The guests' xml didn't change: it is still dated April 2011.
The limiting point is concerning live migration where there is this
hostuuid thing in 6.2 and not in 6.1...
This is all as expected.
You can't live migrate from 6.2 to 6.1. You need to factor this into
your schedule for host upgrades.
You can copy a (shut down) VM from 6.2 to older RHEL releases and it
will work, provided the VM doesn't use any of the new features of RHEL 6.2.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top
5:41 a.m.
New subject: Span Port
Hi all,
i need your help!
Scenario: 3 Virtual Machine, one of this is an IDS system, so is there a
way to implement a span port (like those on bare-metal Cisco switches)
using some "Virtual Network" feature?
Any Suggestons?
Thanks in advantage.
My Server: Fedora 15, kernel i386 Linux server.virt-demo.it
2.6.41.9-1.fc15.i686.PAE, KVM
--
Andrea Modesto Rossi
Fedora Ambassador
8:36 a.m.
New subject: Span Port
----- Original Message -----
From: "Andrea Modesto Rossi" <amrossi(a)linux.it>
To: virt(a)lists.fedoraproject.org
Sent: Thursday, January 19, 2012 6:41:28 AM
Subject: [fedora-virt] Span Port
Hi all,
i need your help!
Scenario: 3 Virtual Machine, one of this is an IDS system, so is
there a
way to implement a span port (like those on bare-metal Cisco
switches)
using some "Virtual Network" feature?
Any Suggestons?
Have a look at the code here, it's a "custom hook" from VDSM, the management
agent in RHEV/oVirt that setups up a virtual mirror/span port for a virtual machine.
http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=tree;f=vdsm_hooks/promisc;h=0...
Thanks in advantage.
My Server: Fedora 15, kernel i386 Linux server.virt-demo.it
2.6.41.9-1.fc15.i686.PAE, KVM
You'll do much better with 64bit if your hardware supports it.
--
Andrea Modesto Rossi
Fedora Ambassador
_______________________________________________
virt mailing list
virt(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/virt
4513
days inactive
4518
days old
7 comments
5 participants
participants (5)
-
Andrea Modesto Rossi -
Andrew Cathrow -
Eric Blake -
Gianluca Cecchi -
Richard W.M. Jones