7:02 a.m.
Meta question: What is the best upstream list for discussing ARM & KVM
issues? Or ARM & virt issues in general?
I'm trying to get stock qemu from git or qemu 1.6 to boot a guest using KVM.
My host has a 3.11.0 LPAE kernel, and it exposes a /dev/kvm device.
qemu can open this device, it gets as far as creating a KVM guest, and
that is all apparently successful. But the ioctl KVM_RUN returns -ENXIO.
ENXIO is not a documented return from the KVM_RUN API but I've looked
at the kernel code and I can't see where KVM_RUN would return ENXIO.
All places in general where ENXIO might be returned are related to the
function "irqchip_in_kernel" and the mysterious "VGIC" hardware
(which
I guess is doing interrupt routing like APIC on PCs?)
Do I have to add one of these devices to the emulation perhaps?
Does anyone have a qemu command line which reliably boots a guest using KVM?
Rich.
/usr/bin/qemu-system-arm \
-global virtio-blk-device.scsi=off \
-nodefconfig \
-nodefaults \
-nographic \
-M vexpress-a9 \
-machine accel=kvm:tcg \
-m 500 \
-no-reboot \
-kernel /home/rjones/d/libguestfs/tmp/.guestfs-1001/kernel.3268 \
-dtb /home/rjones/d/libguestfs/tmp/.guestfs-1001/dtb.3268 \
-initrd /home/rjones/d/libguestfs/tmp/.guestfs-1001/initrd.3268 \
-device virtio-scsi-device,id=scsi \
-drive
file=/home/rjones/d/libguestfs/tmp/libguestfsFZio45/scratch.1,cache=unsafe,format=raw,id=hd0,if=none
\
-device scsi-hd,drive=hd0 \
-drive
file=/home/rjones/d/libguestfs/tmp/.guestfs-1001/root.3268,snapshot=on,id=appliance,cache=unsafe,if=none
\
-device scsi-hd,drive=appliance \
-device virtio-serial-device \
-serial stdio \
-chardev
socket,path=/home/rjones/d/libguestfs/tmp/libguestfsFZio45/guestfsd.sock,id=channel0 \
-device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
-append 'panic=1 console=ttyAMA0 udevtimeout=600 no_timer_check acpi=off
printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1
TERM=xterm-256color'
error: kvm run failed No such device or address
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
7:16 a.m.
On Sat, Sep 07, 2013 at 01:02:39PM +0100, Richard W.M. Jones wrote:
All places in general where ENXIO might be returned are related to
the
function "irqchip_in_kernel" and the mysterious "VGIC" hardware
(which
I guess is doing interrupt routing like APIC on PCs?)
The solution was some creative grepping of the qemu sources:
-machine accel=kvm:tcg,kernel_irqchip=off
Works great (except of course virtio-serial is still broken). I seem
to be getting there, slowly ...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
11:27 a.m.
On 09/07/2013 08:16 AM, Richard W.M. Jones wrote:
On Sat, Sep 07, 2013 at 01:02:39PM +0100, Richard W.M. Jones wrote:
> All places in general where ENXIO might be returned are related to the
> function "irqchip_in_kernel" and the mysterious "VGIC" hardware
(which
> I guess is doing interrupt routing like APIC on PCs?)
The solution was some creative grepping of the qemu sources:
-machine accel=kvm:tcg,kernel_irqchip=off
Works great (except of course virtio-serial is still broken). I seem
to be getting there, slowly ...
Rich.
Hi Rich,
Thanks for the tip above. I was able to get virtualization running on my Samsung ARM
chromebook (using a nv-uboot that turns on HYP and a locally built 3.11 kernel with a
patch).
https://fedoraproject.org/wiki/Architectures/ARM/F18/Versatile_Express
This particular image uses a 3.6.10 linux kernel. The URL above mentions that DTB files
are needed for newer kernels. Is there a particular DTB file that would be best to use?
There seems to be several vexpress*.dtb files to choose from.
-Will
11:41 a.m.
On Tue, Sep 10, 2013 at 12:27:48PM -0400, William Cohen wrote:
On 09/07/2013 08:16 AM, Richard W.M. Jones wrote:
> On Sat, Sep 07, 2013 at 01:02:39PM +0100, Richard W.M. Jones wrote:
>> All places in general where ENXIO might be returned are related to the
>> function "irqchip_in_kernel" and the mysterious "VGIC"
hardware (which
>> I guess is doing interrupt routing like APIC on PCs?)
>
> The solution was some creative grepping of the qemu sources:
>
> -machine accel=kvm:tcg,kernel_irqchip=off
>
> Works great (except of course virtio-serial is still broken). I seem
> to be getting there, slowly ...
>
> Rich.
>
Hi Rich,
Thanks for the tip above. I was able to get virtualization running on my Samsung ARM
chromebook (using a nv-uboot that turns on HYP and a locally built 3.11 kernel with a
patch).
Interesting .. which/what is "nv-uboot" and where did you get it from?
https://fedoraproject.org/wiki/Architectures/ARM/F18/Versatile_Express
This particular image uses a 3.6.10 linux kernel. The URL above mentions that DTB files
are needed for newer kernels. Is there a particular DTB file that would be best to use?
There seems to be several vexpress*.dtb files to choose from.
I used:
qemu-system-arm \
-M vexpress-a9 \
-dtb /boot/dtb-....../vexpress-v2p-ca9.dtb \
-machine accel=kvm:tcg,kernel_irqchip=off \
[etc]
You can find the full command line that libguestfs passes here:
https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c#...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
1:32 p.m.
On 09/10/2013 12:41 PM, Richard W.M. Jones wrote:
On Tue, Sep 10, 2013 at 12:27:48PM -0400, William Cohen wrote:
> On 09/07/2013 08:16 AM, Richard W.M. Jones wrote:
>> On Sat, Sep 07, 2013 at 01:02:39PM +0100, Richard W.M. Jones wrote:
>>> All places in general where ENXIO might be returned are related to the
>>> function "irqchip_in_kernel" and the mysterious "VGIC"
hardware (which
>>> I guess is doing interrupt routing like APIC on PCs?)
>>
>> The solution was some creative grepping of the qemu sources:
>>
>> -machine accel=kvm:tcg,kernel_irqchip=off
>>
>> Works great (except of course virtio-serial is still broken). I seem
>> to be getting there, slowly ...
>>
>> Rich.
>>
>
>
> Hi Rich,
>
> Thanks for the tip above. I was able to get virtualization running on my Samsung ARM
chromebook (using a nv-uboot that turns on HYP and a locally built 3.11 kernel with a
patch).
Interesting .. which/what is "nv-uboot" and where did you get it from?
Hi Rich,
I used the u-boot described in
http://www.virtualopensystems.com/media/chromebook/chromebook.pdf:
git://github.com/virtualopensystems/u-boot.git
This u-boot doesn't seem to understand initramfs, so the kernel I am using has some
things compiled in rather than modules.
For the linux-3.11 kernel I also had to use a patch based on:
https://github.com/virtualopensystems/linux-kvm-arm/commit/32682304c9935d...
Peter Robinson mentioned there are some other patches submitted for review that are
suppose to accomplished the same.
> https://fedoraproject.org/wiki/Architectures/ARM/F18/Versatile_Express
>
> This particular image uses a 3.6.10 linux kernel. The URL above mentions that DTB
files are needed for newer kernels. Is there a particular DTB file that would be best to
use? There seems to be several vexpress*.dtb files to choose from.
I used:
qemu-system-arm \
-M vexpress-a9 \
-dtb /boot/dtb-....../vexpress-v2p-ca9.dtb \
-machine accel=kvm:tcg,kernel_irqchip=off \
[etc]
Okay I will try "-M vexpress-a9 -dtb ..vexpress-v2p-ca9.dtb"
You can find the full command line that libguestfs passes here:
https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c#...
Rich.
I tried to run a couple of the systemtap virtualization examples on the host
(https://sourceware.org/systemtap/examples/keyword-index.html#VIRTUALIZATION). However,
it looks like the some of the kernel tracepoint that are available for x86:
kernel.trace("kvm_entry") and kernel.trace("kvm_exit") are not
available on my kernel. However, I do see the trace points defined on:
http://lxr.linux.no/#linux+v3.11/arch/arm/kvm/trace.h#L12 And look to be used in the 3.11
kernel.
trace_kvm_entry: http://lxr.linux.no/#linux+v3.11/arch/arm/kvm/arm.c#L568
trace_kvm_exit: http://lxr.linux.no/#linux+v3.11/arch/arm/kvm/arm.c#L577
Maybe the kernel doesn't have some config set correctly. There are some
kernel.trace("kvm_*") that are listed on the host kernel:
$ sudo stap -L 'kernel.trace("kvm_*")'
kernel.trace("kvm_ack_irq") $irqchip:unsigned int $pin:unsigned int
kernel.trace("kvm_age_page") $hva:ulong $slot:struct kvm_memory_slot* $ref:int
kernel.trace("kvm_fpu") $load:int
kernel.trace("kvm_mmio") $type:int $len:int $gpa:u64 $val:u64
kernel.trace("kvm_set_irq") $gsi:unsigned int $level:int $irq_source_id:int
kernel.trace("kvm_userspace_exit") $reason:__u32 $errno:int
The kvm trace event that were found are defined in
http://lxr.linux.no/#linux+v3.11/include/trace/events/kvm.h
-Will
3:30 p.m.
On Tue, Sep 10, 2013 at 02:32:55PM -0400, William Cohen wrote:
[...]
Thanks for the links. I will get back to ARM once I've got libguestfs
working on PPC64 ...
In the meantime:
I tried to run a couple of the systemtap virtualization examples on
the host
(https://sourceware.org/systemtap/examples/keyword-index.html#VIRTUALIZATION).
However, it looks like the some of the kernel tracepoint that are
available for x86: kernel.trace("kvm_entry") and
kernel.trace("kvm_exit") are not available on my kernel. However, I
do see the trace points defined on:
http://lxr.linux.no/#linux+v3.11/arch/arm/kvm/trace.h#L12 And look
to be used in the 3.11 kernel.
ARM KVM seems to be completely different from x86 KVM (and
incidentally different from PPC's 2 different types of KVM). They all
seem to just share a rough outline of userspace API.
Anyway, I was going to suggest that you use libguestfs to test KVM on
ARM. It's extremely easy to fire up a VM (eg: 'libguestfs-test-tool'
or 'guestfish run ...').
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
3874
days inactive
3877
days old
5 comments
2 participants
participants (2)
-
Richard W.M. Jones -
William Cohen