Here is my askfedora post that didn't receive any reply.
https://ask.fedoraproject.org/en/question/103915/what-kernel-relates-to-irq-...
Description of the problem: In a network test setup, FC-23 (kernel-4.8) acts as bridge passing network traffic. TCP iperf tests are done from external (to the hypervisor) servers and clients for throughput 700Mbps. The bridge setup on the hypervisor is done through openvswitch and the VM relies on those bridge to pass traffic. Number of CPU core assigned to the FC VM is 2.
Two test cases considered: #1) irqbalance not running #2) irqbalance running
For test#1, CPU usage seen for the qemu-pid on the hypervisor stays close to 100% For test#2, CPU usage seen for the qemu-pid on the hypervisor goes to 200%, sometimes even more.
For test#1, `smp_affinity` uses default value 3, where as for #2, it uses different values for three different virtio-input interrupts, 1,1,2.
Question is, why turning on irqbalance (default for FC installation) makes it worse ?
On Wed, Apr 05, 2017 at 05:29:23PM -0000, I AM USER wrote:
Here is my askfedora post that didn't receive any reply.
https://ask.fedoraproject.org/en/question/103915/what-kernel-relates-to-irq-...
Description of the problem: In a network test setup, FC-23 (kernel-4.8) acts as bridge passing network traffic. TCP iperf tests are done from external (to the hypervisor) servers and clients for throughput 700Mbps. The bridge setup on the hypervisor is done through openvswitch and the VM relies on those bridge to pass traffic. Number of CPU core assigned to the FC VM is 2.
Two test cases considered: #1) irqbalance not running #2) irqbalance running
For test#1, CPU usage seen for the qemu-pid on the hypervisor stays close to 100% For test#2, CPU usage seen for the qemu-pid on the hypervisor goes to 200%, sometimes even more.
For test#1, `smp_affinity` uses default value 3, where as for #2, it uses different values for three different virtio-input interrupts, 1,1,2.
Question is, why turning on irqbalance (default for FC installation) makes it worse ?
Well, first and foremost, just because you have higher cpu utilization doesn't actually mean things are 'worse' - the first question I would ask is, are you getting better throughput with irqbalance on than with it off.
That said, given what you describe, and assuming that throughput is unchanged, I'd guess that you are (a) not using sriov (i.e. your using virtio or some other bridged virtual ethernet, for which some irq semantics are not what you would commonly expect, and (b) are not using cpu pinning, which means every time you try to generate an interrupt directed toward that guest, you may be doing it on a different cpu, which is the antithesis of what irqbalance hopes to achieve.
Long story short, given what your setup seems to be, you should disable irqbalance in the guest, theres no need for you to assign interrupt affinity, if you don't have a stable mapping of vcpu to real cpus.
Neil
kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org
Neil,
Thanks a lot for a pedagogical reply. You are correct on (b). I am not using CPU pinning, that's a requirement I can't change. For (a), I didn't understand your point about 'sriov' ? What is that ? I am using virtio driver for the bridged ethernet.
I see what you are saying about irqbalance. Can I ask, if there are any formal recommendation on CPU pinning for Fedora or red hat derivatives ?
In a follow up question, what kernel config options are desirable for non-CPU pinning ?
On Wed, Apr 05, 2017 at 07:06:18PM -0000, I AM USER wrote:
Neil,
Thanks a lot for a pedagogical reply. You are correct on (b). I am not using CPU pinning, that's a requirement I can't change. For (a), I didn't understand your point about 'sriov' ? What is that ? I am using virtio driver for the bridged ethernet.
I figured, I just wanted to ask. SRIOV is a technology that allows you as host administrator to "pass through" a pci function directly to a guess, so as to accelerate network traffic from said guest to the wire. It creates a situation in which irqs can be handled directly by the guest, and is cause to potentially run irqbalance in the guest.
Since you are using virtio though, you really don't need irqbalance there
I see what you are saying about irqbalance. Can I ask, if there are any formal recommendation on CPU pinning for Fedora or red hat derivatives ?
That would be a question for people working on the virt bits. I think the best answer I can give is "it depends on your workload"
In a follow up question, what kernel config options are desirable for non-CPU pinning ?
None that can be asserted universally with the information given.
kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org
On Wed, Apr 05, 2017 at 08:07:02PM -0000, I AM USER wrote:
Since you are using virtio though, you really don't need irqbalance there
Do you have a doc. to go along those lines ? I couldn't get anything helpful from the web. Thanks a lot.
only the whitepapers on access.redhat.com
kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org
kernel@lists.fedoraproject.org
-
I AM USER -
Neil Horman