1:51 p.m.
The other day I was running the stock fedora kernel on my ip
forwarding setup, to see what the performance was, and the performance
wasn't very good.
system is S5520HC dual socket 2.93GHz Xeon 5570 (Nehalem) with 3 quad
port 82580 adapters (12 ports). Traffic is bidirectional 64 byte
packets being forwarded and received on each port, basically port to
port routing. I am only using 12 flows currently.
The driver is igb, and I am using an affinity script that lines up
each pair of ports that are forwarding traffic into optimal
configurations for cache locality. I am also disabling
remote_node_defrag_ratio to stop cross node traffic.
With the fedora default kernel from F14 it appears that
CONFIG_NETFILTER=y means that I cannot unload all of netfilter even if
I stop iptables service.
perf showed netfilter being prominent, and removing it gives me much
higher throughput. Is there a reason CONFIG_NETFILTER=y ? Isn't it a
good thing to be able to disable netfilter if you want to?
Jesse
2:12 p.m.
From: Jesse Brandeburg <jesse.brandeburg(a)gmail.com>
Date: Wed, 6 Apr 2011 11:51:06 -0700
perf showed netfilter being prominent, and removing it gives me much
higher throughput. Is there a reason CONFIG_NETFILTER=y ? Isn't it a
good thing to be able to disable netfilter if you want to?
CONFIG_NETFILTER=y is the only possible non-disabled setting, as it's
a boolean.
All that is enabling is the hooks, and it's not possible to provide
netfilter at all without the hooks being compiled into the kernel
image.
Fedora by default is also probably installing several defauly
netfilter rules, make sure to remove them.
2:57 p.m.
Em Wed, Apr 06, 2011 at 12:12:08PM -0700, David Miller escreveu:
From: Jesse Brandeburg <jesse.brandeburg(a)gmail.com>
Date: Wed, 6 Apr 2011 11:51:06 -0700
> perf showed netfilter being prominent, and removing it gives me much
> higher throughput. Is there a reason CONFIG_NETFILTER=y ? Isn't it a
> good thing to be able to disable netfilter if you want to?
CONFIG_NETFILTER=y is the only possible non-disabled setting, as it's
a boolean.
All that is enabling is the hooks, and it's not possible to provide
netfilter at all without the hooks being compiled into the kernel
image.
Fedora by default is also probably installing several defauly
netfilter rules, make sure to remove them.
Was there ever any attempt to try to have netfilter built-in but not
causing the performance impact?
Something like ftrace code changing when the user inserts the first
rule?
People wanting top performance disable it in the build, but thos wanting
to stick to vendor provided kernels don't have that choice :)
- Arnaldo
3:02 p.m.
From: Arnaldo Carvalho de Melo <acme(a)ghostprotocols.net>
Date: Wed, 6 Apr 2011 16:57:19 -0300
Something like ftrace code changing when the user inserts the first
rule?
People wanting top performance disable it in the build, but thos wanting
to stick to vendor provided kernels don't have that choice :)
Using ftrace-like stubs would be an interesting idea, and I highly encourage
people to work on something like that.
However I want to reiterate that I think that real rules are installed
in Jesse's case, and once he removes those the majority of the
overhead will disappear. The FC14 workstation I'm using right now, on
which I've made no modifications to the installer's netfilter settings,
has the following rules:
--------------------
[root@ilbolle davem]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
ACCEPT udp -- anywhere anywhere state NEW udp dpt:ipp
ACCEPT udp -- anywhere 224.0.0.251 state NEW udp dpt:mdns
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ipp
ACCEPT udp -- anywhere anywhere state NEW udp dpt:ipp
REJECT all -- anywhere anywhere reject-with
icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- anywhere anywhere reject-with
icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root@ilbolle davem]#
--------------------
I suspect Jesse has something similar on his test box.
When no rules are loaded, all the stubs make happen is a function call
plus a list_empty() check. Nothing more. I really can't see that, all
by itself, obliterating routing performance.
In fact I've done udp flood tests, as recently as a month ago, with just
NETFILTER=y and no rules installed, and the impact was minimal.
And that was on sparc64 where function calls are expensive :)
3:13 p.m.
Em Wed, Apr 06, 2011 at 01:02:39PM -0700, David Miller escreveu:
From: Arnaldo Carvalho de Melo <acme(a)ghostprotocols.net>
Date: Wed, 6 Apr 2011 16:57:19 -0300
> Something like ftrace code changing when the user inserts the first
> rule?
>
> People wanting top performance disable it in the build, but thos wanting
> to stick to vendor provided kernels don't have that choice :)
Using ftrace-like stubs would be an interesting idea, and I highly encourage
people to work on something like that.
cool, these code modification and JIT mechanizms open up a lot of
possibilities indeed ;-)
However I want to reiterate that I think that real rules are
installed
in Jesse's case, and once he removes those the majority of the
overhead will disappear. The FC14 workstation I'm using right now, on
which I've made no modifications to the installer's netfilter settings,
has the following rules:
<SNIP>
I suspect Jesse has something similar on his test box.
When no rules are loaded, all the stubs make happen is a function call
plus a list_empty() check. Nothing more. I really can't see that, all
by itself, obliterating routing performance.
Yeah, would be nice, since he is playing with it, for him to post
numbers about the overheads.
In fact I've done udp flood tests, as recently as a month ago,
with just
NETFILTER=y and no rules installed, and the impact was minimal.
And that was on sparc64 where function calls are expensive :)
:-)
- Arnaldo
3:18 p.m.
Le mercredi 06 avril 2011 à 13:02 -0700, David Miller a écrit :
From: Arnaldo Carvalho de Melo <acme(a)ghostprotocols.net>
Date: Wed, 6 Apr 2011 16:57:19 -0300
> Something like ftrace code changing when the user inserts the first
> rule?
>
> People wanting top performance disable it in the build, but thos wanting
> to stick to vendor provided kernels don't have that choice :)
Using ftrace-like stubs would be an interesting idea, and I highly encourage
people to work on something like that.
However I want to reiterate that I think that real rules are installed
in Jesse's case, and once he removes those the majority of the
overhead will disappear. The FC14 workstation I'm using right now, on
which I've made no modifications to the installer's netfilter settings,
has the following rules:
--------------------
[root@ilbolle davem]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh
ACCEPT udp -- anywhere anywhere state NEW udp dpt:ipp
ACCEPT udp -- anywhere 224.0.0.251 state NEW udp dpt:mdns
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ipp
ACCEPT udp -- anywhere anywhere state NEW udp dpt:ipp
REJECT all -- anywhere anywhere reject-with
icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- anywhere anywhere reject-with
icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
[root@ilbolle davem]#
--------------------
I suspect Jesse has something similar on his test box.
I suspect problem is worse than that.
I remember last time I work on a fedora kernel, it had conntrack enabled
And yes, conntrack can really slowdown a router, because of default
parameters.
cat /proc/sys/net/nf_conntrack_max
3:29 p.m.
From: Eric Dumazet <eric.dumazet(a)gmail.com>
Date: Wed, 06 Apr 2011 22:18:56 +0200
I remember last time I work on a fedora kernel, it had conntrack
enabled
And yes, conntrack can really slowdown a router, because of default
parameters.
Yes, if conntrack is enabled performance will indeed tank.
3:32 p.m.
Le mercredi 06 avril 2011 à 13:29 -0700, David Miller a écrit :
From: Eric Dumazet <eric.dumazet(a)gmail.com>
Date: Wed, 06 Apr 2011 22:18:56 +0200
> I remember last time I work on a fedora kernel, it had conntrack enabled
>
> And yes, conntrack can really slowdown a router, because of default
> parameters.
Yes, if conntrack is enabled performance will indeed tank.
I just check on latest fedora14 (yum updated) :
- conntrack is statically included in kernel, you cannot remove it.
Hmm...
11:40 p.m.
On Wed, 6 Apr 2011, Eric Dumazet wrote:
Le mercredi 06 avril 2011 à 13:29 -0700, David Miller a écrit :
> From: Eric Dumazet <eric.dumazet(a)gmail.com>
> Date: Wed, 06 Apr 2011 22:18:56 +0200
>
> > I remember last time I work on a fedora kernel, it had conntrack enabled
> >
> > And yes, conntrack can really slowdown a router, because of default
> > parameters.
>
> Yes, if conntrack is enabled performance will indeed tank.
I just check on latest fedora14 (yum updated) :
- conntrack is statically included in kernel, you cannot remove it.
Hmm...
Thanks for the replies,
Yes indeed, I bet that conntrack was the item that caused the grief I saw.
In fact turning off CONFIG_NETFILTER disables conntrack and solved my
problem. :-)
Now the question is how to get the netfiltery/conntracky goodness without
impacting those who want to go fast, maybe without a helmet.
4:11 p.m.
Le mercredi 06 avril 2011 à 14:08 -0700, Brandeburg, Jesse a écrit :
Thanks for the replies,
Yes indeed, I bet that conntrack was the item that caused the grief I saw.
In fact turning off CONFIG_NETFILTER disables conntrack and solved my
problem. :-)
Now the question is how to get the netfiltery/conntracky goodness without
impacting those who want to go fast, maybe without a helmet.
Thats pretty simple, just load netfilter modules if necessary.
If not loaded, cost is pretty small, being predicted branches.
3246
days inactive
4767
days old
kernel@lists.fedoraproject.org
9 comments
5 participants
participants (5)
-
Arnaldo Carvalho de Melo -
Brandeburg, Jesse -
David Miller -
Eric Dumazet -
Jesse Brandeburg