Neil

Thanks for your speedy reply! Comments in-line.

Neil Horman wrote:

On Wed, May 13, 2009 at 12:34:36PM -0400, Richard Achmatowicz wrote:

...

Hello

I'm using Fedora 8 but I have the same problem on RHEL 5. Before I submit this issue as a bug, I wanted to check if my understanding is not flawed in some way.

For some reason, when I create an IPv6 address (global or link-local) on interface eth0, three related routs are added:

3ffe:ffff:100:f101::/64 * U 256 0 0 eth0 3ffe:ffff:100:f101::/128 * U 0 0 1 lo 3ffe:ffff:100:f101::1/128 * U 0 0 1 lo

The latter of these, the most specific, drives all datagrams onto lo instead of eth0. I'm trying to find out why. I didn't ask for a route to lo to be created, so why is it being created?

This behavior is causing Sun JDK 6 to behave badly when working with IPv6 addresses in certain contexts (Sun bug 6800096).

Any ideas appreciated...A full example of what is happening is listed below.

Richard

This I think looks fairly normal. Its the mask value that makes all the difference. The first entry says anything going to the 3ffe:ffff:100:f101 subnet (with a 64 bit netmask) should go out go out eth0. The second and third entries say that anyting going to the addresses 3ffe:ffff:100:f101:: and 3ffe:ffff:100:f101::1 should go through lo. Since those two addresses are local to the system, they can be routed through the lo interface. No other addresses on that 64 bit network should match on that route however, since they're both masked at 128 bits. If anything but traffic to your local interfaces is matching on those routes, its a bug, but having traffic bound for your local addresses go through lo is fine.

What exactly is the behavior that you're seeing which is leading you to think that these routes are the cause?

I think I understand now why you have these 128 length prefix rules in the routing table. Thanks for the explanation. But is it really right to equate messages arriving at host X on lo with messages arriving at host X on eth0, which seems to be what the additional lo rules seem to assume? Processes can listen on either interface...and if I am listening on eth0 and messages arrive on lo, i'm not going to get them. Which seems to be what is happening below.

My original problem occurs with the Sun JDK and the handling of IPv6 zone ids for link-local addresses. I'm quoting now from Sun bug #6800096: <quote> In Linux, for link-local addresses, scope id is evaluated from the ipv6 routing table on the proc filesystem. /proc/net/ipv6_route has the entries for the ipv6 address and these entries have a mapping to the respective interface through which the traffic for that ip has to be routed through. Java is comparing the ipv6 address and gets its respective routing device name from these entries. The corresponding device index is retrieved from /proc/net/if_inet6. Problem here is that, on Linux, link-local address traffic is routed through loopback interface and hence the routing table entry for link-local address is mapped to "lo" device instead of "eth0". </quote> What happens in practice is if I have link-local address, say fe80::215:58ff:fec8:81a8 defined on interface eth0, when I try to create a Java socket to connect to a process listening on that address, say at port 1234, something like this:

InetAddress bind_addr=InetAddress.getByName("fe80::215:58ff:fec8:81a8%eth0") ; int port = 1234 ; Socket sock=new Socket(); sock.connect(new InetSocketAddress(bind_addr, port), sock_conn_timeout);

the Sun JDK seems to ignore the link-local address' scope id I pass it and instead tries to pick it up from the route as described above (which isn't such a good idea as as far as I understand, a single host can have two interfaces with the same link-local address anyway, so using the link-local address to identify a scope id shouldn't work in general). So my socket tries to connect via the interface lo instead of the interface eth0, and it finds no process listening on lo (my server process is listening on eth0). Running the program with strace, you see something like this when the socket tries to connect:

[pid 8997] socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = 4 [pid 8997] connect(4, {sa_family=AF_INET6, sin6_port=htons(1234), inet_pton(AF_INET6, "fe80::215:58ff:fec8:81a8", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=if_nametoindex("lo")}, 28) = -1 EINPROGRESS (Operation now in progress) [pid 8997] shutdown(4, 2 /* send and receive */) = 0 exception connecting to host: /fe80:0:0:0:215:58ff:fec8:81a8%eth0, port: 1234java.net.SocketTimeoutException: connect timed out java.net.SocketTimeoutException: connect timed out

You can see that the scope id lo is getting picked up, even though I passed a scoped link-local address. Sun admits this is a bug, but just haven't gotten around to fixing it yet. :-(

If I had a means to remove the routes

3ffe:ffff:100:f101::/128 * U 0 0 1 lo 3ffe:ffff:100:f101::1/128 * U 0 0 1 lo

from my routing table, I could most likely work around the problem. But when I try to remove them I get # route -A inet6 del 3ffe:ffff:100:f101::1/128 dev lo SIOCDELRT: No such process

Is there anyway I can get rid of these two lo routes?

Richard

Regards Neil

...

https://www.redhat.com/mailman/listinfo/fedora-kernel-list%eth0

Neil Horman wrote:

On Wed, May 13, 2009 at 03:11:10PM -0400, Richard Achmatowicz wrote:

...

Neil

Thanks for your speedy reply! Comments in-line.

Neil Horman wrote:

...

On Wed, May 13, 2009 at 12:34:36PM -0400, Richard Achmatowicz wrote:

...

Hello

I'm using Fedora 8 but I have the same problem on RHEL 5. Before I submit this issue as a bug, I wanted to check if my understanding is not flawed in some way.

For some reason, when I create an IPv6 address (global or link-local) on interface eth0, three related routs are added:

3ffe:ffff:100:f101::/64 * U 256 0 0 eth0 3ffe:ffff:100:f101::/128 * U 0 0 1 lo 3ffe:ffff:100:f101::1/128 * U 0 0 1 lo

The latter of these, the most specific, drives all datagrams onto lo instead of eth0. I'm trying to find out why. I didn't ask for a route to lo to be created, so why is it being created?

This behavior is causing Sun JDK 6 to behave badly when working with IPv6 addresses in certain contexts (Sun bug 6800096).

Any ideas appreciated...A full example of what is happening is listed below.

Richard

This I think looks fairly normal. Its the mask value that makes all the difference. The first entry says anything going to the 3ffe:ffff:100:f101 subnet (with a 64 bit netmask) should go out go out eth0. The second and third entries say that anyting going to the addresses 3ffe:ffff:100:f101:: and 3ffe:ffff:100:f101::1 should go through lo. Since those two addresses are local to the system, they can be routed through the lo interface. No other addresses on that 64 bit network should match on that route however, since they're both masked at 128 bits. If anything but traffic to your local interfaces is matching on those routes, its a bug, but having traffic bound for your local addresses go through lo is fine.

What exactly is the behavior that you're seeing which is leading you to think that these routes are the cause?

I think I understand now why you have these 128 length prefix rules in the routing table. Thanks for the explanation. But is it really right to equate messages arriving at host X on lo with messages arriving at host X on eth0, which seems to be what the additional lo rules seem to assume? Processes can listen on either interface...and if I am listening on eth0 and messages arrive on lo, i'm not going to get them. Which seems to be what is happening below.

I'm glad the explination helped. To answer your subsequent question, its fine for the routing table to do what its doing above. You're application should be able to receive them just fine regardless of which interface they arrive on. If thats not happening, that may be a bug. I assume that you are binding your applications sockets to INADDR_ANY, or are you selecting a specific address to listen on? If so, that may be your problem.

I'm going to have to think about this one a little more before I answer. :-)

...

My original problem occurs with the Sun JDK and the handling of IPv6 zone ids for link-local addresses. I'm quoting now from Sun bug #6800096:

<quote> In Linux, for link-local addresses, scope id is evaluated from the ipv6 routing table on the proc filesystem. /proc/net/ipv6_route has the entries for the ipv6 address and these entries have a mapping to the respective interface through which the traffic for that ip has to be routed through. Java is comparing the ipv6 address and gets its respective routing device name from these entries. The corresponding device index is retrieved from /proc/net/if_inet6. Problem here is that, on Linux, link-local address traffic is routed through loopback interface and hence the routing table entry for link-local address is mapped to "lo" device instead of "eth0". </quote> What happens in practice is if I have link-local address, say fe80::215:58ff:fec8:81a8 defined on interface eth0, when I try to create a Java socket to connect to a process listening on that address, say at port 1234, something like this:

InetAddress bind_addr=InetAddress.getByName("fe80::215:58ff:fec8:81a8%eth0") ; int port = 1234 ; Socket sock=new Socket(); sock.connect(new InetSocketAddress(bind_addr, port), sock_conn_timeout);

the Sun JDK seems to ignore the link-local address' scope id I pass it and instead tries to pick it up from the route as described above (which isn't such a good idea as as far as I understand, a single host can have two interfaces with the same link-local address anyway, so using the link-local address to identify a scope id shouldn't work in general). So

Yeah, thats broken, the JDK shouldn't do that, for exactly the reason you describe.

...

my socket tries to connect via the interface lo instead of the interface eth0, and it finds no process listening on lo (my server process is listening on eth0). Running the program with strace, you see something like this when the socket tries to connect:

[pid 8997] socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = 4 [pid 8997] connect(4, {sa_family=AF_INET6, sin6_port=htons(1234), inet_pton(AF_INET6, "fe80::215:58ff:fec8:81a8", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=if_nametoindex("lo")}, 28) = -1 EINPROGRESS (Operation now in progress) [pid 8997] shutdown(4, 2 /* send and receive */) = 0 exception connecting to host: /fe80:0:0:0:215:58ff:fec8:81a8%eth0, port: 1234java.net.SocketTimeoutException: connect timed out java.net.SocketTimeoutException: connect timed out

You can see that the scope id lo is getting picked up, even though I passed a scoped link-local address. Sun admits this is a bug, but just haven't gotten around to fixing it yet. :-(

If I had a means to remove the routes

3ffe:ffff:100:f101::/128 * U 0 0 1 lo 3ffe:ffff:100:f101::1/128 * U 0 0 1 lo

from my routing table, I could most likely work around the problem. But when I try to remove them I get # route -A inet6 del 3ffe:ffff:100:f101::1/128 dev lo SIOCDELRT: No such process

Is there anyway I can get rid of these two lo routes?

Thats odd, that should work, can you try to remove them with the iproute2 utility? Something like this should work: /sbin/ip -6 route del 3ffe:ffff:100:f101::/128

Getting the same response from iproute2:

[root@localhost nrla]# /sbin/ip -V ip utility, iproute2-ss080725

[root@localhost nrla]# route -A inet6 Kernel IPv6 routing table Destination Next Hop Flags Metric Ref Use Iface 3ffe:ffff:100:f101::/64 * U 256 0 0 eth0 fe80::/64 * U 256 0 0 eth0 localhost6.localdomain6/128 * U 0 1 1 lo 3ffe:ffff:100:f101::/128 * U 0 0 1 lo 3ffe:ffff:100:f101::1/128 * U 0 0 1 lo fe80::/128 * U 0 0 1 lo lenovo6/128 * U 0 0 1 lo ff00::/8 * U 256 0 0 eth0

[root@localhost nrla]# /sbin/ip -6 route del 3ffe:ffff:100:f101::/128 RTNETLINK answers: No such process

[root@localhost nrla]# uname -a Linux localhost.localdomain 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 19:19:45 EST 2008 i686 i686 i386 GNU/Linux

And the routing table givcn by iproute2 looks a lot different from that give by route:

[root@localhost nrla]# ip -6 route show unreachable ::/96 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295 unreachable ::ffff:0.0.0.0/96 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295 unreachable 2002:a00::/24 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295 unreachable 2002:7f00::/24 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295 unreachable 2002:a9fe::/32 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295 unreachable 2002:ac10::/28 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295 unreachable 2002:c0a8::/32 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295 unreachable 2002:e000::/19 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295 3ffe:ffff:100:f101::/64 dev eth0 metric 256 mtu 1500 advmss 1440 hoplimit 4294967295 unreachable 3ffe:ffff::/32 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295 fe80::/64 dev eth0 metric 256 mtu 1500 advmss 1440 hoplimit 4294967295

I get the same behaviour on a RHEL machine:

-bash-3.2$ uname -a Linux jawa01.englab.brq.redhat.com 2.6.18-53.1.14.el5xen #1 SMP Tue Feb 19 07:33:17 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

...

Richard

...

Regards Neil

...

https://www.redhat.com/mailman/listinfo/fedora-kernel-list%eth0

On Thu, May 14, 2009 at 01:46:42PM -0400, Neil Horman wrote:

...

3ffe:ffff:100:f101::/64 dev eth0 metric 256 mtu 1500 advmss 1440 hoplimit 4294967295 unreachable 3ffe:ffff::/32 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 4294967295

3ffe::/16 is phased out as of June 6, 2006:

http://tools.ietf.org/html/rfc3701

Additionally, 3ffe:ffff::/32 was a "documentation prefix" to be used in documentation examples, and hence never used on actual networks. Linux is enforcing that with the unreachable route. Unfortunately, all I can find about this is an old expired IETF draft from 2001:

http://www.hexago.com/4105/file.asp?file_id=421 http://mailman.isi.edu/pipermail/6bone/2001-June/004262.html