Today Salman Zafar and I observed something that we hadn't noticed
before (not sure if it didn't happen, or we didn't notice it) -- the
PandaBoards in the build farm have high latency, with occasional packet
drops, when pinged at 1-second intervals (the default for the 'ping'
command).
Oddly, when flood-pinged, they don't drop any packets, and the latency
improves. This sounds a lot like the storage performance bug, without
the storage :-) Perhaps an IRQ is being missed, and a subsequent IRQ
(or polling event or something else) is causing the driver to pick up
the data late? (Just speculating)
Regular ping ...
-------------------------------------------
$ ping -c20 5-1
PING cdot-panda-5-1 (192.168.1.151) 56(84) bytes of data.
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=1 ttl=64
time=0.753 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=2 ttl=64
time=49.8 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=3 ttl=64 time=128
ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=4 ttl=64
time=8.21 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=5 ttl=64 time=233
ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=6 ttl=64 time=646
ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=7 ttl=64
time=0.609 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=8 ttl=64 time=248
ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=9 ttl=64 time=318
ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=10 ttl=64
time=267 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=11 ttl=64
time=267 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=12 ttl=64
time=268 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=13 ttl=64
time=266 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=14 ttl=64
time=265 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=15 ttl=64
time=153 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=16 ttl=64
time=0.632 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=17 ttl=64
time=61.6 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=18 ttl=64
time=0.628 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=19 ttl=64
time=75.9 ms
64 bytes from cdot-panda-5-1 (192.168.1.151): icmp_req=20 ttl=64
time=23.6 ms
--- cdot-panda-5-1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19021ms
rtt min/avg/max/mdev = 0.609/164.416/646.835/159.032 ms
-------------------------------------------
Notice the 164 mS average latency. No packet loss in this particular
sample, though.
Now a flood ping:
-------------------------------------------
# ping -f -c20 5-1
PING cdot-panda-5-1 (192.168.1.151) 56(84) bytes of data.
--- cdot-panda-5-1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 92ms
rtt min/avg/max/mdev = 0.373/4.666/19.813/6.597 ms, pipe 2, ipg/ewma
4.883/3.134 ms
-------------------------------------------
Results vary but are generally repeatable.
This kernel (which is a 2.6.35.3) doesn't appear to be built with the
appropriate /sys/kernel/debug/usbmon bits for tracing with tcpdump.
(Anyone know offhand the kernel flags for that?)
-Chris