8:46 a.m.
Hi,
I have an E3-1240 fedora37 postfix system using SSDs connected to a cable
modem that's having problems with dropped packets. There is one other
fedora37 server (E5-1650) directly connected to the cable modem that is not
having the same problem, although it's just routing packets, not really
doing much processing of data. The server with the problem is using
libreswan to create a VPN between itself and an i7-7700K with fedora37
managed at OVH, thinking it would be more resilient than the cable
connection itself. The problem also happens without the VPN, but perhaps
not to the same degree.
I think the server is certainly powerful enough to process the amount of
DNS queries, but there's also a lot of timeouts.
How do I troubleshoot this? It's the bridge experiencing the dropped
packets, not the interface itself, it seems?
They're not packet errors - just dropped packets. Perhaps the processor
can't handle the traffic?
Do you have a server connected to a cable modem? Is there something about
it being a cable connection that could be causing this?
There's about 27k dropped packets in about 12 hours of uptime.
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 68.195.111.45 netmask 255.255.255.248 broadcast 68.195.111.47
ether ae:64:2c:25:b5:44 txqueuelen 1000 (Ethernet)
RX packets 11355786 bytes 21634260431 (20.1 GiB)
RX errors 0 dropped 26349 overruns 0 frame 0
TX packets 7374430 bytes 993860294 (947.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 14:da:e9:97:ab:72 txqueuelen 1000 (Ethernet)
RX packets 16224042 bytes 22179550934 (20.6 GiB)
RX errors 0 dropped 165 overruns 0 frame 0
TX packets 7493927 bytes 1031987928 (984.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17 memory 0xdf100000-df120000
10:35 a.m.
Hi,
Doing a bit more searching, I've located this from nearly 10 years ago that
describes my problem exactly:
https://hardforum.com/threads/linux-network-bridge-dropping-rx-packets.17...
I've confirmed that ipv6 isn't disabled:
sysctl -a|grep ipv6|grep disable
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.disable_policy = 0
net.ipv6.conf.br0.disable_ipv6 = 0
net.ipv6.conf.br0.disable_policy = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.disable_policy = 0
net.ipv6.conf.enp4s0.disable_ipv6 = 0
net.ipv6.conf.enp4s0.disable_policy = 0
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.eth0.disable_policy = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.disable_policy = 0
Equally interesting is that this problem doesn't happen when tcpdump is
running. I've also disabled my iptables packet filter and it still happens.
Any ideas greatly appreciated.
On Sat, Jun 3, 2023 at 9:46 AM Alex <mysqlstudent(a)gmail.com> wrote:
Hi,
I have an E3-1240 fedora37 postfix system using SSDs connected to a cable
modem that's having problems with dropped packets. There is one other
fedora37 server (E5-1650) directly connected to the cable modem that is not
having the same problem, although it's just routing packets, not really
doing much processing of data. The server with the problem is using
libreswan to create a VPN between itself and an i7-7700K with fedora37
managed at OVH, thinking it would be more resilient than the cable
connection itself. The problem also happens without the VPN, but perhaps
not to the same degree.
I think the server is certainly powerful enough to process the amount of
DNS queries, but there's also a lot of timeouts.
How do I troubleshoot this? It's the bridge experiencing the dropped
packets, not the interface itself, it seems?
They're not packet errors - just dropped packets. Perhaps the processor
can't handle the traffic?
Do you have a server connected to a cable modem? Is there something about
it being a cable connection that could be causing this?
There's about 27k dropped packets in about 12 hours of uptime.
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 68.195.111.45 netmask 255.255.255.248 broadcast
68.195.111.47
ether ae:64:2c:25:b5:44 txqueuelen 1000 (Ethernet)
RX packets 11355786 bytes 21634260431 (20.1 GiB)
RX errors 0 dropped 26349 overruns 0 frame 0
TX packets 7374430 bytes 993860294 (947.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 14:da:e9:97:ab:72 txqueuelen 1000 (Ethernet)
RX packets 16224042 bytes 22179550934 (20.6 GiB)
RX errors 0 dropped 165 overruns 0 frame 0
TX packets 7493927 bytes 1031987928 (984.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17 memory 0xdf100000-df120000
10:58 a.m.
On Sat, 2023-06-03 at 09:46 -0400, Alex wrote:
I have an E3-1240 fedora37 postfix system using SSDs connected to a
cable modem that's having problems with dropped packets. There is one
other fedora37 server (E5-1650) directly connected to the cable modem
that is not having the same problem, although it's just routing
packets, not really doing much processing of data. The server with
the problem is using libreswan to create a VPN between itself and
an i7-7700K with fedora37 managed at OVH, thinking it would be more
resilient than the cable connection itself. The problem also happens
without the VPN, but perhaps not to the same degree.
Asking the obvious questions:
Have you swapped the ports around on the cable modem between the two
computers, to see if the cable modem has a bad port?
Have you swapped cables, to see if you have a bad cable?
--
uname -rsvp
Linux 3.10.0-1160.90.1.el7.x86_64 #1 SMP Thu May 4 15:21:22 UTC 2023 x86_64
Boilerplate: All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.
11:56 a.m.
Hi,
On Sat, Jun 3, 2023 at 11:58 AM Tim via users <users(a)lists.fedoraproject.org>
wrote:
On Sat, 2023-06-03 at 09:46 -0400, Alex wrote:
> I have an E3-1240 fedora37 postfix system using SSDs connected to a
> cable modem that's having problems with dropped packets. There is one
> other fedora37 server (E5-1650) directly connected to the cable modem
> that is not having the same problem, although it's just routing
> packets, not really doing much processing of data. The server with
> the problem is using libreswan to create a VPN between itself and
> an i7-7700K with fedora37 managed at OVH, thinking it would be more
> resilient than the cable connection itself. The problem also happens
> without the VPN, but perhaps not to the same degree.
Asking the obvious questions:
Have you swapped the ports around on the cable modem between the two
computers, to see if the cable modem has a bad port?
Have you swapped cables, to see if you have a bad cable?
Yes, great questions, and I realized that after I posted.
The server has two ethernet ports - I've tried connecting a completely
different ethernet cable to the other port on the server to a completely
different port on the cable modem and the problem persists.
To add to the complexity of this, I have a SuperMicro E5-1650 2U that I
just brought out of retirement and booted with a USB stick running
sysrescue and it also has the same "problem," without even doing anything.
I don't understand how now these two servers are any different than the
other server on the same cable modem that doesn't experience any of these
issues.
As I posted in my follow-up, the problem doesn't happen when tcpdump is
running (now on both servers).
Maybe it is a software problem? iptables/etables? ethtool settings? I'm
using shorewall on the server without the problem, but the problem persists
even with disabling all firewall rules.
The "real" problem I'm trying to address is why I'm seeing so many DNS
timeouts:
Jun 3 10:07:34 mail03 postfix/postscreen[413190]: warning: dnsblog reply
timeout 10s for bb.barracudacentral.org
Jun 3 10:07:34 mail03 postfix/postscreen[413190]: warning: dnsblog reply
timeout 10s for list.dnswl.org
Jun 3 10:07:29 mail03 postfix/dnsblog[413191]: warning: dnsblog_query:
lookup error for DNS query 109.178.102.199.bl.mailspike.net: Host or domain
name not found. Name service error for name=109.178.102.199.bl.mailspike.net
type=A: Host not found, try again
Thanks,
Alex
--
uname -rsvp
Linux 3.10.0-1160.90.1.el7.x86_64 #1 SMP Thu May 4 15:21:22 UTC 2023 x86_64
Boilerplate: All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.
_______________________________________________
users mailing list -- users(a)lists.fedoraproject.org
To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
12:17 p.m.
On Sat, Jun 3, 2023 at 12:56 PM Alex <mysqlstudent(a)gmail.com> wrote:
On Sat, Jun 3, 2023 at 11:58 AM Tim via users <users(a)lists.fedoraproject.org>
wrote:
>
> On Sat, 2023-06-03 at 09:46 -0400, Alex wrote:
> > I have an E3-1240 fedora37 postfix system using SSDs connected to a
> > cable modem that's having problems with dropped packets. There is one
> > other fedora37 server (E5-1650) directly connected to the cable modem
> > that is not having the same problem, although it's just routing
> > packets, not really doing much processing of data. The server with
> > the problem is using libreswan to create a VPN between itself and
> > an i7-7700K with fedora37 managed at OVH, thinking it would be more
> > resilient than the cable connection itself. The problem also happens
> > without the VPN, but perhaps not to the same degree.
>
> Asking the obvious questions:
>
> Have you swapped the ports around on the cable modem between the two
> computers, to see if the cable modem has a bad port?
>
> Have you swapped cables, to see if you have a bad cable?
Yes, great questions, and I realized that after I posted.
The server has two ethernet ports - I've tried connecting a completely different
ethernet cable to the other port on the server to a completely different port on the cable
modem and the problem persists.
To add to the complexity of this, I have a SuperMicro E5-1650 2U that I just brought out
of retirement and booted with a USB stick running sysrescue and it also has the same
"problem," without even doing anything. I don't understand how now these two
servers are any different than the other server on the same cable modem that doesn't
experience any of these issues.
As I posted in my follow-up, the problem doesn't happen when tcpdump is running (now
on both servers).
tcpdump likely ignores the checksum because it is usually offloaded
and calculated by the NIC. Since the NIC performs the calculation,
tcpdump only sees the memory with the header fields and checksum = 0.
The OS sets it to 0 because the NIC will calculate it.
I think you can disable TPC checksum offloads, and have the OS perform
the calculation. See
https://serverfault.com/questions/421995/disable-tcp-offloading-completel...
.
Maybe it is a software problem? iptables/etables? ethtool settings?
I'm using shorewall on the server without the problem, but the problem persists even
with disabling all firewall rules.
The "real" problem I'm trying to address is why I'm seeing so many DNS
timeouts:
Jun 3 10:07:34 mail03 postfix/postscreen[413190]: warning: dnsblog reply timeout 10s for
bb.barracudacentral.org
Jun 3 10:07:34 mail03 postfix/postscreen[413190]: warning: dnsblog reply timeout 10s for
list.dnswl.org
Jun 3 10:07:29 mail03 postfix/dnsblog[413191]: warning: dnsblog_query: lookup error for
DNS query 109.178.102.199.bl.mailspike.net: Host or domain name not found. Name service
error for name=109.178.102.199.bl.mailspike.net type=A: Host not found, try again
You might also be interwsted in
https://unix.stackexchange.com/questions/205141/what-exactly-is-an-ifconf...
.
When I read your first post, I thought you might need to increase the
number of receive buffers or the size of the receive buffers. Change
them from 256kb to 512kb or 1024kb. Also see
https://www.cyberciti.biz/faq/linux-tcp-tuning/ .
Jeff
12:43 p.m.
On 3 Jun 2023, at 14:46, Alex <mysqlstudent(a)gmail.com> wrote:
There's about 27k dropped packets in about 12 hours of uptime.
This maybe an app issue. If this is tcp make sure that the listening socket is using a
large backlog value. I use 2048 for high tps services.
If its udp then there the app fixes for that too.
Barry
349
days inactive
349
days old
5 comments
4 participants
participants (4)
-
Alex -
Barry -
Jeffrey Walton -
Tim