Re: spammy log messages with loadbalance runner
by Hangbin Liu
On Mon, 28 Oct 2019 at 14:37, Hangbin Liu <liuhangbin(a)gmail.com> wrote:
>
> On Fri, 25 Oct 2019 at 17:55, Hangbin Liu <liuhangbin(a)gmail.com> wrote:
>
> >
> > And after commit e47d5db538736 ("teamd: add an option to force log
> > output to stdout, stderr or syslog").
> > NetworkManager will force teamd use DAEMON_LOG_SYSLOG by default. So
> > either the user force to
> > use option "-l stderr" or we need to implement an own filter ourselves.
>
> Hi Jiri,
>
> To fix the spammy log messages, there are three ways
>
> 1. Ask NetworkManager reset TEAM_LOG_OUTPUT to stderr by default
> 2. Ask libdaemon also filter DEBUG messages when using DAEMON_LOG_SYSLOG
> 3. Move most teamd_log_dbg to teamd_log_dbgx
>
> Which way do you prefer?
On the other hand, this issue should be there from the start. Even
without NetworkManager,
If we run teamd with -d option, we will use DAEMON_LOG_SYSLOG and all
logs(including
debug info) will print to syslog.
People start to notice this issue as the user side(NM) changes the log
from stderr to syslog.
It doesn't make sense to ask user side(NM) to change their code. This
feature also lands
in libdaemon for a long time.
So either we ignore this issue, as all looks intend to be. Or we try
to move most debug messages
to teamd_log_dbgx, to make the user/customer happy.
What do you think?
Thanks
Hangbin
4 years, 5 months
spammy log messages with loadbalance runner
by Daniel Arena
Hello,
I have a loadbalance runner for 2 interfaces setup through
NetworkManager on fedora 30. I noticed in the system journal the teamd
messages have been quite spammy with stats updates/rebalance
information from teamd. See example log lines below. I tried looking
at `man teamd.conf` but only found the debug_level setting, which is
defaulted to 0. Is there some other way to turn these messages off? I
tried adjusting NetworkManager's log levels also, but even after
shutting off all logging, the messages from teamd still get through.
Is this actually an issue for NetworkManager, maybe it should be
capturing these log messages and filtering them with NetworkManager's
log levels?
My teamd config looks like this:
{"runner": {
"name": "loadbalance",
"tx_hash": ["eth","ipv4", "ipv6"],
"tx_balancer": {"name": "basic"}
},
"ports":{"eno1": {}, "eno2": {}}
}
Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for port
eno2: "3257841132330".
Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for port
eno1: "6585330648".
Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for hash
"21": "1242456011".
Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for hash
"16": "1270449453".
Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for hash
"9": "3761543798".
Jul 03 15:41:57 server1 teamd_nm-team[32311]: stats update for hash
"1": "626647946".
Jul 03 15:41:57 server1 teamd_nm-team[32311]: Remapped hash "21"
(delta 447647) to port eno2.
Jul 03 15:41:57 server1 teamd_nm-team[32311]: Remapped hash "16"
(delta 5301) to port eno1.
Jul 03 15:41:57 server1 teamd_nm-team[32311]: Port eno2 rebalanced,
delta: 447647
Jul 03 15:41:57 server1 teamd_nm-team[32311]: Port eno1 rebalanced, delta: 6205
Thanks,
Dan
4 years, 5 months
[PATCH libteam] teamd: lacp: use teamd_log_dbgx to print lacp info state
by Hangbin Liu
When we set TEAM_LOG_OUTPUT=syslog in the environment, all the logs,
including debug info will be printed to syslog, even though we set
daemon_verbosity_level = LOG_INFO.
The reason is the function daemon_logv() of libdaemon only check the
daemon_verbosity_level for LOG_STDOUT and LOG_STDERR, not LOG_SYSLOG.
Also here is the quote for daemon_set_verbosity() from libdaemon/dlog.h
"""
Allows to decide which messages to output on standard output/error
streams. All messages are logged to syslog and this setting does
not influence that.
"""
So libdaemon is intend to print all logs to syslog. But when we also
enabled lacp fast_rate, the lacp state info may get a little annoying
as it will print the state info every second, even the log level is
LOG_INFO.
So let's use teamd_log_dbgx() to print lacp info state, which would
only print the state log message with -g option.
Signed-off-by: Hangbin Liu <liuhangbin(a)gmail.com>
---
teamd/teamd_runner_lacp.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/teamd/teamd_runner_lacp.c b/teamd/teamd_runner_lacp.c
index d292d69..179b954 100644
--- a/teamd/teamd_runner_lacp.c
+++ b/teamd/teamd_runner_lacp.c
@@ -929,8 +929,8 @@ static void lacp_port_actor_update(struct lacp_port *lacp_port)
state |= INFO_STATE_DEFAULTED;
if (teamd_port_count(lacp_port->ctx) > 0)
state |= INFO_STATE_AGGREGATION;
- teamd_log_dbg("%s: lacp info state: 0x%02X.", lacp_port->tdport->ifname,
- state);
+ teamd_log_dbgx(lacp_port->ctx, 1, "%s: lacp info state: 0x%02X.",
+ lacp_port->tdport->ifname, state);
lacp_port->actor.state = state;
}
--
2.19.2
4 years, 5 months
Team interface drops packets with kernel 4.14
by Yevhen Kondrashyn
Hi All,
After upgrading from 3.10.104 to kernel 4.14 I see constant drops
(around 10 packets/sec) on team interface in active-backup mode,
increasing txqueuelen on a physical port helps, however, drops still
occur from time to time.
I've added debug printk in team driver where dev_kfree_skb_any is
called, however, I do not see messages in dmesg until I reset physical
interface by ethtool, therefore packets are being dropped somewhere
else, however, for now, I have no clue how to find the exact cause.
Do you have any ideas/suggestions on how to debug this?
Thanks!
net7 Link encap:Ethernet HWaddr 00:1E:67:B5:7F:76
inet addr:192.168.221.203 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::21e:67ff:feb5:7f76/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:263 errors:0 dropped:151 overruns:0 frame:0
TX packets:83944096 errors:0 dropped:57692 overruns:0 carrier:0
collisions:0 txqueuelen:10000
RX bytes:27531 (26.8 KiB) TX bytes:113995992408 (106.1 GiB)
# teamnl net7 ports
5: eth6: up 1000Mbit FD
# ethtool -S eth6
NIC statistics:
rx_packets: 113
tx_packets: 86938454
rx_bytes: 22848
tx_bytes: 118410084622
rx_broadcast: 68
tx_broadcast: 6
rx_multicast: 45
tx_multicast: 86938448
multicast: 45
collisions: 0
rx_crc_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 68
rx_flow_control_xoff: 96
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 22848
tx_dma_out_of_sync: 0
lro_aggregated: 0
lro_flushed: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
os2bmc_rx_by_bmc: 0
os2bmc_tx_by_bmc: 0
os2bmc_tx_by_host: 0
os2bmc_rx_by_host: 0
tx_hwtstamp_timeouts: 0
rx_hwtstamp_cleared: 0
rx_errors: 0
tx_errors: 0
tx_dropped: 0
rx_length_errors: 0
rx_over_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_queue_0_packets: 86938440
tx_queue_0_bytes: 118062311560
tx_queue_0_restart: 0
rx_queue_0_packets: 277
rx_queue_0_bytes: 32236
rx_queue_0_drops: 0
rx_queue_0_csum_err: 0
rx_queue_0_alloc_failed: 0
# teamdctl net7 s d
{
"ports": {
"eth6": {
"ifinfo": {
"dev_addr": "00:1e:67:b5:7f:76",
"dev_addr_len": 6,
"ifindex": 5,
"ifname": "eth6"
},
"link": {
"duplex": "full",
"speed": 1000,
"up": true
},
"link_watches": {
"list": {
"link_watch_0": {
"delay_down": 0,
"delay_up": 0,
"down_count": 1,
"name": "ethtool",
"up": true
}
},
"up": true
}
}
},
"runner": {
"active_port": "eth6"
},
"setup": {
"daemonized": true,
"dbus_enabled": false,
"debug_level": 0,
"kernel_team_mode_name": "activebackup",
"pid": 5533,
"pid_file": "/var/run/teamd/net7.pid",
"runner_name": "activebackup",
"zmq_enabled": false
},
"team_device": {
"ifinfo": {
"dev_addr": "00:1e:67:b5:7f:76",
"dev_addr_len": 6,
"ifindex": 13,
"ifname": "net7"
}
}
}
4 years, 6 months