I've been made aware that it takes two minutes for systemd-networkd-wait- online.service to spin its wheels, before giving up with a squeal:
Apr 09 22:03:30 shorty.email-scan.com systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured… Apr 09 22:05:30 shorty.email-scan.com systemd-networkd-wait-online[1282]: Timeout occurred while waiting for network connectivity. Apr 09 22:05:30 shorty.email-scan.com systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Everything comes up normally. Network connectivity on this box is normal. Originally I was looking into why it took a long time for keepalived to come up on this box and grab its virtual IP address, and I determined that keepalived simply started after quite a bit of time elapsed before systemd deemed it necessary to start keepalived. And when I tried to figure out why's that, I discovered this to be the culprit.
I did not find the man pages to be illumanating, just a bunch of systemd man pages referring to each other, with no clear indication as to what the issue is.
At the time I obtained the above logs I was ssh-ed into the box in question, which was fully up. But systemd-networkd-wait-online was under the impression that it wasn't, apparently, and decided to croak after two minutes elapsed.
So, how do I troubleshoot this?
On Tue, 2024-04-09 at 22:12 -0400, Sam Varshavchik wrote:
I've been made aware that it takes two minutes for systemd-networkd-wait- online.service to spin its wheels, before giving up with a squeal:
Apr 09 22:03:30 shorty.email-scan.com systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured… Apr 09 22:05:30 shorty.email-scan.com systemd-networkd-wait-online[1282]: Timeout occurred while waiting for network connectivity. Apr 09 22:05:30 shorty.email-scan.com systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Everything comes up normally. Network connectivity on this box is normal. Originally I was looking into why it took a long time for keepalived to come up on this box and grab its virtual IP address, and I determined that keepalived simply started after quite a bit of time elapsed before systemd deemed it necessary to start keepalived. And when I tried to figure out why's that, I discovered this to be the culprit.
You say that networking is working, but is whatever doling out IP addresses working well (i.e. responding rapidly).
And since you mention virtual IP addresses, is it two things? Local IP address allocation, and a secondary one?
Many years ago I had something happen like that for the least obvious reason. All the computers were fine, but a network switch had gone crazy. Toggling its power off and on again reset it. Eventually I replaced it because it did it more and more often.
Tim via users writes:
On Tue, 2024-04-09 at 22:12 -0400, Sam Varshavchik wrote:
Everything comes up normally. Network connectivity on this box is normal. Originally I was looking into why it took a long time for keepalived to
come
up on this box and grab its virtual IP address, and I determined that keepalived simply started after quite a bit of time elapsed before systemd deemed it necessary to start keepalived. And when I tried to figure out why's that, I discovered this to be the culprit.
You say that networking is working, but is whatever doling out IP addresses working well (i.e. responding rapidly).
Yes it is. It is the very same box. I haven't looked closely, but dhcpd should be up.
This box has static IP addresses. It is my LAN's router, with static IP addresses defined for all of its interfaces.
And since you mention virtual IP addresses, is it two things? Local IP address allocation, and a secondary one?
Yes, two things. It has static IP addresses on its network interfaces, then keepalived attaches a virtual IP address to one of them, which failed over to a secondary while it is rebooting.
On 4/9/24 19:12, Sam Varshavchik wrote:
I've been made aware that it takes two minutes for systemd-networkd-wait-online.service to spin its wheels, before giving up with a squeal:
Apr 09 22:03:30 shorty.email-scan.com systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured… Apr 09 22:05:30 shorty.email-scan.com systemd-networkd-wait-online[1282]: Timeout occurred while waiting for network connectivity. Apr 09 22:05:30 shorty.email-scan.com systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Everything comes up normally. Network connectivity on this box is normal. Originally I was looking into why it took a long time for keepalived to come up on this box and grab its virtual IP address, and I determined that keepalived simply started after quite a bit of time elapsed before systemd deemed it necessary to start keepalived. And when I tried to figure out why's that, I discovered this to be the culprit.
I did not find the man pages to be illumanating, just a bunch of systemd man pages referring to each other, with no clear indication as to what the issue is.
At the time I obtained the above logs I was ssh-ed into the box in question, which was fully up. But systemd-networkd-wait-online was under the impression that it wasn't, apparently, and decided to croak after two minutes elapsed.
So, how do I troubleshoot this?
I have a similar problem where the wait-online service suddenly started taking a very long time and then failing. My system that used to boot in a few seconds now takes over a minute. So I have two questions.
What is it waiting for? My main ethernet card gets an address right away. It's possibly because I was playing with VMs and created a whole bunch of virtual interfaces. I think it started after that. I just found an empty bridge that would try to use DHCP, so that could have been a problem. I won't know until I next reboot. However, that shouldn't be relevant. The system is online once the ethernet is connected.
Perhaps more importantly, why can't that happen in the background? Why does gdm care if the network is connected? From the analyze output, I don't think it's a direct dependency, but that seems like there is some dependency that really shouldn't be there.
On Tue, 2024-04-09 at 23:20 -0700, Samuel Sieb wrote:
Perhaps more importantly, why can't that happen in the background? Why does gdm care if the network is connected?
It'd need to be *if* people are using a network share for their homespace, or other "expected to be there" directories.
And, *if* you were using some kind of networked user authentication.
Samuel Sieb writes:
I have a similar problem where the wait-online service suddenly started taking a very long time and then failing. My system that used to boot in a few seconds now takes over a minute. So I have two questions.
What is it waiting for? My main ethernet card gets an address right away.
Some stuff that comes up doesn't really like it if there is no network connection, at the time. That's what this whole house of cards is for: to sit and wait until the box has at least one network connection "up".
Run this: systemd-analyze critical-chain network-online.target
On Wed, Apr 10, 2024 at 6:20 AM Sam Varshavchik mrsam@courier-mta.com wrote:
Samuel Sieb writes:
I have a similar problem where the wait-online service suddenly started taking a very long time and then failing. My system that used to boot in a few seconds now takes over a minute. So I have two questions.
What is it waiting for? My main ethernet card gets an address right away.
Some stuff that comes up doesn't really like it if there is no network connection, at the time. That's what this whole house of cards is for: to sit and wait until the box has at least one network connection "up".
-- _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Roger Heflin writes:
Run this: systemd-analyze critical-chain network-online.target
Can you explain why you think inspecting dependencies and starting times of different systemd units would have any bearing on why a single unit, systemd- network, is blowing chunks? Wasn't it clear from my explanation that the issue is a single systemd unit, systemd-networkd that times out and fails to start, on its own merits? Unless you're claiming that its upstream dependencies somehow affect it, I don't see any relevance of this.
What I do see, though, is that we have both NetworkManager-wait-online and systemd-networkd-wait-online. Why do we need both of them?
I simply disabled systemd-networkd-wait-online, and that seems to solve the problem. NetworkManager-wait-online does its job, now, as it always did. It's still enabled, as it always is. From what I could tell, systemd- networkd-wait-online did nothing at all, for me, whatsoever, except to delay other units from starting for a couple of minutes.
On Wed, Apr 10, 2024 at 6:20 AM Sam Varshavchik mrsam@courier-mta.com wrote:
Samuel Sieb writes:
I have a similar problem where the wait-online service suddenly started taking a very long time and then failing. My system that used to boot
in a
few seconds now takes over a minute. So I have two questions.
What is it waiting for? My main ethernet card gets an address right
away.
Some stuff that comes up doesn't really like it if there is no network connection, at the time. That's what this whole house of cards is for: to sit and wait until the box has at least one network connection "up".
On Thu, 2024-04-11 at 07:59 -0400, Sam Varshavchik wrote:
What I do see, though, is that we have both NetworkManager-wait- online and systemd-networkd-wait-online. Why do we need both of them?
I simply disabled systemd-networkd-wait-online, and that seems to solve the problem. NetworkManager-wait-online does its job, now, as it always did. It's still enabled, as it always is. From what I could tell, systemd-networkd-wait-online did nothing at all, for me, whatsoever, except to delay other units from starting for a couple of minutes.
I dunno why there's two wait-online services. I could hazard a guess that one of them will be the new approved way to do it (everything to be a systemd in the future, even "less" will become a systemd service), and the other being the old way. And while arguments ensue about the best way to do things we get stuck with two things doing the almost the same thing...
Looking into systemd-networkd-wait-online I see:
https://www.freedesktop.org/software/systemd/man/latest/systemd-networkd-wai...
Which, in part, says:
"systemd-networkd-wait-online is a oneshot system service (see systemd.service(5)), that waits for the network to be configured. By default, it will wait for all links it is aware of and which are managed by systemd-networkd.service(8) to be fully configured or failed, and for at least one link to be online. Here, online means that the link's operational state is equal or higher than "degraded". The threshold can be configured by --operational-state= option.
"The service systemd-networkd-wait-online.service invokes systemd- networkd-wait-online without any options. Thus, it waits for all managed interfaces to be configured or failed, and for at least one to be online."
Could it be that you have some additional interfaces configured, or half-configured, that it thinks it has to wait for?
In other parts mentioned on that link, there's options that can narrow it down to only caring about a specific interface (such as your real ethernet interface), specifying interfaces to ignore, and timeout parameters.
Another thing that springs to mind, is do you have IPv6? And if not, is it waiting for it in vain?
Bye, Tim.
Tim via users writes:
"The service systemd-networkd-wait-online.service invokes systemd- networkd-wait-online without any options. Thus, it waits for all managed interfaces to be configured or failed, and for at least one to be online."
Could it be that you have some additional interfaces configured, or half-configured, that it thinks it has to wait for?
I have two network interfaces. Both of them fully come up.
This system predates the introduction of systemd-networkd. It uses NetworkManager, which shows both interfaces as fully up:
# nmcli conn show NAME UUID TYPE DEVICE wan0 71e6ac4b-c693-4c20-aa0d-e1a63b7373fe ethernet eno2 lan0 d1a1ee90-f006-43bb-9cbf-175ad32f6565 ethernet eno1 lo 65db7135-f18c-4598-b5ad-61a1475f41ee loopback lo
Another thing that springs to mind, is do you have IPv6? And if not, is it waiting for it in vain?
Both of these are IPv4 only. Both of these have static IP address assignments. There are no non-default IPv6 addresses on either of them. They do have the stock link-local IPv6 address that all interfaces get. That's it. There is no reason, whatsoever, why these network interfaces would not come up immediately. They do. I ssh into the box on one of these immediately after boot, and the Intertubes are there.
"dnf remove systemd-networkd" says that it can be removed by itself, and it has no other dependencies. I have no idea how it got installed. This system's original install predated even NetworkManager's existence. Some dependency must've pulled in systemd-networkd at some point.
Another data point is that the systemd-networkd main service is disabled. The real problem is that systemd-networkd-wait-online was enabled. I have no recollection of enabling it, I was completely unaware of it, until I noticed this breakage.
So, it looks to me like the main issue is systemd-networkd-wait-online, and only systemd-networkd-wait-online was erroneously enabled, and – and this is the guesswork part – the systemd-verse had no network interfaces defined (all the network interfaces are managed by NetworkManager) and systemd- networkd-wait-online will, apparently, hang until it times out, waiting for network interfaces to come up, but none will ever come up because none have been configured.
On Thu, Apr 11, 2024 at 6:01 PM Sam Varshavchik mrsam@courier-mta.com wrote:
Tim via users writes:
"The service systemd-networkd-wait-online.service invokes systemd- networkd-wait-online without any options. Thus, it waits for all managed interfaces to be configured or failed, and for at least one to be online."
Could it be that you have some additional interfaces configured, or half-configured, that it thinks it has to wait for?
I have two network interfaces. Both of them fully come up.
This system predates the introduction of systemd-networkd. It uses NetworkManager, which shows both interfaces as fully up:
# nmcli conn show NAME UUID TYPE DEVICE wan0 71e6ac4b-c693-4c20-aa0d-e1a63b7373fe ethernet eno2 lan0 d1a1ee90-f006-43bb-9cbf-175ad32f6565 ethernet eno1 lo 65db7135-f18c-4598-b5ad-61a1475f41ee loopback lo
Another thing that springs to mind, is do you have IPv6? And if not, is it waiting for it in vain?
Both of these are IPv4 only. Both of these have static IP address assignments. There are no non-default IPv6 addresses on either of them. They do have the stock link-local IPv6 address that all interfaces get. That's it. There is no reason, whatsoever, why these network interfaces would not come up immediately. They do. I ssh into the box on one of these immediately after boot, and the Intertubes are there.
"dnf remove systemd-networkd" says that it can be removed by itself, and it has no other dependencies. I have no idea how it got installed. This system's original install predated even NetworkManager's existence. Some dependency must've pulled in systemd-networkd at some point.
Another data point is that the systemd-networkd main service is disabled. The real problem is that systemd-networkd-wait-online was enabled. I have no recollection of enabling it, I was completely unaware of it, until I noticed this breakage.
DIsabled means the service can be manually started. You should mask the service if you don't want it to start automatically or manually.
So, it looks to me like the main issue is systemd-networkd-wait-online, and only systemd-networkd-wait-online was erroneously enabled, and – and this is the guesswork part – the systemd-verse had no network interfaces defined (all the network interfaces are managed by NetworkManager) and systemd- networkd-wait-online will, apparently, hang until it times out, waiting for network interfaces to come up, but none will ever come up because none have been configured.
Jeff
On 4/11/24 15:08, Jeffrey Walton wrote:
On Thu, Apr 11, 2024 at 6:01 PM Sam Varshavchik mrsam@courier-mta.com wrote:
Tim via users writes:
"The service systemd-networkd-wait-online.service invokes systemd- networkd-wait-online without any options. Thus, it waits for all managed interfaces to be configured or failed, and for at least one to be online."
Could it be that you have some additional interfaces configured, or half-configured, that it thinks it has to wait for?
I have two network interfaces. Both of them fully come up.
This system predates the introduction of systemd-networkd. It uses NetworkManager, which shows both interfaces as fully up:
# nmcli conn show NAME UUID TYPE DEVICE wan0 71e6ac4b-c693-4c20-aa0d-e1a63b7373fe ethernet eno2 lan0 d1a1ee90-f006-43bb-9cbf-175ad32f6565 ethernet eno1 lo 65db7135-f18c-4598-b5ad-61a1475f41ee loopback lo
Another thing that springs to mind, is do you have IPv6? And if not, is it waiting for it in vain?
Both of these are IPv4 only. Both of these have static IP address assignments. There are no non-default IPv6 addresses on either of them. They do have the stock link-local IPv6 address that all interfaces get. That's it. There is no reason, whatsoever, why these network interfaces would not come up immediately. They do. I ssh into the box on one of these immediately after boot, and the Intertubes are there.
"dnf remove systemd-networkd" says that it can be removed by itself, and it has no other dependencies. I have no idea how it got installed. This system's original install predated even NetworkManager's existence. Some dependency must've pulled in systemd-networkd at some point.
Another data point is that the systemd-networkd main service is disabled. The real problem is that systemd-networkd-wait-online was enabled. I have no recollection of enabling it, I was completely unaware of it, until I noticed this breakage.
DIsabled means the service can be manually started. You should mask the service if you don't want it to start automatically or manually.
Because I prefer systemd-networkd for my management system I shutdown NetworkManager: disabled, stopped, masked, and all of a sudden my networks disappeared. Iiiiiit's baaaack! Reason is because .socket files will activate their corresponding service files REGARDLESS of masking or disabling. You have to check for sockets, too, and shut those down.
So, it looks to me like the main issue is systemd-networkd-wait-online, and only systemd-networkd-wait-online was erroneously enabled, and – and this is the guesswork part – the systemd-verse had no network interfaces defined (all the network interfaces are managed by NetworkManager) and systemd- networkd-wait-online will, apparently, hang until it times out, waiting for network interfaces to come up, but none will ever come up because none have been configured.
Jeff
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 12 Apr 2024, at 05:42, Mike Wright nobody@nospam.hostisimo.com wrote:
Reason is because .socket files will activate their corresponding service files REGARDLESS of masking or disabling.
A masked service cannot do anything as masking replaced it with /dev/null.
Barry
On 11 Apr 2024, at 12:59, Sam Varshavchik mrsam@courier-mta.com wrote:
What I do see, though, is that we have both NetworkManager-wait-online and systemd-networkd-wait-online. Why do we need both of them?
It's odd that both are enabled.
On my kde plasma Vm only NetworkManager-wait-online.service is enabled systemd-networkd-wait-online.service is not enabled.
On my router that I use systemd-networkd and removed NetworkManger I have enabled systemd-networkd-wait-online.service.
It would seem that you must have enabled systemd-networkd-wait-online.service at some point in time? disabling it should be sufficient to fix things.
Barry
-
Barry Scott -
Jeffrey Walton -
Mike Wright -
Roger Heflin -
Sam Varshavchik -
Samuel Sieb -
Tim