Hi all,
Fixed it, thanks for the tip Rob
!
Certmonger was to blame or my rather slow Udooboard Celeron processor. Anyway, instead of hacking the upgrade script, I modified the certmonger.serivce file by adding a 180 secs (!!) sleep and extra Timeout: (The modified certmonger.service was removed after the upgrade)
[Unit]
Description=Certificate monitoring and PKI enrollment
After=syslog.target network.target dbus.service
[Service]
Type=dbus
PIDFile=/var/run/certmonger.pid
EnvironmentFile=-/etc/sysconfig/certmonger
ExecStart=/usr/sbin/certmonger -S -p /var/run/certmonger.pid -n $OPTS
ExecStartPost=/bin/sleep 180
TimeoutSec=240
BusName=org.fedorahosted.certmonger
Runing "ipa-server-upgrade" finished OK now. Certmonger takes itś time when it's (restarted, some dogtag-ipa-ca-r(enew ?) processes eating most of the cpu:
top - 16:00:24 up 18:51, 3 users, load average: 2.41, 1.87, 1.37
Tasks: 261 total, 6 running, 221 sleeping, 0 stopped, 34 zombie
%Cpu0 : 90.2 us, 7.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.6 hi, 0.3 si, 0.0 st
%Cpu1 : 92.4 us, 6.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.0 hi, 0.0 si, 0.0 st
%Cpu2 : 95.1 us, 3.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.3 hi, 0.0 si, 0.0 st
%Cpu3 : 88.6 us, 9.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.3 hi, 1.0 si, 0.0 st
MiB Mem : 3847.2 total, 335.4 free, 2154.9 used, 1356.9 buff/cache
MiB Swap: 3968.0 total, 3968.0 free, 0.0 used. 1452.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21750 root 20 0 401244 85296 22612 R 85.9 2.2 0:13.36 dogtag-ipa-ca-r
21764 root 20 0 386700 72880 22508 R 78.4 1.8 0:06.93 dogtag-ipa-ca-r
21771 root 20 0 161788 27332 10812 R 74.5 0.7 0:03.65 dogtag-ipa-ca-r
21758 root 20 0 394512 78340 22436 R 67.3 2.0 0:10.65 dogtag-ipa-ca-r
21746 root 20 0 0 0 0 Z 51.6 0.0 0:15.36 dogtag-ipa-ca-r
21778 root 20 0 106004 1220 0 R 24.8 0.0 0:00.76 certmonger
This seems like a new issue for me... Certainly, the Udoo x86 isn't the fasted in the world, but was running IPA bravely the last year... Am I hitting the bug Rob mentioned? Is there a bug report somewhere to track... I'll like to see it fixed in CentOS 8.
"getcert list" showed "/var/lib/ipa/private/httpd.key" and "/var/lib/ipa/certs/httpd.crt" wating for PIN. Running "ipa-getcert resubmit -i 20200126151811 -p /var/lib/ipa/passwds/ipa.xxx-443-RSA" fixed it.
Winfried
-----Oorspronkelijk bericht-----
Onderwerp: Re: [Freeipa-users] Re: ipa-server-upgrade failed
Datum: Sat, 25 Jan 2020 17:04:39 -0500
Winfried de Heiden via FreeIPA-users wrote:
Hi all,
/var/lib/ipa/private/httpd.key was in a status "waiting for PIN", but I
did brong is back to life using "ipa-getcert resubmit -i 20200117075404
-p /var/lib/ipa/passwds/xxxx-443-RSA. All certss look fine now.
"getcert list" works, although it's a bit slow the first time (running
on a Udoo x86 board with a celeron....)
Just to be shure about dbus, I restarted the entire machine; no success. :-(
Timing issue and/or casued by my rather slow Udoo board.....?
It is very possible. I fixed an issue in certmonger where every time it
forked (and it forks a LOT) it closed ALL the fds it knew about. On
containers this was 1M. It took a LONG time. The default is a more
modest 1k but can still take a while given the amount of forks that
certmonger does. This is fixed upstream, and I don't know of a
workaround, but this can definitely lead to timeout issues if certmonger
is being restarted immediately before this failure.
To diagnose it see what the load on the system is and what processes are
running. If you see dozens of certmonger processes with high load then
that's probably it. You'd have to hack the update script to do a sleep
to give things a chance to settle down.
rob
Winfried
Rob Crittenden schreef op za 25-01-2020 om 14:53 [-0500]:
Winfried de Heiden via FreeIPA-users wrote:
Hi all,
Using CentOS Linux release 8.1.1911 and the Stream repositories,
upgrading IPA fails:
( Upgrade ipa-server-common-4.8.0-13.module_el8.1.0+265+e1e65be4.noarch
@AppStream
Upgraded
ipa-server-common-4.8.0-11.module_el8.1.0+253+3b90c921.noarch @@System )
Running ipa-server-upgrade manually will result in:
[Upgrading CA schema]
CA schema update complete (no changes)
[Verifying that CA audit signing cert has 2 year validity]
[Update certmonger certificate renewal configuration]
Introspect error on :1.417:/org/fedorahosted/certmonger:
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did
not receive a reply. Possible causes include: the remote application did
not send a reply, the message bus security policy blocked the reply, the
reply timeout expired, or the network connection was broken.
I assume certmonger and dbus services are running?
Does `getcert list` work?
The dbus service sometimes isn't too fond of being restarted but you
could try that.
rob
_______________________________________________
FreeIPA-users mailing list --
freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org