Hi all,
Fixed it, thanks for the tip Rob :-)!
Certmonger was to blame or my rather slow Udooboard Celeron processor.
Anyway, instead of hacking the upgrade script, I modified the
certmonger.serivce file by adding a 180 secs (!!) sleep and extra
Timeout: (The modified certmonger.service was removed after the
upgrade)
[Unit]
Description=Certificate monitoring and PKI enrollment
After=syslog.target network.target dbus.service
[Service]
Type=dbus
PIDFile=/var/run/certmonger.pid
EnvironmentFile=-/etc/sysconfig/certmonger
ExecStart=/usr/sbin/certmonger -S -p /var/run/certmonger.pid -n $OPTS
ExecStartPost=/bin/sleep 180
TimeoutSec=240
BusName=org.fedorahosted.certmonger
Runing "ipa-server-upgrade" finished OK now. Certmonger takes itś time
when it's (restarted, some dogtag-ipa-ca-r(enew ?) processes eating
most of the cpu:
top - 16:00:24 up 18:51, 3 users, load average: 2.41, 1.87, 1.37
Tasks: 261 total, 6 running, 221 sleeping, 0 stopped, 34 zombie
%Cpu0 : 90.2 us, 7.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.6 hi, 0.3
si, 0.0 st
%Cpu1 : 92.4 us, 6.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.0 hi, 0.0
si, 0.0 st
%Cpu2 : 95.1 us, 3.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.3 hi, 0.0
si, 0.0 st
%Cpu3 : 88.6 us, 9.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 1.3 hi, 1.0
si, 0.0 st
MiB Mem : 3847.2 total, 335.4 free, 2154.9 used, 1356.9
buff/cache
MiB Swap: 3968.0 total, 3968.0 free, 0.0 used. 1452.0 avail
Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
21750 root 20 0 401244 85296 22612 R 85.9 2.2 0:13.36
dogtag-ipa-ca-r
21764 root 20 0 386700 72880 22508 R 78.4 1.8 0:06.93
dogtag-ipa-ca-r
21771 root 20 0 161788 27332 10812 R 74.5 0.7 0:03.65
dogtag-ipa-ca-r
21758 root 20 0 394512 78340 22436 R 67.3 2.0 0:10.65
dogtag-ipa-ca-r
21746 root 20 0 0 0 0 Z 51.6 0.0 0:15.36
dogtag-ipa-ca-r
21778 root 20 0 106004 1220 0 R 24.8 0.0 0:00.76
certmonger
This seems like a new issue for me... Certainly, the Udoo x86 isn't the
fasted in the world, but was running IPA bravely the last year... Am I
hitting the bug Rob mentioned? Is there a bug report somewhere to
track... I'll like to see it fixed in CentOS 8.
"getcert list" showed "/var/lib/ipa/private/httpd.key" and
"/var/lib/ipa/certs/httpd.crt" wating for PIN. Running "ipa-getcert
resubmit -i 20200126151811 -p /var/lib/ipa/passwds/ipa.xxx-443-RSA"
fixed it.
Winfried
-----Oorspronkelijk bericht-----
Van: Rob Crittenden <rcritten(a)redhat.com>
Aan: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
Cc: Winfried de Heiden <wdh(a)dds.nl>
Onderwerp: Re: [Freeipa-users] Re: ipa-server-upgrade failed
Datum: Sat, 25 Jan 2020 17:04:39 -0500
Winfried de Heiden via FreeIPA-users wrote:
Hi all,
/var/lib/ipa/private/httpd.key was in a status "waiting for PIN", but
Idid brong is back to life using "ipa-getcert resubmit -i
20200117075404-p /var/lib/ipa/passwds/xxxx-443-RSA. All certss look
fine now. "getcert list" works, although it's a bit slow the first
time (runningon a Udoo x86 board with a celeron....)
Just to be shure about dbus, I restarted the entire machine; no
success. :-(
Timing issue and/or casued by my rather slow Udoo board.....?
It is very possible. I fixed an issue in certmonger where every time
itforked (and it forks a LOT) it closed ALL the fds it knew about.
Oncontainers this was 1M. It took a LONG time. The default is a
moremodest 1k but can still take a while given the amount of forks
thatcertmonger does. This is fixed upstream, and I don't know of
aworkaround, but this can definitely lead to timeout issues if
certmongeris being restarted immediately before this failure.
To diagnose it see what the load on the system is and what processes
arerunning. If you see dozens of certmonger processes with high load
thenthat's probably it. You'd have to hack the update script to do a
sleepto give things a chance to settle down.
rob
Winfried
Rob Crittenden schreef op za 25-01-2020 om 14:53 [-0500]:
> Winfried de Heiden via FreeIPA-users wrote:
> > Hi all,
> > Using CentOS Linux release 8.1.1911 and the Stream
> > repositories,upgrading IPA fails:
> > ( Upgrade
> > ipa-server-common-4.8.0-13.module_el8.1.0+265+e1e65be4.noarch@AppStream
> > Upgradedipa-server-common-4.8.0-
> > 11.module_el8.1.0+253+3b90c921.noarch @@System )
> > Running ipa-server-upgrade manually will result in:
> > [Upgrading CA schema]CA schema update complete (no
> > changes)[Verifying that CA audit signing cert has 2 year
> > validity][Update certmonger certificate renewal
> > configuration]Introspect error on
> > :1.417:/org/fedorahosted/certmonger:dbus.exceptions.DBusException
> > : org.freedesktop.DBus.Error.NoReply: Didnot receive a reply.
> > Possible causes include: the remote application didnot send a
> > reply, the message bus security policy blocked the reply,
> > thereply timeout expired, or the network connection was broken.
>
> I assume certmonger and dbus services are running?
> Does `getcert list` work?
> The dbus service sometimes isn't too fond of being restarted but
> youcould try that.
> rob
_______________________________________________FreeIPA-users mailing
list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...