On Wed, May 15, 2019 at 05:15:38PM -0400, Rob Crittenden via FreeIPA-users wrote:
> Jonathan Vaughn via FreeIPA-users wrote:
> > I previously had tested FreeIPA running on a Raspberry Pi 3B+ and as
> > long as I didn't run the Dogtag server on it performance seemed
> > acceptable for the purpose. These are only being used as local
> > DNS/LDAP/Krb5 replicas, everything also runs on both physical x86_64 and
> > VM x86_64 servers as well in more than one location.
>
> It is STRONGLY not recommended to run IPA in production on *Pi. If you
> have you and your wife on some local LAN then maybe.
>
> > However now that I'm trying to set up Pis for actual use (previously had
> > set up a test environment to validate using them) I'm running into major
> > performance issues once certmonger starts. Using a systemd timer to
> > delay start until everything else starts at least lets everything else
> > FreeIPA related start up and work, but once certmonger starts it still
> > hammers the system using tons of memory and causing lots of swapping.
> >
> > Is there any reason for it to spawn so many processes all at once,
> > versus doing them in a more serial fashion? And did something change in
> > FreeIPA/certmonger behavior in the last year that would cause such a
> > performance regression in memory limited scenarios? Previously I just
> > had zram swap and it was fine, now I have to replace that with actual
> > swap on storage.
>
> Hard to say since you include no version information.
>
> > Also, there's currently no certs needing renewal or anything on this
> > system, so why does it even spawn so many processes ?
> >
> > root 1699 1 0 03:55 ? 00:00:00 /usr/sbin/certmonger -S
> > -p /var/run/certmonger.pid -n
> > root 1720 1699 0 03:55 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
> > root 1721 1699 0 03:55 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
> > root 1722 1699 0 03:55 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
> > root 1723 1699 0 03:55 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
> > root 1724 1699 0 03:55 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
> > root 1725 1699 0 03:55 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
> > root 1726 1699 0 03:55 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
> > root 1727 1699 0 03:55 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
> > root 1742 1699 0 03:55 ? 00:00:00
> > /usr/libexec/certmonger/dogtag-ipa-renew-agent-submit
> > root 1759 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
> > root 1761 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
> > root 1762 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
> > root 1763 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
> > root 1764 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
> > root 1765 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
> > root 1767 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
> > root 1768 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
> > root 1769 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
> > root 1770 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
> > root 1771 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
> > root 1772 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
> > root 1773 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
> > root 1774 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
> > root 1775 1699 0 03:56 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
> > root 1776 1699 0 03:57 ? 00:00:00 /usr/bin/python3 -E
> > /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
> >
> > Eventually these complete and things settle down but it takes a very
> > long time, and without delaying certmonger until after the rest of
> > FreeIPA it can cause various IPA services to take so long that they die
> > and fail to start.
>
> On startup certmonger examines all the certs to see if, for example, the
> roots have changed. There are all the processes because there is one per
> tracked cert I assume. There is serialization in the IPA certmonger
> config (ipa-server-guard) so they go one at at time.
>
Do they busy-wait on the lock? Maybe that is why the load is so
high?
I echo Rob's comments about Raspberry Pi. For sure there is room to
improve performance, but a future where FreeIPA runs well on such
low-spec machines... it is hard to imagine, and not something we're
aiming for.
Thanks,
Fraser
> rob
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org