Not necessarily that the time was off, but that replication was not happening. The error logs had this error in them, but the healthcheck script was not picking up on it:

"[27/Jan/2022:11:35:58.368835716 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meTofreeipa.us.ep.corp.local" (freeipa:389) - Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) ()"

On Fri, Jan 28, 2022 at 9:23 AM Rob Crittenden <rcritten@redhat.com> wrote:
Russell Jones via FreeIPA-users wrote:
> Thanks,
>
> I ended up finding the issue from another mailing list post. ntpd was
> not running on this host and the time got skewed too much from the other
> masters.
>
> For what it's worth, the ipa-healthcheck script did not catch this
> issue. Might be something to add?

It would be nice but syncing time can be quite slow and, AFAIK, there is
no way in advance to know if there is a time source available. So check
against what?

rob

>
> On Fri, Jan 28, 2022 at 2:49 AM Florence Blanc-Renaud <flo@redhat.com
> <mailto:flo@redhat.com>> wrote:
>
>     Hi,
>     you can find troubleshooting tips in
>     https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/trouble-gen-replication
>
>     HTH,
>     flo
>
>     On Thu, Jan 27, 2022 at 6:54 PM Russell Jones via FreeIPA-users
>     <freeipa-users@lists.fedorahosted.org
>     <mailto:freeipa-users@lists.fedorahosted.org>> wrote:
>
>         Hi all,
>
>         I have a setup of 4 FreeIPA servers, version 4.6.5, all on CentOS 7.
>
>         I've discovered that #4 is not syncing a new "video" group I
>         created, while the other 3 all have the group.
>
>         When looking at dirsrv error log, I am seeing the following
>         after running an ipactl stop / ipactl start:
>
>         [27/Jan/2022:11:35:55.158724429 -0600] - ERR - set_krb5_creds -
>         Could not get initial credentials for principal
>         [ldap/freeipa4.cluster@US.EP.CORP.LOCAL] in keytab
>         [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any
>         KDC for requested realm)
>         [27/Jan/2022:11:35:55.169790450 -0600] - INFO - slapd_daemon -
>         slapd started.  Listening on All Interfaces port 389 for LDAP
>         requests
>         [27/Jan/2022:11:35:55.173079823 -0600] - INFO - slapd_daemon -
>         Listening on All Interfaces port 636 for LDAPS requests
>         [27/Jan/2022:11:35:55.175096801 -0600] - INFO - slapd_daemon -
>         Listening on /var/run/slapd-US-EP-CORP-LOCAL.socket for LDAPI
>         requests
>         [27/Jan/2022:11:35:55.235218894 -0600] - ERR -
>         schema-compat-plugin - schema-compat-plugin tree scan will start
>         in about 5 seconds!
>         [27/Jan/2022:11:35:58.368835716 -0600] - ERR -
>         NSMMReplicationPlugin - bind_and_check_pwp -
>         agmt="cn=meTofreeipa.us.ep.corp.local" (freeipa:389) -
>         Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid
>         credentials) ()
>
>
>         I am unsure what the issue is or how to resolve this. Could I
>         get some assistance with being pointed in the right direction?
>
>         Thank you!
>         _______________________________________________
>         FreeIPA-users mailing list --
>         freeipa-users@lists.fedorahosted.org
>         <mailto:freeipa-users@lists.fedorahosted.org>
>         To unsubscribe send an email to
>         freeipa-users-leave@lists.fedorahosted.org
>         <mailto:freeipa-users-leave@lists.fedorahosted.org>
>         Fedora Code of Conduct:
>         https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>         List Guidelines:
>         https://fedoraproject.org/wiki/Mailing_list_guidelines
>         List Archives:
>         https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>         Do not reply to spam on the list, report it:
>         https://pagure.io/fedora-infrastructure
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>