Hi,

On Tue, Aug 30, 2022 at 7:32 PM Simon Matthews via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:

Thanks for your reply.

>>> You can find a few things to check in
>>> https://www.freeipa.org/page/Troubleshooting/Directory_Server#Replication...
]# ldapsearch -Y GSSAPI -h ipa1.sj.bps -b "" -s base
SASL/GSSAPI authentication started
SASL username: ldap/ipa2.sj.bps@IPA.<MY COMPANY>.COM
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#

#
dn:
objectClass: top
namingContexts: cn=changelog
namingContexts: dc=ipa,dc=<my company>,dc=com
namingContexts: o=ipaca
defaultnamingcontext: dc=ipa,dc=<my company>,dc=com
supportedExtension: 2.16.840.1.113730.3.5.7
supportedExtension: 2.16.840.1.113730.3.5.8
supportedExtension: 2.16.840.1.113730.3.5.10
supportedExtension: 2.16.840.1.113730.3.8.10.3
supportedExtension: 2.16.840.1.113730.3.8.10.4
supportedExtension: 2.16.840.1.113730.3.8.10.4.1
supportedExtension: 2.16.840.1.113730.3.8.10.4.2
supportedExtension: 1.3.6.1.4.1.4203.1.11.1
supportedExtension: 2.16.840.1.113730.3.8.10.1
supportedExtension: 2.16.840.1.113730.3.8.10.5
supportedExtension: 2.16.840.1.113730.3.5.3
supportedExtension: 2.16.840.1.113730.3.5.12
supportedExtension: 2.16.840.1.113730.3.5.5
supportedExtension: 2.16.840.1.113730.3.5.6
supportedExtension: 2.16.840.1.113730.3.5.9
supportedExtension: 2.16.840.1.113730.3.5.4
supportedExtension: 2.16.840.1.113730.3.6.5
supportedExtension: 2.16.840.1.113730.3.6.6
supportedExtension: 2.16.840.1.113730.3.6.7
supportedExtension: 2.16.840.1.113730.3.6.8
supportedExtension: 1.3.6.1.4.1.4203.1.11.3
supportedExtension: 1.3.6.1.4.1.1466.20037
supportedControl: 2.16.840.1.113730.3.4.2
supportedControl: 2.16.840.1.113730.3.4.3
supportedControl: 2.16.840.1.113730.3.4.4
supportedControl: 2.16.840.1.113730.3.4.5
supportedControl: 1.2.840.113556.1.4.473
supportedControl: 2.16.840.1.113730.3.4.9
supportedControl: 2.16.840.1.113730.3.4.16
supportedControl: 2.16.840.1.113730.3.4.15
supportedControl: 2.16.840.1.113730.3.4.17
supportedControl: 2.16.840.1.113730.3.4.19
supportedControl: 1.3.6.1.1.13.1
supportedControl: 1.3.6.1.1.13.2
supportedControl: 1.3.6.1.4.1.42.2.27.8.5.1
supportedControl: 1.3.6.1.4.1.42.2.27.9.5.2
supportedControl: 1.2.840.113556.1.4.319
supportedControl: 1.3.6.1.4.1.42.2.27.9.5.8
supportedControl: 1.3.6.1.4.1.4203.666.5.16
supportedControl: 2.16.840.1.113730.3.8.10.6
supportedControl: 2.16.840.1.113730.3.8.10.7
supportedControl: 2.16.840.1.113730.3.4.14
supportedControl: 2.16.840.1.113730.3.4.20
supportedControl: 1.3.6.1.4.1.1466.29539.12
supportedControl: 2.16.840.1.113730.3.4.12
supportedControl: 2.16.840.1.113730.3.4.18
supportedControl: 2.16.840.1.113730.3.4.13
supportedControl: 1.3.6.1.4.1.4203.1.9.1.1
supportedSASLMechanisms: EXTERNAL
supportedSASLMechanisms: GSS-SPNEGO
supportedSASLMechanisms: GSSAPI
supportedSASLMechanisms: DIGEST-MD5
supportedSASLMechanisms: CRAM-MD5
supportedSASLMechanisms: LOGIN
supportedSASLMechanisms: PLAIN
supportedSASLMechanisms: ANONYMOUS
supportedLDAPVersion: 2
supportedLDAPVersion: 3
vendorName: 389 Project
vendorVersion: 389-Directory/1.3.10.2 B2022.179.1527
dataversion: 020220830001452020220830001452020220830001452
netscapemdsuffix: cn=ldap://dc=ipa1,dc=sj,dc=bps:389
lastusn: 1222591
changeLog: cn=changelog
firstchangenumber: 151
lastchangenumber: 153
ipatopologypluginversion: 1.0
ipatopologyismanaged: on
ipaDomainLevel: 1

# search result
search: 4
result: 0 Success

# numResponses: 2
# numEntries: 1

>>> If they are configured as DNS servers, is there a forwarder configured?
Yes:
]# ipa dnsserver-show ipa1.sj.bps
Server name: ipa1.sj.bps
SOA mname override: ipa1.sj.bps.
Forwarders: 192.168.254.10, 192.168.254.2
Forward policy: only
[root@ipa1 ~]# ipa dnsserver-show ipa2.sj.bps
Server name: ipa2.sj.bps
SOA mname override: ipa2.sj.bps.
Forwarders: 192.168.254.2
Forward policy: only

The lack of 192.168.254.10 for ipa2 should not matter since this is a secondary/slave nameserver on the network.

>>> Are there any errors related to replication in
>>> /var/log/dirsrv/slapd-<YOUR-DOMAIN>/errors?

I see these errors.

[29/Aug/2022:19:12:53.869825394 -0400] - ERR - schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup!
[29/Aug/2022:19:12:54.686756883 -0400] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=ipa,dc=<my company>,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[29/Aug/2022:19:12:54.870607368 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.sj.bps@IPA.<MY COMPANY>.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328324 (Generic error (see e-text))
[29/Aug/2022:19:12:55.002346083 -0400] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa1.sj.bps" (ipa1:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[29/Aug/2022:19:12:55.058525909 -0400] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=caToipa1.sj.bps" (ipa1:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[29/Aug/2022:19:12:55.116643453 -0400] - ERR - schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds!
[29/Aug/2022:19:13:00.254585526 -0400] - ERR - schema-compat-plugin - warning: no entries set up under ou=sudoers,dc=ipa,dc=<my company>,dc=com
[29/Aug/2022:19:13:00.325746557 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=ipa,dc=<my company>,dc=com
[29/Aug/2022:19:13:00.625350394 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=ipa,dc=<my company>,dc=com
[29/Aug/2022:19:13:00.747736017 -0400] - ERR - schema-compat-plugin - Finished plugin initialization.
[29/Aug/2022:19:19:26.447086663 -0400] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=ipa,dc=<my company>,dc=com--no CoS Templates found, which should be added before the CoS Definition.
[29/Aug/2022:19:19:26.616760756 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.sj.bps@IPA.<MY COMPANY>.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
[29/Aug/2022:19:19:26.652053902 -0400] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa1.sj.bps" (ipa1:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[29/Aug/2022:19:19:26.705855975 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.sj.bps@IPA.<MY COMPANY>.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
[29/Aug/2022:19:19:26.732413212 -0400] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=caToipa1.sj.bps" (ipa1:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[29/Aug/2022:19:19:29.093106968 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa2.sj.bps@IPA.<MY COMPANY>.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
....
[30/Aug/2022:13:14:58.254029634 -0400] - ERR - agmt="cn=meToipa1.sj.bps" (ipa1:389) - clcache_load_buffer - Can't locate CSN 620693cb000200050000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[30/Aug/2022:13:14:58.285772035 -0400] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meToipa1.sj.bps" (ipa1:389): CSN 620693cb000200050000 not found, we aren't as up to date, or we purged
[30/Aug/2022:13:14:58.302465482 -0400] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1.sj.bps" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[30/Aug/2022:13:15:01.355096020 -0400] - ERR - agmt="cn=meToipa1.sj.bps" (ipa1:389) - clcache_load_buffer - Can't locate CSN 620693cb000200050000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[30/Aug/2022:13:15:01.393991242 -0400] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meToipa1.sj.bps" (ipa1:389): CSN 620693cb000200050000 not found, we aren't as up to date, or we purged
[30/Aug/2022:13:15:01.410581481 -0400] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1.sj.bps" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.

It looks like the replication was broken (or stopped) for too long, the changelog got purged and lost part of the updates that should be replicated. If you want to understand about the changelog and purge concepts, please refer to [1].

Depending on your domain level, you can use either
- ipa-replica-manage re-initialize and ipa-csreplica-manage reinitialize
(domain-level 0) [2]
or
- ipa topologysegment-reinitialize (domain level 1). For more
information refer to "ipa help topologysegment-reinitialize".

The command "ipa domainlevel-get" will provide you with the current
domain level. The reinitialize command forces a full synchronization of
the content from the specified source to the replica.

HTH,
flo

[1] https://access.redhat.com/documentation/en-us/red_hat_directory_server/12/html/configuring_and_managing_replication/assembly_trimming-the-replication-changelog_configuring-and-managing-replication#proc_configuring-replication-changelog-trimming-using-the-command-line_assembly_trimming-the-replication-changelog

[2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/linux_domain_identity_authentication_and_policy_guide/index#initialize

_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue