3:58 p.m.
I'm running 5 ipa servers with (the latest on CentOS 8) 4.9.2.
Synchronization had stopped yesterday and also 3 days ago. It actually
stopped yesterday after I stopped / modified / started "ipa1" to
configure rotating logs longer so I could track down what happened 3
days ago.
2021-07-27 17:22:46 ipactl stop
2021-07-27 17:22:59 emacs dse.ldif # Modify to access and error log
rotation values
2021-07-27 17:23:45 ipactl start
Below seems to be what kicked off the bad behavior. I've seen a few
posts about removing the keys out of dse.ldif when this happens. I'm a
bit leery of doing this, as I don't fully understand what is going on.
(is it comparable to clearing out known_host entries when using ssh?)
[27/Jul/2021:17:23:49.818525015 -0600] - ERR - attrcrypt_unwrap_key -
Failed to unwrap key for cipher AES
[27/Jul/2021:17:23:49.820422259 -0600] - ERR - attrcrypt_cipher_init -
Symmetric key failed to unwrap with the private key; Cert might have
been renewed since the
key is wrapped. To recover the encrypted contents, keep the wrapped
symmetric key value.
[27/Jul/2021:17:23:50.040967207 -0600] - ERR - attrcrypt_unwrap_key -
Failed to unwrap key for cipher 3DES
[27/Jul/2021:17:23:50.043074553 -0600] - ERR - attrcrypt_cipher_init -
Symmetric key failed to unwrap with the private key; Cert might have
been renewed since the
key is wrapped. To recover the encrypted contents, keep the wrapped
symmetric key value.
[27/Jul/2021:17:23:50.044268421 -0600] - ERR - attrcrypt_init - All
prepared ciphers are not available. Please disable attribute encryption.
[27/Jul/2021:17:23:50.263786473 -0600] - ERR - attrcrypt_unwrap_key -
Failed to unwrap key for cipher AES
[27/Jul/2021:17:23:50.266090934 -0600] - ERR - attrcrypt_cipher_init -
Symmetric key failed to unwrap with the private key; Cert might have
been renewed since the key is wrapped. To recover the encrypted
contents, keep the wrapped symmetric key value.
[27/Jul/2021:17:23:50.470918523 -0600] - ERR - attrcrypt_unwrap_key -
Failed to unwrap key for cipher 3DES
[27/Jul/2021:17:23:50.472915669 -0600] - ERR - attrcrypt_cipher_init -
Symmetric key failed to unwrap with the private key; Cert might have
been renewed since the key is wrapped. To recover the encrypted
contents, keep the wrapped symmetric key value.
[27/Jul/2021:17:23:50.474282471 -0600] - ERR - attrcrypt_init - All
prepared ciphers are not available. Please disable attribute encryption.
[27/Jul/2021:17:23:50.891048127 -0600] - ERR - schema-compat-plugin -
scheduled schema-compat-plugin tree scan in about 5 seconds after the
server startup!
Then ipa1 can't talk to the replicas (ipa2,ipa3,ipa5,ipa6) shown below:
[27/Jul/2021:17:23:51.081696109 -0600] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/ipa1.hpc.example.com(a)HPC.EXAMPLE.COM] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
[27/Jul/2021:17:23:51.086755379 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=meToipa4.hpc.example.com" (ipa4:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[27/Jul/2021:17:23:51.091748474 -0600] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/ipa1.hpc.example.com(a)HPC.EXAMPLE.COM] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
[27/Jul/2021:17:23:51.093430455 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp -
agmt="cn=ipa1.hpc.example.com-to-ipa6.hpc.example.com" (ipa6:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[27/Jul/2021:17:23:51.094725291 -0600] - ERR - schema-compat-plugin -
schema-compat-plugin tree scan will start in about 5 seconds!
[27/Jul/2021:17:23:51.096059194 -0600] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/ipa1.hpc.example.com(a)HPC.EXAMPLE.COM] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
[27/Jul/2021:17:23:51.097152619 -0600] - INFO - slapd_daemon - slapd
started. Listening on All Interfaces port 389 for LDAP requests
[27/Jul/2021:17:23:51.098356748 -0600] - INFO - slapd_daemon - Listening
on All Interfaces port 636 for LDAPS requests
[27/Jul/2021:17:23:51.099577958 -0600] - INFO - slapd_daemon - Listening
on /var/run/slapd-HPC-EXAMPLE-COM.socket for LDAPI requests
[27/Jul/2021:17:23:51.100701349 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=caToipa3.hpc.example.com" (ipa3:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[27/Jul/2021:17:23:51.101782194 -0600] - ERR - set_krb5_creds - Could
not get initial credentials for principal
[ldap/ipa1.hpc.example.com(a)HPC.EXAMPLE.COM] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
[27/Jul/2021:17:23:51.103848248 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=caToipa5.hpc.example.com" (ipa5:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[27/Jul/2021:17:23:58.152621025 -0600] - ERR - schema-compat-plugin -
Finished plugin initialization.
[27/Jul/2021:17:24:21.201225830 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=meToipa2.hpc.example.com" (ipa2:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[27/Jul/2021:17:24:21.203158794 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp -
agmt="cn=ipa1.hpc.example.com-to-ipa6.hpc.example.com" (ipa6:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[27/Jul/2021:17:24:21.204833314 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=meToipa3.hpc.example.com" (ipa3:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[27/Jul/2021:17:24:21.206099975 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=meToipa5.hpc.example.com" (ipa5:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
[27/Jul/2021:17:54:03.675297221 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=caToipa2.hpc.example.com" (ipa2:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ()
After realizing I had a problem this morning, I rebooted ipa1 but it did
not help syncing. I re-initialized ipa1 from ipa3, this got them all
authenticating to each other and in sync.
[28/Jul/2021:08:09:10.347094254 -0600] - INFO - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=caToipa3.hpc.inl.gov" (ipa3:389):
Replication bind with GSSAPI auth resumed
[28/Jul/2021:08:09:10.449170075 -0600] - INFO - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=meToipa3.hpc.inl.gov" (ipa3:389):
Replication bind with GSSAPI auth resumed
[....]
I changed the Data Manager password with "dsconf" -- but that was
between the first failure and the second. Could that be causing
problems? What direction to go from here? Thank you!
Scott