Re: ipa-healthcheck: ReplicationCheck ERROR
by Rob Crittenden
Dungan, Scott A. via FreeIPA-users wrote:
> We have 3 ipa servers, one of which is throwing an ERROR condition
> during ipa-healthcheck for the "ReplicationCheck" test. Ipa-healthcheck
> shows no errors when run from the other two replicas. Looking back at
> the logs, it appears this started about ten days ago, so it is not a
> transient issue as the output suggests:
>
>
>
> [root(a)ipa1.id.example.com]# ipa-healthcheck --failures-only
>
>
>
> [
>
> {
>
> "source": "ipahealthcheck.ds.replication",
>
> "check": "ReplicationCheck",
>
> "result": "ERROR",
>
> "uuid": "2b971ca3-678e-4c26-86a0-5b352027e7e8",
>
> "when": "20211201180013Z",
>
> "duration": "0.687812",
>
> "kw": {
>
> "key": "DSREPLLE0003",
>
> "items": [
>
> "Replication",
>
> "Agreement"
>
> ],
>
> "msg": "The replication agreement (catoipa2.id.example.com) under
> \"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't
> acquire replica (incremental update transient warning. backing off,
> will retry update later.)"
>
> }
>
> },
>
> {
>
> "source": "ipahealthcheck.ds.replication",
>
> "check": "ReplicationCheck",
>
> "result": "ERROR",
>
> "uuid": "99436870-bc98-4ce8-84b1-c0b0806945c8",
>
> "when": "20211201180013Z",
>
> "duration": "0.687829",
>
> "kw": {
>
> "key": "DSREPLLE0003",
>
> "items": [
>
> "Replication",
>
> "Agreement"
>
> ],
>
> "msg": "The replication agreement (catoipa3.id.example.com) under
> \"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't
> acquire replica (incremental update transient warning. backing off,
> will retry update later.)"
>
> }
>
> }
>
> ]
>
>
>
> 389-ds error logs show a slew of these:
>
>
>
> [30/Nov/2021:23:41:35.277399980 -0800] - ERR - NSMMReplicationPlugin -
> send_updates - agmt="cn=caToipa3.id.example.com" (ipa2:389): Missing
> data encountered. If the error persists the replica must be reinitialized.
>
> [30/Nov/2021:23:41:38.288003253 -0800] - ERR -
> agmt="cn=caToipa3.id.example.com" (ipa3:389) - clcache_load_buffer -
> Can't locate CSN 6197e149000000060000 in the changelog (DB rc=-30988).
> If replication stops, the consumer may need to be reinitialized.
>
> [30/Nov/2021:23:41:38.289713999 -0800] - ERR - NSMMReplicationPlugin -
> send_updates - agmt="cn=caToipa3.id.example.com" (ipa3:389): Missing
> data encountered. If the error persists the replica must be reinitialized.
>
>
>
> That would seem to suggest running a "ipa-replica-manage re-initialize
> --from $SERVER_TO_PULL_FROM" may resolve the issue, but before we try
> that, is there anything else we should look at?
You use ipa-csreplica-manage to manage the CA replication agreements.
But yes, it looks like you need to re-initialize some of them.
I'd suggest dump the ldif of the two to be re-inited and see if there
are any entries there not recorded in the one you will re-init from to
see if there is any potential data loss.
rob
2 years, 4 months
Another pki-tomcatd failing to start due to expired certs
by Jacob Block
Hi all,
I have read through pretty much every thread on this topic and unfortunately will be starting a new one. I am trying to upgrade an older IPA server that has had all the cert-pki-ca certs expired. Some other history, the initial master used to be on a VPS and was moved on-site several years ago by spinning up a replica on-site, promoting it to the new master, and shutting down the master. I am not entirely convinced there wasn't some issue also before the expired certs. There is also no other replica. I'd like to get this working, create a replica, and start upgrading to the latest.
# ipa --version
VERSION: 4.6.4, API_VERSION: 2.230
# getcert list
Number of certificates and requests being tracked: 9.
Request ID '20190405192115':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-IPA-COMPANY-COM/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
expires: 2023-03-09 22:30:53 UTC
dns: ipa.internal.company.com
principal name: ldap/ipa.internal.company.com(a)IPA.COMPANY.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv IPA-COMPANY-COM
track: yes
auto-renew: yes
Request ID '20190405192140':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
expires: 2023-03-09 22:31:53 UTC
dns: ipa.internal.company.com
principal name: HTTP/ipa.internal.company.com(a)IPA.COMPANY.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID '20190405192207':
status: NEED_GUIDANCE
stuck: yes
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
subject: CN=IPA RA,O=IPA.COMPANY.COM
expires: 2021-09-05 16:48:11 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20190405192208':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: IPA
issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
expires: 2023-03-09 22:30:44 UTC
principal name: krbtgt/IPA.COMPANY.COM(a)IPA.COMPANY.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20190405204557':
status: NEED_GUIDANCE
stuck: yes
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
subject: CN=CA Audit,O=IPA.COMPANY.COM
expires: 2021-09-05 16:48:31 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190405204558':
status: GENERATING_CSR
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
subject: CN=OCSP Subsystem,O=IPA.COMPANY.COM
expires: 2021-09-05 16:49:41 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190405204559':
status: NEED_GUIDANCE
stuck: yes
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
subject: CN=CA Subsystem,O=IPA.COMPANY.COM
expires: 2021-09-05 16:48:21 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190405204600':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
subject: CN=Certificate Authority,O=IPA.COMPANY.COM
expires: 2041-09-01 05:41:44 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190405204601':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=IPA.COMPANY.COM
subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM
expires: 2023-02-15 22:30:43 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
The renewal master used to be the remote VPS master that no longer exists. I've since updated that:
# ipa config-show | grep renewal
IPA CA renewal master: ipa.internal.company.com
One thing I am confused by is seeing four entries for "caSigningCert cert-pki-ca" (I also have a tenuous understanding of CAs and certs)
# certutil -L -d /var/lib/pki/pki-tomcat/alias/
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
subsystemCert cert-pki-ca u,u,u
caSigningCert cert-pki-ca CTu,Cu,Cu
DSTRootCAX3 C,,
CN=R3,O=Let's Encrypt,C=US C,,
CN=E1,O=Let's Encrypt,C=US C,,
auditSigningCert cert-pki-ca u,u,Pu
ocspSigningCert cert-pki-ca u,u,u
Server-Cert cert-pki-ca u,u,u
caSigningCert cert-pki-ca CTu,Cu,Cu
caSigningCert cert-pki-ca CTu,Cu,Cu
caSigningCert cert-pki-ca CTu,Cu,Cu
ISRGRootCAX3 C,,
ISRGRootCAX3 C,,
ISRGRootCAX1 C,,
CN=ISRG Root X2,O=Internet Security Research Group,C=US C,,
CN=R4,O=Let's Encrypt,C=US C,,
CN=E2,O=Let's Encrypt,C=US C,,
I've tried rolling back the clock to before 2021-09-05 but pki-tomcatd still doesn't start:
Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore() begins
Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore(): tag=internaldb
Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore(): tag=replicationdb
Jun 01 05:15:45 ipa.internal.company.com server[919212]: Internal Database Error encountered: Could not connect to LDAP server host ipa.internal.company.com port 636 Error netscape.ldap.LDAPException: Unable to create socket: org.mozilla.jss.ssl.SSLSocketException: org.mozilla.jss.ssl.SSLSocketException: SSL_ForceHandshake failed: (-8172) Peer's certificate issuer has been marked as not trusted by the user. (-1)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@70aacdbc background process
Jun 01 05:15:55 ipa.internal.company.com server[919212]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Jun 01 05:15:55 ipa.internal.company.com server[919212]: at java.lang.Thread.run(Thread.java:748)
Maybe its pki certs + https certs are both having a problem? Maybe this is related to a recent LE CA?
Any thoughts would be greatly appreciated. Thank you!
2 years, 4 months
ipa-healthcheck: ReplicationCheck ERROR
by Dungan, Scott A.
We have 3 ipa servers, one of which is throwing an ERROR condition during ipa-healthcheck for the "ReplicationCheck" test. Ipa-healthcheck shows no errors when run from the other two replicas. Looking back at the logs, it appears this started about ten days ago, so it is not a transient issue as the output suggests:
[root(a)ipa1.id.example.com]# ipa-healthcheck --failures-only
[
{
"source": "ipahealthcheck.ds.replication",
"check": "ReplicationCheck",
"result": "ERROR",
"uuid": "2b971ca3-678e-4c26-86a0-5b352027e7e8",
"when": "20211201180013Z",
"duration": "0.687812",
"kw": {
"key": "DSREPLLE0003",
"items": [
"Replication",
"Agreement"
],
"msg": "The replication agreement (catoipa2.id.example.com) under \"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't acquire replica (incremental update transient warning. backing off, will retry update later.)"
}
},
{
"source": "ipahealthcheck.ds.replication",
"check": "ReplicationCheck",
"result": "ERROR",
"uuid": "99436870-bc98-4ce8-84b1-c0b0806945c8",
"when": "20211201180013Z",
"duration": "0.687829",
"kw": {
"key": "DSREPLLE0003",
"items": [
"Replication",
"Agreement"
],
"msg": "The replication agreement (catoipa3.id.example.com) under \"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't acquire replica (incremental update transient warning. backing off, will retry update later.)"
}
}
]
389-ds error logs show a slew of these:
[30/Nov/2021:23:41:35.277399980 -0800] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caToipa3.id.example.com" (ipa2:389): Missing data encountered. If the error persists the replica must be reinitialized.
[30/Nov/2021:23:41:38.288003253 -0800] - ERR - agmt="cn=caToipa3.id.example.com" (ipa3:389) - clcache_load_buffer - Can't locate CSN 6197e149000000060000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[30/Nov/2021:23:41:38.289713999 -0800] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caToipa3.id.example.com" (ipa3:389): Missing data encountered. If the error persists the replica must be reinitialized.
That would seem to suggest running a "ipa-replica-manage re-initialize --from $SERVER_TO_PULL_FROM" may resolve the issue, but before we try that, is there anything else we should look at?
Thanks,
Scott
2 years, 4 months
EMC Isilon and IPA - Kerberos
by thing.thing＠gmail.com
I have the Isilon talking to IPA for LDAP. What I cannot yet do is run the Isilon command to make kerberos work.
=====
tststocoiso-1# kinit admin(a)ODSTEST.VUWTEST.AC.NZ
Password for admin(a)ODSTEST.VUWTEST.AC.NZ:
tststocoiso-1# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin(a)ODSTEST.VUWTEST.AC.NZ
Valid starting Expires Service principal
11/30/21 16:44:56 12/01/21 16:10:10 krbtgt/ODSTEST.VUWTEST.AC.NZ(a)ODSTEST.VUWTEST.AC.NZ
tststocoiso-1# isi auth krb5 spn fix --provider-name=ODSTEST.VUWTEST.AC.NZ --user=admin
password:
Attempting to add missing SPNs:
HTTP/tststocoisnfs01.odstest.vuwtest.ac.nz(a)ODSTEST.VUWTEST.AC.NZ
hdfs/tststocoisnfs01.odstest.vuwtest.ac.nz(a)ODSTEST.VUWTEST.AC.NZ
host/tststocoisnfs01.odstest.vuwtest.ac.nz(a)ODSTEST.VUWTEST.AC.NZ
nfs/tststocoisnfs01.odstest.vuwtest.ac.nz(a)ODSTEST.VUWTEST.AC.NZ
Failed to join realm: (LW_ERROR_KADM5_AUTH_ADD) Operation requires ``add'' privilege
tststocoiso-1#
====
What is the add privilege? how do I grant it to admin?
TY.
2 years, 4 months