FreeIPA-users December 2021

freeipa-users@lists.fedorahosted.org

58 participants
104 discussions

Re: ipa-healthcheck: ReplicationCheck ERROR
by Rob Crittenden 02 Dec '21

02 Dec '21

Dungan, Scott A. via FreeIPA-users wrote: > We have 3 ipa servers, one of which is throwing an ERROR condition > during ipa-healthcheck for the "ReplicationCheck" test. Ipa-healthcheck > shows no errors when run from the other two replicas. Looking back at > the logs, it appears this started about ten days ago, so it is not a > transient issue as the output suggests: > > > > [root(a)ipa1.id.example.com]# ipa-healthcheck --failures-only > > > > [ > > { > > "source": "ipahealthcheck.ds.replication", > > "check": "ReplicationCheck", > > "result": "ERROR", > > "uuid": "2b971ca3-678e-4c26-86a0-5b352027e7e8", > > "when": "20211201180013Z", > > "duration": "0.687812", > > "kw": { > > "key": "DSREPLLE0003", > > "items": [ > > "Replication", > > "Agreement" > > ], > > "msg": "The replication agreement (catoipa2.id.example.com) under > \"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't > acquire replica (incremental update transient warning. backing off, > will retry update later.)" > > } > > }, > > { > > "source": "ipahealthcheck.ds.replication", > > "check": "ReplicationCheck", > > "result": "ERROR", > > "uuid": "99436870-bc98-4ce8-84b1-c0b0806945c8", > > "when": "20211201180013Z", > > "duration": "0.687829", > > "kw": { > > "key": "DSREPLLE0003", > > "items": [ > > "Replication", > > "Agreement" > > ], > > "msg": "The replication agreement (catoipa3.id.example.com) under > \"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't > acquire replica (incremental update transient warning. backing off, > will retry update later.)" > > } > > } > > ] > > > > 389-ds error logs show a slew of these: > > > > [30/Nov/2021:23:41:35.277399980 -0800] - ERR - NSMMReplicationPlugin - > send_updates - agmt="cn=caToipa3.id.example.com" (ipa2:389): Missing > data encountered. If the error persists the replica must be reinitialized. > > [30/Nov/2021:23:41:38.288003253 -0800] - ERR - > agmt="cn=caToipa3.id.example.com" (ipa3:389) - clcache_load_buffer - > Can't locate CSN 6197e149000000060000 in the changelog (DB rc=-30988). > If replication stops, the consumer may need to be reinitialized. > > [30/Nov/2021:23:41:38.289713999 -0800] - ERR - NSMMReplicationPlugin - > send_updates - agmt="cn=caToipa3.id.example.com" (ipa3:389): Missing > data encountered. If the error persists the replica must be reinitialized. > > > > That would seem to suggest running a "ipa-replica-manage re-initialize > --from $SERVER_TO_PULL_FROM" may resolve the issue, but before we try > that, is there anything else we should look at? You use ipa-csreplica-manage to manage the CA replication agreements. But yes, it looks like you need to re-initialize some of them. I'd suggest dump the ldif of the two to be re-inited and see if there are any entries there not recorded in the one you will re-init from to see if there is any potential data loss. rob

2 1

Another pki-tomcatd failing to start due to expired certs
by Jacob Block 02 Dec '21

02 Dec '21

Hi all, I have read through pretty much every thread on this topic and unfortunately will be starting a new one. I am trying to upgrade an older IPA server that has had all the cert-pki-ca certs expired. Some other history, the initial master used to be on a VPS and was moved on-site several years ago by spinning up a replica on-site, promoting it to the new master, and shutting down the master. I am not entirely convinced there wasn't some issue also before the expired certs. There is also no other replica. I'd like to get this working, create a replica, and start upgrading to the latest. # ipa --version VERSION: 4.6.4, API_VERSION: 2.230 # getcert list Number of certificates and requests being tracked: 9. Request ID '20190405192115': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-IPA-COMPANY-COM/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-IPA-COMPANY-COM',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=IPA.COMPANY.COM subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM expires: 2023-03-09 22:30:53 UTC dns: ipa.internal.company.com principal name: ldap/ipa.internal.company.com(a)IPA.COMPANY.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv IPA-COMPANY-COM track: yes auto-renew: yes Request ID '20190405192140': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt' certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=IPA.COMPANY.COM subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM expires: 2023-03-09 22:31:53 UTC dns: ipa.internal.company.com principal name: HTTP/ipa.internal.company.com(a)IPA.COMPANY.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20190405192207': status: NEED_GUIDANCE stuck: yes key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key' certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.COMPANY.COM subject: CN=IPA RA,O=IPA.COMPANY.COM expires: 2021-09-05 16:48:11 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20190405192208': status: MONITORING stuck: no key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key' certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt' CA: IPA issuer: CN=Certificate Authority,O=IPA.COMPANY.COM subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM expires: 2023-03-09 22:30:44 UTC principal name: krbtgt/IPA.COMPANY.COM(a)IPA.COMPANY.COM key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-pkinit-KPKdc pre-save command: post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20190405204557': status: NEED_GUIDANCE stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.COMPANY.COM subject: CN=CA Audit,O=IPA.COMPANY.COM expires: 2021-09-05 16:48:31 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190405204558': status: GENERATING_CSR stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.COMPANY.COM subject: CN=OCSP Subsystem,O=IPA.COMPANY.COM expires: 2021-09-05 16:49:41 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190405204559': status: NEED_GUIDANCE stuck: yes key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.COMPANY.COM subject: CN=CA Subsystem,O=IPA.COMPANY.COM expires: 2021-09-05 16:48:21 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190405204600': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.COMPANY.COM subject: CN=Certificate Authority,O=IPA.COMPANY.COM expires: 2041-09-01 05:41:44 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20190405204601': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=IPA.COMPANY.COM subject: CN=ipa.internal.company.com,O=IPA.COMPANY.COM expires: 2023-02-15 22:30:43 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes The renewal master used to be the remote VPS master that no longer exists. I've since updated that: # ipa config-show | grep renewal IPA CA renewal master: ipa.internal.company.com One thing I am confused by is seeing four entries for "caSigningCert cert-pki-ca" (I also have a tenuous understanding of CAs and certs) # certutil -L -d /var/lib/pki/pki-tomcat/alias/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu DSTRootCAX3 C,, CN=R3,O=Let's Encrypt,C=US C,, CN=E1,O=Let's Encrypt,C=US C,, auditSigningCert cert-pki-ca u,u,Pu ocspSigningCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu caSigningCert cert-pki-ca CTu,Cu,Cu caSigningCert cert-pki-ca CTu,Cu,Cu ISRGRootCAX3 C,, ISRGRootCAX3 C,, ISRGRootCAX1 C,, CN=ISRG Root X2,O=Internet Security Research Group,C=US C,, CN=R4,O=Let's Encrypt,C=US C,, CN=E2,O=Let's Encrypt,C=US C,, I've tried rolling back the clock to before 2021-09-05 but pki-tomcatd still doesn't start: Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore() begins Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore(): tag=internaldb Jun 01 05:15:44 ipa.internal.company.com server[919212]: CMSEngine.initializePasswordStore(): tag=replicationdb Jun 01 05:15:45 ipa.internal.company.com server[919212]: Internal Database Error encountered: Could not connect to LDAP server host ipa.internal.company.com port 636 Error netscape.ldap.LDAPException: Unable to create socket: org.mozilla.jss.ssl.SSLSocketException: org.mozilla.jss.ssl.SSLSocketException: SSL_ForceHandshake failed: (-8172) Peer's certificate issuer has been marked as not trusted by the user. (-1) Jun 01 05:15:55 ipa.internal.company.com server[919212]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@70aacdbc background process Jun 01 05:15:55 ipa.internal.company.com server[919212]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Jun 01 05:15:55 ipa.internal.company.com server[919212]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) Jun 01 05:15:55 ipa.internal.company.com server[919212]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) Jun 01 05:15:55 ipa.internal.company.com server[919212]: at java.lang.Thread.run(Thread.java:748) Maybe its pki certs + https certs are both having a problem? Maybe this is related to a recent LE CA? Any thoughts would be greatly appreciated. Thank you!

3 3

ipa-healthcheck: ReplicationCheck ERROR
by Dungan, Scott A. 02 Dec '21

02 Dec '21

We have 3 ipa servers, one of which is throwing an ERROR condition during ipa-healthcheck for the "ReplicationCheck" test. Ipa-healthcheck shows no errors when run from the other two replicas. Looking back at the logs, it appears this started about ten days ago, so it is not a transient issue as the output suggests: [root(a)ipa1.id.example.com]# ipa-healthcheck --failures-only [ { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "ERROR", "uuid": "2b971ca3-678e-4c26-86a0-5b352027e7e8", "when": "20211201180013Z", "duration": "0.687812", "kw": { "key": "DSREPLLE0003", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (catoipa2.id.example.com) under \"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't acquire replica (incremental update transient warning. backing off, will retry update later.)" } }, { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "ERROR", "uuid": "99436870-bc98-4ce8-84b1-c0b0806945c8", "when": "20211201180013Z", "duration": "0.687829", "kw": { "key": "DSREPLLE0003", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (catoipa3.id.example.com) under \"o=ipaca\" is not in synchronization.\nStatus message: error (18) can't acquire replica (incremental update transient warning. backing off, will retry update later.)" } } ] 389-ds error logs show a slew of these: [30/Nov/2021:23:41:35.277399980 -0800] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caToipa3.id.example.com" (ipa2:389): Missing data encountered. If the error persists the replica must be reinitialized. [30/Nov/2021:23:41:38.288003253 -0800] - ERR - agmt="cn=caToipa3.id.example.com" (ipa3:389) - clcache_load_buffer - Can't locate CSN 6197e149000000060000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized. [30/Nov/2021:23:41:38.289713999 -0800] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caToipa3.id.example.com" (ipa3:389): Missing data encountered. If the error persists the replica must be reinitialized. That would seem to suggest running a "ipa-replica-manage re-initialize --from $SERVER_TO_PULL_FROM" may resolve the issue, but before we try that, is there anything else we should look at? Thanks, Scott

1 0

EMC Isilon and IPA - Kerberos
by thing.thing＠gmail.com 01 Dec '21

01 Dec '21

I have the Isilon talking to IPA for LDAP. What I cannot yet do is run the Isilon command to make kerberos work. ===== tststocoiso-1# kinit admin(a)ODSTEST.VUWTEST.AC.NZ Password for admin(a)ODSTEST.VUWTEST.AC.NZ: tststocoiso-1# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: admin(a)ODSTEST.VUWTEST.AC.NZ Valid starting Expires Service principal 11/30/21 16:44:56 12/01/21 16:10:10 krbtgt/ODSTEST.VUWTEST.AC.NZ(a)ODSTEST.VUWTEST.AC.NZ tststocoiso-1# isi auth krb5 spn fix --provider-name=ODSTEST.VUWTEST.AC.NZ --user=admin password: Attempting to add missing SPNs: HTTP/tststocoisnfs01.odstest.vuwtest.ac.nz(a)ODSTEST.VUWTEST.AC.NZ hdfs/tststocoisnfs01.odstest.vuwtest.ac.nz(a)ODSTEST.VUWTEST.AC.NZ host/tststocoisnfs01.odstest.vuwtest.ac.nz(a)ODSTEST.VUWTEST.AC.NZ nfs/tststocoisnfs01.odstest.vuwtest.ac.nz(a)ODSTEST.VUWTEST.AC.NZ Failed to join realm: (LW_ERROR_KADM5_AUTH_ADD) Operation requires ``add'' privilege tststocoiso-1# ==== What is the add privilege? how do I grant it to admin? TY.

3 2

2024

2023

2022

2021

2020

2019

2018

2017

FreeIPA-users December 2021