On 8/7/17 1:44 AM, thierry bordaz wrote:
On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
> On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
>> On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
>>
>>> On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
>>>> On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
>>>>> On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
>>>>>> On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
>>>>>>> On 08/02/2017 01:43 AM, Ian Harding wrote:
>>>>>>>> On 08/01/2017 12:03 PM, Rob Crittenden wrote:
>>>>>>>>> Ian Harding wrote:
>>>>>>>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud
wrote:
>>>>>>>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote:
>>>>>>>>>>>> On 08/01/2017 01:48 AM, Florence
Blanc-Renaud wrote:
>>>>>>>>>>>>> On 08/01/2017 01:32 AM, Ian Harding
via FreeIPA-users wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 07/31/2017 11:34 AM, Rob
Crittenden wrote:
>>>>>>>>>>>>>>> Ian Harding via FreeIPA-users
wrote:
>>>>>>>>>>>>>>>> I had an unexpected
restart of an IPA server that had
>>>>>>>>>>>>>>>> apparently had
>>>>>>>>>>>>>>>> updates run but had not
been restarted. ipactl says
>>>>>>>>>>>>>>>> pki-tomcatd
>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>> not start.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Strangely, the actual
service appears to be running:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> dogtag is an application
within tomcat so tomcat can
>>>>>>>>>>>>>>> run without
>>>>>>>>>>>>>>> dogtag
>>>>>>>>>>>>>>> running.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We need to see more of the
dogtag debug log to see what
>>>>>>>>>>>>>>> is going on.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It looks like an authentication
problem...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL
>>>>>>>>>>>>>> handshake happened
>>>>>>>>>>>>>> Could not connect to LDAP server
host
>>>>>>>>>>>>>> seattlenfs.bpt.rocks port 636
>>>>>>>>>>>>>> Error
netscape.ldap.LDAPException: Authentication failed
>>>>>>>>>>>>>> (49)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> dogtag stores its internal data in
the LDAP server and
>>>>>>>>>>>>> needs to
>>>>>>>>>>>>> establish a secure LDAP connection.
You can check how this
>>>>>>>>>>>>> connection is configured in
>>>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg, look
for
>>>>>>>>>>>>> the lines:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
internaldb.ldapauth.authtype=SslClientAuth
>>>>>>>>>>>>>
internaldb.ldapauth.bindDN=cn=Directory Manager
>>>>>>>>>>>>>
internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>>>>>>>>>
internaldb.ldapauth.clientCertNickname=subsystemCert
>>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>> internaldb.ldapconn.host=vm-...
>>>>>>>>>>>>> internaldb.ldapconn.port=636
>>>>>>>>>>>>> internaldb.ldapconn.secureConn
>>>>>>>>>>>>>
>>>>>>>>>>>>> authtype can be SslClientAuth
(authentication with a ssl
>>>>>>>>>>>>> certificate) or BasicAuth
(authentication with a bind DN and
>>>>>>>>>>>>> password stored in
>>>>>>>>>>>>>
/var/lib/pki/pki-tomcat/conf/password.conf).
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can use this information to
manually check the
>>>>>>>>>>>>> credentials. For
>>>>>>>>>>>>> instance with sslclientauth:
>>>>>>>>>>>>>
>>>>>>>>>>>>> export
LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
>>>>>>>>>>>>> export
LDAPTLS_CERT='subsystemCert cert-pki-ca'
>>>>>>>>>>>>>
>>>>>>>>>>>>> ldapsearch -H ldaps://`hostname`:636
-b "" -s base -Y
>>>>>>>>>>>>> EXTERNAL
>>>>>>>>>>>>> (provide the password from
>>>>>>>>>>>>>
/etc/pki/pki-tomcat/alias/pwdfile.txt)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I found this:
>>>>>>>>>>>>
>>>>>>>>>>>>
internaldb.ldapauth.authtype=SslClientAuth
>>>>>>>>>>>>
internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>>>>
internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>>>>>>>>
internaldb.ldapauth.clientCertNickname=subsystemCert
>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>
internaldb.ldapconn.cloneReplicationPort=389
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>> and when I try the ldapsearch I am
presented with a prompt
>>>>>>>>>>>> to provide
>>>>>>>>>>>> a pin/password
>>>>>>>>>>>>
>>>>>>>>>>>> Please enter pin, password, or pass
phrase for security
>>>>>>>>>>>> token 'ldap(0)':
>>>>>>>>>>>>
>>>>>>>>>>>> but there is no password file...
>>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> you are right, in 4.4. there is no
pwdfile.txt and the
>>>>>>>>>>> password can be
>>>>>>>>>>> found in
/var/lib/pki/pki-tomcat/conf/password.conf (with
>>>>>>>>>>> the tag
>>>>>>>>>>> internal=...)
>>>>>>>>>>>
>>>>>>>>>>> Can you check if the password with the tag
internal=...
>>>>>>>>>>> allows to read
>>>>>>>>>>> the keys from the NSS db?
>>>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>>>>> (provide password)
>>>>>>>>>>
>>>>>>>>>> That works...
>>>>>>>>>>
>>>>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>>>> certutil: Checking token "NSS Certificate
DB" in slot "NSS
>>>>>>>>>> User Private
>>>>>>>>>> Key and Certificate Services"
>>>>>>>>>> Enter Password or Pin for "NSS Certificate
DB":
>>>>>>>>>> < 0> rsa
0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan)
>>>>>>>>>> < 1> rsa
b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS
>>>>>>>>>> Certificate
>>>>>>>>>> DB:auditSigningCert cert-pki-ca
>>>>>>>>>> < 2> rsa
881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert
>>>>>>>>>> cert-pki-ca
>>>>>>>>>> < 3> rsa
fa9a255a1d15585ac28064c0f4986e416bc48403 NSS
>>>>>>>>>> Certificate
>>>>>>>>>> DB:ocspSigningCert cert-pki-ca
>>>>>>>>>> < 4> rsa
3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert
>>>>>>>>>> cert-pki-ca
>>>>>>>>>> < 5> rsa
1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS
>>>>>>>>>> Certificate
>>>>>>>>>> DB:subsystemCert cert-pki-ca
>>>>>>>>>>
>>>>>>>>>> But this doesn't (with the same password from
password.conf)
>>>>>>>>>>
>>>>>>>>>> # ldapsearch -H ldaps://`hostname`:636 -b
"" -s base -Y
>>>>>>>>>> EXTERNAL
>>>>>>>>>> Please enter pin, password, or pass phrase for
security
>>>>>>>>>> token 'ldap(0)':
>>>>>>>>>> SASL/EXTERNAL authentication started
>>>>>>>>>> ldap_sasl_interactive_bind_s: Invalid credentials
(49)
>>>>>>>>>>
>>>>>>>>>> That password is getting me somewhere though,
since if I put
>>>>>>>>>> in a
>>>>>>>>>> nonsense or incorrect password it just prompts
over and over.
>>>>>>>>>
>>>>>>>>> Let's step back a second. You upgraded from what
to what?
>>>>>>>>
>>>>>>>> There wasn't much of a change... I just assumed
someone ran
>>>>>>>> yum upgrade and didn't restart, then the power
outage... it
>>>>>>>> looks like not much of a version change though.
>>>>>>>>
>>>>>>>> # grep ipa /var/log/yum.log
>>>>>>>> Jan 08 04:45:32 Installed:
>>>>>>>> ipa-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>> Jan 08 04:45:32 Installed:
>>>>>>>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>> Jan 08 04:46:06 Updated:
libipa_hbac-1.14.0-43.el7_3.4.x86_64
>>>>>>>> Jan 08 04:46:07 Updated:
>>>>>>>> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64
>>>>>>>> Jan 08 04:46:08 Installed:
python-ipaddress-1.0.16-2.el7.noarch
>>>>>>>> Jan 08 04:47:04 Installed:
>>>>>>>> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>> Jan 08 04:47:05 Installed:
>>>>>>>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>> Jan 08 04:47:05 Updated:
>>>>>>>> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>> Jan 08 04:47:05 Installed:
>>>>>>>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>> Jan 08 04:47:06 Installed:
>>>>>>>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>> Jan 08 04:47:07 Updated:
sssd-ipa-1.14.0-43.el7_3.4.x86_64
>>>>>>>> Jan 08 04:47:17 Updated:
>>>>>>>> ipa-client-4.4.0-14.el7.centos.1.1.x86_64
>>>>>>>> Jan 08 04:47:23 Updated:
>>>>>>>> ipa-server-4.4.0-14.el7.centos.1.1.x86_64
>>>>>>>> Jan 08 04:47:27 Updated:
>>>>>>>> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>> Jan 08 04:47:36 Installed:
>>>>>>>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>> Jan 08 04:50:07 Erased:
>>>>>>>> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64
>>>>>>>> Jul 28 09:55:41 Updated:
ipa-common-4.4.0-14.el7.centos.7.noarch
>>>>>>>> Jul 28 09:55:42 Updated:
>>>>>>>> ipa-client-common-4.4.0-14.el7.centos.7.noarch
>>>>>>>> Jul 28 09:55:43 Updated:
>>>>>>>> ipa-server-common-4.4.0-14.el7.centos.7.noarch
>>>>>>>> Jul 28 09:55:43 Updated:
>>>>>>>> python2-ipalib-4.4.0-14.el7.centos.7.noarch
>>>>>>>> Jul 28 09:55:44 Updated:
>>>>>>>> python2-ipaclient-4.4.0-14.el7.centos.7.noarch
>>>>>>>> Jul 28 09:55:45 Updated:
>>>>>>>> python2-ipaserver-4.4.0-14.el7.centos.7.noarch
>>>>>>>> Jul 28 09:55:45 Updated:
>>>>>>>> ipa-admintools-4.4.0-14.el7.centos.7.noarch
>>>>>>>> Jul 28 09:55:46 Updated:
ipa-client-4.4.0-14.el7.centos.7.x86_64
>>>>>>>> Jul 28 09:55:48 Updated:
ipa-server-4.4.0-14.el7.centos.7.x86_64
>>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>>> ipa-server-dns-4.4.0-14.el7.centos.7.noarch
>>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>>> ipa-python-compat-4.4.0-14.el7.centos.7.noarch
>>>>>>>>
>>>>>>>> # grep pki /var/log/yum.log
>>>>>>>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch
>>>>>>>> Jan 08 04:46:09 Updated:
krb5-pkinit-1.14.1-27.el7_3.x86_64
>>>>>>>> Jan 08 04:47:19 Installed:
pki-base-java-10.3.3-14.el7_3.noarch
>>>>>>>> Jan 08 04:47:19 Updated:
pki-tools-10.3.3-14.el7_3.x86_64
>>>>>>>> Jan 08 04:47:21 Updated:
pki-server-10.3.3-14.el7_3.noarch
>>>>>>>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch
>>>>>>>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch
>>>>>>>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch
>>>>>>>> Jul 28 10:13:17 Updated:
pki-base-java-10.3.3-19.el7_3.noarch
>>>>>>>> Jul 28 10:13:17 Updated:
pki-tools-10.3.3-19.el7_3.x86_64
>>>>>>>> Jul 28 10:13:21 Updated:
pki-server-10.3.3-19.el7_3.noarch
>>>>>>>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch
>>>>>>>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Are you running in SELinux enforcing mode?
>>>>>>>>
>>>>>>>> # getenforce
>>>>>>>> Enforcing
>>>>>>>>
>>>>>>>>>
>>>>>>>>> If so can you run:
>>>>>>>>>
>>>>>>>>> # ausearch -m AVC
>>>>>>>>>
>>>>>>>>
>>>>>>>> # ausearch -m AVC
>>>>>>>> The NOLOG option to log_format is deprecated. Please use
the
>>>>>>>> write_logs option.
>>>>>>>> The NOLOG option is overriding the write_logs current
setting.
>>>>>>>> ----
>>>>>>>> time->Mon Mar 14 10:39:01 2016
>>>>>>>> type=SYSCALL msg=audit(1457977141.767:17537):
arch=c000003e
>>>>>>>> syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1
>>>>>>>> a3=7fbca60f42e0 items=0 ppid=1406 pid=12965
auid=4294967295
>>>>>>>> uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001
>>>>>>>> egid=581400001 sgid=0 fsgid=581400001 tty=(none)
>>>>>>>> ses=4294967295 comm="sshd"
exe="/usr/sbin/sshd"
>>>>>>>> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null)
>>>>>>>> type=AVC msg=audit(1457977141.767:17537): avc: denied {
read }
>>>>>>>> for pid=12965 comm="sshd"
name="authorized_keys" dev="dm-2"
>>>>>>>> ino=116393517
scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
>>>>>>>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
>>>>>>>>
>>>>>>>>
>>>>>>>>> I suspect that the subsystem cert was renewed and
everything
>>>>>>>>> wasn't
>>>>>>>>> updated properly. If I'm right this is unrelated
to the
>>>>>>>>> upgrade it just
>>>>>>>>> shone a spotlight on it.
>>>>>>>>>
>>>>>>>>> Can you run these commands:
>>>>>>>>>
>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager'
-W -b
>>>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate
description
>>>>>>>>>
>>>>>>>>
>>>>>>>> This returns
>>>>>>>>
>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
>>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate
description
>>>>>>>> Enter LDAP Password:
>>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>>> userCertificate::
>>>>>>>>
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>>> ... stuff ...
>>>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>>> description: 2;4;CN=Certificate
Authority,O=BPT.ROCKS;CN=CA
>>>>>>>> Subsystem,O=BPT.ROCKS
>>>>>>>>
>>>>>>>>> and
>>>>>>>>>
>>>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n
'subsystemCert
>>>>>>>>> cert-pki-ca' | grep Serial
>>>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n
'subsystemCert
>>>>>>>>> cert-pki-ca' -a
>>>>>>>>>
>>>>>>>>
>>>>>>>> These both return
>>>>>>>>
>>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n
'subsystemCert
>>>>>>>> cert-pki-ca' | grep Serial
>>>>>>>> Enter Password or Pin for "NSS Certificate
DB":
>>>>>>>> certutil: problem listing keys:
SEC_ERROR_UNRECOGNIZED_OID:
>>>>>>>> Unrecognized Object Identifier.
>>>>>>>>
>>>>>>>>
>>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n
'subsystemCert
>>>>>>>> cert-pki-ca' -a
>>>>>>>> certutil: Checking token "NSS Certificate DB"
in slot "NSS
>>>>>>>> User Private Key and Certificate Services"
>>>>>>>> Enter Password or Pin for "NSS Certificate
DB":
>>>>>>>> certutil: problem listing keys:
SEC_ERROR_UNRECOGNIZED_OID:
>>>>>>>> Unrecognized Object Identifier.
>>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> you made a typo and requested the keys (-K) instead of the
>>>>>>> certs (-L). Please retry with -L and you should be able to
see
>>>>>>> the certificate serial.
>>>>>>>
>>>>>>
>>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory
>>>>>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca
userCertificate
>>>>>> description
>>>>>> Enter LDAP Password:
>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>> userCertificate::
>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>> ...
>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>>>>>> Subsystem,O=BPT.ROCKS
>>>>>>
>>>>>> So the serial is 4?
>>>>>>
>>>>>> [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias
>>>>>> -n 'subsystemCert cert-pki-ca' | grep Serial
>>>>>> Serial Number: 1341718541 (0x4ff9000d)
>>>>>>
>>>>>> Which is NOT 4...
>>>>>>
>>>>>> [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias
>>>>>> -n 'subsystemCert cert-pki-ca' -a
>>>>>> -----BEGIN CERTIFICATE-----
>>>>>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>> ...
>>>>>> sjeiFbUFtoMnfmHLIYpe5QAR
>>>>>> -----END CERTIFICATE-----
>>>>>>
>>>>>> And this cert is NOT the same.
>>>>>>
>>>>> Hi,
>>>>>
>>>>> when certmonger renews a certificate, it runs pre-save and
>>>>> post-save commands (which you can see in the output of getcert
>>>>> list). In your case, the post-save command for subsystemCert
>>>>> cert-pki-ca probably failed.
>>>>>
>>>>> This command (on the renewal master) is responsible for copying
>>>>> the new cert from the NSS db to the LDAP server. You need first
>>>>> to check which IPA server is the renewal master (as the serial
>>>>> numbers are in a completely different range, I assume that you
>>>>> have multiple IPA servers configured with a CA instance):
>>>>>
>>>>> $ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-`
>>>>> $ kinit admin
>>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b
>>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN"
>>>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
>>>>> dn:
>>>>>
cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
>>>>>
>>>>>
>>>>> In this example the renewal master is vm-ipaserver.
>>>>>
>>>>
>>>> I get this
>>>>
>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b
>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN"
>>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
>>>> dn:
>>>> cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
>>>>
>>>>
>>>> Which is the other FreeIPA server.
>>>>
>>> Hi Ian,
>>>
>>> it's getting difficult to follow this thread, so could we step back
>>> and summarize?
>> Yes, it is. Sorry.
>>>
>>> - On server freeipa-sea (=renewal master), the CA component is
>>> installed and 'subsystemCert cert-pki-ca' has Serial number 4 in
>>> the NSSDB /etc/pki/pki-tomcat/alias
>> No?
>>
>> [root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n
>> 'subsystemCert cert-pki-ca' | grep Serial
>> Serial Number: 1341718541 (0x4ff9000d)
>>> - On server seattlenfs (where PKI fails to restart), the CA
>>> component is installed and 'subsystemCert cert-pki-ca' has Serial
>>> number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
>> Yes
>>
>> [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n
>> 'subsystemCert cert-pki-ca' | grep Serial
>> Serial Number: 1341718541 (0x4ff9000d)
>>
>>>
>>> - In the LDAP servers, the cert is stored in the entry
>>> uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is
>>> the cert from freeipa-sea).
>> This is where the difference is. There are two certificates on the
>> master and only one on the other. I wonder if this is a result of
>> the slow motion replication trainwreck I have going on due to a
>> zombie replica in the ldap database that no longer exists in the
>> real world and I can't seem to get rid of.
>>
> Hi,
>
> the remaining issue is a replication problem. There is a
> "Troubleshooting replication" section [1] in IdM Admin guide that may
> help you.
>
> Regarding your zombie replica, which commands did you use to try to
> remove it, and what was the output?
>
> Flo
>
> [1]
>
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/...
>
Hi,
To debug replication latency or problem we would need several info but
starting with the results of the following commands
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W
-b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
[root@freeipa-sea
ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
"uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi:
entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005:
75274897
nscpentrywsi:
modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042
90004: 20170608055334Z
nscpentrywsi:
modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290
003: cn=Directory Manager
nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074
nscpentrywsi: cn: pkidbuser
nscpentrywsi: createTimestamp: 20150716125146Z
nscpentrywsi: creatorsName: cn=directory manager
nscpentrywsi: description;vucsn-5938e688000104290001:
2;1341718541;CN=Certific
ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
nscpentrywsi: description;vdcsn-5938e688000104290002;deleted:
2;4;CN=Certifica
te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
nscpentrywsi: mail:
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: person
nscpentrywsi: objectClass: organizationalPerson
nscpentrywsi: objectClass: inetOrgPerson
nscpentrywsi: objectClass: cmsuser
nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS
nscpentrywsi: sn: pkidbuser
nscpentrywsi: uid: pkidbuser
nscpentrywsi: userCertificate::
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip>
cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
nscpentrywsi: userCertificate;vucsn-5938e688000104290000::
MIIDbjCCAlagAwIBAgI
<snip>
AR
nscpentrywsi: userPassword:: <snip>
nscpentrywsi: userstate: 1
nscpentrywsi: usertype: agentType
nscpentrywsi: parentid: 2
nscpentrywsi: entryid: 51
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W
-b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS
nscpentrywsi: userCertificate::
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip>
cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
nscpentrywsi: description: 2;4;CN=Certificate
Authority,O=BPT.ROCKS;CN=CA Subs
ystem,O=BPT.ROCKS
nscpentrywsi: entryid: 51
nscpentrywsi: parentid: 2
nscpentrywsi: modifyTimestamp: 20150716125146Z
nscpentrywsi: createTimestamp: 20150716125146Z
nscpentrywsi: modifiersName: cn=directory manager
nscpentrywsi: creatorsName: cn=directory manager
nscpentrywsi: userPassword:: <snip>
nscpentrywsi: userstate: 1
nscpentrywsi: usertype: agentType
nscpentrywsi: mail:
nscpentrywsi: cn: pkidbuser
nscpentrywsi: sn: pkidbuser
nscpentrywsi: uid: pkidbuser
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: person
nscpentrywsi: objectClass: organizationalPerson
nscpentrywsi: objectClass: inetOrgPerson
nscpentrywsi: objectClass: cmsuser
nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074
nscpentrywsi: entryusn: 86
From the output you may only copy/past the ones related to userCertificate
Also to dump the current replication status, send the result of
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W
-b
"o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
"o=ipaca"
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
Enter LDAP Password:
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=Replication Manager
cloneAgreement1-freeipa-sea.bpt.roc
ks-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaBindDN: cn=Replication Manager
masterAgreement1-seattlenfs.bpt.roc
ks-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaId: 1065
nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaType: 3
nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA==
nsds5replicabinddngroup: cn=replication
managers,cn=sysaccounts,cn=etc,dc=bpt,
dc=rocks
nsds5replicabinddngroupcheckinterval: 60
objectClass: top
objectClass: nsDS5Replica
objectClass: extensibleobject
nsds50ruv: {replicageneration} 57c291d9000004290000
nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
57f840bf00000429000
0 598a1c40000104290000
nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
nsds5agmtmaxcsn:
o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat
tlenfs.bpt.rocks;389;unavailable
nsds5agmtmaxcsn:
o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat
tlenfs.bpt.rocks;389;unavailable
nsruvReplicaLastModified: {replica 1065
ldap://freeipa-sea.bpt.rocks:389} 598a
1c16
nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
00000
000
nsds5ReplicaChangeCount: 885
nsds5replicareapactive: 0
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W
-b
"o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
"o=ipaca"
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
Enter LDAP Password:
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=Replication Manager
cloneAgreement1-seattlenfs.bpt.rock
s-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaId: 1290
nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaType: 3
nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA==
nsds5replicabinddngroup: cn=replication
managers,cn=sysaccounts,cn=etc,dc=bpt,
dc=rocks
nsds5replicabinddngroupcheckinterval: 60
objectClass: top
objectClass: nsDS5Replica
objectClass: extensibleobject
nsds50ruv: {replicageneration} 55c8f3ae000000600000
nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
57be804c0000050a0000
587236150000050a0000
nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
57b103d400000429000
0 57be8043000704290000
nsds5agmtmaxcsn:
o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei
pa-sea.bpt.rocks;389;1065;587236150000050a0000
nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
00000
000
nsruvReplicaLastModified: {replica 1065
ldap://freeipa-sea.bpt.rocks:389} 0000
0000
nsds5ReplicaChangeCount: 3
nsds5replicareapactive: 0
Regards
thierry
>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager'
>> -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
>> Enter LDAP Password:
>> dn: uid=pkidbuser,ou=people,o=ipaca
>> userCertificate::
>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>> <truncated>
>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>> userCertificate::
>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK
>> <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR
>> description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>> Subsystem
>> ,O=BPT.ROCKS
>>
>>
>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W
>> -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
>> Enter LDAP Password:
>> dn: uid=pkidbuser,ou=people,o=ipaca
>> userCertificate::
>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>> <truncated>
>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>> Subsystem,O=BPT.RO
>> CKS
>>
>>> If you check the certificates dates, I guess that the most recent
>>> one is on server seattlenfs (because of serial numbers). Can you
>>> confirm this (because our goal is to have the most recent cert
>>> stored in all the NSSDBs and in LDAP)?
>>>
>>> Thanks,
>>> Flo
>>>
>>>
>>>>> If your renewal master is the machine on which the NSS DB
>>>>> /etc/pki/pki-tomcat/alias contains the newer cert, then you can
>>>>> manually run the scripts:
>>>>
>>>> If I'm interpreting this correctly, then it is... so I ran this on
>>>> the broken machine (seattlenfs)
>>>>
>>>>> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad
>>>>> $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert
>>>>> cert-pki-ca"
>>>>>
>>>>
>>>> I ran this, it took a while, then I re-checked and the serial
>>>> still seems to be 4
>>>>
>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description
>>>> Enter LDAP Password:
>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>> userCertificate::
>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>> ...
>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>>>> Subsystem,O=BPT.ROCKS
>>>>
>>>> Here's what was in /var/log/pki/pki-tomcat/ca/debug
>>>>
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>> ============================================
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG
>>>> SUBSYSTEM INITIALIZED =======
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>> ============================================
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart
>>>> at autoShutdown? false
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>> autoShutdown crumb file path?
>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to
>>>> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found
>>>> cert:auditSigningCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done
>>>> init id=debug
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>> initialized debug
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>> initSubsystem id=log
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to
>>>> init id=log
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart
>>>> at autoShutdown? false
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>> autoShutdown crumb file path?
>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to
>>>> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found
>>>> cert:auditSigningCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done
>>>> init id=log
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>> initialized log
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>> initSubsystem id=jss
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to
>>>> init id=jss
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart
>>>> at autoShutdown? false
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>> autoShutdown crumb file path?
>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to
>>>> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found
>>>> cert:auditSigningCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done
>>>> init id=jss
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>> initialized jss
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>> initSubsystem id=dbs
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to
>>>> init id=dbs
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init()
>>>> mEnableSerialMgmt=true
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>>> LdapBoundConnFactor(DBSubsystem)
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>> LdapBoundConnFactory: init
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>> LdapBoundConnFactory:doCloning true
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init()
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init
>>>> begins
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init
>>>> ends
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before
>>>> makeConnection errorIfDown is true
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection:
>>>> errorIfDown true
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>> SSLClientCertificateSelectionCB: Setting desired cert nickname to:
>>>> subsystemCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket:
>>>> set client auth cert nickname subsystemCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>> SSLClientCertificatSelectionCB: Entering!
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert:
>>>> ocspSigningCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert:
>>>> subsystemCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>> SSLClientCertificateSelectionCB: desired cert found in list:
>>>> subsystemCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>> SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened
>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()
>>>>
>>>>
>>>> I really appreciate you sticking with me through this. I owe you.
>>>>
>>>> - Ian
>>>>
>>>>> After that, check that the LDAP entry has been updated. If it is
>>>>> the case, you should be able to restart pki.
>>>>>
>>>>> Flo
>>>>>
>>>>>> Thank you again for your time!
>>>>>>
>>>>>> Ian
>>>>>>
>>>>>>> Flo.
>>>>>>>>
>>>>>>>> which seems weird.
>>>>>>>>
>>>>>>>>> The description attribute in the ldapsearch output is
of the
>>>>>>>>> form 2;#;DN;DN
>>>>>>>>>
>>>>>>>>> The # should match the serial number in the first
certutil
>>>>>>>>> output
>>>>>>>>>
>>>>>>>>> The ASCII blob (minus the BEGIN/END headers) should
match one
>>>>>>>>> of the
>>>>>>>>> userCertificate attributes.
>>>>>>>>>
>>>>>>>>> rob
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to
>> freeipa-users-leave(a)lists.fedorahosted.org
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org