On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
> On 08/02/2017 01:43 AM, Ian Harding wrote:
>> On 08/01/2017 12:03 PM, Rob Crittenden wrote:
>>> Ian Harding wrote:
>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote:
>>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
>>>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote:
>>>>>>>>> Ian Harding via FreeIPA-users wrote:
>>>>>>>>>> I had an unexpected restart of an IPA server that
had
>>>>>>>>>> apparently had
>>>>>>>>>> updates run but had not been restarted. ipactl
says pki-tomcatd
>>>>>>>>>> would
>>>>>>>>>> not start.
>>>>>>>>>>
>>>>>>>>>> Strangely, the actual service appears to be
running:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> dogtag is an application within tomcat so tomcat can
run without
>>>>>>>>> dogtag
>>>>>>>>> running.
>>>>>>>>>
>>>>>>>>> We need to see more of the dogtag debug log to see
what is
>>>>>>>>> going on.
>>>>>>>>>
>>>>>>>>
>>>>>>>> It looks like an authentication problem...
>>>>>>>>
>>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL
handshake
>>>>>>>> happened
>>>>>>>> Could not connect to LDAP server host
seattlenfs.bpt.rocks port
>>>>>>>> 636
>>>>>>>> Error netscape.ldap.LDAPException: Authentication failed
(49)
>>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> dogtag stores its internal data in the LDAP server and needs
to
>>>>>>> establish a secure LDAP connection. You can check how this
>>>>>>> connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg,
look for
>>>>>>> the lines:
>>>>>>>
>>>>>>> internaldb.ldapauth.authtype=SslClientAuth
>>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager
>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert
cert-pki-ca
>>>>>>> internaldb.ldapconn.host=vm-...
>>>>>>> internaldb.ldapconn.port=636
>>>>>>> internaldb.ldapconn.secureConn
>>>>>>>
>>>>>>> authtype can be SslClientAuth (authentication with a ssl
>>>>>>> certificate) or BasicAuth (authentication with a bind DN and
>>>>>>> password stored in
/var/lib/pki/pki-tomcat/conf/password.conf).
>>>>>>>
>>>>>>> You can use this information to manually check the
credentials. For
>>>>>>> instance with sslclientauth:
>>>>>>>
>>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
>>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca'
>>>>>>>
>>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base
-Y EXTERNAL
>>>>>>> (provide the password from
/etc/pki/pki-tomcat/alias/pwdfile.txt)
>>>>>>>
>>>>>>
>>>>>> I found this:
>>>>>>
>>>>>> internaldb.ldapauth.authtype=SslClientAuth
>>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca
>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
>>>>>> internaldb.ldapconn.cloneReplicationPort=389
>>>>>> ...
>>>>>>
>>>>>> and when I try the ldapsearch I am presented with a prompt to
>>>>>> provide
>>>>>> a pin/password
>>>>>>
>>>>>> Please enter pin, password, or pass phrase for security token
>>>>>> 'ldap(0)':
>>>>>>
>>>>>> but there is no password file...
>>>>>>
>>>>> Hi,
>>>>>
>>>>> you are right, in 4.4. there is no pwdfile.txt and the password
>>>>> can be
>>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag
>>>>> internal=...)
>>>>>
>>>>> Can you check if the password with the tag internal=... allows to
>>>>> read
>>>>> the keys from the NSS db?
>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>> (provide password)
>>>>
>>>> That works...
>>>>
>>>> # certutil -K -d /etc/pki/pki-tomcat/alias
>>>> certutil: Checking token "NSS Certificate DB" in slot "NSS
User
>>>> Private
>>>> Key and Certificate Services"
>>>> Enter Password or Pin for "NSS Certificate DB":
>>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan)
>>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS
>>>> Certificate
>>>> DB:auditSigningCert cert-pki-ca
>>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e
caSigningCert
>>>> cert-pki-ca
>>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS
>>>> Certificate
>>>> DB:ocspSigningCert cert-pki-ca
>>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f
Server-Cert
>>>> cert-pki-ca
>>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS
>>>> Certificate
>>>> DB:subsystemCert cert-pki-ca
>>>>
>>>> But this doesn't (with the same password from password.conf)
>>>>
>>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y
EXTERNAL
>>>> Please enter pin, password, or pass phrase for security token
>>>> 'ldap(0)':
>>>> SASL/EXTERNAL authentication started
>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49)
>>>>
>>>> That password is getting me somewhere though, since if I put in a
>>>> nonsense or incorrect password it just prompts over and over.
>>>
>>> Let's step back a second. You upgraded from what to what?
>>
>> There wasn't much of a change... I just assumed someone ran yum
>> upgrade and didn't restart, then the power outage... it looks like
>> not much of a version change though.
>>
>> # grep ipa /var/log/yum.log
>> Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch
>> Jan 08 04:45:32 Installed:
>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch
>> Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64
>> Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64
>> Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch
>> Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch
>> Jan 08 04:47:05 Installed:
>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch
>> Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch
>> Jan 08 04:47:05 Installed:
>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch
>> Jan 08 04:47:06 Installed:
>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch
>> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64
>> Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64
>> Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64
>> Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch
>> Jan 08 04:47:36 Installed:
>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch
>> Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64
>> Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch
>> Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch
>> Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch
>> Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch
>> Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch
>> Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch
>> Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch
>> Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64
>> Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64
>> Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch
>> Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
>>
>> # grep pki /var/log/yum.log
>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch
>> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64
>> Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch
>> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64
>> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch
>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch
>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch
>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch
>> Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch
>> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64
>> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch
>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch
>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
>>
>>
>>>
>>> Are you running in SELinux enforcing mode?
>>
>> # getenforce
>> Enforcing
>>
>>>
>>> If so can you run:
>>>
>>> # ausearch -m AVC
>>>
>>
>> # ausearch -m AVC
>> The NOLOG option to log_format is deprecated. Please use the
>> write_logs option.
>> The NOLOG option is overriding the write_logs current setting.
>> ----
>> time->Mon Mar 14 10:39:01 2016
>> type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2
>> success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0
>> items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0
>> euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0
>> fsgid=581400001 tty=(none) ses=4294967295 comm="sshd"
>> exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023
>> key=(null)
>> type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for
>> pid=12965 comm="sshd" name="authorized_keys"
dev="dm-2" ino=116393517
>> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
>>
>>
>>> I suspect that the subsystem cert was renewed and everything wasn't
>>> updated properly. If I'm right this is unrelated to the upgrade it just
>>> shone a spotlight on it.
>>>
>>> Can you run these commands:
>>>
>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description
>>>
>>
>> This returns
>>
>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
>> uid=pkidbuser,ou=people,o=ipaca userCertificate description
>> Enter LDAP Password:
>> dn: uid=pkidbuser,ou=people,o=ipaca
>> userCertificate::
>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>> ... stuff ...
>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>> Subsystem,O=BPT.ROCKS
>>
>>> and
>>>
>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
>>> cert-pki-ca' | grep Serial
>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
>>> cert-pki-ca' -a
>>>
>>
>> These both return
>>
>> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
>> cert-pki-ca' | grep Serial
>> Enter Password or Pin for "NSS Certificate DB":
>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
>> Unrecognized Object Identifier.
>>
>>
>> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
>> cert-pki-ca' -a
>> certutil: Checking token "NSS Certificate DB" in slot "NSS User
>> Private Key and Certificate Services"
>> Enter Password or Pin for "NSS Certificate DB":
>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
>> Unrecognized Object Identifier.
>>
> Hi,
>
> you made a typo and requested the keys (-K) instead of the certs (-L).
> Please retry with -L and you should be able to see the certificate
> serial.
>
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate description
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate::
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
...
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
Subsystem,O=BPT.ROCKS
So the serial is 4?
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n
'subsystemCert cert-pki-ca' | grep Serial
Serial Number: 1341718541 (0x4ff9000d)
Which is NOT 4...
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n
'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
...
sjeiFbUFtoMnfmHLIYpe5QAR
-----END CERTIFICATE-----
And this cert is NOT the same.
Hi,
when certmonger renews a certificate, it runs pre-save and post-save
commands (which you can see in the output of getcert list). In your
case, the post-save command for subsystemCert cert-pki-ca probably failed.
This command (on the renewal master) is responsible for copying the new
cert from the NSS db to the LDAP server. You need first to check which
IPA server is the renewal master (as the serial numbers are in a
completely different range, I assume that you have multiple IPA servers
configured with a CA instance):
$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-`
$ kinit admin
$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b
"cn=masters,cn=ipa,cn=etc,$BASEDN"
'(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
dn:
cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
In this example the renewal master is vm-ipaserver.
If your renewal master is the machine on which the NSS DB
/etc/pki/pki-tomcat/alias contains the newer cert, then you can manually
run the scripts:
$ sudo /usr/libexec/ipa/certmonger/stop_pkicad
$ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
After that, check that the LDAP entry has been updated. If it is the
case, you should be able to restart pki.
Flo
Thank you again for your time!
Ian
> Flo.
>>
>> which seems weird.
>>
>>> The description attribute in the ldapsearch output is of the form
>>> 2;#;DN;DN
>>>
>>> The # should match the serial number in the first certutil output
>>>
>>> The ASCII blob (minus the BEGIN/END headers) should match one of the
>>> userCertificate attributes.
>>>
>>> rob
>>>
>>
>