On 08/02/2017 01:43 AM, Ian Harding wrote:
On 08/01/2017 12:03 PM, Rob Crittenden wrote:
> Ian Harding wrote:
>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
>>> On 08/01/2017 03:11 PM, Ian Harding wrote:
>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:
>>>>>>
>>>>>>
>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote:
>>>>>>> Ian Harding via FreeIPA-users wrote:
>>>>>>>> I had an unexpected restart of an IPA server that had
>>>>>>>> apparently had
>>>>>>>> updates run but had not been restarted. ipactl says
pki-tomcatd
>>>>>>>> would
>>>>>>>> not start.
>>>>>>>>
>>>>>>>> Strangely, the actual service appears to be running:
>>>>>>>>
>>>>>>>
>>>>>>> dogtag is an application within tomcat so tomcat can run
without
>>>>>>> dogtag
>>>>>>> running.
>>>>>>>
>>>>>>> We need to see more of the dogtag debug log to see what is
going
>>>>>>> on.
>>>>>>>
>>>>>>
>>>>>> It looks like an authentication problem...
>>>>>>
>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake
>>>>>> happened
>>>>>> Could not connect to LDAP server host seattlenfs.bpt.rocks port
636
>>>>>> Error netscape.ldap.LDAPException: Authentication failed (49)
>>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> dogtag stores its internal data in the LDAP server and needs to
>>>>> establish a secure LDAP connection. You can check how this
>>>>> connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for
>>>>> the lines:
>>>>>
>>>>> internaldb.ldapauth.authtype=SslClientAuth
>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager
>>>>> internaldb.ldapauth.bindPWPrompt=internaldb
>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
>>>>> internaldb.ldapconn.host=vm-...
>>>>> internaldb.ldapconn.port=636
>>>>> internaldb.ldapconn.secureConn
>>>>>
>>>>> authtype can be SslClientAuth (authentication with a ssl
>>>>> certificate) or BasicAuth (authentication with a bind DN and
>>>>> password stored in /var/lib/pki/pki-tomcat/conf/password.conf).
>>>>>
>>>>> You can use this information to manually check the credentials. For
>>>>> instance with sslclientauth:
>>>>>
>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca'
>>>>>
>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y
EXTERNAL
>>>>> (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)
>>>>>
>>>>
>>>> I found this:
>>>>
>>>> internaldb.ldapauth.authtype=SslClientAuth
>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca
>>>> internaldb.ldapauth.bindPWPrompt=internaldb
>>>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
>>>> internaldb.ldapconn.cloneReplicationPort=389
>>>> ...
>>>>
>>>> and when I try the ldapsearch I am presented with a prompt to provide
>>>> a pin/password
>>>>
>>>> Please enter pin, password, or pass phrase for security token
>>>> 'ldap(0)':
>>>>
>>>> but there is no password file...
>>>>
>>> Hi,
>>>
>>> you are right, in 4.4. there is no pwdfile.txt and the password can be
>>> found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag
>>> internal=...)
>>>
>>> Can you check if the password with the tag internal=... allows to read
>>> the keys from the NSS db?
>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>> (provide password)
>>
>> That works...
>>
>> # certutil -K -d /etc/pki/pki-tomcat/alias
>> certutil: Checking token "NSS Certificate DB" in slot "NSS User
Private
>> Key and Certificate Services"
>> Enter Password or Pin for "NSS Certificate DB":
>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan)
>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS Certificate
>> DB:auditSigningCert cert-pki-ca
>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert
>> cert-pki-ca
>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS Certificate
>> DB:ocspSigningCert cert-pki-ca
>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert
>> cert-pki-ca
>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS Certificate
>> DB:subsystemCert cert-pki-ca
>>
>> But this doesn't (with the same password from password.conf)
>>
>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL
>> Please enter pin, password, or pass phrase for security token 'ldap(0)':
>> SASL/EXTERNAL authentication started
>> ldap_sasl_interactive_bind_s: Invalid credentials (49)
>>
>> That password is getting me somewhere though, since if I put in a
>> nonsense or incorrect password it just prompts over and over.
>
> Let's step back a second. You upgraded from what to what?
There wasn't much of a change... I just assumed someone ran yum upgrade
and didn't restart, then the power outage... it looks like not much of a
version change though.
# grep ipa /var/log/yum.log
Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:45:32 Installed: ipa-client-common-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64
Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64
Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch
Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:05 Installed: python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:05 Installed: ipa-server-common-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:06 Installed: python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64
Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64
Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64
Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:36 Installed: ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64
Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64
Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64
Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
# grep pki /var/log/yum.log
Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch
Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64
Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch
Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64
Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch
Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch
Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch
Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch
Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch
Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64
Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch
Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch
Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
>
> Are you running in SELinux enforcing mode?
# getenforce
Enforcing
>
> If so can you run:
>
> # ausearch -m AVC
>
# ausearch -m AVC
The NOLOG option to log_format is deprecated. Please use the write_logs
option.
The NOLOG option is overriding the write_logs current setting.
----
time->Mon Mar 14 10:39:01 2016
type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2
success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0
ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001 suid=0
fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001 tty=(none)
ses=4294967295 comm="sshd" exe="/usr/sbin/sshd"
subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for
pid=12965 comm="sshd" name="authorized_keys" dev="dm-2"
ino=116393517
scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
> I suspect that the subsystem cert was renewed and everything wasn't
> updated properly. If I'm right this is unrelated to the upgrade it just
> shone a spotlight on it.
>
> Can you run these commands:
>
> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
> uid=pkidbuser,ou=people,o=ipaca userCertificate description
>
This returns
$ ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate description
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate::
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
... stuff ...
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
Subsystem,O=BPT.ROCKS
> and
>
> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
> cert-pki-ca' | grep Serial
> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
> cert-pki-ca' -a
>
These both return
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' | grep Serial
Enter Password or Pin for "NSS Certificate DB":
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized
Object Identifier.
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' -a
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized
Object Identifier.
Hi,
you made a typo and requested the keys (-K) instead of the certs (-L).
Please retry with -L and you should be able to see the certificate serial.
Flo.
which seems weird.
> The description attribute in the ldapsearch output is of the form
> 2;#;DN;DN
>
> The # should match the serial number in the first certutil output
>
> The ASCII blob (minus the BEGIN/END headers) should match one of the
> userCertificate attributes.
>
> rob
>