On 8/10/17 2:45 AM, thierry bordaz wrote:
On 08/09/2017 09:30 PM, Ian Harding via FreeIPA-users wrote:
>
> On 8/9/17 3:05 AM, thierry bordaz wrote:
>
>> Hi Ian,
>>
>> Thanks for having gather those data.
>>
>> #
>> # So pkidbuser entries have a same (old) userCertificate likely
>> generated during install
>> # But only freeipa-sea has a new one created on freeipa-sea
>> around Jun 8th 2017 05:54:16
>> # This recent certificate is identified by 5938e688000104290000
>> #
>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
>> manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
>> dn: uid=pkidbuser,ou=people,o=ipaca
>> ...
>> nscpentrywsi: userCertificate::
>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip>
>> nscpentrywsi: userCertificate;vucsn-5938e688000104290000::
>> MIIDbjCCAlagAwIBAgI <snip>
>>
>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory
>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
>> dn: uid=pkidbuser,ou=people,o=ipaca
>> nscpentrywsi: userCertificate::
>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip>
>>
>>
>>
>> #
>> # why 5938e688000104290000 value was not propagated to seattlenfs ?
>> # The most recent update (from freeipa-sea) that was replicated
>> to seattlenfs
>> # is 1 year old (57be8043000704290000 - Aug 25th 2016 05:21:07).
>> # In addition seattlenfs received direct update (last one was in
>> Jan 1017) that were not
>> # replicated to freeipa-sea
>> #
>> # The two servers have diverged because they can not replicate
>> to eachother because
>> # they were not correctly initialize.
>> # They have different "replicageneration" (57c291d9000004290000
>> vs 55c8f3ae000000600000)
>> #
>> # It is looking like freeipa-sea was created one or two years
>> ago and used to initialized seattlenfs.
>> # But later freeipa-sea was recreated.
>>
> That's about right.
>>
>> #
>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
>> manager' -W -b "o=ipaca"
>>
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>> Enter LDAP Password:
>> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>> nsds50ruv: {replicageneration} 57c291d9000004290000
>> nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
>> 57f840bf000004290000 598a1c40000104290000
>> nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
>> nsruvReplicaLastModified: {replica 1065
>> ldap://freeipa-sea.bpt.rocks:389} 598a1c16
>> nsruvReplicaLastModified: {replica 1290
>> ldap://seattlenfs.bpt.rocks:389} 00000000
>>
>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory
>> manager' -W -b "o=ipaca"
>>
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>> nsds50ruv: {replicageneration} 55c8f3ae000000600000
>> nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
>> 57b103d4000004290000 57be8043000704290000
>> nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
>> 57be804c0000050a0000 587236150000050a0000
>> nsruvReplicaLastModified: {replica 1290
>> ldap://seattlenfs.bpt.rocks:389} 00000000
>> nsruvReplicaLastModified: {replica 1065
>> ldap://freeipa-sea.bpt.rocks:389} 00000000
>>
>>
>>
>> In conclusion:
>> From replication pov, the two instances can not communicate.
>> One solution would be to identify which instance is the good
>> one, you want to keep.
>> and reinit the second one from that reference.
>>
> What exactly does reinit mean? I have run
>
> ipa-replica-manage re-initialize --from freeipa-sea.bpt.rocks
This triggers the reinit of 'dc=ipadomain,dc=com' suffix but not for
o=ipaca suffix. To reinit o=ipaca you may use
ipa-cs-replica-manager re-initialize --from freeipa-sea.bpt.rocks.
Note that it will overwrite all data of ipaca on seattlenfs with the
data from freeipa-sea. This is why it is important to know which
server is the valid one.
Hmm.. This looks odd.
[root@seattlenfs ianh]# ipa-csreplica-manage re-initialize --from
freeipa-sea.bpt.rocks
Directory Manager password:
ipa: ERROR: Found multiple agreements for seattlenfs.bpt.rocks. Only
initializing the first one returned:
cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\=ipaca,cn=mapping
tree,cn=config
Update in progress, 15 seconds elapsed
[freeipa-sea.bpt.rocks] reports: Update failed! Status: [32 - LDAP
error: No such object]
On freeipa-sea and seattlenfs you may check the ipaca replica
agreements status before/after reinit. I would guess they are
currently reporting failures between these two replicas.
ldapsearch -LLL -D 'cn=directory manager' -W -b
"cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"
"objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
"cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"
"objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
Enter LDAP Password:
dn:
cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac
a,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: Error (32) Problem connecting to replica -
LDAP
error: No such object (connection error)
dn:
cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac
a,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring
replica:
Replica has different database generation ID, remote replica may need
to be i
nitialized (RUV error)
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
"cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"
"objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
Enter LDAP Password:
dn:
cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipaca
,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring
replica:
Replica has different database generation ID, remote replica may need
to be i
nitialized (RUV error)
>
> several times over the years when replication has stopped.
>
> Replication actually is working, as far as user and machine accounts
> and attributes are concerned anyway, and has been for a while.
>
> There is a zombie server freeipa-dal.bpt.rocks that I can't get rid
> of... it shows up in the GUI on the Topology page but generates a
> Server not found error. I don't know if that's related.
Zombie servers have been discussed a lot on this mailing list. You may
get rid of them with list-ruv/clean-ruv subcommand. Replication
usually manage to work fine even if some zombie entries exist but it
is a good practice to clean them.
Yeah, I've been through every possible
combination of
list-ruv/clean-ruv. I have something odd going on in that no
replication agreements show up anywhere for freeipa-dal.
regards
thierry
>>
>> On 08/08/2017 10:33 PM, Ian Harding via FreeIPA-users wrote:
>>>
>>> On 8/7/17 1:44 AM, thierry bordaz wrote:
>>>
>>>>
>>>>
>>>> On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
>>>>> On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
>>>>>> On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
>>>>>>
>>>>>>> On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
>>>>>>>> On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
>>>>>>>>> On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users
wrote:
>>>>>>>>>> On 08/02/2017 12:11 AM, Florence Blanc-Renaud
wrote:
>>>>>>>>>>> On 08/02/2017 01:43 AM, Ian Harding wrote:
>>>>>>>>>>>> On 08/01/2017 12:03 PM, Rob Crittenden
wrote:
>>>>>>>>>>>>> Ian Harding wrote:
>>>>>>>>>>>>>> On 08/01/2017 07:39 AM, Florence
Blanc-Renaud wrote:
>>>>>>>>>>>>>>> On 08/01/2017 03:11 PM, Ian
Harding wrote:
>>>>>>>>>>>>>>>> On 08/01/2017 01:48 AM,
Florence Blanc-Renaud wrote:
>>>>>>>>>>>>>>>>> On 08/01/2017 01:32
AM, Ian Harding via FreeIPA-users
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 07/31/2017
11:34 AM, Rob Crittenden wrote:
>>>>>>>>>>>>>>>>>>> Ian Harding
via FreeIPA-users wrote:
>>>>>>>>>>>>>>>>>>>> I had an
unexpected restart of an IPA server that
>>>>>>>>>>>>>>>>>>>> had
apparently had
>>>>>>>>>>>>>>>>>>>> updates
run but had not been restarted. ipactl
>>>>>>>>>>>>>>>>>>>> says
pki-tomcatd
>>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>> not
start.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
Strangely, the actual service appears to be running:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> dogtag is an
application within tomcat so tomcat
>>>>>>>>>>>>>>>>>>> can run
without
>>>>>>>>>>>>>>>>>>> dogtag
>>>>>>>>>>>>>>>>>>> running.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We need to
see more of the dogtag debug log to see
>>>>>>>>>>>>>>>>>>> what is going
on.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It looks like an
authentication problem...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL
>>>>>>>>>>>>>>>>>> handshake
happened
>>>>>>>>>>>>>>>>>> Could not connect
to LDAP server host
>>>>>>>>>>>>>>>>>>
seattlenfs.bpt.rocks port 636
>>>>>>>>>>>>>>>>>> Error
netscape.ldap.LDAPException: Authentication
>>>>>>>>>>>>>>>>>> failed (49)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> dogtag stores its
internal data in the LDAP server
>>>>>>>>>>>>>>>>> and needs to
>>>>>>>>>>>>>>>>> establish a secure
LDAP connection. You can check how
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> connection is
configured in
>>>>>>>>>>>>>>>>>
/etc/pki/pki-tomcat/ca/CS.cfg, look for
>>>>>>>>>>>>>>>>> the lines:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
internaldb.ldapauth.authtype=SslClientAuth
>>>>>>>>>>>>>>>>>
internaldb.ldapauth.bindDN=cn=Directory Manager
>>>>>>>>>>>>>>>>>
internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>>>>>>>>>>>>>
internaldb.ldapauth.clientCertNickname=subsystemCert
>>>>>>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>>>>>>
internaldb.ldapconn.host=vm-...
>>>>>>>>>>>>>>>>>
internaldb.ldapconn.port=636
>>>>>>>>>>>>>>>>>
internaldb.ldapconn.secureConn
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> authtype can be
SslClientAuth (authentication with a ssl
>>>>>>>>>>>>>>>>> certificate) or
BasicAuth (authentication with a bind
>>>>>>>>>>>>>>>>> DN and
>>>>>>>>>>>>>>>>> password stored in
>>>>>>>>>>>>>>>>>
/var/lib/pki/pki-tomcat/conf/password.conf).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You can use this
information to manually check the
>>>>>>>>>>>>>>>>> credentials. For
>>>>>>>>>>>>>>>>> instance with
sslclientauth:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> export
LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
>>>>>>>>>>>>>>>>> export
LDAPTLS_CERT='subsystemCert cert-pki-ca'
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ldapsearch -H
ldaps://`hostname`:636 -b "" -s base -Y
>>>>>>>>>>>>>>>>> EXTERNAL
>>>>>>>>>>>>>>>>> (provide the password
from
>>>>>>>>>>>>>>>>>
/etc/pki/pki-tomcat/alias/pwdfile.txt)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I found this:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
internaldb.ldapauth.authtype=SslClientAuth
>>>>>>>>>>>>>>>>
internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>>>>>>>>>>>>
internaldb.ldapauth.clientCertNickname=subsystemCert
>>>>>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>>>>>
internaldb.ldapconn.cloneReplicationPort=389
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and when I try the
ldapsearch I am presented with a
>>>>>>>>>>>>>>>> prompt to provide
>>>>>>>>>>>>>>>> a pin/password
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please enter pin,
password, or pass phrase for
>>>>>>>>>>>>>>>> security token
'ldap(0)':
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> but there is no password
file...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> you are right, in 4.4. there
is no pwdfile.txt and the
>>>>>>>>>>>>>>> password can be
>>>>>>>>>>>>>>> found in
/var/lib/pki/pki-tomcat/conf/password.conf
>>>>>>>>>>>>>>> (with the tag
>>>>>>>>>>>>>>> internal=...)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you check if the password
with the tag internal=...
>>>>>>>>>>>>>>> allows to read
>>>>>>>>>>>>>>> the keys from the NSS db?
>>>>>>>>>>>>>>> certutil -K -d
/etc/pki/pki-tomcat/alias
>>>>>>>>>>>>>>> (provide password)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That works...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # certutil -K -d
/etc/pki/pki-tomcat/alias
>>>>>>>>>>>>>> certutil: Checking token
"NSS Certificate DB" in slot
>>>>>>>>>>>>>> "NSS User Private
>>>>>>>>>>>>>> Key and Certificate
Services"
>>>>>>>>>>>>>> Enter Password or Pin for
"NSS Certificate DB":
>>>>>>>>>>>>>> < 0> rsa
0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan)
>>>>>>>>>>>>>> < 1> rsa
b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS
>>>>>>>>>>>>>> Certificate
>>>>>>>>>>>>>> DB:auditSigningCert cert-pki-ca
>>>>>>>>>>>>>> < 2> rsa
881b7254c40fa40bc50681bcc8d37bb3eb49937e
>>>>>>>>>>>>>> caSigningCert
>>>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>>> < 3> rsa
fa9a255a1d15585ac28064c0f4986e416bc48403 NSS
>>>>>>>>>>>>>> Certificate
>>>>>>>>>>>>>> DB:ocspSigningCert cert-pki-ca
>>>>>>>>>>>>>> < 4> rsa
3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f
>>>>>>>>>>>>>> Server-Cert
>>>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>>> < 5> rsa
1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS
>>>>>>>>>>>>>> Certificate
>>>>>>>>>>>>>> DB:subsystemCert cert-pki-ca
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But this doesn't (with the
same password from
>>>>>>>>>>>>>> password.conf)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # ldapsearch -H
ldaps://`hostname`:636 -b "" -s base -Y
>>>>>>>>>>>>>> EXTERNAL
>>>>>>>>>>>>>> Please enter pin, password, or
pass phrase for security
>>>>>>>>>>>>>> token 'ldap(0)':
>>>>>>>>>>>>>> SASL/EXTERNAL authentication
started
>>>>>>>>>>>>>> ldap_sasl_interactive_bind_s:
Invalid credentials (49)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That password is getting me
somewhere though, since if I
>>>>>>>>>>>>>> put in a
>>>>>>>>>>>>>> nonsense or incorrect password it
just prompts over and
>>>>>>>>>>>>>> over.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let's step back a second. You
upgraded from what to what?
>>>>>>>>>>>>
>>>>>>>>>>>> There wasn't much of a change... I
just assumed someone
>>>>>>>>>>>> ran yum upgrade and didn't restart,
then the power
>>>>>>>>>>>> outage... it looks like not much of a
version change though.
>>>>>>>>>>>>
>>>>>>>>>>>> # grep ipa /var/log/yum.log
>>>>>>>>>>>> Jan 08 04:45:32 Installed:
>>>>>>>>>>>>
ipa-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>> Jan 08 04:45:32 Installed:
>>>>>>>>>>>>
ipa-client-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>> Jan 08 04:46:06 Updated:
libipa_hbac-1.14.0-43.el7_3.4.x86_64
>>>>>>>>>>>> Jan 08 04:46:07 Updated:
>>>>>>>>>>>>
python-libipa_hbac-1.14.0-43.el7_3.4.x86_64
>>>>>>>>>>>> Jan 08 04:46:08 Installed:
>>>>>>>>>>>> python-ipaddress-1.0.16-2.el7.noarch
>>>>>>>>>>>> Jan 08 04:47:04 Installed:
>>>>>>>>>>>>
python2-ipalib-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>> Jan 08 04:47:05 Installed:
>>>>>>>>>>>>
python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>> Jan 08 04:47:05 Updated:
>>>>>>>>>>>>
ipa-admintools-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>> Jan 08 04:47:05 Installed:
>>>>>>>>>>>>
ipa-server-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>> Jan 08 04:47:06 Installed:
>>>>>>>>>>>>
python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>> Jan 08 04:47:07 Updated:
sssd-ipa-1.14.0-43.el7_3.4.x86_64
>>>>>>>>>>>> Jan 08 04:47:17 Updated:
>>>>>>>>>>>>
ipa-client-4.4.0-14.el7.centos.1.1.x86_64
>>>>>>>>>>>> Jan 08 04:47:23 Updated:
>>>>>>>>>>>>
ipa-server-4.4.0-14.el7.centos.1.1.x86_64
>>>>>>>>>>>> Jan 08 04:47:27 Updated:
>>>>>>>>>>>>
ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>> Jan 08 04:47:36 Installed:
>>>>>>>>>>>>
ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>> Jan 08 04:50:07 Erased:
>>>>>>>>>>>>
ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64
>>>>>>>>>>>> Jul 28 09:55:41 Updated:
>>>>>>>>>>>> ipa-common-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>> Jul 28 09:55:42 Updated:
>>>>>>>>>>>>
ipa-client-common-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>> Jul 28 09:55:43 Updated:
>>>>>>>>>>>>
ipa-server-common-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>> Jul 28 09:55:43 Updated:
>>>>>>>>>>>>
python2-ipalib-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>> Jul 28 09:55:44 Updated:
>>>>>>>>>>>>
python2-ipaclient-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>> Jul 28 09:55:45 Updated:
>>>>>>>>>>>>
python2-ipaserver-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>> Jul 28 09:55:45 Updated:
>>>>>>>>>>>>
ipa-admintools-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>> Jul 28 09:55:46 Updated:
>>>>>>>>>>>> ipa-client-4.4.0-14.el7.centos.7.x86_64
>>>>>>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>>>>>>> ipa-server-4.4.0-14.el7.centos.7.x86_64
>>>>>>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>>>>>>>
ipa-server-dns-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>>>>>>>
ipa-python-compat-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>
>>>>>>>>>>>> # grep pki /var/log/yum.log
>>>>>>>>>>>> Jan 08 04:46:05 Updated:
pki-base-10.3.3-14.el7_3.noarch
>>>>>>>>>>>> Jan 08 04:46:09 Updated:
krb5-pkinit-1.14.1-27.el7_3.x86_64
>>>>>>>>>>>> Jan 08 04:47:19 Installed:
>>>>>>>>>>>> pki-base-java-10.3.3-14.el7_3.noarch
>>>>>>>>>>>> Jan 08 04:47:19 Updated:
pki-tools-10.3.3-14.el7_3.x86_64
>>>>>>>>>>>> Jan 08 04:47:21 Updated:
pki-server-10.3.3-14.el7_3.noarch
>>>>>>>>>>>> Jan 08 04:47:22 Updated:
pki-kra-10.3.3-14.el7_3.noarch
>>>>>>>>>>>> Jan 08 04:47:22 Updated:
pki-ca-10.3.3-14.el7_3.noarch
>>>>>>>>>>>> Jul 28 10:13:16 Updated:
pki-base-10.3.3-19.el7_3.noarch
>>>>>>>>>>>> Jul 28 10:13:17 Updated:
pki-base-java-10.3.3-19.el7_3.noarch
>>>>>>>>>>>> Jul 28 10:13:17 Updated:
pki-tools-10.3.3-19.el7_3.x86_64
>>>>>>>>>>>> Jul 28 10:13:21 Updated:
pki-server-10.3.3-19.el7_3.noarch
>>>>>>>>>>>> Jul 28 10:13:21 Updated:
pki-kra-10.3.3-19.el7_3.noarch
>>>>>>>>>>>> Jul 28 10:13:21 Updated:
pki-ca-10.3.3-19.el7_3.noarch
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Are you running in SELinux enforcing
mode?
>>>>>>>>>>>>
>>>>>>>>>>>> # getenforce
>>>>>>>>>>>> Enforcing
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If so can you run:
>>>>>>>>>>>>>
>>>>>>>>>>>>> # ausearch -m AVC
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> # ausearch -m AVC
>>>>>>>>>>>> The NOLOG option to log_format is
deprecated. Please use
>>>>>>>>>>>> the write_logs option.
>>>>>>>>>>>> The NOLOG option is overriding the
write_logs current
>>>>>>>>>>>> setting.
>>>>>>>>>>>> ----
>>>>>>>>>>>> time->Mon Mar 14 10:39:01 2016
>>>>>>>>>>>> type=SYSCALL
msg=audit(1457977141.767:17537):
>>>>>>>>>>>> arch=c000003e syscall=2 success=no
exit=-13
>>>>>>>>>>>> a0=7fbcab038d10 a1=800 a2=1
a3=7fbca60f42e0 items=0
>>>>>>>>>>>> ppid=1406 pid=12965 auid=4294967295 uid=0
gid=0
>>>>>>>>>>>> euid=581400001 suid=0 fsuid=581400001
egid=581400001
>>>>>>>>>>>> sgid=0 fsgid=581400001 tty=(none)
ses=4294967295
>>>>>>>>>>>> comm="sshd"
exe="/usr/sbin/sshd"
>>>>>>>>>>>>
subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null)
>>>>>>>>>>>> type=AVC msg=audit(1457977141.767:17537):
avc: denied {
>>>>>>>>>>>> read } for pid=12965
comm="sshd" name="authorized_keys"
>>>>>>>>>>>> dev="dm-2" ino=116393517
>>>>>>>>>>>>
scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
>>>>>>>>>>>> tcontext=system_u:object_r:unlabeled_t:s0
tclass=file
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I suspect that the subsystem cert was
renewed and
>>>>>>>>>>>>> everything wasn't
>>>>>>>>>>>>> updated properly. If I'm right
this is unrelated to the
>>>>>>>>>>>>> upgrade it just
>>>>>>>>>>>>> shone a spotlight on it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can you run these commands:
>>>>>>>>>>>>>
>>>>>>>>>>>>> $ ldapsearch -LLL -D
'cn=directory manager' -W -b
>>>>>>>>>>>>> uid=pkidbuser,ou=people,o=ipaca
userCertificate description
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This returns
>>>>>>>>>>>>
>>>>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory
manager' -W -b
>>>>>>>>>>>> uid=pkidbuser,ou=people,o=ipaca
userCertificate description
>>>>>>>>>>>> Enter LDAP Password:
>>>>>>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>>>> userCertificate::
>>>>>>>>>>>>
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>>>>>>> ... stuff ...
>>>>>>>>>>>>
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>>>>>>> description: 2;4;CN=Certificate
>>>>>>>>>>>> Authority,O=BPT.ROCKS;CN=CA
Subsystem,O=BPT.ROCKS
>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>>> # certutil -L -d
/etc/pki/pki-tomcat/alias -n 'subsystemCert
>>>>>>>>>>>>> cert-pki-ca' | grep Serial
>>>>>>>>>>>>> # certutil -L -d
/etc/pki/pki-tomcat/alias -n
>>>>>>>>>>>>> 'subsystemCert cert-pki-ca'
-a
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> These both return
>>>>>>>>>>>>
>>>>>>>>>>>> # certutil -K -d
/etc/pki/pki-tomcat/alias -n
>>>>>>>>>>>> 'subsystemCert cert-pki-ca' |
grep Serial
>>>>>>>>>>>> Enter Password or Pin for "NSS
Certificate DB":
>>>>>>>>>>>> certutil: problem listing keys:
>>>>>>>>>>>> SEC_ERROR_UNRECOGNIZED_OID: Unrecognized
Object Identifier.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> # certutil -K -d
/etc/pki/pki-tomcat/alias -n
>>>>>>>>>>>> 'subsystemCert cert-pki-ca' -a
>>>>>>>>>>>> certutil: Checking token "NSS
Certificate DB" in slot "NSS
>>>>>>>>>>>> User Private Key and Certificate
Services"
>>>>>>>>>>>> Enter Password or Pin for "NSS
Certificate DB":
>>>>>>>>>>>> certutil: problem listing keys:
>>>>>>>>>>>> SEC_ERROR_UNRECOGNIZED_OID: Unrecognized
Object Identifier.
>>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> you made a typo and requested the keys (-K)
instead of the
>>>>>>>>>>> certs (-L). Please retry with -L and you
should be able to
>>>>>>>>>>> see the certificate serial.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D
'cn=directory
>>>>>>>>>> manager' -W -b
uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>> userCertificate description
>>>>>>>>>> Enter LDAP Password:
>>>>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>> userCertificate::
>>>>>>>>>>
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>>>>> ...
>>>>>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>>>>> description: 2;4;CN=Certificate
Authority,O=BPT.ROCKS;CN=CA
>>>>>>>>>> Subsystem,O=BPT.ROCKS
>>>>>>>>>>
>>>>>>>>>> So the serial is 4?
>>>>>>>>>>
>>>>>>>>>> [root@seattlenfs ianh]# certutil -L -d
>>>>>>>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' |
>>>>>>>>>> grep Serial
>>>>>>>>>> Serial Number: 1341718541 (0x4ff9000d)
>>>>>>>>>>
>>>>>>>>>> Which is NOT 4...
>>>>>>>>>>
>>>>>>>>>> [root@seattlenfs ianh]# certutil -L -d
>>>>>>>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' -a
>>>>>>>>>> -----BEGIN CERTIFICATE-----
>>>>>>>>>>
MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>> sjeiFbUFtoMnfmHLIYpe5QAR
>>>>>>>>>> -----END CERTIFICATE-----
>>>>>>>>>>
>>>>>>>>>> And this cert is NOT the same.
>>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> when certmonger renews a certificate, it runs
pre-save and
>>>>>>>>> post-save commands (which you can see in the output
of
>>>>>>>>> getcert list). In your case, the post-save command
for
>>>>>>>>> subsystemCert cert-pki-ca probably failed.
>>>>>>>>>
>>>>>>>>> This command (on the renewal master) is responsible
for
>>>>>>>>> copying the new cert from the NSS db to the LDAP
server. You
>>>>>>>>> need first to check which IPA server is the renewal
master
>>>>>>>>> (as the serial numbers are in a completely different
range, I
>>>>>>>>> assume that you have multiple IPA servers configured
with a
>>>>>>>>> CA instance):
>>>>>>>>>
>>>>>>>>> $ export BASEDN=`grep basedn /etc/ipa/default.conf |
cut -d=
>>>>>>>>> -f2-`
>>>>>>>>> $ kinit admin
>>>>>>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI
-b
>>>>>>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN"
>>>>>>>>>
'(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
>>>>>>>>> dn:
>>>>>>>>>
cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In this example the renewal master is vm-ipaserver.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I get this
>>>>>>>>
>>>>>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b
>>>>>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN"
>>>>>>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))'
dn
>>>>>>>> dn:
>>>>>>>>
cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
>>>>>>>>
>>>>>>>>
>>>>>>>> Which is the other FreeIPA server.
>>>>>>>>
>>>>>>> Hi Ian,
>>>>>>>
>>>>>>> it's getting difficult to follow this thread, so could we
step
>>>>>>> back and summarize?
>>>>>> Yes, it is. Sorry.
>>>>>>>
>>>>>>> - On server freeipa-sea (=renewal master), the CA component
is
>>>>>>> installed and 'subsystemCert cert-pki-ca' has Serial
number 4
>>>>>>> in the NSSDB /etc/pki/pki-tomcat/alias
>>>>>> No?
>>>>>>
>>>>>> [root@freeipa-sea ianh]# certutil -L -d
>>>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'
| grep
>>>>>> Serial
>>>>>> Serial Number: 1341718541 (0x4ff9000d)
>>>>>>> - On server seattlenfs (where PKI fails to restart), the CA
>>>>>>> component is installed and 'subsystemCert
cert-pki-ca' has
>>>>>>> Serial number 1341718541 in the NSSDB
/etc/pki/pki-tomcat/alias.
>>>>>> Yes
>>>>>>
>>>>>> [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias
>>>>>> -n 'subsystemCert cert-pki-ca' | grep Serial
>>>>>> Serial Number: 1341718541 (0x4ff9000d)
>>>>>>
>>>>>>>
>>>>>>> - In the LDAP servers, the cert is stored in the entry
>>>>>>> uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it
>>>>>>> is the cert from freeipa-sea).
>>>>>> This is where the difference is. There are two certificates on
>>>>>> the master and only one on the other. I wonder if this is a
>>>>>> result of the slow motion replication trainwreck I have going on
>>>>>> due to a zombie replica in the ldap database that no longer
>>>>>> exists in the real world and I can't seem to get rid of.
>>>>>>
>>>>> Hi,
>>>>>
>>>>> the remaining issue is a replication problem. There is a
>>>>> "Troubleshooting replication" section [1] in IdM Admin
guide that
>>>>> may help you.
>>>>>
>>>>> Regarding your zombie replica, which commands did you use to try
>>>>> to remove it, and what was the output?
>>>>>
>>>>> Flo
>>>>>
>>>>> [1]
>>>>>
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/...
>>>>>
>>>>
>>>> Hi,
>>>>
>>>> To debug replication latency or problem we would need several info
>>>> but starting with the results of the following commands
>>>>
>>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
manager'
>>>> -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager'
>>> -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
>>> Enter LDAP Password:
>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>> nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca
>>> nscpentrywsi:
>>> entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005:
>>> 75274897
>>> nscpentrywsi:
>>> modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042
>>> 90004: 20170608055334Z
>>> nscpentrywsi:
>>> modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290
>>> 003: cn=Directory Manager
>>> nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074
>>> nscpentrywsi: cn: pkidbuser
>>> nscpentrywsi: createTimestamp: 20150716125146Z
>>> nscpentrywsi: creatorsName: cn=directory manager
>>> nscpentrywsi: description;vucsn-5938e688000104290001:
>>> 2;1341718541;CN=Certific
>>> ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
>>> nscpentrywsi: description;vdcsn-5938e688000104290002;deleted:
>>> 2;4;CN=Certifica
>>> te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
>>> nscpentrywsi: mail:
>>> nscpentrywsi: objectClass: top
>>> nscpentrywsi: objectClass: person
>>> nscpentrywsi: objectClass: organizationalPerson
>>> nscpentrywsi: objectClass: inetOrgPerson
>>> nscpentrywsi: objectClass: cmsuser
>>> nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS
>>> nscpentrywsi: sn: pkidbuser
>>> nscpentrywsi: uid: pkidbuser
>>> nscpentrywsi: userCertificate::
>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
>>> <snip>
>>> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>> nscpentrywsi: userCertificate;vucsn-5938e688000104290000::
>>> MIIDbjCCAlagAwIBAgI
>>> <snip>
>>> AR
>>> nscpentrywsi: userPassword:: <snip>
>>> nscpentrywsi: userstate: 1
>>> nscpentrywsi: usertype: agentType
>>> nscpentrywsi: parentid: 2
>>> nscpentrywsi: entryid: 51
>>>> and
>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager'
>>>> -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
>>>
>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager'
>>> -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
>>> Enter LDAP Password:
>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>> nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca
>>> nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS
>>> nscpentrywsi: userCertificate::
>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
>>> <snip>
>>> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>> nscpentrywsi: description: 2;4;CN=Certificate
>>> Authority,O=BPT.ROCKS;CN=CA Subs
>>> ystem,O=BPT.ROCKS
>>> nscpentrywsi: entryid: 51
>>> nscpentrywsi: parentid: 2
>>> nscpentrywsi: modifyTimestamp: 20150716125146Z
>>> nscpentrywsi: createTimestamp: 20150716125146Z
>>> nscpentrywsi: modifiersName: cn=directory manager
>>> nscpentrywsi: creatorsName: cn=directory manager
>>> nscpentrywsi: userPassword:: <snip>
>>> nscpentrywsi: userstate: 1
>>> nscpentrywsi: usertype: agentType
>>> nscpentrywsi: mail:
>>> nscpentrywsi: cn: pkidbuser
>>> nscpentrywsi: sn: pkidbuser
>>> nscpentrywsi: uid: pkidbuser
>>> nscpentrywsi: objectClass: top
>>> nscpentrywsi: objectClass: person
>>> nscpentrywsi: objectClass: organizationalPerson
>>> nscpentrywsi: objectClass: inetOrgPerson
>>> nscpentrywsi: objectClass: cmsuser
>>> nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074
>>> nscpentrywsi: entryusn: 86
>>>>
>>>> From the output you may only copy/past the ones related to
>>>> userCertificate
>>>>
>>>>
>>>> Also to dump the current replication status, send the result of
>>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
manager'
>>>> -W -b
>>>>
"o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager'
>>> -W -b "o=ipaca"
>>>
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>> Enter LDAP Password:
>>> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>>> cn: replica
>>> nsDS5Flags: 1
>>> nsDS5ReplicaBindDN: cn=Replication Manager
>>> cloneAgreement1-freeipa-sea.bpt.roc
>>> ks-pki-tomcat,ou=csusers,cn=config
>>> nsDS5ReplicaBindDN: cn=Replication Manager
>>> masterAgreement1-seattlenfs.bpt.roc
>>> ks-pki-tomcat,ou=csusers,cn=config
>>> nsDS5ReplicaId: 1065
>>> nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1
>>> nsDS5ReplicaRoot: o=ipaca
>>> nsDS5ReplicaType: 3
>>> nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA==
>>> nsds5replicabinddngroup: cn=replication
>>> managers,cn=sysaccounts,cn=etc,dc=bpt,
>>> dc=rocks
>>> nsds5replicabinddngroupcheckinterval: 60
>>> objectClass: top
>>> objectClass: nsDS5Replica
>>> objectClass: extensibleobject
>>> nsds50ruv: {replicageneration} 57c291d9000004290000
>>> nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
>>> 57f840bf00000429000
>>> 0 598a1c40000104290000
>>> nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
>>> nsds5agmtmaxcsn:
>>> o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat
>>> tlenfs.bpt.rocks;389;unavailable
>>> nsds5agmtmaxcsn:
>>> o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat
>>> tlenfs.bpt.rocks;389;unavailable
>>> nsruvReplicaLastModified: {replica 1065
>>> ldap://freeipa-sea.bpt.rocks:389} 598a
>>> 1c16
>>> nsruvReplicaLastModified: {replica 1290
>>> ldap://seattlenfs.bpt.rocks:389} 00000
>>> 000
>>> nsds5ReplicaChangeCount: 885
>>> nsds5replicareapactive: 0
>>>> and
>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager'
>>>> -W -b
>>>>
"o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>>
>>>
>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager'
>>> -W -b "o=ipaca"
>>>
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>> Enter LDAP Password:
>>> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>>> cn: replica
>>> nsDS5Flags: 1
>>> nsDS5ReplicaBindDN: cn=Replication Manager
>>> cloneAgreement1-seattlenfs.bpt.rock
>>> s-pki-tomcat,ou=csusers,cn=config
>>> nsDS5ReplicaId: 1290
>>> nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323
>>> nsDS5ReplicaRoot: o=ipaca
>>> nsDS5ReplicaType: 3
>>> nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA==
>>> nsds5replicabinddngroup: cn=replication
>>> managers,cn=sysaccounts,cn=etc,dc=bpt,
>>> dc=rocks
>>> nsds5replicabinddngroupcheckinterval: 60
>>> objectClass: top
>>> objectClass: nsDS5Replica
>>> objectClass: extensibleobject
>>> nsds50ruv: {replicageneration} 55c8f3ae000000600000
>>> nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
>>> 57be804c0000050a0000
>>> 587236150000050a0000
>>> nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
>>> 57b103d400000429000
>>> 0 57be8043000704290000
>>> nsds5agmtmaxcsn:
>>> o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei
>>> pa-sea.bpt.rocks;389;1065;587236150000050a0000
>>> nsruvReplicaLastModified: {replica 1290
>>> ldap://seattlenfs.bpt.rocks:389} 00000
>>> 000
>>> nsruvReplicaLastModified: {replica 1065
>>> ldap://freeipa-sea.bpt.rocks:389} 0000
>>> 0000
>>> nsds5ReplicaChangeCount: 3
>>> nsds5replicareapactive: 0
>>>>
>>>> Regards
>>>> thierry
>>>>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
>>>>>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca
userCertificate
>>>>>> description
>>>>>> Enter LDAP Password:
>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>> userCertificate::
>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>> <truncated>
>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>> userCertificate::
>>>>>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK
>>>>>> <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR
>>>>>> description: 2;1341718541;CN=Certificate
>>>>>> Authority,O=BPT.ROCKS;CN=CA Subsystem
>>>>>> ,O=BPT.ROCKS
>>>>>>
>>>>>>
>>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory
>>>>>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca
userCertificate
>>>>>> description
>>>>>> Enter LDAP Password:
>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>> userCertificate::
>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>> <truncated>
>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>>>>>> Subsystem,O=BPT.RO
>>>>>> CKS
>>>>>>
>>>>>>> If you check the certificates dates, I guess that the most
>>>>>>> recent one is on server seattlenfs (because of serial
numbers).
>>>>>>> Can you confirm this (because our goal is to have the most
>>>>>>> recent cert stored in all the NSSDBs and in LDAP)?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Flo
>>>>>>>
>>>>>>>
>>>>>>>>> If your renewal master is the machine on which the
NSS DB
>>>>>>>>> /etc/pki/pki-tomcat/alias contains the newer cert,
then you
>>>>>>>>> can manually run the scripts:
>>>>>>>>
>>>>>>>> If I'm interpreting this correctly, then it is... so
I ran
>>>>>>>> this on the broken machine (seattlenfs)
>>>>>>>>
>>>>>>>>> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad
>>>>>>>>> $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert
>>>>>>>>> "subsystemCert cert-pki-ca"
>>>>>>>>>
>>>>>>>>
>>>>>>>> I ran this, it took a while, then I re-checked and the
serial
>>>>>>>> still seems to be 4
>>>>>>>>
>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
>>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate
description
>>>>>>>> Enter LDAP Password:
>>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>>> userCertificate::
>>>>>>>>
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>>> ...
>>>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>>> description: 2;4;CN=Certificate
Authority,O=BPT.ROCKS;CN=CA
>>>>>>>> Subsystem,O=BPT.ROCKS
>>>>>>>>
>>>>>>>> Here's what was in /var/log/pki/pki-tomcat/ca/debug
>>>>>>>>
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> ============================================
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: =====
DEBUG
>>>>>>>> SUBSYSTEM INITIALIZED =======
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> ============================================
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> restart at autoShutdown? false
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> autoShutdown crumb file path?
>>>>>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> about to look for cert for auto-shutdown
>>>>>>>> support:auditSigningCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> found cert:auditSigningCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
done
>>>>>>>> init id=debug
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> initialized debug
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> initSubsystem id=log
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> ready to init id=log
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>>>>>>>
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
>>>>>>>>
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>>>>>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>>>>>>>
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> restart at autoShutdown? false
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> autoShutdown crumb file path?
>>>>>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> about to look for cert for auto-shutdown
>>>>>>>> support:auditSigningCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> found cert:auditSigningCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
done
>>>>>>>> init id=log
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> initialized log
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> initSubsystem id=jss
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> ready to init id=jss
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> restart at autoShutdown? false
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> autoShutdown crumb file path?
>>>>>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> about to look for cert for auto-shutdown
>>>>>>>> support:auditSigningCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> found cert:auditSigningCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
done
>>>>>>>> init id=jss
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> initialized jss
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> initSubsystem id=dbs
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>>>>>>> ready to init id=dbs
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
DBSubsystem:
>>>>>>>> init() mEnableSerialMgmt=true
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>>>>>>> LdapBoundConnFactor(DBSubsystem)
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> LdapBoundConnFactory: init
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> LdapBoundConnFactory:doCloning true
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
LdapAuthInfo:
>>>>>>>> init()
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
LdapAuthInfo:
>>>>>>>> init begins
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
LdapAuthInfo:
>>>>>>>> init ends
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: init:
before
>>>>>>>> makeConnection errorIfDown is true
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
makeConnection:
>>>>>>>> errorIfDown true
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP
Keep-Alive:
>>>>>>>> true
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> SSLClientCertificateSelectionCB: Setting desired cert
nickname
>>>>>>>> to: subsystemCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> LdapJssSSLSocket: set client auth cert nickname
subsystemCert
>>>>>>>> cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> SSLClientCertificatSelectionCB: Entering!
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate
cert:
>>>>>>>> ocspSigningCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate
cert:
>>>>>>>> subsystemCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> SSLClientCertificateSelectionCB: desired cert found in
list:
>>>>>>>> subsystemCert cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> SSLClientCertificateSelectionCB: returning: subsystemCert
>>>>>>>> cert-pki-ca
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL
handshake
>>>>>>>> happened
>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>> CMSEngine.shutdown()
>>>>>>>>
>>>>>>>>
>>>>>>>> I really appreciate you sticking with me through this. I
owe
>>>>>>>> you.
>>>>>>>>
>>>>>>>> - Ian
>>>>>>>>
>>>>>>>>> After that, check that the LDAP entry has been
updated. If it
>>>>>>>>> is the case, you should be able to restart pki.
>>>>>>>>>
>>>>>>>>> Flo
>>>>>>>>>
>>>>>>>>>> Thank you again for your time!
>>>>>>>>>>
>>>>>>>>>> Ian
>>>>>>>>>>
>>>>>>>>>>> Flo.
>>>>>>>>>>>>
>>>>>>>>>>>> which seems weird.
>>>>>>>>>>>>
>>>>>>>>>>>>> The description attribute in the
ldapsearch output is of
>>>>>>>>>>>>> the form 2;#;DN;DN
>>>>>>>>>>>>>
>>>>>>>>>>>>> The # should match the serial number
in the first
>>>>>>>>>>>>> certutil output
>>>>>>>>>>>>>
>>>>>>>>>>>>> The ASCII blob (minus the BEGIN/END
headers) should match
>>>>>>>>>>>>> one of the
>>>>>>>>>>>>> userCertificate attributes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> rob
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>>>>> To unsubscribe send an email to
>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>> _______________________________________________
>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>> To unsubscribe send an email to
>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list --freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email tofreeipa-users-leave(a)lists.fedorahosted.org
>>
>
>
>
> _______________________________________________
> FreeIPA-users mailing list --freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email tofreeipa-users-leave(a)lists.fedorahosted.org