FreeIPA Upgrade F31 -> F32: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
by Anthony Joseph Messina
After upgrading FreeIPA from F31 to F32, on startup I now see a lot of these errors from certmonger, ns-slapd, java, etc.
May 08 17:57:28 certmonger[38]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
May 08 17:57:30 ns-slapd[67]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
May 08 17:57:33 dogtag-ipa-renew-agent-submit[143]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
May 08 17:57:42 java[640]: usr/lib/api/apiutil.c Could not open /run/lock/opencryptoki/LCK..APIlock
The server seems to come up without issue, but can you point me in the right direction to resolve these errors?
freeipa-server-4.8.6-1.fc32.x86_64
opendnssec-2.1.6-5.fc32.x86_64
opencryptoki-3.13.0-1.fc32.x86_64
I've installed a fresh F32 freeipa-server (on a test domain) and I don't see these errors.
Thanks. -A
--
Anthony - https://messinet.com
F9B6 560E 68EA 037D 8C3D D1C9 FF31 3BDB D9D8 99B6
2 years, 9 months
dirsrv hangs soon after reboot
by Kees Bakker
Hey,
I'm looking for advice how to analyse/debug this.
On one of the masters the dirsrv is unresponsive. It runs, but every
attempt to connect it hangs.
The command "systemctl status" does not show anything alarming
● dirsrv(a)EXAMPLE-COM.service - 389 Directory Server EXAMPLE-COM.
Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
Active: active (running) since vr 2020-04-17 13:46:25 CEST; 1h 33min ago
Process: 3123 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS)
Main PID: 3134 (ns-slapd)
Status: "slapd started: Ready to process requests"
CGroup: /system.slice/system-dirsrv.slice/dirsrv(a)EXAMPLE-COM.service
└─3134 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-EXAMPLE-COM -i /var/run/dirsrv/slapd-EXAMPLE-COM.pid
apr 17 15:13:54 linge.example.com ns-slapd[3134]: GSSAPI client step 1
apr 17 15:13:54 linge.example.com ns-slapd[3134]: GSSAPI client step 1
apr 17 15:13:54 linge.example.com ns-slapd[3134]: GSSAPI client step 1
apr 17 15:13:54 linge.example.com ns-slapd[3134]: GSSAPI client step 1
apr 17 15:13:54 linge.example.com ns-slapd[3134]: GSSAPI client step 2
apr 17 15:18:54 linge.example.com ns-slapd[3134]: GSSAPI client step 1
apr 17 15:18:54 linge.example.com ns-slapd[3134]: GSSAPI client step 1
apr 17 15:18:55 linge.example.com ns-slapd[3134]: GSSAPI client step 1
apr 17 15:18:55 linge.example.com ns-slapd[3134]: GSSAPI client step 1
apr 17 15:18:55 linge.example.com ns-slapd[3134]: GSSAPI client step 2
However, an ldapsearch command hangs forever
[root@rotte ~]# ldapsearch -H ldaps://linge.example.com -D uid=keesbtest,cn=users,cn=accounts,dc=example,dc=com -W -LLL -o ldif-wrap=no -b cn=users,cn=accounts,dc=example,dc=com '(&(objectClass=person)(memberOf=cn=admins,cn=groups,cn=accounts,dc=example,dc=com))' uid
Enter LDAP Password:
Even if I use the socket (ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket) the ldapsearch
command hangs.
"ipactl status" hangs
"kinit" hangs
--
Kees Bakker
2 years, 11 months
Unable to install ipa client centos 7.5.1804 (Core)
by William Graboyes
Hello List,
I have been searching around for the day and have found an answer for
the error I am getting when I am trying to install the client on a brand
new install:
Version:
ipa-client-4.5.4-10.el7.centos.3.x86_64
ipa-client-common-4.5.4-10.el7.centos.3.noarch
The error is below (run as root, not via sudo):
ipa-client-install
Traceback (most recent call last):
File "/sbin/ipa-client-install", line 22, in <module>
from ipaclient.install import ipa_client_install
File
"/usr/lib/python2.7/site-packages/ipaclient/install/ipa_client_install.py",
line 5, in <module>
from ipaclient.install import client
File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py",
line 34, in <module>
from ipalib import api, errors, x509
File "/usr/lib/python2.7/site-packages/ipalib/x509.py", line 45, in
<module>
from pyasn1_modules import rfc2315, rfc2459
File "/usr/lib/python2.7/site-packages/pyasn1_modules/rfc2315.py",
line 67, in <module>
class DigestedData(univ.Sequence):
File "/usr/lib/python2.7/site-packages/pyasn1_modules/rfc2315.py",
line 72, in DigestedData
namedtype.NamedType('digest', Digest)
File "/usr/lib/python2.7/site-packages/pyasn1/type/namedtype.py",
line 115, in __init__
self.__ambiguousTypes = 'terminal' not in kwargs and
self.__computeAmbiguousTypes() or {}
File "/usr/lib/python2.7/site-packages/pyasn1/type/namedtype.py",
line 232, in __computeAmbiguousTypes
ambigiousTypes[idx] = NamedTypes(*partialAmbigiousTypes,
**dict(terminal=True))
File "/usr/lib/python2.7/site-packages/pyasn1/type/namedtype.py",
line 114, in __init__
self.__tagToPosMap = self.__computeTagToPosMap()
File "/usr/lib/python2.7/site-packages/pyasn1/type/namedtype.py",
line 205, in __computeTagToPosMap
for _tagSet in tagMap.presentTypes:
AttributeError: 'property' object has no attribute 'presentTypes'
Any help would be greatly appreciated.
Thanks,
Bill G.
3 years, 2 months
FreeIPA certificate doesn't validate in iOS
by Jochen Kellner
Hello,
I'm running IPA on current Fedora 32, freeipa-server-4.8.9-2 and pki-server-10.9.0-0.4
Today the certificate of my IMAP server (running on Debian Buster) was
automatically refreshed:
,----
| Request ID '20181003215953':
| status: MONITORING
| stuck: no
| key pair storage: type=FILE,location='/etc/ssl/private/imap.jochen.org.key'
| certificate: type=FILE,location='/etc/ssl/certs/imap.jochen.org.crt'
| CA: IPA
| issuer: CN=Certificate Authority,O=JOCHEN.ORG
| subject: CN=imap.jochen.org,O=JOCHEN.ORG
| expires: 2022-09-07 09:30:16 CEST
| dns: imap.jochen.org
| principal name: imap/jupiter.jochen.org(a)JOCHEN.ORG
| key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
| eku: id-kp-serverAuth,id-kp-clientAuth
| pre-save command:
| post-save command: /root/refresh_cyrus_certificate.sh
| track: yes
| auto-renew: yes
`----
On an iPhone one of my users gets a message that the certificate is not valid.
Reason seems to be this: https://7402.org/blog/2019/new-self-signed-ssl-cert-ios-13.html
When I look at the certificate with openssl I see:
,----
| X509v3 extensions:
| X509v3 Authority Key Identifier:
| keyid:4F:F8:45:3D:E8:06:4B:8D:BB:9D:D2:D1:8B:00:43:A1:07:16:A1:17
|
| Authority Information Access:
| OCSP - URI:http://ipa-ca.jochen.org/ca/ocsp
|
| X509v3 Key Usage: critical
| Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
| X509v3 Extended Key Usage:
| TLS Web Server Authentication, TLS Web Client Authentication
`----
My current guess is that the "Key Usage: critical" is the reason for the iOS error.
I've looked for the certprofiles and found these files:
,----
| [root@freeipa3 /]# find . -name \*caIPAserviceCert\* -ls
| 8510694 8 -rw-rw---- 1 pkiuser pkiuser 6218 Mär 4 2020 ./var/lib/pki/pki-tomcat/ca/profiles/ca/caIPAserviceCert.cfg
| 9332162 4 -rw-r--r-- 1 root root 229 Aug 20 12:38 ./usr/lib/python3.8/site-packages/ipaclient/csrgen/profiles/caIPAserviceCert.json
| 26138015 8 -rw-r--r-- 1 root root 7014 Aug 20 12:37 ./usr/share/ipa/profiles/caIPAserviceCert.UPGRADE.cfg
| 26138016 8 -rw-r--r-- 1 root root 7294 Aug 20 12:37 ./usr/share/ipa/profiles/caIPAserviceCert.cfg
| 9323278 8 -rw-r--r-- 1 root root 6272 Jun 25 23:53 ./usr/share/pki/ca/profiles/ca/caIPAserviceCert.cfg
`----
These files contain:
,----
| policyset.serverCertSet.6.constraint.params.keyUsageCritical=true
| policyset.serverCertSet.6.constraint.params.keyUsageDigitalSignature=true
| policyset.serverCertSet.6.constraint.params.keyUsageNonRepudiation=true
| policyset.serverCertSet.6.constraint.params.keyUsageDataEncipherment=true
| policyset.serverCertSet.6.constraint.params.keyUsageKeyEncipherment=true
| policyset.serverCertSet.6.constraint.params.keyUsageKeyAgreement=false
`----
So I think this is where the critical comes from and the keyUsage defaults come from.
What I could use help with is the following:
1. I didn't find reports about the problem in pagure or the mailing
list. Am I really alone with this?
2. My FreeIPA has been installed years ago on Fedora, moved to CentOS
and this year back to Fedora by creating replicas. Has there been a
problem with upgrading the certprofiles?
3. How can I remove the options from the certificate request so that
certmonger gets a valid certificate?
Do I miss something else?
--
This space is intentionally left blank.
3 years, 2 months
Stateless Machines and Force Join
by Mark Potter
We boot everything stateless in our environment and are using FreeIPA for
authentication. I started discussing this a while ago but ended up with
other things taking priority. The number of machines we have make managing
keys an untenable solution so we are using
ipa-client-install -U -q -p <join user> -w <password --domain=domain.com
--server=ipaserver.domain.com --fixed-primary --force-join
called from rc.local during boot to rejoin machines to the FreeIPA
environment (we will be moving away from --fixed-primary but aren't there
yet). While this works it, potentially, exposes a password. I am looking
for a better way to handle machines that need to re-join at every boot.
We have access to ansible as well a decent, in house, templating system for
configuration. Please forgive my starting this discussion anew and not
resurrecting a zombie and thanks in advance for your help!
--
*Mark Potter*
Senior Linux Administrator
3 years, 4 months
Allocation of a new value for DNA range failed
by Ronald Wimmer
After upgrading to OL 8.1 and replacing all of my 8 IPA servers I ran
into this particular problem.
Is it right that I need to have an ID range where all DNA ranges have to
fit in? And that the DNA range of each IPA server has to be distinct
from the ranges of the other IPA servers?
I will start by checking each IPA server with
ldapsearch -x -D 'cn=Directory Manager' -W -b 'cn=Posix
IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config'
(according to what Rob wrote on his blog some years ago
https://rcritten.wordpress.com/2015/01/05/freeipa-and-no-dna-range/ )
Cheers,
Ronald
3 years, 4 months
FreeIPA, OSX, DockerDesktop
by james liu
PREP
====
git clone https://github.com/freeipa/freeipa-container.git
cd freeipa-container
mkdir /tmp/ipa-data
docker run --name freeipa-server-container -ti -h ipa.example.test --read-only -v /tmp/ip-data :/data:Z freeipa-server --sysctl net.ipv6.conf.all.disable_ipv6=1
RESULT
======
tar: etc/sysconfig/selinux: Cannot utime: No such file or directory
tar: Exiting with failure status due to previous errors
QUESTION
=========
I'm running DockerDesktop 2.0.4, OSX 10.13.6.
Is there a set of commands that will work?
Thanks
3 years, 6 months
yum update problem
by Natxo Asenjo
hi,
after patching our centos 7 hosts to the latest version today, one of the
two replicas is having trouble.
[root@kdc2 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: STOPPED
kadmin Service: STOPPED
named Service: STOPPED
httpd Service: RUNNING
ipa-custodia Service: STOPPED
ntpd Service: STOPPED
pki-tomcatd Service: RUNNING
smb Service: STOPPED
winbind Service: STOPPED
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
ipa: INFO: The ipactl command was successful
and after digging in the logs I come across this in /var/log/ipaupgrade.log:
2019-11-20T18:18:29Z DEBUG stderr=
2019-11-20T18:18:31Z INFO Certmonger certificate renewal configuration
already up-to-date
2019-11-20T18:18:31Z INFO [Enable PKIX certificate path discovery and
validation]
2019-11-20T18:18:31Z DEBUG Loading StateFile from
'/var/lib/ipa/sysupgrade/sysupgrade.state'
2019-11-20T18:18:31Z INFO PKIX already enabled
2019-11-20T18:18:31Z INFO [Authorizing RA Agent to modify profiles]
2019-11-20T18:18:31Z INFO [Authorizing RA Agent to manage lightweight CAs]
2019-11-20T18:18:31Z INFO [Ensuring Lightweight CAs container exists in
Dogtag database]
2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_139740162547472
2019-11-20T18:18:31Z DEBUG flushing
ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache
2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17cc24b638>
2019-11-20T18:18:31Z DEBUG Destroyed connection
context.ldap2_139740162547472
2019-11-20T18:18:31Z INFO [Adding default OCSP URI configuration]
2019-11-20T18:18:31Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2019-11-20T18:18:31Z INFO [Migrating certificate profiles to LDAP]
2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_139740160021648
2019-11-20T18:18:31Z DEBUG flushing
ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache
2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache
url=ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket
conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17cc289b00>
2019-11-20T18:18:31Z DEBUG Destroyed connection
context.ldap2_139740160021648
2019-11-20T18:18:31Z DEBUG request GET
https://kdc2.l.domain.it:8443/ca/rest/account/login
2019-11-20T18:18:31Z DEBUG request body ''
2019-11-20T18:18:31Z DEBUG response status 401
2019-11-20T18:18:31Z DEBUG response headers Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Thu, 01 Jan 1970 01:00:00 CET
WWW-Authenticate: Basic realm="Certificate Authority"
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 951
Date: Wed, 20 Nov 2019 18:18:31 GMT
2019-11-20T18:18:31Z DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 401 - </h1><HR size="1"
noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b>
<u></u></p><p><b>description</b> <u>This request requires HTTP
authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
2019-11-20T18:18:31Z ERROR IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2019-11-20T18:18:31Z DEBUG File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
line 54, in run
server.upgrade()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2146, in upgrade
upgrade_configuration()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 2018, in upgrade_configuration
ca_enable_ldap_profile_subsystem(ca)
File
"/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 406, in ca_enable_ldap_profile_subsystem
cainstance.migrate_profiles_to_ldap()
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
line 2027, in migrate_profiles_to_ldap
_create_dogtag_profile(profile_id, profile_data, overwrite=False)
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py",
line 2033, in _create_dogtag_profile
with api.Backend.ra_certprofile as profile_api:
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line
1315, in __enter__
raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA
REST API'))
2019-11-20T18:18:31Z DEBUG The ipa-server-upgrade command failed,
exception: RemoteRetrieveError: Failed to authenticate to CA REST API
2019-11-20T18:18:31Z ERROR Unexpected error - see /var/log/ipaupgrade.log
for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
In this kdc I see these errors in getcert list:
Request ID '20190220182014':
status: MONITORING
ca-error: Invalid cookie: u''
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT
subject: CN=CA Audit,O=L.DOMAIN.IT
expires: 2019-12-05 13:58:24 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190220182015':
status: MONITORING
ca-error: Invalid cookie: u''
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT
subject: CN=OCSP Subsystem,O=L.DOMAIN.IT
expires: 2019-12-05 13:58:24 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190220182016':
status: MONITORING
ca-error: Invalid cookie: u''
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT
subject: CN=CA Subsystem,O=L.DOMAIN.IT
expires: 2019-12-05 13:58:24 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20190220182018':
status: MONITORING
ca-error: Invalid cookie: u''
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT
subject: CN=IPA RA,O=L.DOMAIN.IT
expires: 2019-12-05 13:58:44 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20190220182019':
status: MONITORING
ca-error: Server at "
https://kdc2.l.domain.it:8443/ca/agent/ca/profileProcess" replied: 1:
Invalid Credential.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=L.DOMAIN.IT
subject: CN=kdc2.l.domain.it,O=L.DOMAIN.IT
expires: 2019-12-10 10:57:52 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
I still have a working replica, so I could just reinstall and have a
working set in a couple of minutes, but I would like to find out what has
gone wrong.
The systems are running ipa-server-4.6.5-11.el7.centos.3.x86_64
Any help welcome ;-)
Thanks,
--
Groeten,
natxo
3 years, 6 months
Re: Renewing a failed to auto-renewal certificate
by Stuart McRobert
Dear flo,
Thanks for the helpful links.
To check whether replication is possible between the three freeipa
servers, via the web interface on each, I have successfully created three
new users:
+ On server 1 create a new user 1 and check they appear on servers 2 & 3 - yes
+ On server 2 create a new user 2 and check they appear on servers 1 & 3 - yes
+ On server 3 create a new user 3 and check they appear on servers 1 & 2 - yes
Then for each user remove them from a different server to where they were
created. This worked as expected.
However I do appear to have one user whose information failed to
correctly replicate some time ago, causing an inconsistency that I am
uncertain on the best way to fix.
On server 1 via web attempt to view this specific user:
Operations Error
Some operations failed.
XXX: user not found
which should be displayed as a preserved user entry but lacks any details
shown other than the "user login".
On server 2 this user appears as an active user, with details, status
disabled.
On server 3 this user appears as a preserved user, with full details, and
a note I added to the Class field a few days ago to see whether it would
then just replicate but failed.
In logs, searching for the user in question on server 1
./httpd/error_log:[Sun Sep 20 10:34:14.256333 2020] [wsgi:error] [pid 29637] [remote IP_ADDRESS:56930] ipa: INFO: [jsonserver_session] sm@OUR_DOMAIN: user_find(u'USER_XXX', sizelimit=0, version=u'2.215', pkey_only=True): SUCCESS
./httpd/error_log:[Sun Sep 20 10:34:28.721951 2020] [wsgi:error] [pid 29637] [remote IP_ADDRESS:56932] ipa: INFO: [jsonserver_session] sm@OUR_DOMAIN: user_find(u'USER_XXX', preserved=True, sizelimit=0, version=u'2.215', pkey_only=True): SUCCESS
./httpd/error_log:[Sun Sep 20 10:34:28.738772 2020] [wsgi:error] [pid 29636] [remote IP_ADDRESS:56932] ipa: INFO: sm@OUR_DOMAIN: batch: user_show(u'USER_XXX', no_members=True): NotFound
./httpd/error_log:[Sun Sep 20 10:34:28.738952 2020] [wsgi:error] [pid 29636] [remote IP_ADDRESS:56932] ipa: INFO: [jsonserver_session] sm@OUR_DOMAIN: batch(({u'params': ((u'USER_XXX',), {u'no_members': True}), u'method': u'user_show'},), version=u'2.215'): SUCCESS
Which just confirms what was seen via the web interface.
Next go looking in /var/log/dirsrv/slapd-OUR-DOMAIN/errors and find lots
of these messages:
[20/Sep/2020:11:18:12.449932996 +0100] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=caToserver2" (server2:389): CSN 5bb473ef000000070000 not found, we aren't as up to date, or we purged
[20/Sep/2020:11:18:12.452051936 +0100] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caToserver2" (server2:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[20/Sep/2020:11:18:15.479226915 +0100] - ERR - agmt="cn=caToserver2" (server2:389) - clcache_load_buffer - Can't locate CSN 5bb473ef000000070000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[20/Sep/2020:11:18:15.481677959 +0100] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=caToserver2" (server2:389): CSN 5bb473ef000000070000 not found, we aren't as up to date, or we purged
[20/Sep/2020:11:18:15.483458741 +0100] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=caToserver2" (server2:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
But is that the only CSN found to be missing, seems to be based on this
search:
grep -v 5bb473ef000000070000 ./dirsrv/slapd-OUR-DOMAIN/errors | grep -v "If the error persists"
389-Directory/1.3.6.6 B2017.145.2031
server1.OUR-DOMAIN:636 (/etc/dirsrv/slapd-OUR-DOMAIN)
[18/Sep/2020:05:57:26.038958649 +0100] - ERR - NSMMReplicationPlugin - release_replica - agmt="cn=meToserver3" (server3:389): Unable to parse the response to the endReplication extended operation.
[18/Sep/2020:17:45:55.236862951 +0100] - ERR - NSMMReplicationPlugin - release_replica - agmt="cn=caToserver2" (server2:389): Unable to parse the response to the endReplication extended operation.
Turning next to server 2 and the dirsrv logs there, lots of messages of the form:
[17/Sep/2020:19:33:24.075845464 +0100] - ERR - DSRetroclPlugin - delete_changerecord: could not delete change record 6773158 (rc: 32)
[17/Sep/2020:19:33:24.076296207 +0100] - ERR - DSRetroclPlugin - delete_changerecord: could not delete change record 6773159 (rc: 32)
[17/Sep/2020:19:33:24.077311010 +0100] - ERR - DSRetroclPlugin - delete_changerecord: could not delete change record 6773160 (rc: 32)
[17/Sep/2020:19:33:24.077830624 +0100] - ERR - DSRetroclPlugin - delete_changerecord: could not delete change record 6773161 (rc: 32)
[19/Sep/2020:19:59:28.091654453 +0100] - ERR - NSMMReplicationPlugin - release_replica - agmt="cn=meToserver1" (server1:389): Unable to parse the response to the endReplication extended operation.
Finally for server 3, dirsrv logs appear to be more about getting started
after certificate problems the other day, rather than issues seen on the
other servers, but this server does appear to display the user fully as
would be desired everywhere.
Our old version appears to lack the 389 command dsconf so not been able to
use that to help.
Summary:
+ Replication appears to work for e.g. new users
+ I guess one entry has failed to be replicated correctly causing the
inconsistent state on the other two servers
+ This should ideally be corrected to avoid further issues???
+ Uncertain about the best way to do this, suggestions very welcome as
first time fixing such problems and I don't want to make it any worse,
after all replications seems to be working for everything else.
Thanks
Best wishes
Stuart
3 years, 6 months
