1:08 p.m.
We have experienced several cases of end users not being able to authenticate. While
investigating I've found that I can not obtain kinit credentials on the local freeipa
replicaipactl however shows all processes including Directory Server as running. Doing
ipactl restart hangs but service ipa stop/start does help.
In the logs I find the following:cat errors | grep
"28/Oct/2017"[28/Oct/2017:01:30:46.931199685 +0000] NSMMReplicationPlugin -
agmt="cn=meTomaster.pop1.domain.company" (master:389): Unable to receive the
response for a startReplication extended operation to consumer (Can't contact LDAP
server). Will retry later.[28/Oct/2017:01:37:08.323949440 +0000] NSMMReplicationPlugin -
agmt="cn=meTomaster.pop1.domain.company" (master:389): Replication bind with
GSSAPI auth resumed[28/Oct/2017:10:51:48.025975201 +0000] ipa-topology-plugin -
ipa_topo_be_state_changebackend userRoot is going offline; inactivate
plugin[28/Oct/2017:10:51:48.026935974 +0000] NSMMReplicationPlugin -
multimaster_be_state_change: replica dc=domain,dc=company is going offline; disabling
replication[28/Oct/2017:10:51:48.263462882 +0000] WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to access the
database[28/Oct/2017:10:52:08.300485142 +0000] import userRoot: Processed 2042 entries --
average rate 102.1/sec, recent rate 102.0/sec, hit ratio 0%[28/Oct/2017:10:52:28.330367817
+0000] import userRoot: Processed 7749 entries -- average rate 193.7/sec, recent rate
193.7/sec, hit ratio 100%[28/Oct/2017:10:52:48.360876924 +0000] import userRoot: Processed
9921 entries -- average rate 165.3/sec, recent rate 197.0/sec, hit ratio
100%[28/Oct/2017:10:53:08.391322582 +0000] import userRoot: Processed 15853 entries --
average rate 198.2/sec, recent rate 202.6/sec, hit ratio
100%[28/Oct/2017:10:53:14.802005648 +0000] import userRoot: Workers finished; cleaning
up...[28/Oct/2017:10:53:15.002839240 +0000] import userRoot: Workers cleaned
up.[28/Oct/2017:10:53:15.003167651 +0000] import userRoot: Indexing
complete. Post-processing...[28/Oct/2017:10:53:15.003384044 +0000] import userRoot:
Generating numsubordinates (this may take several minutes to
complete)...[28/Oct/2017:10:53:15.043991058 +0000] import userRoot: Generating
numSubordinates complete.[28/Oct/2017:10:53:15.045232248 +0000] import userRoot: Gathering
ancestorid non-leaf IDs...[28/Oct/2017:10:53:15.045698245 +0000] import userRoot: Finished
gathering ancestorid non-leaf IDs.[28/Oct/2017:10:53:15.046529835 +0000] import userRoot:
Creating ancestorid index (new idl)...[28/Oct/2017:10:53:15.175418711 +0000] import
userRoot: Created ancestorid index (new idl).[28/Oct/2017:10:53:15.175659600 +0000] import
userRoot: Flushing caches...[28/Oct/2017:10:53:15.175818325 +0000] import userRoot:
Closing files...[28/Oct/2017:10:53:15.243592429 +0000] import userRoot: Import
complete. Processed 16676 entries in 87 seconds. (191.68
entries/sec)[28/Oct/2017:10:53:15.252306744 +0000] ipa-topology-plugin -
ipa_topo_be_state_change - backend userRoot is coming online; checking domain level and
init shared topology[28/Oct/2017:10:53:15.256378790 +0000] NSMMReplicationPlugin -
multimaster_be_state_change: replica dc=domain,dc=company is coming online; enabling
replication[28/Oct/2017:10:53:15.267602128 +0000] NSMMReplicationPlugin -
replica_reload_ruv: Warning: new data for replica dc=domain,dc=company does not match the
data in the changelog.[28/Oct/2017:10:53:15.284118756 +0000] NSMMReplicationPlugin -
changelog program - _cl5NewDBFile: PR_DeleteSemaphore:
/var/lib/dirsrv/slapd-domain-company/cldb/c96bdb0c-7d1a11e7-9c2f9351-ba1966ca.sema; NSPR
error - -5943[28/Oct/2017:11:08:04.961514521 +0000] slapd shutting down - signaling
operation threads - op stack size 81 max work q size 52 max work q stack size
52[28/Oct/2017:11:08:04.962208885 +0000] slapd shutting down - waiting for 24 threads to
terminate[28/Oct/2017:11:09:42.503084236 +0000] SSL alert: Sending pin request to SVRCore.
You may need to run systemd-tty-ask-password-agent to provide the
password.[28/Oct/2017:11:09:42.504400971 +0000] SSL alert: Security Initialization:
Enabling default cipher set.[28/Oct/2017:11:09:42.504747723 +0000] SSL alert: Configured
NSS Ciphers[28/Oct/2017:11:09:42.504975400 +0000] SSL
alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384:
enabled[28/Oct/2017:11:09:42.505157282 +0000] SSL
alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled[28/Oct/2017:11:09:42.505371032
+0000] SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256:
enabled[28/Oct/2017:11:09:42.505521550 +0000] SSL
alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled[28/Oct/2017:11:09:42.505686484
+0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:
enabled[28/Oct/2017:11:09:42.505907355 +0000] SSL
alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled[28/Oct/2017:11:09:42.506066798
+0000] SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:
enabled[28/Oct/2017:11:09:42.506207828 +0000] SSL
alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled[28/Oct/2017:11:09:42.506349370
+0000] SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384:
enabled[28/Oct/2017:11:09:42.506492473 +0000] SSL
alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled[28/Oct/2017:11:09:42.506634151
+0000] SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA:
enabled[28/Oct/2017:11:09:42.506810644 +0000] SSL
alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled[28/Oct/2017:11:09:42.506977554
+0000] SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256:
enabled[28/Oct/2017:11:09:42.507120362 +0000] SSL
alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled[28/Oct/2017:11:09:42.507262604
+0000] SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA:
enabled[28/Oct/2017:11:09:42.507402949 +0000] SSL
alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled[28/Oct/2017:11:09:42.507541573
+0000] SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384:
enabled[28/Oct/2017:11:09:42.507722070 +0000] SSL
alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled[28/Oct/2017:11:09:42.507877825 +0000]
SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled[28/Oct/2017:11:09:42.508016421
+0000] SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256:
enabled[28/Oct/2017:11:09:42.508202238 +0000] SSL
alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled[28/Oct/2017:11:09:42.508417061 +0000]
SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled[28/Oct/2017:11:09:42.508653676
+0000] SSL alert: TLS_AES_128_GCM_SHA256: enabled[28/Oct/2017:11:09:42.508834912
+0000] SSL alert: TLS_CHACHA20_POLY1305_SHA256:
enabled[28/Oct/2017:11:09:42.508994238 +0000] SSL alert: TLS_AES_256_GCM_SHA384:
enabled[28/Oct/2017:11:09:42.509136471 +0000] SSL
alert: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256:
enabled[28/Oct/2017:11:09:42.509282307 +0000] SSL
alert: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256:
enabled[28/Oct/2017:11:09:42.509418462 +0000] SSL
alert: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256:
enabled[28/Oct/2017:11:09:42.518209787 +0000] SSL Initialization - Configured SSL version
range: min: TLS1.0, max: TLS1.2[28/Oct/2017:11:09:42.518559355 +0000]
389-Directory/1.3.5.10 B2017.102.203 starting up[28/Oct/2017:11:09:42.532319246 +0000]
default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle
caseExactIA5Match[28/Oct/2017:11:09:42.541075634 +0000] WARNING: userRoot: entry cache
size 10485760 B is less than db size 73367552 B; We recommend to increase the entry cache
size nsslapd-cachememsize.[28/Oct/2017:11:09:42.541255997 +0000] WARNING: changelog: entry
cache size 2097152 B is less than db size 138485760 B; We recommend to increase the entry
cache size nsslapd-cachememsize.[28/Oct/2017:11:09:42.542038907 +0000] Detected Disorderly
Shutdown last time Directory Server was running, recovering
database.[28/Oct/2017:11:09:42.665474196 +0000] schema-compat-plugin - scheduled
schema-compat-plugin tree scan in about 5 seconds after the server
startup![28/Oct/2017:11:09:42.680833311 +0000] NSACLPlugin - The ACL target
cn=groups,cn=compat,dc=domain,dc=company does not exist[28/Oct/2017:11:09:42.681203039
+0000] NSACLPlugin - The ACL target cn=computers,cn=compat,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.681466158 +0000] NSACLPlugin - The ACL target
cn=ng,cn=compat,dc=domain,dc=company does not exist[28/Oct/2017:11:09:42.681742228 +0000]
NSACLPlugin - The ACL target ou=sudoers,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.682008654 +0000] NSACLPlugin - The ACL target
cn=users,cn=compat,dc=domain,dc=company does not exist[28/Oct/2017:11:09:42.682628758
+0000] NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.682919339 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=domain,dc=company does not exist[28/Oct/2017:11:09:42.683179463 +0000]
NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.683434761 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=domain,dc=company does not exist[28/Oct/2017:11:09:42.683692899 +0000]
NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.683955886 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=domain,dc=company does not exist[28/Oct/2017:11:09:42.684214903 +0000]
NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.684467463 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=domain,dc=company does not exist[28/Oct/2017:11:09:42.684727834 +0000]
NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.684981590 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=domain,dc=company does not exist[28/Oct/2017:11:09:42.685241334 +0000]
NSACLPlugin - The ACL target cn=vaults,cn=kra,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.702875810 +0000] NSACLPlugin - The ACL target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.703208704 +0000] NSACLPlugin - The ACL target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=domain,dc=company does not
exist[28/Oct/2017:11:09:42.815182267 +0000] NSACLPlugin - The ACL target cn=automember
rebuild membership,cn=tasks,cn=config does not exist[28/Oct/2017:11:09:42.822681438 +0000]
auto-membership-plugin - automember_parse_regex_rule: Unable to parse regex rule (invalid
regex). Error "nothing to repeat".[28/Oct/2017:11:09:42.865610767 +0000]
schema-compat-plugin - schema-compat-plugin tree scan will start in about 5
seconds![28/Oct/2017:11:09:42.873896378 +0000] slapd started. Listening on All Interfaces
port 389 for LDAP requests[28/Oct/2017:11:09:42.874123907 +0000] Listening on All
Interfaces port 636 for LDAPS requests[28/Oct/2017:11:09:42.874279887 +0000] Listening on
/var/run/slapd-domain-company.socket for LDAPI requests[28/Oct/2017:11:09:54.727083945
+0000] schema-compat-plugin - warning: no entries set up under cn=computers,
cn=compat,dc=domain,dc=company[28/Oct/2017:11:09:54.727502733 +0000] schema-compat-plugin
- Finished plugin initialization.
Does this server need re-installing/re-initializing or can I do anything to troubleshot
this further.