Hi Jan,

Would you be able to run those commands:

dbscan -f /var/lib/dirsrv/<instance>/db/cesnet_cz /nsuniqueid.db -k =fffffff-fffffff-fffffff-fffffff -r =fffffff-fffffff-fffffff-fffffff

then for each ID
dbscan -f /var/lib/dirsrv/<instance>/db/cesnet_cz /id2entry.db -K <ID>

thanks it could help to diagnose.
regards
thierry

On 1/15/21 9:56 AM, Jan Tomasek wrote:

Hi Mark,
On 13. 01. 21 17:33, Mark Reynolds wrote:

This is definitely an older version of the server, I would highly suggest to get onto the latest 1.4.x version that you can. 1.4.0 has not been maintained in a very long time, and is missing important fixes.

I've upgraded to version 1.4.4.9 which is present in upcomming Debian stable (Bullseye). The result is even worse.

After reindexing:

[14/Jan/2021:16:47:17.866756854 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: aci
[14/Jan/2021:16:47:17.867468013 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: cn
[14/Jan/2021:16:47:17.868599567 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing entryrdn
[14/Jan/2021:16:47:17.869738128 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: entryusn
[14/Jan/2021:16:47:17.870441180 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: givenName
[14/Jan/2021:16:47:17.873042341 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: mail
[14/Jan/2021:16:47:17.874059327 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: mailAlternateAddress
[14/Jan/2021:16:47:17.874611626 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: mailHost
[14/Jan/2021:16:47:17.875048657 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: member
[14/Jan/2021:16:47:17.875445102 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: memberOf
[14/Jan/2021:16:47:17.876030086 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: nsCertSubjectDN
[14/Jan/2021:16:47:17.876506348 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: nscpEntryDN
[14/Jan/2021:16:47:17.877063072 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: nsds5ReplConflict
[14/Jan/2021:16:47:17.877463986 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: nsTombstoneCSN
[14/Jan/2021:16:47:17.877869952 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: nsuniqueid
[14/Jan/2021:16:47:17.878523994 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: ntUniqueId
[14/Jan/2021:16:47:17.878950068 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: ntUserDomainId
[14/Jan/2021:16:47:17.879324003 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: numsubordinates
[14/Jan/2021:16:47:17.879937054 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: objectclass
[14/Jan/2021:16:47:17.880652956 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: owner
[14/Jan/2021:16:47:17.881298947 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: parentid
[14/Jan/2021:16:47:17.881917015 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: seeAlso
[14/Jan/2021:16:47:17.882350399 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: sn
[14/Jan/2021:16:47:17.883025762 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: targetuniqueid
[14/Jan/2021:16:47:17.883440145 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: telephoneNumber
[14/Jan/2021:16:47:17.884134175 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: uid
[14/Jan/2021:16:47:17.884760406 +0100] - INFO - bdb_db2index - cesnet_cz: Indexing attribute: uniquemember
[14/Jan/2021:16:47:18.749491112 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 1000 entries (9%).
[14/Jan/2021:16:47:19.528484588 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 2000 entries (19%).
[14/Jan/2021:16:47:20.040531342 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 3000 entries (28%).
[14/Jan/2021:16:47:20.769555937 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 4000 entries (38%).
[14/Jan/2021:16:47:21.403762300 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 5000 entries (47%).
[14/Jan/2021:16:47:22.134055315 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 6000 entries (57%).
[14/Jan/2021:16:47:22.861718595 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 7000 entries (66%).
[14/Jan/2021:16:47:23.455932352 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 8000 entries (76%).
[14/Jan/2021:16:47:24.105353501 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 9000 entries (86%).
[14/Jan/2021:16:47:24.603336412 +0100] - INFO - bdb_db2index - cesnet_cz: Indexed 10000 entries (95%).
[14/Jan/2021:16:47:24.749496424 +0100] - INFO - bdb_db2index - cesnet_cz: Finished indexing.
[14/Jan/2021:16:47:47.778127870 +0100] - ERR - _entryrdn_insert_key - Same DN (dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=cesnet,dc=cz) is already in the entryrdn file with different ID 10454. Expected ID is 10456.
[14/Jan/2021:16:47:47.778321609 +0100] - ERR - index_addordel_entry - database index operation failed BAD 1023, err=9999 Unknown error 9999
[14/Jan/2021:16:47:47.800034893 +0100] - ERR - NSMMReplicationPlugin - _replica_configure_ruv - Failed to create replica ruv tombstone entry (dc=cesnet,dc=cz); LDAP error - 1

After restarting, server crashes:

Jan 14 16:55:44 cml3 ns-slapd[18736]: [14/Jan/2021:16:55:44.123358593 +0100] - ERR - _entryrdn_insert_key - Same DN (dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=cesnet,dc=cz) is already in the entryrdn file with different ID 10454. Expected ID is 10456.
Jan 14 16:55:44 cml3 ns-slapd[18736]: [14/Jan/2021:16:55:44.123685788 +0100] - ERR - index_addordel_entry - database index operation failed BAD 1023, err=9999 Unknown error 9999
Jan 14 16:55:44 cml3 ns-slapd[18736]: [14/Jan/2021:16:55:44.145147700 +0100] - ERR - NSMMReplicationPlugin - _replica_configure_ruv - Failed to create replica ruv tombstone entry (dc=cesnet,dc=cz); LDAP error - 1
Jan 14 16:55:44 cml3 ns-slapd[18736]: [14/Jan/2021:16:55:44.145445953 +0100] - ERR - NSMMReplicationPlugin - replica_new - Unable to configure replica dc=cesnet,dc=cz:
Jan 14 16:55:44 cml3 kernel: ns-slapd[18736]: segfault at 110 ip 00007fc7b6210a20 sp 00007ffea3966728 error 4
Jan 14 16:55:44 cml3 kernel: Code: 1f 84 00 00 00 00 00 0f 1f 00 48 8b 7f 70 e9 d7 09 fd ff 0f 1f 80 00 00 00 00 48 8b 7f 70 e9 97 1b fd ff 0f 1f 80 00 00 00 00 <48> 8b 87 10 01 00 00 c3 0f 1f 84 00 00 00 00 00 41 54 45 31 e4 55
Jan 14 16:55:44 cml3 systemd[1]: dirsrv@cml3.service: Main process exited, code=killed, status=11/SEGV

and is unable to start.

Solution is to run:

dsctl cml3 db2index cesnet_cz

where I didn't discovered how to index only cesnet_cz backend, it reindexs all. And causes replication stop working:

[15/Jan/2021:09:35:14.406675323 +0100] - ERR - agmt="cn=cesnet_cz-ldap31" (ldap31:636) - clcache_load_buffer - Can't locate CSN 60014363000000010000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[15/Jan/2021:09:35:14.406966190 +0100] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=cesnet_cz-ldap31" (ldap31:636): CSN 60014363000000010000 not found, we aren't as up to date, or we purged
[15/Jan/2021:09:35:14.407091288 +0100] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=cesnet_cz-ldap31" (ldap31:636): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.

So reinitialize all suffixes on all replicas. And server is again working.

At this moment are replicas 1.4.0 upgraded is only master.

I will try to reduce amount of data in suffix and try elaborate if this problem is causing some our data. :(
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org