Hello,
In our set-up, we have a DB with all the users and groups, which we use as ground truth for provisioning the forementioned objects in FreeIPA (2 master servers + replicas). We are continuously synchronizing entries (~60000 users and 60000 groups, where groups might have 0 to 20000 members) from the DB to FreeIPA. In each cycle of synch, we are figuring out the differences and add, delete, or change existing entries.
The first sync (through which we had to import all 120000 objects) clogged the server totally, and after tweaking the 389DS we ended up disabling the memberOf plugin where it finally worked (we followed the FreeIPA documentation[1]). One of the advice to follow is to do the sync and then run the fixup task in the server where the provisioning happened. The fixup still clogs the server after some point and stops. The errors we get in the log are the following:
``` [06/May/2020:18:16:59.862308719 +0200] - INFO - memberof-plugin - memberof_fixup_task_thread - Memberof task starts (filter: "(|(objectclass=inetuser)(objectclass=inetadmin)(objectclass=nsmemberof))") ... [06/May/2020:20:07:49.545606214 +0200] - ERR - libdb - BDB2055 Lock table is out of available lock entries [06/May/2020:20:07:49.547921580 +0200] - ERR - idl_new_delete_key - idl_new.c BAD 22, err=12 Cannot allocate memory [06/May/2020:20:07:49.548930035 +0200] - ERR - addordel_values_sv - database index operation failed BAD 1130, err=12 Cannot allocate memory [06/May/2020:20:07:49.549779631 +0200] - ERR - addordel_values_sv - database index operation failed BAD 1140, err=12 Cannot allocate memory [06/May/2020:20:07:49.550612745 +0200] - ERR - index_addordel_values_ext_sv - database index operation failed BAD 1230, err=12 Cannot allocate memory [06/May/2020:20:07:49.551444741 +0200] - ERR - index_add_mods - database index operation failed BAD 1041, err=12 Cannot allocate memory [06/May/2020:20:07:49.552457769 +0200] - ERR - index_add_mods - database index operation failed BAD 1040, err=12 Cannot allocate memory [06/May/2020:20:07:49.553305019 +0200] - ERR - ldbm_back_modify - index_add_mods failed, err=12 Cannot allocate memory ```
We increased the number of DB locks and set the `nsslapd-cache-autosize` to 50% (server has currently 13G of memory). The only thing we saw was that one thread was using 100% of one of the CPUs.
Any advice on how to deal with this? We would really need to have memberOf attribute. Thank you in advance! Best Regards Mary Georgiou
[1] https://www.freeipa.org/page/V4/Performance_Improvements
On 5/7/20 4:38 AM, Mary Georgiou via FreeIPA-users wrote:
Hello,
In our set-up, we have a DB with all the users and groups, which we use as ground truth for provisioning the forementioned objects in FreeIPA (2 master servers + replicas). We are continuously synchronizing entries (~60000 users and 60000 groups, where groups might have 0 to 20000 members) from the DB to FreeIPA. In each cycle of synch, we are figuring out the differences and add, delete, or change existing entries.
The first sync (through which we had to import all 120000 objects) clogged the server totally, and after tweaking the 389DS we ended up disabling the memberOf plugin where it finally worked (we followed the FreeIPA documentation[1]). One of the advice to follow is to do the sync and then run the fixup task in the server where the provisioning happened. The fixup still clogs the server after some point and stops. The errors we get in the log are the following:
[06/May/2020:18:16:59.862308719 +0200] - INFO - memberof-plugin - memberof_fixup_task_thread - Memberof task starts (filter: "(|(objectclass=inetuser)(objectclass=inetadmin)(objectclass=nsmemberof))") ... [06/May/2020:20:07:49.545606214 +0200] - ERR - libdb - BDB2055 Lock table is out of available lock entries [06/May/2020:20:07:49.547921580 +0200] - ERR - idl_new_delete_key - idl_new.c BAD 22, err=12 Cannot allocate memory [06/May/2020:20:07:49.548930035 +0200] - ERR - addordel_values_sv - database index operation failed BAD 1130, err=12 Cannot allocate memory [06/May/2020:20:07:49.549779631 +0200] - ERR - addordel_values_sv - database index operation failed BAD 1140, err=12 Cannot allocate memory [06/May/2020:20:07:49.550612745 +0200] - ERR - index_addordel_values_ext_sv - database index operation failed BAD 1230, err=12 Cannot allocate memory [06/May/2020:20:07:49.551444741 +0200] - ERR - index_add_mods - database index operation failed BAD 1041, err=12 Cannot allocate memory [06/May/2020:20:07:49.552457769 +0200] - ERR - index_add_mods - database index operation failed BAD 1040, err=12 Cannot allocate memory [06/May/2020:20:07:49.553305019 +0200] - ERR - ldbm_back_modify - index_add_mods failed, err=12 Cannot allocate memory
We just saw this in a different case.
First do a ldapsearch as follows:
# ldapsearch -D "cn=directory manager" -W -b "YOUR_DB SUFFIX" (|(objectclass=inetuser)(objectclass=inetadmin)(objectclass=nsmemberof))
How many entries are turned, save this value add 10000 to it and use it below:
For now try setting this attribute under cn=config
ldapmodify -D "cn=directory manager" -W dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-idlistscanlimit nsslapd-idlistscanlimit: YOUR_NUMBER_FROM_THE SEARCH
Then try the fixup task again.
HTH, Mark
We increased the number of DB locks and set the `nsslapd-cache-autosize` to 50% (server has currently 13G of memory). The only thing we saw was that one thread was using 100% of one of the CPUs. Any advice on how to deal with this? We would really need to have memberOf attribute. Thank you in advance! Best Regards Mary Georgiou [1] https://www.freeipa.org/page/V4/Performance_Improvements _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Hi Mark, Thank you very much for your quick and detailed answer. I'm a bit confused with some details I do not understand. I ran this query and I got back only the user entires. Arent's also the groups part of this (since they can be members of other groups etc)? In case this is supposed to be the expected behavior, after checking the attribute 'nsslapd-idlistscanlimit' I saw that it is already set to 100.000 (and the entries that we got back from the query were 63000). I noticed that none of the entries had the 'nsmemberof' object class explicitly. How is this class used by default in freeIPA concerning users and groups? Again I'm super curious about the groups that do have the memberOf attribute but were not returned in the aforementioned query.
At this point, I would really like to mention, how much we appreciate the help and the very quick answers we've got so far from this mail-list. Thank you! Best Regards Mary Georgiou
On to, 07 touko 2020, Mary Georgiou via FreeIPA-users wrote:
Hi Mark, Thank you very much for your quick and detailed answer. I'm a bit confused with some details I do not understand. I ran this query and I got back only the user entires. Arent's also the groups part of this (since they can be members of other groups etc)? In case this is supposed to be the expected behavior, after checking the attribute 'nsslapd-idlistscanlimit' I saw that it is already set to 100.000 (and the entries that we got back from the query were 63000). I noticed that none of the entries had the 'nsmemberof' object class explicitly. How is this class used by default in freeIPA concerning users and groups? Again I'm super curious about the groups that do have the memberOf attribute but were not returned in the aforementioned query.
You would be interested in memberOf attribute, not the object class. IPA has a number of obejct classes that allow memberOf attribute:
(2.16.840.1.113730.3.8.4.1 NAME 'ipaHost' AUXILIARY MUST ( fqdn ) MAY ( userPassword $ ipaClientVersion $ enrolledBy $ memberOf $ userClass $ ipaAssignedIDView ) X-ORIGIN 'IPA v2' ) (2.16.840.1.113730.3.8.4.2 NAME 'ipaService' DESC 'IPA service objectclass' AUXILIARY MAY ( memberOf $ managedBy $ ipaKrbAuthzData) X-ORIGIN 'IPA v2' ) (2.16.840.1.113730.3.8.4.3 NAME 'nestedGroup' DESC 'Group that supports nesting' SUP groupOfNames STRUCTURAL MAY memberOf X-ORIGIN 'IPA v2' ) (2.16.840.1.113730.3.8.4.8 NAME 'ipaNISNetgroup' DESC 'IPA version of NIS netgroup' SUP ipaAssociation STRUCTURAL MAY ( externalHost $ nisDomainName $ member $ memberOf ) X-ORIGIN 'IPA v2' ) (2.16.840.1.113730.3.8.4.10 NAME 'ipaHBACService' STRUCTURAL MUST ( cn ) MAY ( description $ memberOf ) X-ORIGIN 'IPA v2' ) (2.16.840.1.113730.3.8.12.1 NAME 'ipaExternalGroup' SUP top STRUCTURAL MUST ( cn ) MAY ( ipaExternalMember $ memberOf $ description $ owner) X-ORIGIN 'IPA v3' ) (2.16.840.1.113730.3.8.8.2 NAME 'ipaSudoCmd' DESC 'IPA object class for SUDO command' STRUCTURAL MUST ( ipaUniqueID $ sudoCmd ) MAY ( memberOf $ description ) X-ORIGIN 'IPA v2' )
They all can be used for group membership. nestedGroup is the one you are interested in -- any nestedGroup object can be a member of any other group.
For example, on my test deployment I have a regular group 'idmgroup', it has the following structure: # ipa group-show idmgroup --raw --all dn: cn=idmgroup,cn=groups,cn=accounts,dc=ipa,dc=test cn: idmgroup gidnumber: 1908200005 member: uid=idmuser,cn=users,cn=accounts,dc=ipa,dc=test ipaNTSecurityIdentifier: S-1-5-21-3787809381-104084847-3373960542-1005 ipaUniqueID: 141f0f32-8ea7-11ea-8724-fa163efaafc4 objectClass: top objectClass: groupofnames objectClass: nestedgroup objectClass: ipausergroup objectClass: ipaobject objectClass: posixgroup objectClass: ipantgroupattrs
as you can see, it is nestedGroup, as any IPA group that contains users.
freeipa-users@lists.fedorahosted.org
-
Alexander Bokovoy -
Mark Reynolds -
Mary Georgiou