I've forgotten Linked Attributes plugin, you could also disable it.

Don't you have some exotic type of index activated for uniqueMember (like substring)? The default value is only the equality index in dse.ldif

dn: cn=uniquemember,cn=default indexes,cn=config,cn=ldbm database,cn=plugins,c
 n=config
objectClass: top
objectClass: nsIndex
cn: uniquemember
nsSystemIndex: false
nsIndexType: eq


In any case, batch write loads are quite particular. You could try to play with nsslapd-db-checkpoint-interval and nsslapd-db-durable-transaction config attributes while you run your batch uniqueMember modifications. You could also try disabling completely or limiting logging intensity (http://directory.fedoraproject.org/wiki/Named_Pipe_Log_Script) .

In my VM tests (RHEL5, 1 vCPU Xeon E5640  @ 2.67GHz, 1Gb mem, 389DS v1.2.10.6) with our production environment (~20k users, groups of ~6000 members created as in your case with perl scripts using ldapmodify running on the same VM)  a group of 6000 uniqueMembers is created in 3 minutes 10 sec (190s) from scratch. Using "dstat" i see that the main problem is disk writes (transaction logs of db4):
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw
 91   0   0   9   0   0|   0    40M| 120B  978B|   0     0 | 316   325
 90   3   0   7   0   0|   0    40M|  60B  310B|   0     0 | 184   294
 92   2   0   5   0   1|   0    40M|  60B  310B|   0     0 | 160   297
 93   1   0   5   0   1|   0    40M|  60B  310B|   0     0 | 159   312
 93   3   1   1   0   2|   0    24M|  60B  310B|   0     0 | 206   372
 76   4   2  18   0   0|   0    40M|  60B  310B|   0     0 | 165   265
 94   0   0   6   0   0|   0    40M|  60B  310B|   0     0 | 221   275
 90   1   0   7   1   1|   0    41M|  60B  310B|   0     0 | 479   313
 86   2   0  11   1   0|   0    40M|  60B  310B|   0     0 | 403   306
 93   0   0   6   0   1|   0    20M| 120B  364B|   0     0 | 489   298
 90   1   0   9   0   0|   0    40M|  60B  310B|   0     0 | 389   296
 88   1   0  11   0   0|   0    40M|  60B  310B|   0     0 | 358   319
 76   0   0  23   1   0|   0    41M|  60B  310B|   0     0 | 403   303

@+



Le 19 avril 2012 18:50, Russell Beall <beall@usc.edu> a écrit :
Thanks for the tips.  I scanned the dse.ldif for those plugins and I found definitions for them all, but they all have nsslapd-pluginEnabled: off.

There is something special about the uniquemember attribute that requires additional processing different from other attributes...  Ldapmodify of other attributes runs pretty quick.

Regards,
Russ.

On Apr 19, 2012, at 2:20 AM, Andrey Ivanov wrote:

Hi Russel,


Le 18 avril 2012 23:06, Russell Beall <beall@usc.edu> a écrit :
On Apr 18, 2012, at 11:15 AM, Rich Megginson wrote:
Yeah, this particular operation has not been optimized.  I believe SunDS added explicit optimizations for this particular case.


It is becoming painfully apparent as I write more detailed tests.  389 takes time to add or delete uniquemember values proportionate to the number of values being operated on and is using about twice as much time to delete as it does to add.  Sun DS appears to have perhaps an almost O(1) algorithm in play on both adding and deleting values.

Is there perhaps some kind of referential integrity setting that is being used and forcing some kind of lookup of each value, one that we could perhaps turn off?  We wouldn't need such a check because our metadirectory process handles the integrity/consistency checking already.

There is memberOf plugin that maintains the memberOf attribute for groups. I don't know whether  it is activated by default or not. You could try to disable it. There is also referential integrity plugin, attribute uniqueness plugin, maybe USN plugin or custom indexes that could consume a lot of CPU. Make sure you've disabled them if you don't need them.

@+
--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users


--
389 users mailing list
389-users@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users