Correct, set them to 0 for autotuning to take effect

nsslapd-cache-autosize This settings controls if auto-sizing is enabled for the database and entry cache. Auto-sizing is enabled: For both the database and entry cache, if the nsslapd-cache-autosize parameter is set to a value greater than 0. For the database cache, if the nsslapd-cache-autosize and nsslapd-dbcachesize parameters are set to 0. For the entry cache, if the nsslapd-cache-autosize and nsslapd-cachememsizeparameters are set to 0.

I don't believe autotuning exists in 1.3.5, it was only added to 1.3.6 - sorry :-/

> Also, is there a way to check that auto-tuning is working normally? Is dbmon.sh the right way? The error log at startup will tell you what the server sets the caches to. ldapsearch on nsslapd-dbcachesize will also return the adjusted size if I am correct, but I haven't tried it. But like I said before... this feature is not present in 389-ds-base-1.3.5

Sorry to resurrect this thread, but I’ve upgraded to 1.3.6 and so my question is back on the table :) My cn=userRoot,cn=ldbm database,cn=plugins,cn=config is currently: ... nsslapd-cachesize: -1 nsslapd-cachememsize: 1543503872 nsslapd-readonly: off nsslapd-require-index: off nsslapd-dncachememsize: 500000000 … But cn=config,cn=ldbm database,cn=plugins,cn=config has these settings: ... nsslapd-cache-autosize: 10 nsslapd-cache-autosize-split: 40 … If you remember, according to the docs, setting nsslapd-cache- autosize to a value > 0, enables autosizing for all backends regardless of their cache size settings. Is that accurate or must I set them to 0s after all? It looks like autosizing is enforced because I got an error when trying to modify the changelog backend: modifying entry "cn=changelog,cn=ldbm database,cn=plugins,cn=config" ldap_modify: Server is unwilling to perform (53) additional info: Error: "nsslapd-cachememsize" can not be updated while "nsslapd-cache-autosize" is set in "cn=config,cn=ldbm database,cn=plugins,cn=config".

Thanks! > On Mar 1, 2018, at 2:55 PM, Sergei Gerasenko <gerases(a)gmail.com&gt; > wrote: > > > I don't believe autotuning exists in 1.3.5, it was only added to > > 1.3.6 - sorry :-/ > > Ah, that makes my life a bit simpler for now :) > > > > Also, is there a way to check that auto-tuning is working > > > normally? Is dbmon.sh the right way? > > The error log at startup will tell you what the server sets the > > caches to. ldapsearch on nsslapd-dbcachesize will also return > > the adjusted size if I am correct, but I haven't tried it. But > > like I said before... this feature is not present in 389-ds-base- > > 1.3.5 > > Got it. Thanks, Mark!! _______________________________________________ 389-users mailing list -- 389-users(a)lists.fedoraproject.org To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.o rg

Hi William, With autosizing on I configured the changelog backend: dn: cn=changelog,cn=ldbm database,cn=plugins,cn=config cn: changelog objectClass: top objectClass: extensibleObject objectClass: nsBackendInstance nsslapd-suffix: cn=changelog nsslapd-cachesize: -1 nsslapd-cachememsize: 8858370048 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-CNVR-NET/db/changelog nsslapd-dncachememsize: 3000000000 Here’s the ldif: dn: cn=changelog,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-dncachememsize nsslapd-dncachememsize: 3000000000 - replace: nsslapd-cachememsize nsslapd-cachememsize: 3000000000 The server refused to change nsslapd-cachememsize because of autosizing but nsslapd-dncachememsize did get changed. So it seems that I can still control nsslapd-dncachememsize? When I restart 389- ds, the 3G persists as well. Does that make sense?

Thank you for a quick response!

Sergei > On Mar 13, 2018, at 7:40 PM, William Brown <william(a)blackhats.net.a > u> wrote: > > If autosize > 0, we write a new entrychace/cachememsize every start > up. > > So you only need to set autosize to between 1 and 99 for it to > work. > > There is some other logic in there to account for other scenarioes > = > for example, if you set autosize to 0 AND you set the entry > cachesize > to 0, we'll autosize it at start up anyway. BUT if you have > autosize = > 0, and entry cachesize > 0, we won't touch it. _______________________________________________ 389-users mailing list -- 389-users(a)lists.fedoraproject.org To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.o rg

Thank you for the detailed response, William. That’s great info. You mentioned FreeIPA in passing and that’s actually what I use 389-ds for. You mentioned dogtag eating memory. You mean it has a memory leak or some other memory mismanagement issue?

I have 125G of RAM on my systems. So, I can allocate quite a bit of RAM to the caches. My plan is to set nsslapd-cache-autosize to 20% and set the dncache size to 5G for all three backends. Does that sound reasonable to you? These are dedicated FreeIPA machines, so it’s the main consumer of the memory.

Also, can I set the dncachesize once at the cn=config,cn=ldbm database,cn=plugins,cn=config level and have all the other backends inherit that? Would I have to remove that parameter from the other backends for it to become inherited?

Thanks again, William! Sergei > On Mar 18, 2018, at 8:06 PM, William Brown <william(a)blackhats.net.a > u> wrote: > > On Tue, 2018-03-13 at 21:10 -0500, Sergei Gerasenko wrote: > > Hi William, > > > > With autosizing on I configured the changelog backend: > > > > dn: cn=changelog,cn=ldbm database,cn=plugins,cn=config > > cn: changelog > > objectClass: top > > objectClass: extensibleObject > > objectClass: nsBackendInstance > > nsslapd-suffix: cn=changelog > > nsslapd-cachesize: -1 > > nsslapd-cachememsize: 8858370048 > > nsslapd-readonly: off > > nsslapd-require-index: off > > nsslapd-directory: /var/lib/dirsrv/slapd-CNVR-NET/db/changelog > > nsslapd-dncachememsize: 3000000000 > > > > Here’s the ldif: > > > > dn: cn=changelog,cn=ldbm database,cn=plugins,cn=config > > changetype: modify > > replace: nsslapd-dncachememsize > > nsslapd-dncachememsize: 3000000000 > > - > > replace: nsslapd-cachememsize > > nsslapd-cachememsize: 3000000000 > > > > The server refused to change nsslapd-cachememsize because of > > autosizing but nsslapd-dncachememsize did get changed. So it > > seems > > that I can still control nsslapd-dncachememsize? When I restart > > 389- > > ds, the 3G persists as well. > > > > Does that make sense? > > Yes it does. > > > > > Thank you for a quick response! > > At the current point in time, dncachememsize is NOT controlled by > autosizing. There is some ideas in my head about how to improve > this, > but the issue is legacy. We have one control: autosize percentage. > But > this doesn't really represent a complete picture and story about > how we > size our backends and data. > > We have to continue to support the current variables, so I can't > easily > change this from it's current form. > > *IF* I was given unlimited power (I never will be ;) ) I would > probably > make the interface something like: > > > > backend-foo: > autosize-max-memory: <value in bytes/mb/gb> > entrycache-slice: A% > dncache-slice: B% > dbcache-slice: C% > ndncache-slice: D% > > So how would this work? > > You have autosize max that is the "total ram" we want to use. Lets > say > 2GB. Each backend would be given it's own "limit". We'd try to > detect > some sane values on your system of course, but we can only do so > much > :) So say ... each backend gets 5-10% of system ram limit. (We have > to > be super conservative due to dogtag in freeipa which eats ram). > > Then we slice that up. So entrycache-slice + dncache-slice + > dbcache- > slice + ndncache-slice = 100%. So by default we might do something > like: > > entrycache-slice: 75 > dbcache-slice: 10 > dncache-slice: 10 > ndncache-slice: 5 > > So this on a 1GB backend translates to: > > 750MB of entrycache > 100MB of dncache > 100MB of dbcache > 50MB of ndncache > > of course some tests would be needed to find the right slice values > by > default. > > > Then it's as simple as "if you want more from your backend" just up > the > one memlimit value and "everything" increases in a correct > proportional > rate. Rather than thinking about how to hand tune everything, we > make > informed design decisions, you just say "here's mem max". You have > control to say each backend gets a different value (say FreeIPA you > may > want 4GB to userRoot, 1GB to CA system backend). you still have > lots of > flexibility as an admin, but you don't have to worry about so many > moving interacting parts. > > > Hope that helps, > > > > > Sergei > > > > > > > On Mar 13, 2018, at 7:40 PM, William Brown <william(a)blackhats.n > > > et.a > > > u> wrote: > > > > > > If autosize > 0, we write a new entrychace/cachememsize every > > > start > > > up. > > > > > > So you only need to set autosize to between 1 and 99 for it to > > > work. > > > > > > There is some other logic in there to account for other > > > scenarioes > > > = > > > for example, if you set autosize to 0 AND you set the entry > > > cachesize > > > to 0, we'll autosize it at start up anyway. BUT if you have > > > autosize = > > > 0, and entry cachesize > 0, we won't touch it. > > > > _______________________________________________ > > 389-users mailing list -- 389-users(a)lists.fedoraproject.org > > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproje > > ct.o > > rg > > -- > Thanks, > > William Brown > _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject > .org _______________________________________________ 389-users mailing list -- 389-users(a)lists.fedoraproject.org To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.o rg

> Dogtag is Java/Tomcat. It's well known for consuming large volumes > of > ram! Ah, I thought it was something besides that :)

> Sure, sounds reasonable to me - I'd want to see your database sizes > to > make a complete assesment, but it seems pretty reasonable to me. I will get that for you. I’ve used dbmon.sh on a live system and things looked pretty well (>=92%). You would like to see the number of entries in the db, right?

> Additionally, check that cn=config,cn=ldbm ... is actually > d'B'cachesize not d’N'cachesize. Now I’m confused a bit. I thought the dncachesize was what was supposed to be set manually while everything else was autosized. Sorry for being slow on this.

_______________________________________________ 389-users mailing list -- 389-users(a)lists.fedoraproject.org To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.o rg

> dbmon.sh Tue Mar 20 02:16:04 UTC 2018 dbcachefree 432873472 free% 80.629 roevicts 0 hit% 99 pagein 1267 pageout 23191 changelog:ent 61 6509293897 100.0 4359.2 changelog:dn 2926 2999793332 100.0 70.6 userroot:ent 11777 6408341275 98.4 8594.6 userroot:dn 11777 2998722450 100.0 108.5 > ls -lh /var/lib/dirsrv/slapd-XXXX-XXX/db/userRoot total 121M