Hi,
Is it just ordinary behavior with 389 DS that search results may take a very loong time just after starting the server when there are no entries in the cache yet? And when the cache is fully saturated (enough cache configured for all the entries) results become dramatically down - for instance from 4 minutes to 4 seconds.
If this this is so, is there anything that could be done to fill in the cache automatically after startup?
We have some 60 000 entries, RHEL 7.1, 389-Directory/1.3.3.1 B2015.267.1737 on VMWare. We have quite a heavy use of roles, and this seems to be a significant issue especially with them - or at least with them. We have used the Sun DSEE previously and are quite new to 389 DS. The technology seems very similar although.
Thanks a lot in advance!
Regards, Petteri Jekunen, Tampere University of Applied Sciences
On Wed, 2015-11-25 at 09:35 +0200, Petteri Jekunen wrote:
Hi,
Is it just ordinary behavior with 389 DS that search results may take a very loong time just after starting the server when there are no entries in the cache yet? And when the cache is fully saturated (enough cache configured for all the entries) results become dramatically down - for instance from 4 minutes to 4 seconds.
If this this is so, is there anything that could be done to fill in the cache automatically after startup?
We have some 60 000 entries, RHEL 7.1, 389-Directory/1.3.3.1 B2015.267.1737 on VMWare. We have quite a heavy use of roles, and this seems to be a significant issue especially with them - or at least with them. We have used the Sun DSEE previously and are quite new to 389 DS. The technology seems very similar although.
Thanks a lot in advance!
Can you post:
rpm -qa | grep 389-ds-base for the rpm version
As well, can we see:
ldapsearch -b cn=monitor,cn=ldbm database,cn=plugins,cn=config -s base ldapsearch -b cn=userRoot,cn=ldbm database,cn=plugins,cn=config -s base ldapsearch -b cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config -s base
This will show what the cache hit rates and sizing are.
You may find that the issue is a lack of key indexes, and that once the cache is primed that is masking the issue. Perhaps look in the access log for notes=U ?
Hi,
Thank you for your answer! Here the result of the rpm command: 389-ds-base-1.3.3.1-23.el7_1.x86_64 389-ds-base-1.3.4.0-19.el7.x86_64 389-ds-base-libs-1.3.4.0-19.el7.x86_64 389-ds-base-devel-1.3.4.0-19.el7.x86_64
The ldapsearch results are enclosed. You have to take into account that this is not yet in production - will be very soon and that's why I'm a little bit concerned about the performance :)
I was just thinking how well indexing works at all for searches that contain nsrole-attribute conditions, since nsrole is a dynamic computed attribute - we are using roles a lot and searches may include logical combinations of many roles.
Regards, Petteri
On 25.11.2015 23:20, William Brown wrote:
On Wed, 2015-11-25 at 09:35 +0200, Petteri Jekunen wrote:
Hi,
Is it just ordinary behavior with 389 DS that search results may take a very loong time just after starting the server when there are no entries in the cache yet? And when the cache is fully saturated (enough cache configured for all the entries) results become dramatically down - for instance from 4 minutes to 4 seconds.
If this this is so, is there anything that could be done to fill in the cache automatically after startup?
We have some 60 000 entries, RHEL 7.1, 389-Directory/1.3.3.1 B2015.267.1737 on VMWare. We have quite a heavy use of roles, and this seems to be a significant issue especially with them - or at least with them. We have used the Sun DSEE previously and are quite new to 389 DS. The technology seems very similar although.
Thanks a lot in advance!
Can you post:
rpm -qa | grep 389-ds-base for the rpm version
As well, can we see:
ldapsearch -b cn=monitor,cn=ldbm database,cn=plugins,cn=config -s base ldapsearch -b cn=userRoot,cn=ldbm database,cn=plugins,cn=config -s base ldapsearch -b cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config -s base
This will show what the cache hit rates and sizing are.
You may find that the issue is a lack of key indexes, and that once the cache is primed that is masking the issue. Perhaps look in the access log for notes=U ?
-- 389 users mailing list 389-users@%(host_name)s http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
On Thu, 2015-11-26 at 22:30 +0200, Petteri Jekunen wrote:
Hi,
Thank you for your answer! Here the result of the rpm command: 389-ds-base-1.3.3.1-23.el7_1.x86_64 389-ds-base-1.3.4.0-19.el7.x86_64 389-ds-base-libs-1.3.4.0-19.el7.x86_64 389-ds-base-devel-1.3.4.0-19.el7.x86_64
The ldapsearch results are enclosed. You have to take into account that this is not yet in production - will be very soon and that's why I'm a little bit concerned about the performance :)
That's a very reasonable concern to have, and good to understand.
Remember, slapd tends to run for months at a time, so this is an extraordinary circumstance. If you have a load balancer setup, than you can always takes Rich's advice on the "cold" node before you re-insert to the load balancer pool.
I was just thinking how well indexing works at all for searches that contain nsrole-attribute conditions, since nsrole is a dynamic computed attribute - we are using roles a lot and searches may include logical combinations of many roles.
Provided that the attributes you use the condition are indexed, it should be okay.
However, it depends what you are doing.
I found previously that things like memberOf were better than roles, because you can filter very effectively on memeberOf attributes on an object, rather than have to use nsRole. But it really depends on your object and schema design that you are choosing to use.
Did you check for un-indexed searchs in the logs as well? Can you post the cache sizes that I asked for as well? See below.
ldapsearch -b cn=monitor,cn=ldbm database,cn=plugins,cn=config -s base ldapsearch -b cn=userRoot,cn=ldbm database,cn=plugins,cn=config -s base ldapsearch -b cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config -s base
This will show what the cache hit rates and sizing are.
You may find that the issue is a lack of key indexes, and that once the cache is primed that is masking the issue. Perhaps look in the access log for notes=U ?
On 1.12.2015 3:12, William Brown wrote:
On Thu, 2015-11-26 at 22:30 +0200, Petteri Jekunen wrote:
Hi,
Thank you for your answer! Here the result of the rpm command: 389-ds-base-1.3.3.1-23.el7_1.x86_64 389-ds-base-1.3.4.0-19.el7.x86_64 389-ds-base-libs-1.3.4.0-19.el7.x86_64 389-ds-base-devel-1.3.4.0-19.el7.x86_64
The ldapsearch results are enclosed. You have to take into account that this is not yet in production - will be very soon and that's why I'm a little bit concerned about the performance :)
That's a very reasonable concern to have, and good to understand.
Remember, slapd tends to run for months at a time, so this is an extraordinary circumstance. If you have a load balancer setup, than you can always takes Rich's advice on the "cold" node before you re-insert to the load balancer pool.
I was just thinking how well indexing works at all for searches that contain nsrole-attribute conditions, since nsrole is a dynamic computed attribute - we are using roles a lot and searches may include logical combinations of many roles.
Provided that the attributes you use the condition are indexed, it should be okay.
However, it depends what you are doing.
I found previously that things like memberOf were better than roles, because you can filter very effectively on memeberOf attributes on an object, rather than have to use nsRole. But it really depends on your object and schema design that you are choosing to use.
Did you check for un-indexed searchs in the logs as well? Can you post the cache sizes that I asked for as well? See below.
ldapsearch -b cn=monitor,cn=ldbm database,cn=plugins,cn=config -s base ldapsearch -b cn=userRoot,cn=ldbm database,cn=plugins,cn=config -s base ldapsearch -b cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config -s base
This will show what the cache hit rates and sizing are.
You may find that the issue is a lack of key indexes, and that once the cache is primed that is masking the issue. Perhaps look in the access log for notes=U ?
-- 389 users mailing list 389-users@%(host_name)s http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
Hi,
The ldapsearch numbers are down below. Not all the numbers to all the indexed attributes are there ... but all the "cachemiss" -numbers for them were 0's.
In the log there are not many "notes=U" lines but some "notes=A" lines: # grep "nentries=" access | grep -v "notes=" | wc -l 65674 # grep "nentries=" access | grep "notes=U" | wc -l 173 # grep "nentries=" access | grep "notes=A" | wc -l 2189
Still about the nsrole -attribute. We have index for nsroledn attribute but not for nsrole. I have thought that indexing nsrole-attribute is not reasonable and wouldn't improve search since it is a calculated attribute. But is this so - should the nsrole-attribute be indexed as well?
The discussion if to use either groups and memberOf attributes or roles is valuable. However we have made the decision to use roles long ago and it would be a big effort to change the approach...
Regards,
Petteri
------------------
cn=monitor,cn=ldbm database,cn=plugins,cn=config ----------------------------------------------
dn: cn=monitor,cn=ldbm database,cn=plugins,cn=config objectClass: top objectClass: extensibleObject cn: monitor database: ldbm database dbcachehits: 5903060 dbcachetries: 5903060 dbcachehitratio: 100 dbcachepagein: 0 dbcachepageout: 1 dbcacheroevict: 0 dbcacherwevict: 0
cn=userRoot,cn=ldbm database,cn=plugins,cn=config ----------------------------------------------
dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config objectClass: top objectClass: extensibleObject objectClass: nsBackendInstance cn: userRoot nsslapd-suffix: o=cctpu nsslapd-cachesize: -1 nsslapd-cachememsize: 1073741824 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap90/db/userRoot nsslapd-dncachememsize: 4294966
cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config ----------------------------------------------
dn: cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config objectClass: top objectClass: extensibleObject cn: monitor database: ldbm database readonly: 0 entrycachehits: 89754414 entrycachetries: 89860109 entrycachehitratio: 99 currententrycachesize: 567748839 maxentrycachesize: 1073741824 currententrycachecount: 67334 maxentrycachecount: -1 dncachehits: 0 dncachetries: 67334 dncachehitratio: 0 currentdncachesize: 4294966 maxdncachesize: 4294966 currentdncachecount: 38566 maxdncachecount: -1 normalizeddncachetries: 572563254 normalizeddncachehits: 572495057 normalizeddncachemisses: 68197 normalizeddncachehitratio: 99 currentnormalizeddncachesize: 11568843 maxnormalizeddncachesize: 20971520 currentnormalizeddncachecount: 68194 dbfilename-0: userRoot/entryrdn.db dbfilecachehit-0: 1135364 dbfilecachemiss-0: 0 dbfilepagein-0: 0 dbfilepageout-0: 0
Hi,
The ldapsearch numbers are down below. Not all the numbers to all the indexed attributes are there ... but all the "cachemiss" -numbers for them were 0's.
In the log there are not many "notes=U" lines but some "notes=A" lines: # grep "nentries=" access | grep -v "notes=" | wc -l 65674 # grep "nentries=" access | grep "notes=U" | wc -l 173 # grep "nentries=" access | grep "notes=A" | wc -l 2189
I think these might be the most telling.
notes=U means there was an unindexed component in the search.
notes=A means that all components of the search were unindexed.
I think you should examine these queries closely, look at what is and isn't indexed. Make sure the attributes have an equality syntax, and if they do, add them as an index. This will likely help you a lot.
Still about the nsrole -attribute. We have index for nsroledn attribute but not for nsrole. I have thought that indexing nsrole-attribute is not reasonable and wouldn't improve search since it is a calculated attribute. But is this so - should the nsrole-attribute be indexed as well?
It depends on the structure of your roles. You'll probably pick up the issues in the above exploration.
The discussion if to use either groups and memberOf attributes or roles is valuable. However we have made the decision to use roles long ago and it would be a big effort to change the approach...
I'll leave that decision to you.
On 12/03/2015 05:02 PM, William Brown wrote:
Hi,
The ldapsearch numbers are down below. Not all the numbers to all the indexed attributes are there ... but all the "cachemiss" -numbers for them were 0's.
In the log there are not many "notes=U" lines but some "notes=A" lines: # grep "nentries=" access | grep -v "notes=" | wc -l 65674 # grep "nentries=" access | grep "notes=U" | wc -l 173 # grep "nentries=" access | grep "notes=A" | wc -l 2189
I think these might be the most telling.
notes=U means there was an unindexed component in the search.
notes=A means that all components of the search were unindexed.
I think you should examine these queries closely, look at what is and isn't indexed. Make sure the attributes have an equality syntax, and if they do, add them as an index. This will likely help you a lot.
"logvonv.pl -V <access logs>" will give you detailed info on those unindexed searches, but note that it will take a long time to run, and it will generate a lot of output. I recommend redirecting the output to a file when using it.
Still about the nsrole -attribute. We have index for nsroledn attribute but not for nsrole. I have thought that indexing nsrole-attribute is not reasonable and wouldn't improve search since it is a calculated attribute. But is this so - should the nsrole-attribute be indexed as well?
It depends on the structure of your roles. You'll probably pick up the issues in the above exploration.
The discussion if to use either groups and memberOf attributes or roles is valuable. However we have made the decision to use roles long ago and it would be a big effort to change the approach...
I'll leave that decision to you.
-- 389 users mailing list 389-users@%(host_name)s http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
On 11/25/2015 12:35 AM, Petteri Jekunen wrote:
Hi,
Is it just ordinary behavior with 389 DS that search results may take a very loong time just after starting the server when there are no entries in the cache yet?
Yes.
And when the cache is fully saturated (enough cache configured for all the entries) results become dramatically down - for instance from 4 minutes to 4 seconds.
Yes.
If this this is so, is there anything that could be done to fill in the cache automatically after startup?
Yes. Do
ldapsearch -D "cn=directory manager" -w "password" -b "dc=your,dc=base,dc=suffix" '(|(objectclass=*)(objectclass=ldapsubentry)(objectclass=nstombstone))'
/dev/null
just after the server starts. This will probably take several minutes or more, but after that, every single entry in the database will be cached (assuming you have enough RAM - use dbmon.sh to check).
We have some 60 000 entries, RHEL 7.1, 389-Directory/1.3.3.1 B2015.267.1737 on VMWare. We have quite a heavy use of roles, and this seems to be a significant issue especially with them - or at least with them. We have used the Sun DSEE previously and are quite new to 389 DS. The technology seems very similar although.
Thanks a lot in advance!
Regards, Petteri Jekunen, Tampere University of Applied Sciences -- 389 users mailing list 389-users@%(host_name)s http://lists.fedoraproject.org/admin/lists/389-users@lists.fedoraproject.org
389-users@lists.fedoraproject.org
-
Mark Reynolds -
Petteri Jekunen -
Rich Megginson -
William Brown