Hi, yes, aci processing can become very expensive if you have a lot of acis are used, the code tries to do soem optimizations to cache evaluation, but a small change in acis could prevent the use of cached results. So if you do not have a full set of acis from the "better" times it will be difficult to tell what led to the changed performance. The aci code in RHDS and SunDS is very similar, but both have done over the time bug fixes and attempts to optimize, so for a given set of acis performance could be different. ACI logging slows down a lot, but it could help you to investigate the usage of your acis and which acis lead to the decision and which otehr acis are processed before, so it could help in redesigning your acis, what you probably need to do. Regards, Ludwig On 07/04/2013 12:06 AM, Russell Beall wrote: > I did a lot of work experimenting with 389 for use as a replacement to Sun SJES. Worked really well when I focused my efforts on the backend processing we do with Directory Manager, except for a few performance issues which are being addressed in bug reports. > > I thought sure I had done at least some load testing with service accounts. The service accounts must go through ACL processing, and we have a lot of ACLs. I'm not sure if I changed something, or if I just didn't quite test this feature enough, but now that I am doing more development work with service accounts, I am showing a huge processing hit taken if a service account is used as opposed to Directory Manager. This is on the order of a second and a half to respond to a simple base query, versus instantaneous. Our old SJES servers respond very snappily in comparison for this type of query. > > CPU usage for a single thread maxes out during the time spent waiting and I/O wait is zero, so I know that probably the bulk of time is being spent processing the ACLs. This is especially true if I turn on logging for ACL processing, then it takes a very long time, with one example taking about 9 minutes. > > It seems to be processing and reprocessing the ACLs many many times over. > > I think I must have changed something or done something wrong because I'm pretty sure I remember much quicker response times when using a service account in earlier testing. > > This is with 389-ds-base 1.2.10.14 on RedHat 6.2. > > This was an experimental version downloaded to check out a memory fragmentation option that was coded in, so maybe I just have a version that was mid ACL processing changes? > > Thanks for any help, > Russ. > > > -- > 389 users mailing list > 389-users(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users

On 07/09/2013 12:01 AM, Russell Beall wrote: > Hi Ludwig, > > Thanks for responding on this. > > After further experimentation and re-applying ACI files from earlier times, I find that the condition probably has been present the whole time and I just didn't notice because I was focusing on performance related to our Directory Manager-based scripts. > > We have specific ACI rules for each service account we issue. I found that by commenting out 83 of these rules, I was able to cut down the response time from ~2.5 seconds to ~0.3 seconds. Even further if I limited the returned attributes to a small set. yes, since access control is evaluated for each attribute returned reducing the requested attributes will help (and reducing the number of acis of course helped as well) > > I think a key problem with this is that our rules apply to most entries, but then it is only a very specific userdn which has the "allow" permission at the end of the rule. Is there any way to turn that around so that the userdn might be looked at first rather than processing the whole rule only to then find out that it applies to an irrelevant account? > > Is there any documentation on how to best optimize a complex set of ACIs? no, not that I am aware of, but maybe an understanding of how aci processing works can help you - first for an entry all potential acis are collected and since the target of an aci can extend to teh subtree where it is placed all acis from the entry in question up to the root suffix are copllected - while collecting the acis, they are placed in two buckets, one for acis with deny rules, one for acis with allow rules. - first the deny acis are processed, as soon as a matching aci is found aci processing stops - next the allow list is processed until a matching aci is found when an aci is evaluated, first check if the target rule covers then entry, then if the targetattrs include the attr evaluated, then if it contains the correct permission (read, search,...) and if the target matches then the bind rule part is evaluated, if it matches the bound dn (and other rules like dns or ip or ....) So to optimize aci evaluation, you need to try to minimize the above steps - if you have a directory tree with many nested subtrees and acis affecting only subtrees, place them as close as possible to the subtree, so no unrelated acis will be collected - always all deny acis will be processed, maybe minimizing denies can help -if you have service accounts which will have access to only parts of the directory tree, maybe using macro acis can help to reduce the number of acis > > You suggested the logging would show the processing order. I tried placing the most important rule at the top of the list, but it didn't actually change the processing (and I didn't really expect that anyway). What is it that you are referring to that might help me reorder/redesign the processing of the rules? if you enable aci logging you will get at startup a list of acis and their aci index, later when evaluating access, there is always something like evaluation xxx deny handles, xxx allow handles and referencing the index of the aci evaluated, you could then probably get some insight how many and which acis are processed for each request and in which order and what finally is the decidng aci. It will also log if a cached evaluation could be used. Evaluation tries to remember if the set of acis is the same for the attributes handled and then use the previous evaluation. Caching cannot be used eg if the acis contain rules depending on the entry, eg self rules or entry#attr rules. > > Appreciate any help you can offer. > > Regards, > Russ. > > On Jul 4, 2013, at 6:11 AM, Ludwig Krispenz <lkrispen(a)redhat.com&gt; > wrote: > >> Hi, >> >> yes, aci processing can become very expensive if you have a lot of acis are used, the code tries to do soem optimizations to cache evaluation, but a small change in acis could prevent the use of cached results. So if you do not have a full set of acis from the "better" times it will be difficult to tell what led to the changed performance. >> >> The aci code in RHDS and SunDS is very similar, but both have done over the time bug fixes and attempts to optimize, so for a given set of acis performance could be different. >> >> ACI logging slows down a lot, but it could help you to investigate the usage of your acis and which acis lead to the decision and which otehr acis are processed before, so it could help in redesigning your acis, what you probably need to do. >> >> Regards, >> Ludwig >> >> On 07/04/2013 12:06 AM, Russell Beall wrote: >>> I did a lot of work experimenting with 389 for use as a replacement to Sun SJES. Worked really well when I focused my efforts on the backend processing we do with Directory Manager, except for a few performance issues which are being addressed in bug reports. >>> >>> I thought sure I had done at least some load testing with service accounts. The service accounts must go through ACL processing, and we have a lot of ACLs. I'm not sure if I changed something, or if I just didn't quite test this feature enough, but now that I am doing more development work with service accounts, I am showing a huge processing hit taken if a service account is used as opposed to Directory Manager. This is on the order of a second and a half to respond to a simple base query, versus instantaneous. Our old SJES servers respond very snappily in comparison for this type of query. >>> >>> CPU usage for a single thread maxes out during the time spent waiting and I/O wait is zero, so I know that probably the bulk of time is being spent processing the ACLs. This is especially true if I turn on logging for ACL processing, then it takes a very long time, with one example taking about 9 minutes. >>> >>> It seems to be processing and reprocessing the ACLs many many times over. >>> >>> I think I must have changed something or done something wrong because I'm pretty sure I remember much quicker response times when using a service account in earlier testing. >>> >>> This is with 389-ds-base 1.2.10.14 on RedHat 6.2. >>> >>> This was an experimental version downloaded to check out a memory fragmentation option that was coded in, so maybe I just have a version that was mid ACL processing changes? >>> >>> Thanks for any help, >>> Russ. >>> >>> >>> -- >>> 389 users mailing list >>> 389-users(a)lists.fedoraproject.org >>> https://admin.fedoraproject.org/mailman/listinfo/389-users >> -- >> 389 users mailing list >> 389-users(a)lists.fedoraproject.org >> https://admin.fedoraproject.org/mailman/listinfo/389-users >> > > -- > 389 users mailing list > 389-users(a)lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/389-users -- 389 users mailing list 389-users(a)lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users