Over the past few weeks, I have been testing OpenLDAP as a backend for Samba4.
I've been working with the OpenLDAP team on my requirements, and there has been some really good outcomes - the memberOf module has been improved, as has the refint module.
However, I seem to have hit a brick wall, in the form of (internal) transaction support. I need an LDAP backend to support internal transactions - that is, when for example a 'member' modification is made, all the memberOf attributes must be updated before the call returns. Similarly, if a user or group moves, all the member/memberOf attributes that link the user to their groups must also move, before the modrdn returns.
The Samba4 test ldap.js tests this behaviour extensively, because I want to be sure it works.
As understand the discussion I've had with the OpenLDAP team, OpenLDAP does not support this, and will not support it for perhaps some time.
Similarly, from discussions with the Fedora DS team at the CIFS developer days, I understand that it is similarly very unlikely that Fedora DS will support internal transactions. (It also does not support subtree renames, which we also need).
The fact that LDAP does not expose a transaction API was always going to be a difficult part of having Samba4 use an LDAP backend, but I always assumed that if we pushed the really hard bit - updating linked attributes - into the LDAP server that we could at least always have a consistent DB. (It turns out this is one of the primary uses of transactions anyway.)
But without that consistency, and without knowing as a caller if all the updates succeed, I'm worried about how we can safely move forward.
This is especially disappointing because I was hoping that these free, replicating LDAP servers might solve the backed replication problem for me, without needing to use AD replication.
Does anybody have any ideas or suggestions on how I could get around this?
Should we drop the LDAP backend as a nice idea, but not going to work, and focus on DRS or some other form of replication?
Can someone imagine a sane way to reconstruct the DN links, including subtree renames, without the help of the LDAP backend? Could we ban subtree renames (as Fedora DS does), and try to handle this ourselves (with pre/post checks and a good deal of prayer)?
Thanks,
Andrew Bartlett
Andrew Bartlett wrote:
Over the past few weeks, I have been testing OpenLDAP as a backend for Samba4.
I've been working with the OpenLDAP team on my requirements, and there has been some really good outcomes - the memberOf module has been improved, as has the refint module.
However, I seem to have hit a brick wall, in the form of (internal) transaction support. I need an LDAP backend to support internal transactions - that is, when for example a 'member' modification is made, all the memberOf attributes must be updated before the call returns. Similarly, if a user or group moves, all the member/memberOf attributes that link the user to their groups must also move, before the modrdn returns.
The Samba4 test ldap.js tests this behaviour extensively, because I want to be sure it works.
As understand the discussion I've had with the OpenLDAP team, OpenLDAP does not support this, and will not support it for perhaps some time.
I may have overstated the problem in the previous discussion of our refint module. In fact, RE24 was already changed to work around any potential deadlock issues a long time ago. But to give some context: the refint module was originally written to operate synchronously (back in 2004). Some time later it was changed (2006) to asynchronous mode because users didn't want their clients to be stuck waiting for all the cascaded updates to complete. Most clients don't know or care that a particular change has side-effects. We could introduce a config keyword to select synch vs asynch behavior here, but I have a feeling that will still leave some group of users unhappy no matter how you set it.
Similarly, from discussions with the Fedora DS team at the CIFS developer days, I understand that it is similarly very unlikely that Fedora DS will support internal transactions. (It also does not support subtree renames, which we also need).
The fact that LDAP does not expose a transaction API
You mean draft-zeilenga-ldap-txn ?
was always going to be a difficult part of having Samba4 use an LDAP backend, but I always assumed that if we pushed the really hard bit - updating linked attributes - into the LDAP server that we could at least always have a consistent DB. (It turns out this is one of the primary uses of transactions anyway.)
But without that consistency, and without knowing as a caller if all the updates succeed, I'm worried about how we can safely move forward.
This is especially disappointing because I was hoping that these free, replicating LDAP servers might solve the backed replication problem for me, without needing to use AD replication.
Does anybody have any ideas or suggestions on how I could get around this?
There are other ways to guarantee consistency. The simplest approach is to just not store one end of the linked attributes, and always generate them dynamically when they're referenced.
In the old Symas Connexitor EMS product we used (the equivalent of) a UUIDAndOptionalName syntax for all references. In that case the DN was essentially just window-dressing; we always used the ID to actually reference entries and we updated the DNs whenever we found that they didn't match. As such, referential integrity was pretty simple - you never had anything pointing to the wrong entry; the worst that would happen is that you occasionally had dangling references to deleted entries stored in the DB but no one ever saw them because they were cleaned out whenever the entry containing the reference was read.
Should we drop the LDAP backend as a nice idea, but not going to work, and focus on DRS or some other form of replication?
Can someone imagine a sane way to reconstruct the DN links, including subtree renames, without the help of the LDAP backend? Could we ban subtree renames (as Fedora DS does), and try to handle this ourselves (with pre/post checks and a good deal of prayer)?
Banning subtree renames seems like a non-starter, and it only eliminates one case; the overall problem still remains.
On Wed, 2008-03-19 at 04:26 -0700, Howard Chu wrote:
Andrew Bartlett wrote:
Over the past few weeks, I have been testing OpenLDAP as a backend for Samba4.
I've been working with the OpenLDAP team on my requirements, and there has been some really good outcomes - the memberOf module has been improved, as has the refint module.
However, I seem to have hit a brick wall, in the form of (internal) transaction support. I need an LDAP backend to support internal transactions - that is, when for example a 'member' modification is made, all the memberOf attributes must be updated before the call returns. Similarly, if a user or group moves, all the member/memberOf attributes that link the user to their groups must also move, before the modrdn returns.
The Samba4 test ldap.js tests this behaviour extensively, because I want to be sure it works.
As understand the discussion I've had with the OpenLDAP team, OpenLDAP does not support this, and will not support it for perhaps some time.
I may have overstated the problem in the previous discussion of our refint module. In fact, RE24 was already changed to work around any potential deadlock issues a long time ago. But to give some context: the refint module was originally written to operate synchronously (back in 2004). Some time later it was changed (2006) to asynchronous mode because users didn't want their clients to be stuck waiting for all the cascaded updates to complete. Most clients don't know or care that a particular change has side-effects. We could introduce a config keyword to select synch vs asynch behavior here, but I have a feeling that will still leave some group of users unhappy no matter how you set it.
Great. If run sync, will it error out correctly if I make an invalid modification (say target not present etc).
Similarly, from discussions with the Fedora DS team at the CIFS developer days, I understand that it is similarly very unlikely that Fedora DS will support internal transactions. (It also does not support subtree renames, which we also need).
The fact that LDAP does not expose a transaction API
You mean draft-zeilenga-ldap-txn ?
I suppose I should have said 'The free LDAP implemetnations I'm looking at don't expose a transaction API'. What did end up happening with that draft?
was always going to be a difficult part of having Samba4 use an LDAP backend, but I always assumed that if we pushed the really hard bit - updating linked attributes - into the LDAP server that we could at least always have a consistent DB. (It turns out this is one of the primary uses of transactions anyway.)
But without that consistency, and without knowing as a caller if all the updates succeed, I'm worried about how we can safely move forward.
This is especially disappointing because I was hoping that these free, replicating LDAP servers might solve the backed replication problem for me, without needing to use AD replication.
Does anybody have any ideas or suggestions on how I could get around this?
There are other ways to guarantee consistency. The simplest approach is to just not store one end of the linked attributes, and always generate them dynamically when they're referenced.
In the old Symas Connexitor EMS product we used (the equivalent of) a UUIDAndOptionalName syntax for all references. In that case the DN was essentially just window-dressing; we always used the ID to actually reference entries and we updated the DNs whenever we found that they didn't match. As such, referential integrity was pretty simple - you never had anything pointing to the wrong entry; the worst that would happen is that you occasionally had dangling references to deleted entries stored in the DB but no one ever saw them because they were cleaned out whenever the entry containing the reference was read.
Do you think the LDAP backend could/should handle this, or will Samba4 have to do the GUID -> DN and DN -> GUID translations before passing things to the backend?
Should we drop the LDAP backend as a nice idea, but not going to work, and focus on DRS or some other form of replication?
Can someone imagine a sane way to reconstruct the DN links, including subtree renames, without the help of the LDAP backend? Could we ban subtree renames (as Fedora DS does), and try to handle this ourselves (with pre/post checks and a good deal of prayer)?
Banning subtree renames seems like a non-starter, and it only eliminates one case; the overall problem still remains.
Indeed.
Andrew Bartlett
Andrew Bartlett wrote:
On Wed, 2008-03-19 at 04:26 -0700, Howard Chu wrote:
Most clients don't know or care that a particular change has side-effects. We could introduce a config keyword to select synch vs asynch behavior here, but I have a feeling that will still leave some group of users unhappy no matter how you set it.
Great. If run sync, will it error out correctly if I make an invalid modification (say target not present etc).
No. We specifically don't validate the original modification here - the server's number one job is always to try to do exactly what the client specified. Also, you can't reliably detect an "invalid" name - it may simply be the name of a non-local entry.
The fact that LDAP does not expose a transaction API
You mean draft-zeilenga-ldap-txn ?
I suppose I should have said 'The free LDAP implemetnations I'm looking at don't expose a transaction API'. What did end up happening with that draft?
It's stalled, like a lot of other things... It is partially implemented in OpenLDAP. Eventually we'll have to complete the implementation, it's too important to leave dangling.
Does anybody have any ideas or suggestions on how I could get around this?
There are other ways to guarantee consistency. The simplest approach is to just not store one end of the linked attributes, and always generate them dynamically when they're referenced.
In the old Symas Connexitor EMS product we used (the equivalent of) a UUIDAndOptionalName syntax for all references. In that case the DN was essentially just window-dressing; we always used the ID to actually reference entries and we updated the DNs whenever we found that they didn't match. As such, referential integrity was pretty simple - you never had anything pointing to the wrong entry; the worst that would happen is that you occasionally had dangling references to deleted entries stored in the DB but no one ever saw them because they were cleaned out whenever the entry containing the reference was read.
Do you think the LDAP backend could/should handle this, or will Samba4 have to do the GUID -> DN and DN -> GUID translations before passing things to the backend?
That would only be practical if you had LDAP transaction support....
It strikes me that the best approach is to just dynamically generate the memberOf values instead of storing them statically. But that also depends on your usage patterns.
On Wed, 2008-03-19 at 17:09 -0700, Howard Chu wrote:
Andrew Bartlett wrote:
On Wed, 2008-03-19 at 04:26 -0700, Howard Chu wrote:
Most clients don't know or care that a particular change has side-effects. We could introduce a config keyword to select synch vs asynch behavior here, but I have a feeling that will still leave some group of users unhappy no matter how you set it.
Great. If run sync, will it error out correctly if I make an invalid modification (say target not present etc).
No. We specifically don't validate the original modification here - the server's number one job is always to try to do exactly what the client specified. Also, you can't reliably detect an "invalid" name - it may simply be the name of a non-local entry.
This is going to get painful then...
The fact that LDAP does not expose a transaction API
You mean draft-zeilenga-ldap-txn ?
I suppose I should have said 'The free LDAP implemetnations I'm looking at don't expose a transaction API'. What did end up happening with that draft?
It's stalled, like a lot of other things... It is partially implemented in OpenLDAP. Eventually we'll have to complete the implementation, it's too important to leave dangling.
OK.
Does anybody have any ideas or suggestions on how I could get around this?
There are other ways to guarantee consistency. The simplest approach is to just not store one end of the linked attributes, and always generate them dynamically when they're referenced.
In the old Symas Connexitor EMS product we used (the equivalent of) a UUIDAndOptionalName syntax for all references. In that case the DN was essentially just window-dressing; we always used the ID to actually reference entries and we updated the DNs whenever we found that they didn't match. As such, referential integrity was pretty simple - you never had anything pointing to the wrong entry; the worst that would happen is that you occasionally had dangling references to deleted entries stored in the DB but no one ever saw them because they were cleaned out whenever the entry containing the reference was read.
Do you think the LDAP backend could/should handle this, or will Samba4 have to do the GUID -> DN and DN -> GUID translations before passing things to the backend?
That would only be practical if you had LDAP transaction support....
It strikes me that the best approach is to just dynamically generate the memberOf values instead of storing them statically. But that also depends on your usage patterns.
Well, searches on member and memberOf are common, so it seems we would quickly find ourselves downloading the whole backend DB and filtering locally...
I would hate to give up on this area of work - if only because I've put a lot of time into it, and it seems like a really useful way to use Samba4, but it also seems that I've found a rat-hole, with far more to it than I ever imagined.
Any further thoughts on how we can resurrect this would be most appreciated.
Andrew Bartlett
Andrew Bartlett wrote:
On Wed, 2008-03-19 at 17:09 -0700, Howard Chu wrote:
It strikes me that the best approach is to just dynamically generate the memberOf values instead of storing them statically. But that also depends on your usage patterns.
Well, searches on member and memberOf are common, so it seems we would quickly find ourselves downloading the whole backend DB and filtering locally...
Searching on memberOf doesn't make a lot of sense to me, when you could simply read the group object directly. When is this actually a useful thing to do? An alternative would be to make the memberOf overlay intercept these filters and rewrite them in terms of member.
On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote:
Andrew Bartlett wrote:
On Wed, 2008-03-19 at 17:09 -0700, Howard Chu wrote:
It strikes me that the best approach is to just dynamically generate the memberOf values instead of storing them statically. But that also depends on your usage patterns.
Well, searches on member and memberOf are common, so it seems we would quickly find ourselves downloading the whole backend DB and filtering locally...
Searching on memberOf doesn't make a lot of sense to me, when you could simply read the group object directly. When is this actually a useful thing to do? An alternative would be to make the memberOf overlay intercept these filters and rewrite them in terms of member.
Premise: here I am thinking beyond what AD is doing as I use the memberOf concept in another project.
From my usage memberOf makes it very simple to find all the groups a
member is part of even if that membership derives from nested grouping.
It's very clear that most of the time you have an identity and you want to know what this Identity is part of, not the other way.
So you really want to do a single search on one entry, rather than a huge search on the whole directory to find out (including local calculation for nesting) what groups include that identity as member, by parsing all groups one by one.
It is as simple as that.
Now for what concern the Samba4 problem, I think we should be more creative and first understand in which cases we might hit a problem with plugins like memberOf. I am sure some of these cases are just normal possible inconsistencies that can happen even in a normal AD server if you do many modifications at the same time. For these cases we just have to try not to make them more probable or problematic than what they are now. In other cases we might think of doing aggressive caching/prediction in our internal transactions. It might require some more work, but it could be a viable option, and also drive some more performance as dealing with an external LDAP is necessarily slower. Finally, if caching/prediction is not possible, we can think of writing overlays/slapi plugins directly for the LDAP server of choice be it OpenLDAP or Fedora Directory Server or anything else. This third option would require some more work and will be server specific, and perhaps involve some creative thinking wrt licensing, but it is certainly a viable option we should not discard. After all, these LDAP servers have a plugin system with defined APIs exactly to solve those problems that cannot be solved merely by external interaction.
Simo.
simo wrote:
On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote:
Searching on memberOf doesn't make a lot of sense to me, when you could simply read the group object directly. When is this actually a useful thing to do? An alternative would be to make the memberOf overlay intercept these filters and rewrite them in terms of member.
Premise: here I am thinking beyond what AD is doing as I use the memberOf concept in another project.
From my usage memberOf makes it very simple to find all the groups a
member is part of even if that membership derives from nested grouping.
Ah, an interesting point, but probably a separate discussion. Note that the OpenLDAP memberOf overlay doesn't handle nested groups.
It's very clear that most of the time you have an identity and you want to know what this Identity is part of, not the other way.
No, not clear at all. A very common application of "groups" is for things like email lists. In that case, an MTA knows a specific group (the name of the email list), and needs to know all of the members.
Other times (e.g. access control) you know an identity (current user) and want to know if the identity belongs to a particular group (for an authorization check). In that case, it is an equal amount of work to look in the user's entry for a memberOf value as to look in the group entry for the particular member value. In practice, because groups may be referenced frequently for multiple users, the group entries will be hot in the server cache and so the member lookup is actually cheaper.
The frequency with which the question "what groups do I belong to" is asked is extremely low in most applications. The frequency with which the question "is XX a member of this group" is asked is very high in most applications. E.g., the libc initgroups() call needs to know what groups you belong to; that call typically occurs only once at the beginning of a login session. After that the result is essentially cached by the kernel. The result is cached in the kernel because the subsequent "is XX a member of group YY" questions happen so frequently as a process accesses system resources.
So you really want to do a single search on one entry, rather than a huge search on the whole directory to find out (including local calculation for nesting) what groups include that identity as member, by parsing all groups one by one.
It is as simple as that.
No, not simple at all.
Yes, ideally you would like to be able to look in a single place and get the answer to "what privileges does user X have" but that doesn't actually mean what you're implying. In particular, the "single place" you're looking isn't necessarily that user's own entry. In most cases, that's the worst place to use because users generally have full write privileges to their own data, and the data comprising their set of privileges really belongs to the sysadmin, not to the individual user.
Ideally, you write to the privilege set and read from the privilege set in the same way, in the same place. As an administrator, this simple consistency makes life easier. The memberOf concept is fundamentally broken if you actually rely on it for privilege determination because it is one step removed from how privileges are actually assigned by the sysadmin. (I.e., the sysadmin doesn't assign membership privileges to a user by writing to the memberOf attribute, therefore memberOf is not authoritative.)
Now for what concern the Samba4 problem, I think we should be more creative and first understand in which cases we might hit a problem with plugins like memberOf. I am sure some of these cases are just normal possible inconsistencies that can happen even in a normal AD server if you do many modifications at the same time. For these cases we just have to try not to make them more probable or problematic than what they are now. In other cases we might think of doing aggressive caching/prediction in our internal transactions. It might require some more work, but it could be a viable option, and also drive some more performance as dealing with an external LDAP is necessarily slower. Finally, if caching/prediction is not possible, we can think of writing overlays/slapi plugins directly for the LDAP server of choice be it OpenLDAP or Fedora Directory Server or anything else. This third option would require some more work and will be server specific, and perhaps involve some creative thinking wrt licensing, but it is certainly a viable option we should not discard. After all, these LDAP servers have a plugin system with defined APIs exactly to solve those problems that cannot be solved merely by external interaction.
Agreed. And frankly, there's already an existence proof that this approach is viable.
On Fri, 2008-03-21 at 05:40 -0700, Howard Chu wrote:
simo wrote:
On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote:
Searching on memberOf doesn't make a lot of sense to me, when you could simply read the group object directly. When is this actually a useful thing to do? An alternative would be to make the memberOf overlay intercept these filters and rewrite them in terms of member.
Premise: here I am thinking beyond what AD is doing as I use the memberOf concept in another project.
From my usage memberOf makes it very simple to find all the groups a
member is part of even if that membership derives from nested grouping.
Ah, an interesting point, but probably a separate discussion. Note that the OpenLDAP memberOf overlay doesn't handle nested groups.
Interesting. To be honest, if memberOf does not solve the nesting group problem, I find it rather useless, and that may explain your comments below.
It's very clear that most of the time you have an identity and you want to know what this Identity is part of, not the other way.
No, not clear at all. A very common application of "groups" is for things like email lists. In that case, an MTA knows a specific group (the name of the email list), and needs to know all of the members.
My bad, I should have specified that this is the case I meet/care in my work. It is not, in general, the most common case, I agree.
Other times (e.g. access control) you know an identity (current user) and want to know if the identity belongs to a particular group (for an authorization check). In that case, it is an equal amount of work to look in the user's entry for a memberOf value as to look in the group entry for the particular member value. In practice, because groups may be referenced frequently for multiple users, the group entries will be hot in the server cache and so the member lookup is actually cheaper.
This is true only if you do member checking by looking at groups. But if you don't they won't, so I am not sure this really is a reason unless there is another reason groups are frequently checked. But given the fact that groups may have huge memberships, actually wasting a lot of cache memory to keep around the members could also be a reason not to do it that way. Of course this is just cheap talk of me, without numbers related to a particular use case (and the use case here is more identity management, not much MTAs or such, even if, in the end, the same Directory can be used for many applications, so groups can actually be really in cache already).
The frequency with which the question "what groups do I belong to" is asked is extremely low in most applications. The frequency with which the question "is XX a member of this group" is asked is very high in most applications. E.g., the libc initgroups() call needs to know what groups you belong to; that call typically occurs only once at the beginning of a login session. After that the result is essentially cached by the kernel. The result is cached in the kernel because the subsequent "is XX a member of group YY" questions happen so frequently as a process accesses system resources.
Well, initgroups() is a special case, it asks for all groups. And I agree that the question: "is XX a member of group YY" is the most common.
So you really want to do a single search on one entry, rather than a huge search on the whole directory to find out (including local calculation for nesting) what groups include that identity as member, by parsing all groups one by one.
It is as simple as that.
No, not simple at all.
Points of view :-)
Yes, ideally you would like to be able to look in a single place and get the answer to "what privileges does user X have" but that doesn't actually mean what you're implying.
Checking privileges is a whole different business, you still have the "is XX a member of group YY" to solve before that.
In particular, the "single place" you're looking isn't necessarily that user's own entry. In most cases, that's the worst place to use because users generally have full write privileges to their own data, and the data comprising their set of privileges really belongs to the sysadmin, not to the individual user.
Well obviously memberOf need to be protected from user manipulation. Esp. in Identity Management applications user entries are usually very well constrained as they hold other information about the user that the user is not supposed to be able to modify. That's not a big problem, we have ACIs just for that.
Ideally, you write to the privilege set and read from the privilege set in the same way, in the same place.
This may be theoretically a sound principle, but in this case I think it is not as ideal as it sounds.
As an administrator, this simple consistency makes life easier.
As long as the Admin knows how the mechanism work, I honestly see no problem, and you can still just query the members of a group when member/memberOf is in use.
The memberOf concept is fundamentally broken if you actually rely on it for privilege determination because it is one step removed from how privileges are actually assigned by the sysadmin. (I.e., the sysadmin doesn't assign membership privileges to a user by writing to the memberOf attribute, therefore memberOf is not authoritative.)
I do not agree with the "fundamentally broken", I think that's exaggeration. Esp. when memberOf solves the "unrolling nested memberships" problem, it actually is, I'd say it is "fundamentally better" than requiring clients to do it and for 2 reasons:
1.) The server choose what nested groups mean, if you want to change that mechanism you are free to do it without having to care what legacy clients may be doing. *** <-- This is *extremely* important IMHO *** 1.1.) You also do not have to beg client app implementers to fix bugs, or change the code if you need to change something, and as group membership can be critical it means you make it simpler for client apps not to do the wrong thing.
2.) The operation is done only once, and is verifiable for the admin. Nested groups can be very complicated to grok, but, by querying memberOf, the Administrator can empirically check that what he did actually lead to the expected results. So in this sense I see memberOf as a real aid for admins.
Of course all this discussion is almost meaningless if you do not support group nesting which is, for me, the whole point of using memberOf.
Finally, while I agree that, technically, memberOf is not authoritative, as what admins change is really the groups's "member" attribute, I fail to see how this can be a problem unless you have a buggy implementation. Memberships do not change often enough to consider the mechanism not worth of trust imo, and there are many ways to implement the memberOf attribute, you can even make it a virtual attribute generated by a search, but this would add a substantial cost if you consider nested groups, so you would very quick end up caching it aggressively which is actually what calculating and storing it ends up being. If you think of memberOf as a cached search that tells you the user's set of groups (including nested...) I think it makes a lot of sense.
Now for what concern the Samba4 problem, I think we should be more creative and first understand in which cases we might hit a problem with plugins like memberOf. I am sure some of these cases are just normal possible inconsistencies that can happen even in a normal AD server if you do many modifications at the same time. For these cases we just have to try not to make them more probable or problematic than what they are now. In other cases we might think of doing aggressive caching/prediction in our internal transactions. It might require some more work, but it could be a viable option, and also drive some more performance as dealing with an external LDAP is necessarily slower. Finally, if caching/prediction is not possible, we can think of writing overlays/slapi plugins directly for the LDAP server of choice be it OpenLDAP or Fedora Directory Server or anything else. This third option would require some more work and will be server specific, and perhaps involve some creative thinking wrt licensing, but it is certainly a viable option we should not discard. After all, these LDAP servers have a plugin system with defined APIs exactly to solve those problems that cannot be solved merely by external interaction.
Agreed. And frankly, there's already an existence proof that this approach is viable.
Indeed.
Simo.
On Fri, 21 Mar 2008, simo wrote:
On Fri, 2008-03-21 at 05:40 -0700, Howard Chu wrote:
simo wrote:
On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote:
Searching on memberOf doesn't make a lot of sense to me, when you could simply read the group object directly. When is this actually a useful thing to do? An alternative would be to make the memberOf overlay intercept these filters and rewrite them in terms of member.
Premise: here I am thinking beyond what AD is doing as I use the memberOf concept in another project.
From my usage memberOf makes it very simple to find all the groups a
member is part of even if that membership derives from nested grouping.
Ah, an interesting point, but probably a separate discussion. Note that the OpenLDAP memberOf overlay doesn't handle nested groups.
Interesting. To be honest, if memberOf does not solve the nesting group problem, I find it rather useless, and that may explain your comments below.
It's very clear that most of the time you have an identity and you want to know what this Identity is part of, not the other way.
No, not clear at all. A very common application of "groups" is for things like email lists. In that case, an MTA knows a specific group (the name of the email list), and needs to know all of the members.
My bad, I should have specified that this is the case I meet/care in my work. It is not, in general, the most common case, I agree.
Other times (e.g. access control) you know an identity (current user) and want to know if the identity belongs to a particular group (for an authorization check). In that case, it is an equal amount of work to look in the user's entry for a memberOf value as to look in the group entry for the particular member value. In practice, because groups may be referenced frequently for multiple users, the group entries will be hot in the server cache and so the member lookup is actually cheaper.
This is true only if you do member checking by looking at groups. But if you don't they won't, so I am not sure this really is a reason unless there is another reason groups are frequently checked. But given the fact that groups may have huge memberships, actually wasting a lot of cache memory to keep around the members could also be a reason not to do it that way. Of course this is just cheap talk of me, without numbers related to a particular use case (and the use case here is more identity management, not much MTAs or such, even if, in the end, the same Directory can be used for many applications, so groups can actually be really in cache already).
We run a medium sized directory here at Oregon State University which contains user objects for all students, staff, and faculty (approximately 35,000 entries). It also contains a few static groups (not nested) for categorizing users. We are using Sun Directory Server 5.2.
The documentation for Sun Directory Server 6.2 says:
The isMemberOf attribute is calculated and added to the user entry at the start of the search. It is then removed again after the search has finished. This functionality provides easy management of groups, and fast read access.
To me, this says that the directory does an internal group search to generate the isMemberOf attribute on the fly. I believe this is the way Active Directory handles the memberOf attribute as well.
Most directory servers suggest tuning the caches so that the entire directory fits in RAM, which isn't too hard these days with the price of RAM. Directory entries are typically only a few kilobytes each.
From a performance standpoint, it may not matter which method of group
enumeration you use. Directory servers are optimized for reads, and a simple LDAP query like:
(&(objectclass=groupofuniquenames)(uniquemember=uid=doej))
can be just as fast as:
(&(objectclass=posixaccount)(uid=doej))
I'm not familiar with the background of this whole discussion. It sounds like the memberOf/isMemberOf implementations are relatively new. Unless you want Samba to require the latest and greatest versions of these directory servers, you may need to avoid memberOf/isMemberOf and use the standard methods of querying group membership.
Andy
On Fri, 2008-03-21 at 10:51 -0700, Andrew Morgan wrote:
On Fri, 21 Mar 2008, simo wrote:
On Fri, 2008-03-21 at 05:40 -0700, Howard Chu wrote:
simo wrote:
On Wed, 2008-03-19 at 17:33 -0700, Howard Chu wrote:
Searching on memberOf doesn't make a lot of sense to me, when you could simply read the group object directly. When is this actually a useful thing to do? An alternative would be to make the memberOf overlay intercept these filters and rewrite them in terms of member.
Premise: here I am thinking beyond what AD is doing as I use the memberOf concept in another project.
From my usage memberOf makes it very simple to find all the groups a
member is part of even if that membership derives from nested grouping.
Ah, an interesting point, but probably a separate discussion. Note that the OpenLDAP memberOf overlay doesn't handle nested groups.
Interesting. To be honest, if memberOf does not solve the nesting group problem, I find it rather useless, and that may explain your comments below.
It's very clear that most of the time you have an identity and you want to know what this Identity is part of, not the other way.
No, not clear at all. A very common application of "groups" is for things like email lists. In that case, an MTA knows a specific group (the name of the email list), and needs to know all of the members.
My bad, I should have specified that this is the case I meet/care in my work. It is not, in general, the most common case, I agree.
Other times (e.g. access control) you know an identity (current user) and want to know if the identity belongs to a particular group (for an authorization check). In that case, it is an equal amount of work to look in the user's entry for a memberOf value as to look in the group entry for the particular member value. In practice, because groups may be referenced frequently for multiple users, the group entries will be hot in the server cache and so the member lookup is actually cheaper.
This is true only if you do member checking by looking at groups. But if you don't they won't, so I am not sure this really is a reason unless there is another reason groups are frequently checked. But given the fact that groups may have huge memberships, actually wasting a lot of cache memory to keep around the members could also be a reason not to do it that way. Of course this is just cheap talk of me, without numbers related to a particular use case (and the use case here is more identity management, not much MTAs or such, even if, in the end, the same Directory can be used for many applications, so groups can actually be really in cache already).
We run a medium sized directory here at Oregon State University which contains user objects for all students, staff, and faculty (approximately 35,000 entries). It also contains a few static groups (not nested) for categorizing users. We are using Sun Directory Server 5.2.
The documentation for Sun Directory Server 6.2 says:
The isMemberOf attribute is calculated and added to the user entry at the start of the search. It is then removed again after the search has finished. This functionality provides easy management of groups, and fast read access.
Hmm this seem just a description of a virtual attribute.
To me, this says that the directory does an internal group search to generate the isMemberOf attribute on the fly. I believe this is the way Active Directory handles the memberOf attribute as well.
IIRC in AD memberOf is a linked attribute, stored permanently.
Most directory servers suggest tuning the caches so that the entire directory fits in RAM, which isn't too hard these days with the price of RAM. Directory entries are typically only a few kilobytes each.
Yes it is a good idea to do so.
From a performance standpoint, it may not matter which method of group enumeration you use. Directory servers are optimized for reads, and a simple LDAP query like:
(&(objectclass=groupofuniquenames)(uniquemember=uid=doej))
can be just as fast as:
(&(objectclass=posixaccount)(uid=doej))
It matters if you start thinking of nested groups as in that case the number of searches may increase considerably, although, if all the data is in cache it may not have such a big impact depending on the frequency of such searches.
I'm not familiar with the background of this whole discussion. It sounds like the memberOf/isMemberOf implementations are relatively new. Unless you want Samba to require the latest and greatest versions of these directory servers, you may need to avoid memberOf/isMemberOf and use the standard methods of querying group membership.
Samba4 has very peculiar needs as the aim is to provide an AD compatible interface to the clients. This is because MS clients are tightly coupled and expect certain behaviors that are AD specific. We can't just run a generic directory in this case, we either proxy and translate to a generic directory or we heavily modify one.
So far we have basically worked on 2 fronts. One was to make our own LDAP implementation based on our LDB database so that we could freely experiment and discover exactly what the clients need/want. At the same side we developed LDB to be a proxy to a backend directory and created a module called ldb_map that could do translation on the fly. This second approach is showing the inherent limits it was clear we would have met since we started. The point is to decide to what degree of compatibility we need to get to.
Personally I am ok with just Windows clients being able to join using Kerberos and our Directory, even if some Windows admin tools don't work.
We have reached this point some time ago already, it is just a matter of deciding when we are compatible enough to be satisfied, and start stabilizing the code.
Simo.
To me, this says that the directory does an internal group search to generate the isMemberOf attribute on the fly. I believe this is the way Active Directory handles the memberOf attribute as well.
IIRC in AD memberOf is a linked attribute, stored permanently.
From memory, AD stores the entry IDs of the link tuples in a separate table, so both "member" and "memberOf" are ostensibly generated on the fly. But this is really an implementation decision (although things do start to get interesting when dealing with references across partition boundaries).
-- Luke
On Sat, 22 Mar 2008, Luke Howard wrote:
To me, this says that the directory does an internal group search to generate the isMemberOf attribute on the fly. I believe this is the way Active Directory handles the memberOf attribute as well.
IIRC in AD memberOf is a linked attribute, stored permanently.
From memory, AD stores the entry IDs of the link tuples in a separate table, so both "member" and "memberOf" are ostensibly generated on the fly. But this is really an implementation decision (although things do start to get interesting when dealing with references across partition boundaries).
I noticed that the memberOf tab in AD Users and Computers does not show group membership for groups that are not in the same domain as the user. Access controls were still correctly applied though, so those interfaces must be enumerating group membership some other way.
I've always viewed the memberOf attribute as a convenient tool for human use, but not something I would use programmatically. All the benefits provided by memberOf can be attained other ways by the LDAP client, although the LDAP client may need to know more about the directory structure to do it.
Andy
On Fri, 2008-03-21 at 16:17 -0700, Andrew Morgan wrote:
On Sat, 22 Mar 2008, Luke Howard wrote:
To me, this says that the directory does an internal group search to generate the isMemberOf attribute on the fly. I believe this is the way Active Directory handles the memberOf attribute as well.
IIRC in AD memberOf is a linked attribute, stored permanently.
From memory, AD stores the entry IDs of the link tuples in a separate table, so both "member" and "memberOf" are ostensibly generated on the fly. But this is really an implementation decision (although things do start to get interesting when dealing with references across partition boundaries).
I noticed that the memberOf tab in AD Users and Computers does not show group membership for groups that are not in the same domain as the user. Access controls were still correctly applied though, so those interfaces must be enumerating group membership some other way.
AD clients get user memberships from the PAC (the kerberos extension) when the user authenticates. Windows clients never do LDAP calls to get group membership of a user.
I've always viewed the memberOf attribute as a convenient tool for human use, but not something I would use programmatically. All the benefits provided by memberOf can be attained other ways by the LDAP client, although the LDAP client may need to know more about the directory structure to do it.
This is my point in using memberOf, less knowledge need to be put in the client the better.
Simo.
Andrew Morgan wrote:
I noticed that the memberOf tab in AD Users and Computers does not show group membership for groups that are not in the same domain as the user. Access controls were still correctly applied though, so those interfaces must be enumerating group membership some other way.
How about the 'tokenGroups' attribute? Samba would have to emulate this too.
http://msdn2.microsoft.com/en-us/library/ms680275(VS.85).aspx
Ciao, Michael.
On 22/03/2008, at 10:17 AM, Andrew Morgan wrote:
On Sat, 22 Mar 2008, Luke Howard wrote:
To me, this says that the directory does an internal group search to generate the isMemberOf attribute on the fly. I believe this is the way Active Directory handles the memberOf attribute as well.
IIRC in AD memberOf is a linked attribute, stored permanently.
From memory, AD stores the entry IDs of the link tuples in a separate table, so both "member" and "memberOf" are ostensibly generated on the fly. But this is really an implementation decision (although things do start to get interesting when dealing with references across partition boundaries).
I noticed that the memberOf tab in AD Users and Computers does not show group membership for groups that are not in the same domain as the user. Access controls were still correctly applied though, so those interfaces must be enumerating group membership some other way.
Right, this is because it's expensive (I think they'll show such memberships if you query through the GC port [3268] but that requires you talk to a GC).
When a non-GC KDC builds a PAC, it will contact a GC over RPC to complete the user's token (cf. IDL_DRSGetMemberships).
-- Luke
FYI here's how we did it in XAD, page 19ff.
http://www.openldap.org/conf/odd-wien-2003/luke.pdf
Implemented in OpenLDAP completely at the SLAPI layer. Without help from the underlying DB, though, it wasn't transaction safe; so the approach of storing "member" and computing "memberOf" is probably better.
-- Luke
On 23/03/2008, at 2:25 AM, Luke Howard wrote:
On 22/03/2008, at 10:17 AM, Andrew Morgan wrote:
On Sat, 22 Mar 2008, Luke Howard wrote:
To me, this says that the directory does an internal group search to generate the isMemberOf attribute on the fly. I believe this is the way Active Directory handles the memberOf attribute as well.
IIRC in AD memberOf is a linked attribute, stored permanently.
From memory, AD stores the entry IDs of the link tuples in a separate table, so both "member" and "memberOf" are ostensibly generated on the fly. But this is really an implementation decision (although things do start to get interesting when dealing with references across partition boundaries).
I noticed that the memberOf tab in AD Users and Computers does not show group membership for groups that are not in the same domain as the user. Access controls were still correctly applied though, so those interfaces must be enumerating group membership some other way.
Right, this is because it's expensive (I think they'll show such memberships if you query through the GC port [3268] but that requires you talk to a GC).
When a non-GC KDC builds a PAC, it will contact a GC over RPC to complete the user's token (cf. IDL_DRSGetMemberships).
-- Luke
-- Fedora-directory-devel mailing list Fedora-directory-devel@redhat.com https://www.redhat.com/mailman/listinfo/fedora-directory-devel
-- www.padl.com | www.fghr.net
389-devel@lists.fedoraproject.org
-
Andrew Bartlett -
Andrew Morgan -
Howard Chu -
Luke Howard -
Michael Ströder -
simo