Re: 389-devel] Advice on a problem (syncrepl + entryuuid)
by Howard Chu
>>> >> I still think allowing entryuuid to diverge from nsuniqueid is the right call. It opens more scenarioes up to us for migration.
>> >
>> > Just fyi, when replicating from 389DS to OpenLDAP, we directly map nsUniqueId to entryUuid. They're
>> > both UUIDs and both serve the same purpose on their respective servers, so why is there
>> > any reason to allow proliferation of other IDs?
>
> Ahh I believe that you may be referring to the sundsee syncrepl consumer mode from your 2.5 or git master perhaps? Sadly, this is not the target of my work - if it was, life would be much easier, and this would not be a problem. In this case, we have a customer who wants to create read-only openldap consumers which are able to get data from 389-ds, but they are on version 2.4.41 of openldap which does not appear to perform this mapping from my reading of the code (but i've had some mistakes reading code lately, so perhaps I'm wrong).
Yes, the consumer code is only in 2.5, not 2.4. But the mapping is trivial; the nsUniqueId format
has the same number of digits as the entryUuid format and only differs by having one less hyphen.
So it's quite simple to write an overlay or plugin to convert from one to the other. e.g.
https://git.openldap.org/openldap/openldap/-/blob/master/servers/slapd/sy...
> Their goal is to move from openldap to 389-ds due to support reasons. So this means entryuuid which has been a stable ID for a long time in their fleet is something we need to support - and many external applications do read entryuuid as a primary key to identify objects in the directory, despite the fact it should be an internal replication-only element IMO (vmware and nextcloud come to mind immediately). nsuniqueid (annoyingly) is not the same format as entryuuid, so we can't just ask them to change to reading nsunique (even if the bytes were the same). Every day we get to live with the decisions of the past, and we have the joy of working through and around them.
>
> So I think as much as your suggestion is probably correct technically, it doesn't apply in this situation.
>
> I am considering that the "solution" in this case though, is that when we see an entryuuid on an import/add, we derive nsuniqueid from that, so that entryuuid / nsunique are equivalents in the respective directories, and we can then re-synthesise the entryuuid from nsunique on the syncrepl. We still need to store entryuuid however, because client applications will be expecting it to remain consistent and present during an openldap to 389 migration.
Why wouldn't you just generate entryUuid from nsUniqueId on the fly, as a virtual attribute, when requested in
a search/compare? And yes, derive nsUniqueId from entryUuid on an Add.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
3 years, 9 months
Re: Advice on a problem (syncrepl + entryuuid) (William Brown)
by Howard Chu
> Date: Mon, 22 Jun 2020 15:43:30 +1000
> From: William Brown <wbrown(a)suse.de>
> Subject: [389-devel] Advice on a problem (syncrepl + entryuuid)
> To: "389 Directory server developer discussion."
> <389-devel(a)lists.fedoraproject.org>
> Message-ID: <CEAABAA1-E958-4EA6-8432-B7DF92E4AA2C(a)suse.de>
> Content-Type: text/plain; charset=utf-8
>
> Hi all,
>
> I've been a bit stuck on this problem, and I was hoping for some advice or thoughts.
>
> So, part of the migration from openldap, and in general, is that we support entryuuid. That's all fine. I made a decision to allow entryuuid to diverge from nsuniqueid, as entryuuid is often a primary key in many databases. So we should be able to bring in entryuuids' that already exist.
>
> This creates a dilema in syncrepl though. Syncrepl currently uses the nsuniqueid in place of the entryuuid for sync. It does not appear to be straight forward to change this to use entryuuid if it exists - when we send a delete, that comes from the retro changelog, and it stores a delete as a delete with the nsuniqueid. That also means the entry that held the nsunique in question may no longer exist so we cant reference what the entryuuid was.
>
> So this leads me to a problem of "how to solve it".
>
> I still think allowing entryuuid to diverge from nsuniqueid is the right call. It opens more scenarioes up to us for migration.
Just fyi, when replicating from 389DS to OpenLDAP, we directly map nsUniqueId to entryUuid. They're
both UUIDs and both serve the same purpose on their respective servers, so why is there
any reason to allow proliferation of other IDs?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
3 years, 9 months
Advice on a problem (syncrepl + entryuuid)
by William Brown
Hi all,
I've been a bit stuck on this problem, and I was hoping for some advice or thoughts.
So, part of the migration from openldap, and in general, is that we support entryuuid. That's all fine. I made a decision to allow entryuuid to diverge from nsuniqueid, as entryuuid is often a primary key in many databases. So we should be able to bring in entryuuids' that already exist.
This creates a dilema in syncrepl though. Syncrepl currently uses the nsuniqueid in place of the entryuuid for sync. It does not appear to be straight forward to change this to use entryuuid if it exists - when we send a delete, that comes from the retro changelog, and it stores a delete as a delete with the nsuniqueid. That also means the entry that held the nsunique in question may no longer exist so we cant reference what the entryuuid was.
So this leads me to a problem of "how to solve it".
I still think allowing entryuuid to diverge from nsuniqueid is the right call. It opens more scenarioes up to us for migration.
So some options:
* EntryUUID has to be consistent to nsuniqueid, and when we see a create with an entryUuid, we assign the nsunique from the entryuuid we have rather than generating it. If there is no entryuuid, we generate it from the nsuniqueid.
* Extend the retrochangelog to store the entryuuid as well as the nsunique for deletes/mods/adds.
* Have syncrepl "strip" entryuuid, so that the inconsistent attribute is never sent to clients (but this means that entryuuid between 389 -> syncrepl client diverges, which could have other consequences).
* Rather than send the set of deletes, we can send a list of 'what uuids are still present' that implies all others are removed. It's less efficient on the wire, but it works around the problem. (More risk of breaking IPA parts though).
* Disallow entryuuid and syncrepl plugins at the same time - you have one or the other (probably good for IPA tbh).
* Other thoughts?
I'm honestly not sure whats the best choice - they are all tradeoffs in some way or another, with their own risks. So I'd appreciate your advice here.
—
Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server
SUSE Labs
3 years, 10 months
