We have a setup where we are running 2 servers behind a load balancer (for HA purposes), where each of these servers is bulk-initialized daily (via ldif2db.pl) with a large set of data fed to us via batch extracts from various administrative systems. Up till now, there has been no need to configure replication between these 2 servers, as all of the data is read-only. However, we now have a requirement to update some of the directory data in a "real-time" fashion (e.g., when particular events fire in our PeopleSoft system we want to update the directory)--hence, the need for MMR. The batch extracts will still be our "checkpoints", so we will want to load them in once-per-day, as we do now.
So, the question is: what would be the "recommended" approach for a scenario like this? How do we (can we?) make MMR coexist peacefully with frequent bulk initializations?
TIA, --Gary
-- Gary Windham Senior Enterprise Systems Architect The University of Arizona, UITS +1 520 626 5981
Gary Windham wrote:
We have a setup where we are running 2 servers behind a load balancer (for HA purposes), where each of these servers is bulk-initialized daily (via ldif2db.pl) with a large set of data fed to us via batch extracts from various administrative systems. Up till now, there has been no need to configure replication between these 2 servers, as all of the data is read-only. However, we now have a requirement to update some of the directory data in a "real-time" fashion (e.g., when particular events fire in our PeopleSoft system we want to update the directory)--hence, the need for MMR. The batch extracts will still be our "checkpoints", so we will want to load them in once-per-day, as we do now.
How does the data get from peoplesoft to the directory server?
So, the question is: what would be the "recommended" approach for a scenario like this? How do we (can we?) make MMR coexist peacefully with frequent bulk initializations?
In general, it's not a good idea to do a bulk load daily.
TIA, --Gary
-- Gary Windham Senior Enterprise Systems Architect The University of Arizona, UITS +1 520 626 5981
-- 389 users mailing list 389-users@redhat.com https://www.redhat.com/mailman/listinfo/fedora-directory-users
Hi Gary,
I used to work for a university that does something similar to what you are trying to do. I'll explain their setup and it might give you a few ideas. They have a custom user management database that's the authoritative source of computer account information, a series of FDS servers are used for identification and authentication. A Perl script is used to turn the database contents into LDIF format as it would be used to populate an empty database (like one of your ldif2db batch extracts). They then take a dump of the LDAP directory into LDIF format and compare the database LDIF to directory LDIF and come up with a delta LDIF file. This delta LDIF is then run on the directory server to bring it in line with the database contents.
This update process runs every couple minutes, so the delta never really gets that big and password changes / new users only take a few minutes to propagate around the university. They would never need to batch import the entire database contents unless there was a catastrophe.
So, for your scenario, you might consider scrapping the nightly bulk initialisations, turn your servers into MMR and look at doing more frequent updates with delta files to provide faster synchronisation between your data sources.
If you actually need to do real real-time updates, you can do that with the same setup above, you just need to fire off a specific LDAP update to your load balanced LDAP from Peoplesoft.
Luke Bigum Systems Administrator (p) 1300 661 668 (f) 1300 661 540 (e) lbigum@iseek.com.au http://www.iseek.com.au Level 1, 100 Ipswich Road Woolloongabba QLD 4102
This e-mail and any files transmitted with it may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorised to receive for the recipient), please contact the sender by reply e-mail and delete all copies of this message.
-----Original Message----- From: fedora-directory-users-bounces@redhat.com [mailto:fedora-directory-users-bounces@redhat.com] On Behalf Of Rich Megginson Sent: Wednesday 1 July 2009 1:21 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] bulk initialization with MMR
Gary Windham wrote:
We have a setup where we are running 2 servers behind a load balancer (for HA purposes), where each of these servers is bulk-initialized daily (via ldif2db.pl) with a large set of data fed to us via batch extracts from various administrative systems. Up till now, there has been no need to configure replication between these 2 servers, as all of the data is read-only. However, we now have a requirement to update some of the directory data in a "real-time" fashion (e.g., when particular events fire in our PeopleSoft system we want to update the directory)--hence, the need for MMR. The batch extracts will still be our "checkpoints", so we will want to load them in once-per-day, as we do now.
How does the data get from peoplesoft to the directory server?
So, the question is: what would be the "recommended" approach for a scenario like this? How do we (can we?) make MMR coexist peacefully with frequent bulk initializations?
In general, it's not a good idea to do a bulk load daily.
TIA, --Gary
-- Gary Windham Senior Enterprise Systems Architect The University of Arizona, UITS +1 520 626 5981
-- 389 users mailing list 389-users@redhat.com https://www.redhat.com/mailman/listinfo/fedora-directory-users
Thanks, Luke! Since my original post, I did some more research and arrived at a solution very close to yours. I did turn the servers into MMR, and am using the "ldifsort" and "ldifdiff" utilities that come with the perl-ldap module to generate "delta" LDIF files that are loaded via ldapmodify. For our "real time" updates we have PeopleSoft sending SOAP messages via Integration Broker to some web services which perform the appropriate LDAP operations.
I appreciate your response...it serves as a good "sanity check". :)
--Gary
-- Gary Windham Senior Enterprise Systems Architect The University of Arizona, UITS +1 520 626 5981
On Jul 1, 2009, at 3:44 PM, Luke Bigum wrote:
Hi Gary,
I used to work for a university that does something similar to what you are trying to do. I'll explain their setup and it might give you a few ideas. They have a custom user management database that's the authoritative source of computer account information, a series of FDS servers are used for identification and authentication. A Perl script is used to turn the database contents into LDIF format as it would be used to populate an empty database (like one of your ldif2db batch extracts). They then take a dump of the LDAP directory into LDIF format and compare the database LDIF to directory LDIF and come up with a delta LDIF file. This delta LDIF is then run on the directory server to bring it in line with the database contents.
This update process runs every couple minutes, so the delta never really gets that big and password changes / new users only take a few minutes to propagate around the university. They would never need to batch import the entire database contents unless there was a catastrophe.
So, for your scenario, you might consider scrapping the nightly bulk initialisations, turn your servers into MMR and look at doing more frequent updates with delta files to provide faster synchronisation between your data sources.
If you actually need to do real real-time updates, you can do that with the same setup above, you just need to fire off a specific LDAP update to your load balanced LDAP from Peoplesoft.
Luke Bigum Systems Administrator (p) 1300 661 668 (f) 1300 661 540 (e) lbigum@iseek.com.au http://www.iseek.com.au Level 1, 100 Ipswich Road Woolloongabba QLD 4102
This e-mail and any files transmitted with it may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorised to receive for the recipient), please contact the sender by reply e-mail and delete all copies of this message.
-----Original Message----- From: fedora-directory-users-bounces@redhat.com [mailto:fedora-directory-users-bounces@redhat.com ] On Behalf Of Rich Megginson Sent: Wednesday 1 July 2009 1:21 AM To: General discussion list for the 389 Directory server project. Subject: Re: [389-users] bulk initialization with MMR
Gary Windham wrote:
We have a setup where we are running 2 servers behind a load balancer (for HA purposes), where each of these servers is bulk-initialized daily (via ldif2db.pl) with a large set of data fed to us via batch extracts from various administrative systems. Up till now, there has been no need to configure replication between these 2 servers, as all of the data is read-only. However, we now have a requirement to update some of the directory data in a "real-time" fashion (e.g., when particular events fire in our PeopleSoft system we want to update the directory)--hence, the need for MMR. The batch extracts will still be our "checkpoints", so we will want to load them in once-per-day, as we do now.
How does the data get from peoplesoft to the directory server?
So, the question is: what would be the "recommended" approach for a scenario like this? How do we (can we?) make MMR coexist peacefully with frequent bulk initializations?
In general, it's not a good idea to do a bulk load daily.
TIA, --Gary
-- Gary Windham Senior Enterprise Systems Architect The University of Arizona, UITS +1 520 626 5981
-- 389 users mailing list 389-users@redhat.com https://www.redhat.com/mailman/listinfo/fedora-directory-users
-- 389 users mailing list 389-users@redhat.com https://www.redhat.com/mailman/listinfo/fedora-directory-users
389-users@lists.fedoraproject.org