11:17 p.m.
Hello everyone,
I have a request for advice on how to approach monitoring of replication in an environment
with approximately 30 FreeIPA servers, all in a master-master replication agreement, using
389-ds (389-ds-base-1.4.3.28-6). I am currently looking for ways to reduce the number of
replicas (because there are more to come) and need to justify it to the architecture
department with evidence based on experimental observations.
The problem we are facing is that our installation has started experiencing lags in some
operations, such as adding user groups, HBAC, and SUDO rules and the most heaviest (by the
impact) is automember-rebuild operation.
The number of entities being added is not large, with a maximum of 10 groups and several
sudo and HBAC rules, though for automember-rebuild I don't know for certain cause for
now I didn't figure out what operations are done internally by this. The
"lag" manifests as latency in LDAP operations, leading to timeouts, which in
turn causes some services that rely on Kerberos or DNS (because FreeIPA uses LDAP
directory for everything) to go down. Our monitoring system also shows that the outage
propagates through replicas as replication progresses.
The classic approach of monitoring replication agreements through the
nsds5replicaLastUpdateStatus attribute is not sufficient. We need a more dynamic approach
that can show the "waves" or replication sessions throughout the environment,
which can help in further tuning replication parameters.
I am facing the following problems:
1) The only way to get full replication information currently is to turn on full debug for
error logs. While this can be done in test environments, I cannot rely on it in
production. I thought that BPF could be the answer, but I am not sure if dirsrv has
internal support (predefined probe points) for it. Has anyone from the developers tried to
use BPF to monitor some features in 389-ds?
2) Regardless of BPF support, I can still try to implement monitoring with it, in
conjunction with debug symbols. However, another problem is that I do not know the exact
algorithm of the replication process. I have read this article
(https://www.port389.org/docs/389ds/design/replication_troubleshooting.html), but it is
still obscure for my purposes. Can you shed some light on the approach I should take here?
In my mind, the first step should be very basic - attach to a set of consumer level
functions responsible for receiving replica updates, and monitor the latency, the amount
of incoming connections at a given point in time, and so on. But if you could point me in
the right direction (other than just directly pointing to the repository and searching the
source code), I would greatly appreciate it.
3) This feature
(https://directory.fedoraproject.org/docs/389ds/design/log-operation-stats...) is not
supported for my version of 389-ds, is it? Is there a way to patch my version to support
it?
Thank you in advance for your help.
10:11 p.m.
New subject: A more profound replication monitoring of 389-ds instance
I wrote a really long post but now looking at it again and searching for monitoring
problems in mailing list it seems kind of vague and unrelated.
I found that records under cn=ldbm database,cn=plugins,cn=config expose database metrics
(I only knew about metrics under cn=monitor). So I will try to expose them to monitoring
system and see what I could find during high load tasks.
3:21 a.m.
New subject: A more profound replication monitoring of 389-ds instance
Hi,
I read your first post. I found it very interesting but was not able to
get a clear understanding of your needs. This second post would also
need additional details. The cn=ldbm database monitoring will mainly
return stats about DB activity and are IMHO a bit raw data.
In the posts you also mentioned, replication lag and "waves" of
replication session. More specifically what concerns are you trying to
fix ? Do you want to monitor, for reporting, to anticipate with a
corrective actions ?
best regards
thierry
On 4/20/23 05:11, dweller dweller wrote:
I wrote a really long post but now looking at it again and searching
for monitoring problems in mailing list it seems kind of vague and unrelated.
I found that records under cn=ldbm database,cn=plugins,cn=config expose database metrics
(I only knew about metrics under cn=monitor). So I will try to expose them to monitoring
system and see what I could find during high load tasks.
_______________________________________________
389-users mailing list -- 389-users(a)lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproje...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
9:10 a.m.
New subject: A more profound replication monitoring of 389-ds instance
Yes, I'll try to explain my needs more clearly. As it happens a lot I recently
inherited a FreeIPA installation and am now responsible for managing the service. As
someone who was not previously familiar with FreeIPA, I am in the process of building my
expertise in managing it.
When I started the monitoring setup was represented with node_exporter, process_exporter
for the host and 389ds_exporter (https://github.com/terrycain/389ds_exporter) for the ldap
data. However, as the FreeIPA installation grew in size, we started encountering issues
and realized that we lacked critical information to pinpoint the root causes of these
problems. To address this, I have taken steps to improve the monitoring setup. I have
started monitoring FreeIPA's bind service using a separate_exporter and exporting DNS
queries to opensearch. Additionally, I have rewritten the 389ds_exporter to include
cn=monitor metrics to provide more visibility into the 389 Directory Server.
I recently realized that I could also include 'cn=ldbm database' metrics, which
are low-level but could be useful in troubleshooting the issues we are facing. The
problems we are encountering are related to disk IO, and having these metrics could
provide valuable insights into the following:
1) Excessive paging out and increased swap usage without spikes in load. For example after
restarting of replica the swap usage increases to 30% (of 3GB swap space) over 1-2 days
while there are at least 4GB of availiable RAM present on the host. And the general swap
consumer is ns-slapd service. For now I only tested to configure swappiness parametr to
zero, which did not help, so I guess there are some other factors involved.
2) Spikes in IO latency observed during modifying and adding operations, which were not
present when the cluster was smaller (up to 10 replicas). I need to determine whether the
issue lies with service tuning or with the cloud provider and its SAN, as we recently
migrated to SSD disks without improvement. As I said about "replication lag"
those problems just started more appearing as new replicas were added, but for now we
mostly observe it by outage of services that rely on ldap. The "waves" refers to
the way problem apprear, as different clients VDCs are having problems one after the other
which is looks like replication propagation.
3) Master-master replication just seems to me as a big "black cloud", which I
have no control or knowledge of. When you have couple of hosts it is maybe fine to rely on
documented way of looking up replicationStatus attribute, but when you have couple of
dozens I guess things could get quite not so straitforward, at least relying on intuition
suggests it. When I say about replication observability what I mean and what I'd like
to see is following:
Graph representation...
- ...of time it took to replay a change (or I guess time of full replication session)
- ...of the amount simutanialous connections that Suppliers trying to establish with
Consumer
- ...of time spent waiting to acquire replica access
I just pointed a few of the top of my head. I don't know for sure (and first post was
about it) is it really worth it to try and get those kind of metrics or I just don't
know what I'm talking about and it would be a waste of time and hard to implement. As
I mentioned bpf cause I see it as only option I could get it, the other option is to parse
logs that are in DEBUG mode which is not the option.
With replication metrics besides the ability to see its impact on the problems above,
I'm also trying to solve more administrative task - I need to convince the
architerture departament to change the model of adding new replicas. Right now we kinda
adding two replicas for every new client.
+------------------------------+
| client#1 |
| VDC |
| |
| +--------------+ | +---------------------+
+---------------------+
| | +-------------->+
+---------------->+ | ...
| | replica-01 | | | common-replica-01 | |
common-replica-02 |
| | +<--------------+
+<----------------+ |
| +--------------+ | +---------------------+
+---------------------+
| | ^ | | ^
| ^
| v | | | |
| |
| +--------------+ | | |
| |
| | | | | |
| |
| | replica-02 | | | |
| |
| | | | | |
| |
| +--------------+ | | |
| |
| | | |
| |
| | v |
v |
+------------------------------+ +---------------------+
+---------------------+
| +---------------->+
|
| common-replica-03 | |
common-replica-04 |
| +<----------------+
|
+---------------------+
+---------------------+
Which is not ideal at all (and as I said we started to face problems). From their side the
answer is that they are following documentation restrictions for no more than 4 replica
agreements for replica and no more than 60 simutanialous replicas in master-master
replications. And for now this is indeed being followed and I need to come with deeper
analysis or find that problem lies in fine tuning the service.
So it's kind mishmash of everything at the same time, hope I answered you question.
best regards,
v.zh
12:50 a.m.
New subject: A more profound replication monitoring of 389-ds instance
Oops, I ment to post this diagram
https://pastebin.com/HZZjCtFJ
The vision was, I guess, for each common-replica to have a connection with its own
downstream client's replica, in order to not exceed 4 replication agreements per
replica.
9:23 a.m.
New subject: A more profound replication monitoring of 389-ds instance
On Fri, 2023-04-21 at 05:50 +0000, dweller dweller wrote:
Oops, I ment to post this diagram
https://pastebin.com/HZZjCtFJ
The vision was, I guess, for each common-replica to have a connection with its
own downstream client's replica, in order to not exceed 4 replication
agreements per replica.
You might consider a replication hub and spoke model. Where all replication
agreements point to the replication hub nodes.
perhaps something like this would improve replication performance?
+--+ +--+ +--+ +--+
| | | | | | | |
| | | | | | | |
| | | | | | | |
+--+ +--+ +--+ +--+
| +---+ +---+ |
| | | | | |
+--->| | | |<---+
replication hub
+-------->| | | | <------+
| +---+ +---+ |
| |
+--++ +--+ +--+++--+
| | | | | || |
| | | | | || |
+---+ +--+ +--++---+
3:33 a.m.
New subject: A more profound replication monitoring of 389-ds instance
Hi,
I agree that it is complex task to master such FreeIPA deployment.
FreeIPA enables many components, 389ds is just one of them, and several
of them could contribute when a problem occurs. My main concern here is
that you express a need to monitor (how well FreeIPA deployment works)
rather than pointing a clear misbehavior of the topology that we could
focus on.
Replication is an important functionality of 389ds/FreeIPA and
monitoring replication is a common demand. One common demand is to
monitor replication lag (how much time it takes for the topology to
converge: an update is replicated on all replicas). There are several
ways to monitor that but I think an easy way is to rely on dirsrv access
logs. Each replicated update is identified uniquely with its 'csn' and
you can find csn like 'csn=57eb7dbc000000600000'. Then grepping this
value among all access logs on all instances, could give you an
indication of lag. Lag could typically be in the range of seconds or 1-2
minutes, if it spikes to many minutes or never hit some replica then you
could start investigating why it is slow.
A difficulty with that procedure is that some updates are not replicated
(fractional replication).
Investigation in replication are quite complex and difficult to explain
in general. This forum is a good place to get answers to specific questions.
best regards
Thierry
On 4/20/23 16:10, dweller dweller wrote:
Yes, I'll try to explain my needs more clearly. As it happens a
lot I recently inherited a FreeIPA installation and am now responsible for managing the
service. As someone who was not previously familiar with FreeIPA, I am in the process of
building my expertise in managing it.
When I started the monitoring setup was represented with node_exporter, process_exporter
for the host and 389ds_exporter (https://github.com/terrycain/389ds_exporter) for the ldap
data. However, as the FreeIPA installation grew in size, we started encountering issues
and realized that we lacked critical information to pinpoint the root causes of these
problems. To address this, I have taken steps to improve the monitoring setup. I have
started monitoring FreeIPA's bind service using a separate_exporter and exporting DNS
queries to opensearch. Additionally, I have rewritten the 389ds_exporter to include
cn=monitor metrics to provide more visibility into the 389 Directory Server.
I recently realized that I could also include 'cn=ldbm database' metrics, which
are low-level but could be useful in troubleshooting the issues we are facing. The
problems we are encountering are related to disk IO, and having these metrics could
provide valuable insights into the following:
1) Excessive paging out and increased swap usage without spikes in load. For example
after restarting of replica the swap usage increases to 30% (of 3GB swap space) over 1-2
days while there are at least 4GB of availiable RAM present on the host. And the general
swap consumer is ns-slapd service. For now I only tested to configure swappiness parametr
to zero, which did not help, so I guess there are some other factors involved.
2) Spikes in IO latency observed during modifying and adding operations, which were not
present when the cluster was smaller (up to 10 replicas). I need to determine whether the
issue lies with service tuning or with the cloud provider and its SAN, as we recently
migrated to SSD disks without improvement. As I said about "replication lag"
those problems just started more appearing as new replicas were added, but for now we
mostly observe it by outage of services that rely on ldap. The "waves" refers to
the way problem apprear, as different clients VDCs are having problems one after the other
which is looks like replication propagation.
3) Master-master replication just seems to me as a big "black cloud", which I
have no control or knowledge of. When you have couple of hosts it is maybe fine to rely on
documented way of looking up replicationStatus attribute, but when you have couple of
dozens I guess things could get quite not so straitforward, at least relying on intuition
suggests it. When I say about replication observability what I mean and what I'd like
to see is following:
Graph representation...
- ...of time it took to replay a change (or I guess time of full replication session)
- ...of the amount simutanialous connections that Suppliers trying to establish with
Consumer
- ...of time spent waiting to acquire replica access
I just pointed a few of the top of my head. I don't know for sure (and first post was
about it) is it really worth it to try and get those kind of metrics or I just don't
know what I'm talking about and it would be a waste of time and hard to implement. As
I mentioned bpf cause I see it as only option I could get it, the other option is to parse
logs that are in DEBUG mode which is not the option.
With replication metrics besides the ability to see its impact on the problems above,
I'm also trying to solve more administrative task - I need to convince the
architerture departament to change the model of adding new replicas. Right now we kinda
adding two replicas for every new client.
+------------------------------+
| client#1 |
| VDC |
| |
| +--------------+ | +---------------------+
+---------------------+
| | +-------------->+
+---------------->+ | ...
| | replica-01 | | | common-replica-01 | |
common-replica-02 |
| | +<--------------+
+<----------------+ |
| +--------------+ | +---------------------+
+---------------------+
| | ^ | | ^
| ^
| v | | | |
| |
| +--------------+ | | |
| |
| | | | | |
| |
| | replica-02 | | | |
| |
| | | | | |
| |
| +--------------+ | | |
| |
| | | |
| |
| | v |
v |
+------------------------------+ +---------------------+
+---------------------+
|
+---------------->+ |
| common-replica-03 | |
common-replica-04 |
|
+<----------------+ |
+---------------------+
+---------------------+
Which is not ideal at all (and as I said we started to face problems). From their side
the answer is that they are following documentation restrictions for no more than 4
replica agreements for replica and no more than 60 simutanialous replicas in master-master
replications. And for now this is indeed being followed and I need to come with deeper
analysis or find that problem lies in fine tuning the service.
So it's kind mishmash of everything at the same time, hope I answered you question.
best regards,
v.zh
_______________________________________________
389-users mailing list --389-users(a)lists.fedoraproject.org
To unsubscribe send an email to389-users-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List
Archives:https://lists.fedoraproject.org/archives/list/389-users@lists.fe...
Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
3:34 a.m.
New subject: A more profound replication monitoring of 389-ds instance
Hi,
I agree that it is complex task to master such FreeIPA deployment.
FreeIPA enables many components, 389ds is just one of them, and several
of them could contribute when a problem occurs. My main concern here is
that you express a need to monitor (how well FreeIPA deployment works)
rather than pointing a clear misbehavior of the topology that we could
focus on.
Replication is an important functionality of 389ds/FreeIPA and
monitoring replication is a common demand. One common demand is to
monitor replication lag (how much time it takes for the topology to
converge: an update is replicated on all replicas). There are several
ways to monitor that but I think an easy way is to rely on dirsrv access
logs. Each replicated update is identified uniquely with its 'csn' and
you can find csn like 'csn=57eb7dbc000000600000'. Then grepping this
value among all access logs on all instances, could give you an
indication of lag. Lag could typically be in the range of seconds or 1-2
minutes, if it spikes to many minutes or never hit some replica then you
could start investigating why it is slow.
A difficulty with that procedure is that some updates are not replicated
(fractional replication).
Investigation in replication are quite complex and difficult to explain
in general. This forum is a good place to get answers to specific questions.
best regards
Thierry
On 4/20/23 16:10, dweller dweller wrote:
Yes, I'll try to explain my needs more clearly. As it happens a
lot I recently inherited a FreeIPA installation and am now responsible for managing the
service. As someone who was not previously familiar with FreeIPA, I am in the process of
building my expertise in managing it.
When I started the monitoring setup was represented with node_exporter, process_exporter
for the host and 389ds_exporter (https://github.com/terrycain/389ds_exporter) for the ldap
data. However, as the FreeIPA installation grew in size, we started encountering issues
and realized that we lacked critical information to pinpoint the root causes of these
problems. To address this, I have taken steps to improve the monitoring setup. I have
started monitoring FreeIPA's bind service using a separate_exporter and exporting DNS
queries to opensearch. Additionally, I have rewritten the 389ds_exporter to include
cn=monitor metrics to provide more visibility into the 389 Directory Server.
I recently realized that I could also include 'cn=ldbm database' metrics, which
are low-level but could be useful in troubleshooting the issues we are facing. The
problems we are encountering are related to disk IO, and having these metrics could
provide valuable insights into the following:
1) Excessive paging out and increased swap usage without spikes in load. For example
after restarting of replica the swap usage increases to 30% (of 3GB swap space) over 1-2
days while there are at least 4GB of availiable RAM present on the host. And the general
swap consumer is ns-slapd service. For now I only tested to configure swappiness parametr
to zero, which did not help, so I guess there are some other factors involved.
2) Spikes in IO latency observed during modifying and adding operations, which were not
present when the cluster was smaller (up to 10 replicas). I need to determine whether the
issue lies with service tuning or with the cloud provider and its SAN, as we recently
migrated to SSD disks without improvement. As I said about "replication lag"
those problems just started more appearing as new replicas were added, but for now we
mostly observe it by outage of services that rely on ldap. The "waves" refers to
the way problem apprear, as different clients VDCs are having problems one after the other
which is looks like replication propagation.
3) Master-master replication just seems to me as a big "black cloud", which I
have no control or knowledge of. When you have couple of hosts it is maybe fine to rely on
documented way of looking up replicationStatus attribute, but when you have couple of
dozens I guess things could get quite not so straitforward, at least relying on intuition
suggests it. When I say about replication observability what I mean and what I'd like
to see is following:
Graph representation...
- ...of time it took to replay a change (or I guess time of full replication session)
- ...of the amount simutanialous connections that Suppliers trying to establish with
Consumer
- ...of time spent waiting to acquire replica access
I just pointed a few of the top of my head. I don't know for sure (and first post was
about it) is it really worth it to try and get those kind of metrics or I just don't
know what I'm talking about and it would be a waste of time and hard to implement. As
I mentioned bpf cause I see it as only option I could get it, the other option is to parse
logs that are in DEBUG mode which is not the option.
With replication metrics besides the ability to see its impact on the problems above,
I'm also trying to solve more administrative task - I need to convince the
architerture departament to change the model of adding new replicas. Right now we kinda
adding two replicas for every new client.
+------------------------------+
| client#1 |
| VDC |
| |
| +--------------+ | +---------------------+
+---------------------+
| | +-------------->+
+---------------->+ | ...
| | replica-01 | | | common-replica-01 | |
common-replica-02 |
| | +<--------------+
+<----------------+ |
| +--------------+ | +---------------------+
+---------------------+
| | ^ | | ^
| ^
| v | | | |
| |
| +--------------+ | | |
| |
| | | | | |
| |
| | replica-02 | | | |
| |
| | | | | |
| |
| +--------------+ | | |
| |
| | | |
| |
| | v |
v |
+------------------------------+ +---------------------+
+---------------------+
|
+---------------->+ |
| common-replica-03 | |
common-replica-04 |
|
+<----------------+ |
+---------------------+
+---------------------+
Which is not ideal at all (and as I said we started to face problems). From their side
the answer is that they are following documentation restrictions for no more than 4
replica agreements for replica and no more than 60 simutanialous replicas in master-master
replications. And for now this is indeed being followed and I need to come with deeper
analysis or find that problem lies in fine tuning the service.
So it's kind mishmash of everything at the same time, hope I answered you question.
best regards,
v.zh
_______________________________________________
389-users mailing list --389-users(a)lists.fedoraproject.org
To unsubscribe send an email to389-users-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List
Archives:https://lists.fedoraproject.org/archives/list/389-users@lists.fe...
Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
368
days inactive
370
days old
389-users@lists.fedoraproject.org
7 comments
3 participants
participants (3)
-
Christian, Mark -
dweller dweller -
Thierry Bordaz