Hi German,
Apologies for resurrecting an old thread. We're also experiencing something similar. We're currently running 389-ds-base-1.2.11.15-48.el6_6.x86_64
I'm afraid I don't have login privileges in order to view the details of the bug you linked. Could you please post details of how you defined an entry cache to include the whole db, and why this works?
FYI - moves are afoot re upgrading DS on a set of new servers, but in the meantime, we need to address this issue.
Thanks a lot, Trev
On 2015-02-05, 1:57 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi,
we have had several customer cases showing this behavior. In one of these cases, we have confirmed it was due to memory fragmentation after cache-trashing.
We have stopped seeing this behavior by defining an entry cache which includes the whole db (when possible, of course).
Details can be found at:
https://bugzilla.redhat.com/show_bug.cgi?id=1186512 Apparent memory leak in ns-slapd; OOM-Killer invoked
Regards,
German
----- Original Message -----
From: "David Boreham" david_list@boreham.org To: 389-users@lists.fedoraproject.org Sent: Wednesday, February 4, 2015 8:50:55 PM Subject: Re: [389-users] DS crashed /killed by OS
On 2/4/2015 11:20 AM, ghiureai wrote:
Out of memory: Kill process 2090 (ns-slapd) score 954 or sacrifice child
It wasn't clear to me from your post whether you already have a good understanding of the OOM killer behavior in the kernel. On the chance that you're not yet familiar with its ways, suggest reading, for example this article : http://unix.stackexchange.com/questions/153585/how-oom-killer-decides-which-... I mention this because it may not be the DS that is the problem (not saying that it absolutely is not, but it might not be). The OMM killer picks a process that is using a large amount of memory, and kills it in order to preserve system stability. This does not necessarily imply that the process it kills is the process that is causing the system to run out of memory. You said that the DS "crashed", but in fact the kernel killed it -- not quite the same thing!
It is also possible that the system has insufficient memory for the processes it is running, DS cache size and so on. Certainly it is worthwhile checking that the DS hasn't been inadvertently configured to use more peak memory than the machine has available.
Bottom line : there are a few potential explanations, including but not limited to a memory leak in the DS process. Some analysis will be needed to identify the cause. As a precaution, if you can -- configure more swap space on the box. This will allow more runway before the kernel starts looking for processes to kill, and hence more time to figure out what's using memory and why.
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Hi Trevor,
no problem. In fact, this issue has been investigated by the experts and it's due to fragmentation. A fix is being tested right internally but not delivered yet, to use a different allocator.
The official workaround is different to the one I have proposed. It's finally to define entry cache rather small since the fragmentation could be like
15 * size of entry cache.
So, we need something like (15 * size of entry cache ) < Available memory.
Thanks and regards,
German.
----- Original Message -----
From: "Trevor Fong" trevor.fong@ubc.ca To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Sent: Tuesday, October 20, 2015 7:09:46 PM Subject: Re: [389-users] DS crashed /killed by OS
Hi German,
Apologies for resurrecting an old thread. We're also experiencing something similar. We're currently running 389-ds-base-1.2.11.15-48.el6_6.x86_64
I'm afraid I don't have login privileges in order to view the details of the bug you linked. Could you please post details of how you defined an entry cache to include the whole db, and why this works?
FYI - moves are afoot re upgrading DS on a set of new servers, but in the meantime, we need to address this issue.
Thanks a lot, Trev
On 2015-02-05, 1:57 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi,
we have had several customer cases showing this behavior. In one of these cases, we have confirmed it was due to memory fragmentation after cache-trashing.
We have stopped seeing this behavior by defining an entry cache which includes the whole db (when possible, of course).
Details can be found at:
https://bugzilla.redhat.com/show_bug.cgi?id=1186512 Apparent memory leak in ns-slapd; OOM-Killer invoked
Regards,
German
----- Original Message -----
From: "David Boreham" david_list@boreham.org To: 389-users@lists.fedoraproject.org Sent: Wednesday, February 4, 2015 8:50:55 PM Subject: Re: [389-users] DS crashed /killed by OS
On 2/4/2015 11:20 AM, ghiureai wrote:
Out of memory: Kill process 2090 (ns-slapd) score 954 or sacrifice child
It wasn't clear to me from your post whether you already have a good understanding of the OOM killer behavior in the kernel. On the chance that you're not yet familiar with its ways, suggest reading, for example this article : http://unix.stackexchange.com/questions/153585/how-oom-killer-decides-which-... I mention this because it may not be the DS that is the problem (not saying that it absolutely is not, but it might not be). The OMM killer picks a process that is using a large amount of memory, and kills it in order to preserve system stability. This does not necessarily imply that the process it kills is the process that is causing the system to run out of memory. You said that the DS "crashed", but in fact the kernel killed it -- not quite the same thing!
It is also possible that the system has insufficient memory for the processes it is running, DS cache size and so on. Certainly it is worthwhile checking that the DS hasn't been inadvertently configured to use more peak memory than the machine has available.
Bottom line : there are a few potential explanations, including but not limited to a memory leak in the DS process. Some analysis will be needed to identify the cause. As a precaution, if you can -- configure more swap space on the box. This will allow more runway before the kernel starts looking for processes to kill, and hence more time to figure out what's using memory and why.
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Hi German,
Thanks very much for your reply. Just to make sure I have it straight, I’ve currently got userRoot’s nsslapd-cachememsize = 6 GB on at 16GB machine. I should change that to nsslapd-cachememsize = 6 GB / 15 = 429496730 Do I have that right?
Thanks again, Trev
On 2015-10-20, 10:23 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi Trevor,
no problem. In fact, this issue has been investigated by the experts and it's due to fragmentation. A fix is being tested right internally but not delivered yet, to use a different allocator.
The official workaround is different to the one I have proposed. It's finally to define entry cache rather small since the fragmentation could be like
15 * size of entry cache.
So, we need something like (15 * size of entry cache ) < Available memory.
Thanks and regards,
German.
----- Original Message -----
From: "Trevor Fong" trevor.fong@ubc.ca To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Sent: Tuesday, October 20, 2015 7:09:46 PM Subject: Re: [389-users] DS crashed /killed by OS
Hi German,
Apologies for resurrecting an old thread. We're also experiencing something similar. We're currently running 389-ds-base-1.2.11.15-48.el6_6.x86_64
I'm afraid I don't have login privileges in order to view the details of the bug you linked. Could you please post details of how you defined an entry cache to include the whole db, and why this works?
FYI - moves are afoot re upgrading DS on a set of new servers, but in the meantime, we need to address this issue.
Thanks a lot, Trev
On 2015-02-05, 1:57 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi,
we have had several customer cases showing this behavior. In one of these cases, we have confirmed it was due to memory fragmentation after cache-trashing.
We have stopped seeing this behavior by defining an entry cache which includes the whole db (when possible, of course).
Details can be found at:
https://bugzilla.redhat.com/show_bug.cgi?id=1186512 Apparent memory leak in ns-slapd; OOM-Killer invoked
Regards,
German
----- Original Message -----
From: "David Boreham" david_list@boreham.org To: 389-users@lists.fedoraproject.org Sent: Wednesday, February 4, 2015 8:50:55 PM Subject: Re: [389-users] DS crashed /killed by OS
On 2/4/2015 11:20 AM, ghiureai wrote:
Out of memory: Kill process 2090 (ns-slapd) score 954 or sacrifice child
It wasn't clear to me from your post whether you already have a good understanding of the OOM killer behavior in the kernel. On the chance that you're not yet familiar with its ways, suggest reading, for example this article : http://unix.stackexchange.com/questions/153585/how-oom-killer-decides-which-... I mention this because it may not be the DS that is the problem (not saying that it absolutely is not, but it might not be). The OMM killer picks a process that is using a large amount of memory, and kills it in order to preserve system stability. This does not necessarily imply that the process it kills is the process that is causing the system to run out of memory. You said that the DS "crashed", but in fact the kernel killed it -- not quite the same thing!
It is also possible that the system has insufficient memory for the processes it is running, DS cache size and so on. Certainly it is worthwhile checking that the DS hasn't been inadvertently configured to use more peak memory than the machine has available.
Bottom line : there are a few potential explanations, including but not limited to a memory leak in the DS process. Some analysis will be needed to identify the cause. As a precaution, if you can -- configure more swap space on the box. This will allow more runway before the kernel starts looking for processes to kill, and hence more time to figure out what's using memory and why.
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Hi Trevor,
400Mb could be a more reasonable value. With a cache of 6gb, fragmentation could very quickly provoke the OOM killer error.
Regards,
German.
----- Original Message -----
From: "Trevor Fong" trevor.fong@ubc.ca To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Sent: Tuesday, October 20, 2015 7:44:06 PM Subject: Re: [389-users] DS crashed /killed by OS
Hi German,
Thanks very much for your reply. Just to make sure I have it straight, I’ve currently got userRoot’s nsslapd-cachememsize = 6 GB on at 16GB machine. I should change that to nsslapd-cachememsize = 6 GB / 15 = 429496730 Do I have that right?
Thanks again, Trev
On 2015-10-20, 10:23 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi Trevor,
no problem. In fact, this issue has been investigated by the experts and it's due to fragmentation. A fix is being tested right internally but not delivered yet, to use a different allocator.
The official workaround is different to the one I have proposed. It's finally to define entry cache rather small since the fragmentation could be like
15 * size of entry cache.
So, we need something like (15 * size of entry cache ) < Available memory.
Thanks and regards,
German.
----- Original Message -----
From: "Trevor Fong" trevor.fong@ubc.ca To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Sent: Tuesday, October 20, 2015 7:09:46 PM Subject: Re: [389-users] DS crashed /killed by OS
Hi German,
Apologies for resurrecting an old thread. We're also experiencing something similar. We're currently running 389-ds-base-1.2.11.15-48.el6_6.x86_64
I'm afraid I don't have login privileges in order to view the details of the bug you linked. Could you please post details of how you defined an entry cache to include the whole db, and why this works?
FYI - moves are afoot re upgrading DS on a set of new servers, but in the meantime, we need to address this issue.
Thanks a lot, Trev
On 2015-02-05, 1:57 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi,
we have had several customer cases showing this behavior. In one of these cases, we have confirmed it was due to memory fragmentation after cache-trashing.
We have stopped seeing this behavior by defining an entry cache which includes the whole db (when possible, of course).
Details can be found at:
https://bugzilla.redhat.com/show_bug.cgi?id=1186512 Apparent memory leak in ns-slapd; OOM-Killer invoked
Regards,
German
----- Original Message -----
From: "David Boreham" david_list@boreham.org To: 389-users@lists.fedoraproject.org Sent: Wednesday, February 4, 2015 8:50:55 PM Subject: Re: [389-users] DS crashed /killed by OS
On 2/4/2015 11:20 AM, ghiureai wrote:
Out of memory: Kill process 2090 (ns-slapd) score 954 or sacrifice child
It wasn't clear to me from your post whether you already have a good understanding of the OOM killer behavior in the kernel. On the chance that you're not yet familiar with its ways, suggest reading, for example this article : http://unix.stackexchange.com/questions/153585/how-oom-killer-decides-which-... I mention this because it may not be the DS that is the problem (not saying that it absolutely is not, but it might not be). The OMM killer picks a process that is using a large amount of memory, and kills it in order to preserve system stability. This does not necessarily imply that the process it kills is the process that is causing the system to run out of memory. You said that the DS "crashed", but in fact the kernel killed it -- not quite the same thing!
It is also possible that the system has insufficient memory for the processes it is running, DS cache size and so on. Certainly it is worthwhile checking that the DS hasn't been inadvertently configured to use more peak memory than the machine has available.
Bottom line : there are a few potential explanations, including but not limited to a memory leak in the DS process. Some analysis will be needed to identify the cause. As a precaution, if you can -- configure more swap space on the box. This will allow more runway before the kernel starts looking for processes to kill, and hence more time to figure out what's using memory and why.
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Hi German,
Thanks for your suggestion. I’m happy to confirm that setting userRoot’s nsslapd-cachememsize: 429496730 (1/15th of previous value of 6 GB) has addressed the memory issue for now, and % Mem for the ns-slapd process seems to be at a manageable level.
Thanks very much, Trev
On 2015-10-20, 11:07 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi Trevor,
400Mb could be a more reasonable value. With a cache of 6gb, fragmentation could very quickly provoke the OOM killer error.
Regards,
German.
----- Original Message -----
From: "Trevor Fong" trevor.fong@ubc.ca To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Sent: Tuesday, October 20, 2015 7:44:06 PM Subject: Re: [389-users] DS crashed /killed by OS
Hi German,
Thanks very much for your reply. Just to make sure I have it straight, I’ve currently got userRoot’s nsslapd-cachememsize = 6 GB on at 16GB machine. I should change that to nsslapd-cachememsize = 6 GB / 15 = 429496730 Do I have that right?
Thanks again, Trev
On 2015-10-20, 10:23 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi Trevor,
no problem. In fact, this issue has been investigated by the experts and it's due to fragmentation. A fix is being tested right internally but not delivered yet, to use a different allocator.
The official workaround is different to the one I have proposed. It's finally to define entry cache rather small since the fragmentation could be like
15 * size of entry cache.
So, we need something like (15 * size of entry cache ) < Available memory.
Thanks and regards,
German.
----- Original Message -----
From: "Trevor Fong" trevor.fong@ubc.ca To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Sent: Tuesday, October 20, 2015 7:09:46 PM Subject: Re: [389-users] DS crashed /killed by OS
Hi German,
Apologies for resurrecting an old thread. We're also experiencing something similar. We're currently running 389-ds-base-1.2.11.15-48.el6_6.x86_64
I'm afraid I don't have login privileges in order to view the details of the bug you linked. Could you please post details of how you defined an entry cache to include the whole db, and why this works?
FYI - moves are afoot re upgrading DS on a set of new servers, but in the meantime, we need to address this issue.
Thanks a lot, Trev
On 2015-02-05, 1:57 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi,
we have had several customer cases showing this behavior. In one of these cases, we have confirmed it was due to memory fragmentation after cache-trashing.
We have stopped seeing this behavior by defining an entry cache which includes the whole db (when possible, of course).
Details can be found at:
https://bugzilla.redhat.com/show_bug.cgi?id=1186512 Apparent memory leak in ns-slapd; OOM-Killer invoked
Regards,
German
----- Original Message -----
From: "David Boreham" david_list@boreham.org To: 389-users@lists.fedoraproject.org Sent: Wednesday, February 4, 2015 8:50:55 PM Subject: Re: [389-users] DS crashed /killed by OS
On 2/4/2015 11:20 AM, ghiureai wrote:
Out of memory: Kill process 2090 (ns-slapd) score 954 or sacrifice child
It wasn't clear to me from your post whether you already have a good understanding of the OOM killer behavior in the kernel. On the chance that you're not yet familiar with its ways, suggest reading, for example this article : http://unix.stackexchange.com/questions/153585/how-oom-killer-decides-which-... I mention this because it may not be the DS that is the problem (not saying that it absolutely is not, but it might not be). The OMM killer picks a process that is using a large amount of memory, and kills it in order to preserve system stability. This does not necessarily imply that the process it kills is the process that is causing the system to run out of memory. You said that the DS "crashed", but in fact the kernel killed it -- not quite the same thing!
It is also possible that the system has insufficient memory for the processes it is running, DS cache size and so on. Certainly it is worthwhile checking that the DS hasn't been inadvertently configured to use more peak memory than the machine has available.
Bottom line : there are a few potential explanations, including but not limited to a memory leak in the DS process. Some analysis will be needed to identify the cause. As a precaution, if you can -- configure more swap space on the box. This will allow more runway before the kernel starts looking for processes to kill, and hence more time to figure out what's using memory and why.
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Hi Trevor,
good to know it's working fine. Thanks for your feedback.
Regards,
German.
----- Original Message -----
From: "Trevor Fong" trevor.fong@ubc.ca To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Sent: Thursday, October 22, 2015 7:48:48 PM Subject: Re: [389-users] DS crashed /killed by OS
Hi German,
Thanks for your suggestion. I’m happy to confirm that setting userRoot’s nsslapd-cachememsize: 429496730 (1/15th of previous value of 6 GB) has addressed the memory issue for now, and % Mem for the ns-slapd process seems to be at a manageable level.
Thanks very much, Trev
On 2015-10-20, 11:07 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi Trevor,
400Mb could be a more reasonable value. With a cache of 6gb, fragmentation could very quickly provoke the OOM killer error.
Regards,
German.
----- Original Message -----
From: "Trevor Fong" trevor.fong@ubc.ca To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Sent: Tuesday, October 20, 2015 7:44:06 PM Subject: Re: [389-users] DS crashed /killed by OS
Hi German,
Thanks very much for your reply. Just to make sure I have it straight, I’ve currently got userRoot’s nsslapd-cachememsize = 6 GB on at 16GB machine. I should change that to nsslapd-cachememsize = 6 GB / 15 = 429496730 Do I have that right?
Thanks again, Trev
On 2015-10-20, 10:23 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi Trevor,
no problem. In fact, this issue has been investigated by the experts and it's due to fragmentation. A fix is being tested right internally but not delivered yet, to use a different allocator.
The official workaround is different to the one I have proposed. It's finally to define entry cache rather small since the fragmentation could be like
15 * size of entry cache.
So, we need something like (15 * size of entry cache ) < Available memory.
Thanks and regards,
German.
----- Original Message -----
From: "Trevor Fong" trevor.fong@ubc.ca To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Sent: Tuesday, October 20, 2015 7:09:46 PM Subject: Re: [389-users] DS crashed /killed by OS
Hi German,
Apologies for resurrecting an old thread. We're also experiencing something similar. We're currently running 389-ds-base-1.2.11.15-48.el6_6.x86_64
I'm afraid I don't have login privileges in order to view the details of the bug you linked. Could you please post details of how you defined an entry cache to include the whole db, and why this works?
FYI - moves are afoot re upgrading DS on a set of new servers, but in the meantime, we need to address this issue.
Thanks a lot, Trev
On 2015-02-05, 1:57 AM, "389-users-bounces@lists.fedoraproject.org on behalf of German Parente" <389-users-bounces@lists.fedoraproject.org on behalf of gparente@redhat.com> wrote:
Hi,
we have had several customer cases showing this behavior. In one of these cases, we have confirmed it was due to memory fragmentation after cache-trashing.
We have stopped seeing this behavior by defining an entry cache which includes the whole db (when possible, of course).
Details can be found at:
https://bugzilla.redhat.com/show_bug.cgi?id=1186512 Apparent memory leak in ns-slapd; OOM-Killer invoked
Regards,
German
----- Original Message ----- > From: "David Boreham" david_list@boreham.org > To: 389-users@lists.fedoraproject.org > Sent: Wednesday, February 4, 2015 8:50:55 PM > Subject: Re: [389-users] DS crashed /killed by OS > > On 2/4/2015 11:20 AM, ghiureai wrote: > > > > Out of memory: Kill process 2090 (ns-slapd) score 954 or sacrifice > child > > It wasn't clear to me from your post whether you already have a good > understanding of the OOM killer behavior in the kernel. > On the chance that you're not yet familiar with its ways, suggest > reading, > for example this article : > http://unix.stackexchange.com/questions/153585/how-oom-killer-decides-which-... > I mention this because it may not be the DS that is the problem (not > saying > that it absolutely is not, but it might not be). > The OMM killer picks a process that is using a large amount of > memory, > and > kills it in order to preserve system stability. > This does not necessarily imply that the process it kills is the > process > that > is causing the system to run out of memory. > You said that the DS "crashed", but in fact the kernel killed it -- > not > quite > the same thing! > > It is also possible that the system has insufficient memory for the > processes > it is running, DS cache size and so on. > Certainly it is worthwhile checking that the DS hasn't been > inadvertently > configured to use more peak memory than the machine has available. > > Bottom line : there are a few potential explanations, including but > not > limited to a memory leak in the DS process. > Some analysis will be needed to identify the cause. As a precaution, > if > you > can -- configure more swap space on the box. > This will allow more runway before the kernel starts looking for > processes > to > kill, and hence more time to figure out what's using memory and why. > > > > > > > -- > 389 users mailing list > 389-users@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/389-users
389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
On Thu, 2015-10-22 at 17:48 +0000, Fong, Trevor wrote:
Hi German,
Thanks for your suggestion. I’m happy to confirm that setting userRoot’s nsslapd-cachememsize: 429496730 (1/15th of previous value of 6 GB) has addressed the memory issue for now, and % Mem for the ns -slapd process seems to be at a manageable level.
Thanks very much, Trev
As I understand it, the fragmentation is due to the use of fastbins. see man mallopt M_MXFAST for an explination.
You may be able to reduce fragmentation with the setting nsslapd-malloc -mxfast, but you may see a (potentially severe) degredation in performance. As I understand the value is by default 64 on a 32 bit system, and 128 on a 64bit one, so perhaps try reducing it by half and see if that helps.
I'm not sure if this is a supported option either so you may not wish to enable it. You should always try changes like this on a non -production system first.
Alternatelly, you can set the cachemem to autosize with nsslapd-cache -autosize=50 or something like that. This way the cache will use only 50% of the free ram on the system. I believe this value is determined at server start up, rather than being constantly adjusted through the lifetime of the process.
Remember, that with the caching, there is some good material in the tuning guide which may help you understand the correct values you should set for your cache sizes based on the number of entries you have.
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/ 10/html/Performance_Tuning_Guide/index.html
As Germane said, there is work to reduce the impace of memory fragmentation on process memory size, so these are hopefully temporary solutions.
Sincerely,
William Brown Software Engineer Red Hat, Brisbane
On 11/01/2015 08:50 PM, William Brown wrote:
On Thu, 2015-10-22 at 17:48 +0000, Fong, Trevor wrote:
Hi German,
Thanks for your suggestion. I’m happy to confirm that setting userRoot’s nsslapd-cachememsize: 429496730 (1/15th of previous value of 6 GB) has addressed the memory issue for now, and % Mem for the ns -slapd process seems to be at a manageable level.
Thanks very much, Trev
As I understand it, the fragmentation is due to the use of fastbins. see man mallopt M_MXFAST for an explination.
You may be able to reduce fragmentation with the setting nsslapd-malloc -mxfast, but you may see a (potentially severe) degredation in performance. As I understand the value is by default 64 on a 32 bit system, and 128 on a 64bit one, so perhaps try reducing it by half and see if that helps.
I'm not sure if this is a supported option either so you may not wish to enable it. You should always try changes like this on a non -production system first.
Well we have not seen any significant improvement modifying the fast bins(M_MXFAST). So while it can slightly reduce fragmentation, unfortunately it's not really a solution. Now using a different memory allocator, like jemalloc, has shown significant improvements in memory size/fragmentation. Checkout:
http://www.port389.org/docs/389ds/FAQ/jemalloc-testing.html
The only issue is that jemalloc is not available on all platforms yet(especially older versions of RHEL/fedora).
Mark
Alternatelly, you can set the cachemem to autosize with nsslapd-cache -autosize=50 or something like that. This way the cache will use only 50% of the free ram on the system. I believe this value is determined at server start up, rather than being constantly adjusted through the lifetime of the process.
Remember, that with the caching, there is some good material in the tuning guide which may help you understand the correct values you should set for your cache sizes based on the number of entries you have.
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/ 10/html/Performance_Tuning_Guide/index.html
As Germane said, there is work to reduce the impace of memory fragmentation on process memory size, so these are hopefully temporary solutions.
Sincerely,
William Brown Software Engineer Red Hat, Brisbane
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
389-users@lists.fedoraproject.org
-
Fong, Trevor -
German Parente -
Mark Reynolds -
William Brown