locking performance and scalability (eye candy gnuplots inside!)
by liblfds admin
Hi, everyone.
I am the author of liblfds (a lock-free data structure library), which
is used by nunc-stans, which is used by 389.
By various events of no interest to anyone, I've ended up idling in the
#389 channel on Freenode and earlier today had a conversation with richm
(who used liblfds in nunc-stans), regarding performance, data structures
in use, locking and so on.
I was asked by richm to post the substantive content of our conversation
to this mailing list.
===
I am currently working on a benchmark application for liblfds.
This benchmarks the lock-free data structures in the library, and also
*locking* versions of those same data structures, where the locks tested
are all those available in the platform.
I've not finished yet, and so currently only have two gnuplots
available, from an earlier prototype benchmark application.
Note the 16 core image is I think it's 1800x2400 pixels. The file size
is still small (~250kb).
http://www.liblfds.org/freelist_4core.png
http://www.liblfds.org/freelist_16core.png
The CPU topology diagram can be found at the top of each image.
The four core machine is my laptop, real hardware, real Linux.
The sixteen core machine is a dedicated-hardware Amazon VM, so all bets
are off. It looks nice, but God knows what the Xen configuration is. I
can't even know there's a one-to-one mapping between virtual cores and
logical cores - I've seen cloud providers where this is *not* the case.
Turning first to the four core plot, what we see is that while we are
within the confines of a single physical core, spinlocks win. This is
because the physical core knows what its logical cores are up to.
As soon as we step outside of a single physical core, lock-free wins,
and wins big - an order of magnitude.
We see however, looking at the sad, short bars in the final graph, that
the freelist in its current form does not scale linearly with cores.
This is because of memory contention on the single cache line holding
the freelist head pointer.
There is a method to improve on this, though, something called an
elimination layer, which is based on the insight that when a thread
which pushes at or almost at the same time as a thread which pops, those
two threads can service each others request, without having to access
the freelist head pointer.
Turning to the (questionable - VM, don't forget) sixteen core plot, we
see much the same. Lock-free wins, but it's not scaling. We do see one
or two other items of note; looking at the second graph, we can see a
huge imbalance between the spinlock performances on the two cores; in
both cases, one core dominated the other and stopped it from getting
much work done.
The total amount of work done then is fairly high - because we are in
fact by this behaviour starting to approach a single-threaded scenario!
The pthread mutex absolutely does not display this behaviour, since it
serializes access and I believe internally is maintaining a queue of
requestors; so everyone gets a fair chance.
The spinlocks of course just do their thing, and then we're much more
reliant upon the hardware to provide fairness, and as we see, this may
not be a safe bet.
We also see in the third graph and fifth graphs that the first core
performed badly compared to the others. I do not know why, but it needs
explanation.
Relating to this though, although I don't have other plots, but I did do
quite a lot of benchmarking with the prototype benchmark app, and I
seemed to find on some machines that some cores *always* did badly, and
for *all* lock types. I have a weak suspicion it reflects internal
memory access arrangements in the core itself. However, this was on VMs
- although they were all dedicated hardware, so I had no neighbours on
the box - so it could just be Xen doing strange things.
I need to add RW locks to the benchmark, as they are used by nunc-stans/389.
The recent new release of liblfds included - and this was done for
implementation speed, to at least get some functionality out there ASAP
- an *addonly* list, btree and hash. The list will become normal
(read-write) soon enough, as I have a safe memory reclamation
implementation about to come out. There is now in the literature a
lock-free red-black tree, which was invented last year. Cliff Click has
had a proper lock-free hash out for some time now. Each of these will
take a month to implement, whereas the addonly versions were
straightforward, so I got them out first.
8 years, 1 month
NSACLPlugin warning/error
by ghiureai
Hi List,
I am seeing this warning/error in one of my LDAP log, ( version 1.2.11)
" NSACLPlugin - Can't find the acl in the tree for moddn
operation:olddnuid= " I would like to know more details , is this
related to an aci issue/missing etc?
Isabella
8 years, 1 month
automembership plugin questions
by Frank Munsche
Hi Guys,
I try to get the automembership plugin configured and running in the right way,
but it still causes some headaches.
I'm running CentOS 6.7 and the following 389 packages:
389-ds-base-1.2.11.15-69.el6_7.x86_64
389-ds-base-libs-1.2.11.15-69.el6_7.x86_64
I followed the guide at
http://directory.fedoraproject.org/docs/389ds/design/automember-design.html
and created a group, where the members should be automatically added:
dn: cn=users,ou=app,ou=groups,dc=example,dc=com
description: users group
objectClass: top
objectClass: groupofnames
cn: users
And the corresponding auto membership plugin entry:
dn: cn=users,cn=Auto Membership Plugin,cn=plugins,cn=config
cn: users
objectclass: top
objectclass: autoMemberDefinition
autoMemberScope: dc=example,dc=com
autoMemberFilter: Team=ops
autoMemberDefaultGroup: cn=users,ou=app,ou=groups,dc=example,dc=com
autoMemberGroupingAttr: member:dn
After that, I added a user who got the Team attribute (own schema extension)
set to 'ops'
dn: uid=foo,ou=people,dc=example,dc=com
objectClass: top
objectClass: posixAccount
objectClass: organizationalPerson
objectClass: inetorgPerson
objectClass: person
objectClass: shadowAccount
objectClass: ldapPublicKey
objectClass: bopsUser
objectClass: inetuser
uid: foo
uidNumber: 2222
gidNumber: 2222
homeDirectory: /home/foo
loginShell: /bin/bash
cn: foo bar
givenName: foo
sn: bar
gecos: foo bar
mail: foo.bar(a)example.com
Team: ops
A ldapsearch for the groups member attribute shows the user's dn being added
automatically:
# users, app, groups, bildops.de
dn: cn=users,ou=app,ou=groups,dc=example,dc=com
member: uid=foo,ou=people,dc=example,dc=com
When the user entry is removed, the membership also disappears.
But whenever I just changed the Team attribute of an existing user, the
membership did not change.
I found a page describing a rebuild task implemented in the plugin at:
https://fedorahosted.org/389/ticket/20
It worked for me for any existing users, who got the Team attribute set to
'ops'. The task finds them and puts their dn into the group.
But the opposite way did not work:
I changed the Team's value from 'ops' to 'none' and started the task, but the
user's dn is still a member of the group.
Is this the expected behavior?
thank you very much,
cheers, Frank
8 years, 1 month
ns-slapd memory usage
by Radu Pantiru
Hi,
I am using 1.3.3 and the reserved memory usage it is going up on average
~ 800MB per week and I have a cache hit rate of ~98% on both ldap
userRoot and db_stat
Is this normal behavior or possibly I have a memory leak?
Regards,
Radu
8 years, 1 month
Installation of 389 DS
by wodel youchi
Hi,
I am a newbie on 389 DS, I was following the RDS install document from
RedHat Documentation.
OS: Centos 7.2 x64 latest updates
389 DS :
389-admin-console-1.1.10-1.el7.noarch
389-ds-base-libs-1.3.4.0-26.el7_2.x86_64
389-ds-base-1.3.4.0-26.el7_2.x86_64
389-console-1.1.9-1.el7.noarch
389-ds-console-1.2.12-1.el7.noarch
389-adminutil-1.1.22-1.el7.x86_64
389-admin-1.1.42-1.el7.x86_64
In the consideration before setting up DS, it's mentioned that we need to
add this line to
*/etc/pam.d/system-authsession required /lib/security/$ISA/pam_limits.so*
After adding this line and rebooting the server, I am getting this error
when I try to login into it:
*Unknown module*
in */var/log/secure* I have
*login: PAM unable to dlopen(/lib/security/$ISA/pam_limits.so):
/lib/security/../../lib64/security/pam_limits.so: cannot open shared object
file: No such file or directory*
I did read the */etc/pam.d/system-auth* file again, and I found that there
is a line like this in it
*session required pam_limits.so*
My question is : do I need the
*session required /lib/security/$ISA/pam_limits.so*
for 389 to work properly ?
and if yes, how to avoid the above error?
if no, does
*session required pam_limits.so*
do the work?
Regards.
8 years, 1 month