3:13 p.m.
Jonathan Vaughn wrote:
Here's the output from ipa-replica-install :
# ipa-replica-install
WARNING: conflicting time&date synchronization service 'chronyd' will
be disabled in favor of ntpd
Password for admin(a)COMPANY.INTERNAL:
Run connection check to master
Connection check OK
Configuring NTP daemon (ntpd)
[1/4]: stopping ntpd
[2/4]: writing configuration
[3/4]: configuring ntpd to start on boot
[4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30 seconds
[1/41]: creating directory server instance
[2/41]: enabling ldapi
[3/41]: configure autobind for root
[4/41]: stopping directory server
[5/41]: updating configuration in dse.ldif
[6/41]: starting directory server
[7/41]: adding default schema
[8/41]: enabling memberof plugin
[9/41]: enabling winsync plugin
[10/41]: configuring replication version plugin
[11/41]: enabling IPA enrollment plugin
[12/41]: configuring uniqueness plugin
[13/41]: configuring uuid plugin
[14/41]: configuring modrdn plugin
[15/41]: configuring DNS plugin
[16/41]: enabling entryUSN plugin
[17/41]: configuring lockout plugin
[18/41]: configuring topology plugin
[19/41]: creating indices
[20/41]: enabling referential integrity plugin
[21/41]: configuring certmap.conf
[22/41]: configure new location for managed entries
[23/41]: configure dirsrv ccache
[24/41]: enabling SASL mapping fallback
[25/41]: restarting directory server
[26/41]: creating DS keytab
[27/41]: ignore time skew for initial replication
[28/41]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 11 seconds elapsed
Update succeeded
[29/41]: prevent time skew after initial replication
[30/41]: adding sasl mappings to the directory
[31/41]: updating schema
[32/41]: setting Auto Member configuration
[33/41]: enabling S4U2Proxy delegation
[error] NetworkError: cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
ipapython.admintool: ERROR The ipa-replica-install command failed.
See /var/log/ipareplica-install.log for more information
And here's the /var/log/ipareplica-install.log from just before it fails
to the failure:
2018-05-02T00:22:44Z DEBUG [32/41]: setting Auto Member configuration
2018-05-02T00:22:44Z DEBUG Starting external process
2018-05-02T00:22:44Z DEBUG args=/usr/bin/ldapmodify -v -f
/tmp/tmpt7bxf4x1 -H ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket
-Y EXTERNAL
2018-05-02T00:22:45Z DEBUG Process finished, return code=0
2018-05-02T00:22:45Z DEBUG stdout=add nsslapd-pluginConfigArea:
cn=automember,cn=etc,dc=company,dc=internal
modifying entry "cn=Auto Membership Plugin,cn=plugins,cn=config"
modify complete
2018-05-02T00:22:45Z DEBUG stderr=ldap_initialize(
ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket/??base )
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
2018-05-02T00:22:45Z DEBUG duration: 0 seconds
2018-05-02T00:22:45Z DEBUG [33/41]: enabling S4U2Proxy delegation
2018-05-02T00:22:45Z DEBUG Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
979, in error_handler
yield
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
1572, in update_entry
self.conn.modify_s(str(entry.dn), modlist)
File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 600,
in modify_s
return self.modify_ext_s(dn,modlist,None,None)
File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 573,
in modify_ext_s
resp_type, resp_data, resp_msgid, resp_ctrls =
self.result3(msgid,all=1,timeout=self.timeout)
File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 714,
in result3
resp_ctrl_classes=resp_ctrl_classes
File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 721,
in result4
ldap_result =
self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 294,
in _ldap_call
result = func(*args,**kwargs)
ldap.SERVER_DOWN: {'desc': "Can't contact LDAP server"}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
line 506, in start_creation
run_step(full_msg, method)
File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
line 496, in run_step
method()
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py", line
977, in __setup_s4u2proxy
__add_principal('ipa-http-delegation', 'HTTP', self)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py", line
973, in __add_principal
api.Backend.ldap2.update_entry(entry)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
1572, in update_entry
self.conn.modify_s(str(entry.dn), modlist)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
1028, in error_handler
error=info)
ipalib.errors.NetworkError: cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
2018-05-02T00:22:45Z DEBUG [error] NetworkError: cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
2018-05-02T00:22:45Z DEBUG Destroyed connection context.ldap2_2991558640
2018-05-02T00:22:45Z DEBUG Backing up system configuration file
'/etc/ipa/default.conf'
2018-05-02T00:22:45Z DEBUG Saving Index File to
'/var/lib/ipa/sysrestore/sysrestore.index'
2018-05-02T00:22:45Z DEBUG File
"/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 174, in
execute
return_value = self.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py",
line 319, in run
cfgr.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 364, in run
self.execute()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 388, in execute
for _nothing in self._executor():
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 430, in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 459, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 449, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 420, in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 417, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 654, in _configure
next(executor)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 430, in __runner
exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 459, in _handle_execute_exception
self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 517, in _handle_exception
self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 449, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 514, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 449, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 420, in __runner
step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
line 417, in <lambda>
step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
line 81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
line 59, in run_generator_with_yield_from
value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/common.py",
line 66, in _install
for unused in self._installer(self.parent):
File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py",
line 622, in main
replica_install(self)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
line 388, in decorated
func(installer)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
line 1407, in install
pkcs12_info=dirsrv_pkcs12_info)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
line 110, in install_replica_ds
setup_pkinit=not options.no_pkinit,
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py", line
419, in create_replica
self.start_creation(runtime=30)
File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
line 506, in start_creation
run_step(full_msg, method)
File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
line 496, in run_step
method()
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py", line
977, in __setup_s4u2proxy
__add_principal('ipa-http-delegation', 'HTTP', self)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py", line
973, in __add_principal
api.Backend.ldap2.update_entry(entry)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
1572, in update_entry
self.conn.modify_s(str(entry.dn), modlist)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
1028, in error_handler
error=info)
2018-05-02T00:22:45Z DEBUG The ipa-replica-install command failed,
exception: NetworkError: cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
2018-05-02T00:22:45Z ERROR cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
2018-05-02T00:22:45Z ERROR The ipa-replica-install command failed. See
/var/log/ipareplica-install.log for more information
Maybe it just is trying to soon and it hasn't fully started yet? Because
now I see that the previous step ALSO shows the URL encoded socket path,
and it worked fine - but it looks like the previous step used
ldapmodify, but the failing step was accessing LDAP from Python? Perhaps
it works fine one way and not the other ?
I won't rule it out but I doubt it. Is ns-slapd still running at this
point? This could happen, for example, if ns-slapd crashed after
enabling the automember plugin. The DS error log might have something to
say as well.
rob
On Wed, May 2, 2018 at 3:29 PM, Rob Crittenden <rcritten(a)redhat.com
<mailto:rcritten@redhat.com>> wrote:
Jonathan Vaughn via FreeIPA-users wrote:
Yes, I know, not recommended etc, low performance. I'm not going
to run the CA on it. I just want to have a backup LDAP/Kerberos
server.
Right now I'm just trying to test things out. I've got a master
and a replica (so you could say two masters I suppose) running
in Virtualbox VMs, and I'm trying to set up a 3rd replica on a
Pi. All are Fedors 27. I had to downgrade httpd due to
https://pagure.io/freeipa/issue/7493
<https://pagure.io/freeipa/issue/7493> to even set up the first
VM replica, but this issue is separate.
Currently, the problem is it can't connect to it's own LDAP
instance due to some kind of error ... ipa-replica-install
worked fine on the x86_64 VM but on the armv71 Pi 3B when it
tries to connect to LDAPI instead of
using 'ldapi:///var/run//slapd-COMPANY-INTERNAL.socket' it
uses 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket'.
So it seems there is yet another ARM (or non-x86_64) bug ...
similar to the problem with httpd and passing the KRB5CCNAME
properly https://pagure.io/freeipa/issue/7337
<https://pagure.io/freeipa/issue/7337>
Any ideas on where to look to patch in a fix to this so it uses
the correct filename? The socket file is there ... and (at the
time it tries) LDAP is running.
What makes you think the ldapi URI is the problem?
Can you share the logs?
rob
4:02 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
It's still running.
Here's the error log from slapd during that run:
[01/May/2018:19:22:24.567650453 -0500] - ERR - NSMMReplicationPlugin -
multimaster_extop_StartNSDS50ReplicationRequest - conn=42 op=5
replica="dc=company,dc=internal": Unable to acquire replica: error:
permission denied
[01/May/2018:19:22:24.613143629 -0500] - ERR - NSMMReplicationPlugin -
multimaster_extop_StartNSDS50ReplicationRequest - conn=43 op=5
replica="dc=company,dc=internal": Unable to acquire replica: error:
permission denied
[01/May/2018:19:22:31.445153171 -0500] - INFO - NSMMReplicationPlugin -
repl5_tot_run - Beginning total update of replica
"agmt="cn=meToipa-11.company.internal" (ipa-11:389)".
[01/May/2018:19:22:31.672485209 -0500] - NOTICE - NSMMReplicationPlugin -
replica_subentry_check - Need to create replication keep alive entry
<cn=repl keep alive 7,dc=company,dc=internal>
[01/May/2018:19:22:31.674922201 -0500] - INFO - NSMMReplicationPlugin -
replica_subentry_create - add dn: cn=repl keep alive
7,dc=company,dc=internal
objectclass: top
objectclass: ldapsubentry
objectclass: extensibleObject
cn: repl keep alive 7
[01/May/2018:19:22:36.125109819 -0500] - ERR - NSMMReplicationPlugin -
multimaster_extop_StartNSDS50ReplicationRequest - conn=44 op=5
replica="dc=company,dc=internal": Unable to acquire replica: error:
permission denied
[01/May/2018:19:22:36.598178749 -0500] - INFO - NSMMReplicationPlugin -
repl5_tot_run - Finished total update of replica
"agmt="cn=meToipa-11.company.internal" (ipa-11:389)". Sent 503
entries.
[01/May/2018:19:27:40.482836683 -0500] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=meToipa-11.company.internal" (ipa-11:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP
server) ()
ipa-11 is the Pi I'm trying to set up replication on. Looks like there was
a problem setting up replication due to permissions? But the other replica
started up just fine ...
On Thu, May 3, 2018 at 3:13 PM, Rob Crittenden <rcritten(a)redhat.com> wrote:
Jonathan Vaughn wrote:
> Here's the output from ipa-replica-install :
>
> # ipa-replica-install
> WARNING: conflicting time&date synchronization service 'chronyd' will
> be disabled in favor of ntpd
>
> Password for admin(a)COMPANY.INTERNAL:
> Run connection check to master
> Connection check OK
> Configuring NTP daemon (ntpd)
> [1/4]: stopping ntpd
> [2/4]: writing configuration
> [3/4]: configuring ntpd to start on boot
> [4/4]: starting ntpd
> Done configuring NTP daemon (ntpd).
> Configuring directory server (dirsrv). Estimated time: 30 seconds
> [1/41]: creating directory server instance
> [2/41]: enabling ldapi
> [3/41]: configure autobind for root
> [4/41]: stopping directory server
> [5/41]: updating configuration in dse.ldif
> [6/41]: starting directory server
> [7/41]: adding default schema
> [8/41]: enabling memberof plugin
> [9/41]: enabling winsync plugin
> [10/41]: configuring replication version plugin
> [11/41]: enabling IPA enrollment plugin
> [12/41]: configuring uniqueness plugin
> [13/41]: configuring uuid plugin
> [14/41]: configuring modrdn plugin
> [15/41]: configuring DNS plugin
> [16/41]: enabling entryUSN plugin
> [17/41]: configuring lockout plugin
> [18/41]: configuring topology plugin
> [19/41]: creating indices
> [20/41]: enabling referential integrity plugin
> [21/41]: configuring certmap.conf
> [22/41]: configure new location for managed entries
> [23/41]: configure dirsrv ccache
> [24/41]: enabling SASL mapping fallback
> [25/41]: restarting directory server
> [26/41]: creating DS keytab
> [27/41]: ignore time skew for initial replication
> [28/41]: setting up initial replication
> Starting replication, please wait until this has completed.
> Update in progress, 11 seconds elapsed
> Update succeeded
>
> [29/41]: prevent time skew after initial replication
> [30/41]: adding sasl mappings to the directory
> [31/41]: updating schema
> [32/41]: setting Auto Member configuration
> [33/41]: enabling S4U2Proxy delegation
> [error] NetworkError: cannot connect to 'ldapi://%2Fvar%2Frun%2Fslapd-
> COMPANY-INTERNAL.socket':
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
> ipapython.admintool: ERROR cannot connect to
> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
> ipapython.admintool: ERROR The ipa-replica-install command failed. See
> /var/log/ipareplica-install.log for more information
>
> And here's the /var/log/ipareplica-install.log from just before it fails
> to the failure:
>
> 2018-05-02T00:22:44Z DEBUG [32/41]: setting Auto Member configuration
> 2018-05-02T00:22:44Z DEBUG Starting external process
> 2018-05-02T00:22:44Z DEBUG args=/usr/bin/ldapmodify -v -f
> /tmp/tmpt7bxf4x1 -H ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket
> -Y EXTERNAL
> 2018-05-02T00:22:45Z DEBUG Process finished, return code=0
> 2018-05-02T00:22:45Z DEBUG stdout=add nsslapd-pluginConfigArea:
> cn=automember,cn=etc,dc=company,dc=internal
> modifying entry "cn=Auto Membership Plugin,cn=plugins,cn=config"
> modify complete
>
>
> 2018-05-02T00:22:45Z DEBUG stderr=ldap_initialize(
> ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket/??base )
> SASL/EXTERNAL authentication started
> SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> SASL SSF: 0
>
> 2018-05-02T00:22:45Z DEBUG duration: 0 seconds
> 2018-05-02T00:22:45Z DEBUG [33/41]: enabling S4U2Proxy delegation
> 2018-05-02T00:22:45Z DEBUG Traceback (most recent call last):
> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
> 979, in error_handler
> yield
> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
> 1572, in update_entry
> self.conn.modify_s(str(entry.dn), modlist)
> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 600,
> in modify_s
> return self.modify_ext_s(dn,modlist,None,None)
> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 573,
> in modify_ext_s
> resp_type, resp_data, resp_msgid, resp_ctrls =
> self.result3(msgid,all=1,timeout=self.timeout)
> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 714,
> in result3
> resp_ctrl_classes=resp_ctrl_classes
> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 721,
> in result4
> ldap_result = self._ldap_call(self._l.result
> 4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line 294,
> in _ldap_call
> result = func(*args,**kwargs)
> ldap.SERVER_DOWN: {'desc': "Can't contact LDAP server"}
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
> line 506, in start_creation
> run_step(full_msg, method)
> File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
> line 496, in run_step
> method()
> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
> line 977, in __setup_s4u2proxy
> __add_principal('ipa-http-delegation', 'HTTP', self)
> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
> line 973, in __add_principal
> api.Backend.ldap2.update_entry(entry)
> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
> 1572, in update_entry
> self.conn.modify_s(str(entry.dn), modlist)
> File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
> self.gen.throw(type, value, traceback)
> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
> 1028, in error_handler
> error=info)
> ipalib.errors.NetworkError: cannot connect to
> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>
> 2018-05-02T00:22:45Z DEBUG [error] NetworkError: cannot connect to
> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
> 2018-05-02T00:22:45Z DEBUG Destroyed connection context.ldap2_2991558640
> 2018-05-02T00:22:45Z DEBUG Backing up system configuration file
> '/etc/ipa/default.conf'
> 2018-05-02T00:22:45Z DEBUG Saving Index File to
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2018-05-02T00:22:45Z DEBUG File
"/usr/lib/python3.6/site-packages/ipapython/admintool.py",
> line 174, in execute
> return_value = self.run()
> File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py",
> line 319, in run
> cfgr.run()
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 364, in run
> self.execute()
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 388, in execute
> for _nothing in self._executor():
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 430, in __runner
> exc_handler(exc_info)
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 459, in _handle_execute_exception
> self._handle_exception(exc_info)
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 449, in _handle_exception
> six.reraise(*exc_info)
> File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
> raise value
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 420, in __runner
> step()
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 417, in <lambda>
> step = lambda: next(self.__gen)
> File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
> line 81, in run_generator_with_yield_from
> six.reraise(*exc_info)
> File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
> raise value
> File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
> line 59, in run_generator_with_yield_from
> value = gen.send(prev_value)
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 654, in _configure
> next(executor)
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 430, in __runner
> exc_handler(exc_info)
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 459, in _handle_execute_exception
> self._handle_exception(exc_info)
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 517, in _handle_exception
> self.__parent._handle_exception(exc_info)
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 449, in _handle_exception
> six.reraise(*exc_info)
> File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
> raise value
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 514, in _handle_exception
> super(ComponentBase, self)._handle_exception(exc_info)
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 449, in _handle_exception
> six.reraise(*exc_info)
> File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
> raise value
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 420, in __runner
> step()
> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
> line 417, in <lambda>
> step = lambda: next(self.__gen)
> File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
> line 81, in run_generator_with_yield_from
> six.reraise(*exc_info)
> File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
> raise value
> File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
> line 59, in run_generator_with_yield_from
> value = gen.send(prev_value)
> File "/usr/lib/python3.6/site-packages/ipapython/install/common.py",
> line 66, in _install
> for unused in self._installer(self.parent):
> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py",
> line 622, in main
> replica_install(self)
> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
> line 388, in decorated
> func(installer)
> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
> line 1407, in install
> pkcs12_info=dirsrv_pkcs12_info)
> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
> line 110, in install_replica_ds
> setup_pkinit=not options.no_pkinit,
> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
> line 419, in create_replica
> self.start_creation(runtime=30)
> File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
> line 506, in start_creation
> run_step(full_msg, method)
> File "/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
> line 496, in run_step
> method()
> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
> line 977, in __setup_s4u2proxy
> __add_principal('ipa-http-delegation', 'HTTP', self)
> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
> line 973, in __add_principal
> api.Backend.ldap2.update_entry(entry)
> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
> 1572, in update_entry
> self.conn.modify_s(str(entry.dn), modlist)
> File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
> self.gen.throw(type, value, traceback)
> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
> 1028, in error_handler
> error=info)
>
> 2018-05-02T00:22:45Z DEBUG The ipa-replica-install command failed,
> exception: NetworkError: cannot connect to 'ldapi://%2Fvar%2Frun%2Fslapd-
> COMPANY-INTERNAL.socket':
> 2018-05-02T00:22:45Z ERROR cannot connect to
> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
> 2018-05-02T00:22:45Z ERROR The ipa-replica-install command failed. See
> /var/log/ipareplica-install.log for more information
>
> Maybe it just is trying to soon and it hasn't fully started yet? Because
> now I see that the previous step ALSO shows the URL encoded socket path,
> and it worked fine - but it looks like the previous step used ldapmodify,
> but the failing step was accessing LDAP from Python? Perhaps it works fine
> one way and not the other ?
>
I won't rule it out but I doubt it. Is ns-slapd still running at this
point? This could happen, for example, if ns-slapd crashed after enabling
the automember plugin. The DS error log might have something to say as well.
rob
> On Wed, May 2, 2018 at 3:29 PM, Rob Crittenden <rcritten(a)redhat.com
> <mailto:rcritten@redhat.com>> wrote:
>
> Jonathan Vaughn via FreeIPA-users wrote:
>
> Yes, I know, not recommended etc, low performance. I'm not going
> to run the CA on it. I just want to have a backup LDAP/Kerberos
> server.
>
> Right now I'm just trying to test things out. I've got a master
> and a replica (so you could say two masters I suppose) running
> in Virtualbox VMs, and I'm trying to set up a 3rd replica on a
> Pi. All are Fedors 27. I had to downgrade httpd due to
> https://pagure.io/freeipa/issue/7493
> <https://pagure.io/freeipa/issue/7493> to even set up the first
> VM replica, but this issue is separate.
>
> Currently, the problem is it can't connect to it's own LDAP
> instance due to some kind of error ... ipa-replica-install
> worked fine on the x86_64 VM but on the armv71 Pi 3B when it
> tries to connect to LDAPI instead of
> using 'ldapi:///var/run//slapd-COMPANY-INTERNAL.socket' it
> uses 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket'.
>
> So it seems there is yet another ARM (or non-x86_64) bug ...
> similar to the problem with httpd and passing the KRB5CCNAME
> properly https://pagure.io/freeipa/issue/7337
> <https://pagure.io/freeipa/issue/7337>
>
> Any ideas on where to look to patch in a fix to this so it uses
> the correct filename? The socket file is there ... and (at the
> time it tries) LDAP is running.
>
>
> What makes you think the ldapi URI is the problem?
>
> Can you share the logs?
>
> rob
>
>
>
6:40 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Still trying to figure this out. It looks like slapd is dying, I thought it
was still running for some reason.
slapd is dying to segfault. strace of it happening doesn't seem to reveal
much:
18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", O_WRONLY|O_CREAT|O_TRUNC,
0666) = 32
18:32:41.544907 (+ 0.001195) getpid() = 16014
18:32:41.545269 (+ 0.000329) fstat64(32, {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0
18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) = 6
18:32:41.546603 (+ 0.000818) close(32) = 0
18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", 0644) = 0
18:32:41.547741 (+ 0.000676) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=548030641}) = 0
18:32:41.548324 (+ 0.000587) write(2, "[07/May/2018:18:32:41.548030641
"..., 134) = 134
18:32:41.551096 (+ 0.002840) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=551287555}) = 0
18:32:41.551568 (+ 0.000406) write(4, "[07/May/2018:18:32:41.551287555
"..., 134) = 134
18:32:41.552360 (+ 0.000811) fsync(4) = 0
18:32:41.558499 (+ 0.006170) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=558678099}) = 0
18:32:41.558901 (+ 0.000350) write(2, "[07/May/2018:18:32:41.558678099
"..., 136) = 136
18:32:41.561537 (+ 0.002680) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=561718659}) = 0
18:32:41.562357 (+ 0.000793) write(4, "[07/May/2018:18:32:41.561718659
"..., 136) = 136
18:32:41.563293 (+ 0.001148) fsync(4) = 0
18:32:41.566928 (+ 0.003452) getpid() = 16014
18:32:41.567712 (+ 0.000752) socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC,
0) = 32
18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET, SO_SNDBUF,
[163840], [4]) = 0
18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET, SO_SNDBUFFORCE,
[8388608], 4) = -1 EPERM (Operation not permitted)
18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0
18:32:41.570064 (+ 0.000513) sendmsg(32, {msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"}, msg_namelen=21,
msg_iov=[{iov_base="READY=1\nSTATUS=slapd started: Re"..., iov_len=69}],
msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 69
18:32:41.570845 (+ 0.000789) close(32) = 0
18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8, FUTEX_WAIT, 16016, NULL)
= ?
18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
It doesn't die immediately after hitting that final futex call - it's just
over 20s later. There's still plenty of free memory at the time.
# free -m
total used free shared buff/cache
available
Mem: 974 222 156 0 594
679
Swap: 2535 24 2511
The last entries in the error log is that it is listening on socket / port.
No error logged for the segfault.
I did see that during previous setup steps (before it dies) there is
apparently an error setting up replication:
[07/May/2018:18:17:46.245041807 -0500] - ERR - NSMMReplicationPlugin -
acquire_replica - agmt="cn=meToipa-12.company.internal" (ipa-12:5): Unable
to acquire replica: permission denied. The bind dn "" does not have
permission to supply replication updates to the replica. Will retry later.
Possibly, the failure to configure replication is leading to the segfault?
Any ideas why it would not have permission / what to check to ensure it has
permission ?
On Thu, May 3, 2018 at 4:02 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
It's still running.
Here's the error log from slapd during that run:
[01/May/2018:19:22:24.567650453 -0500] - ERR - NSMMReplicationPlugin -
multimaster_extop_StartNSDS50ReplicationRequest - conn=42 op=5
replica="dc=company,dc=internal": Unable to acquire replica: error:
permission denied
[01/May/2018:19:22:24.613143629 -0500] - ERR - NSMMReplicationPlugin -
multimaster_extop_StartNSDS50ReplicationRequest - conn=43 op=5
replica="dc=company,dc=internal": Unable to acquire replica: error:
permission denied
[01/May/2018:19:22:31.445153171 -0500] - INFO - NSMMReplicationPlugin -
repl5_tot_run - Beginning total update of replica
"agmt="cn=meToipa-11.company.internal" (ipa-11:389)".
[01/May/2018:19:22:31.672485209 -0500] - NOTICE - NSMMReplicationPlugin -
replica_subentry_check - Need to create replication keep alive entry
<cn=repl keep alive 7,dc=company,dc=internal>
[01/May/2018:19:22:31.674922201 -0500] - INFO - NSMMReplicationPlugin -
replica_subentry_create - add dn: cn=repl keep alive
7,dc=company,dc=internal
objectclass: top
objectclass: ldapsubentry
objectclass: extensibleObject
cn: repl keep alive 7
[01/May/2018:19:22:36.125109819 -0500] - ERR - NSMMReplicationPlugin -
multimaster_extop_StartNSDS50ReplicationRequest - conn=44 op=5
replica="dc=company,dc=internal": Unable to acquire replica: error:
permission denied
[01/May/2018:19:22:36.598178749 -0500] - INFO - NSMMReplicationPlugin -
repl5_tot_run - Finished total update of replica
"agmt="cn=meToipa-11.company.internal" (ipa-11:389)". Sent 503
entries.
[01/May/2018:19:27:40.482836683 -0500] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=meToipa-11.company.internal" (ipa-11:389) -
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP
server) ()
ipa-11 is the Pi I'm trying to set up replication on. Looks like there was
a problem setting up replication due to permissions? But the other replica
started up just fine ...
On Thu, May 3, 2018 at 3:13 PM, Rob Crittenden <rcritten(a)redhat.com>
wrote:
> Jonathan Vaughn wrote:
>
>> Here's the output from ipa-replica-install :
>>
>> # ipa-replica-install
>> WARNING: conflicting time&date synchronization service 'chronyd'
will
>> be disabled in favor of ntpd
>>
>> Password for admin(a)COMPANY.INTERNAL:
>> Run connection check to master
>> Connection check OK
>> Configuring NTP daemon (ntpd)
>> [1/4]: stopping ntpd
>> [2/4]: writing configuration
>> [3/4]: configuring ntpd to start on boot
>> [4/4]: starting ntpd
>> Done configuring NTP daemon (ntpd).
>> Configuring directory server (dirsrv). Estimated time: 30 seconds
>> [1/41]: creating directory server instance
>> [2/41]: enabling ldapi
>> [3/41]: configure autobind for root
>> [4/41]: stopping directory server
>> [5/41]: updating configuration in dse.ldif
>> [6/41]: starting directory server
>> [7/41]: adding default schema
>> [8/41]: enabling memberof plugin
>> [9/41]: enabling winsync plugin
>> [10/41]: configuring replication version plugin
>> [11/41]: enabling IPA enrollment plugin
>> [12/41]: configuring uniqueness plugin
>> [13/41]: configuring uuid plugin
>> [14/41]: configuring modrdn plugin
>> [15/41]: configuring DNS plugin
>> [16/41]: enabling entryUSN plugin
>> [17/41]: configuring lockout plugin
>> [18/41]: configuring topology plugin
>> [19/41]: creating indices
>> [20/41]: enabling referential integrity plugin
>> [21/41]: configuring certmap.conf
>> [22/41]: configure new location for managed entries
>> [23/41]: configure dirsrv ccache
>> [24/41]: enabling SASL mapping fallback
>> [25/41]: restarting directory server
>> [26/41]: creating DS keytab
>> [27/41]: ignore time skew for initial replication
>> [28/41]: setting up initial replication
>> Starting replication, please wait until this has completed.
>> Update in progress, 11 seconds elapsed
>> Update succeeded
>>
>> [29/41]: prevent time skew after initial replication
>> [30/41]: adding sasl mappings to the directory
>> [31/41]: updating schema
>> [32/41]: setting Auto Member configuration
>> [33/41]: enabling S4U2Proxy delegation
>> [error] NetworkError: cannot connect to 'ldapi://%2Fvar%2Frun%2Fslapd-
>> COMPANY-INTERNAL.socket':
>> Your system may be partly configured.
>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>
>> ipapython.admintool: ERROR cannot connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>> ipapython.admintool: ERROR The ipa-replica-install command failed.
>> See /var/log/ipareplica-install.log for more information
>>
>> And here's the /var/log/ipareplica-install.log from just before it
>> fails to the failure:
>>
>> 2018-05-02T00:22:44Z DEBUG [32/41]: setting Auto Member configuration
>> 2018-05-02T00:22:44Z DEBUG Starting external process
>> 2018-05-02T00:22:44Z DEBUG args=/usr/bin/ldapmodify -v -f
>> /tmp/tmpt7bxf4x1 -H ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket
>> -Y EXTERNAL
>> 2018-05-02T00:22:45Z DEBUG Process finished, return code=0
>> 2018-05-02T00:22:45Z DEBUG stdout=add nsslapd-pluginConfigArea:
>> cn=automember,cn=etc,dc=company,dc=internal
>> modifying entry "cn=Auto Membership Plugin,cn=plugins,cn=config"
>> modify complete
>>
>>
>> 2018-05-02T00:22:45Z DEBUG stderr=ldap_initialize(
>> ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket/??base )
>> SASL/EXTERNAL authentication started
>> SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>> SASL SSF: 0
>>
>> 2018-05-02T00:22:45Z DEBUG duration: 0 seconds
>> 2018-05-02T00:22:45Z DEBUG [33/41]: enabling S4U2Proxy delegation
>> 2018-05-02T00:22:45Z DEBUG Traceback (most recent call last):
>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
>> 979, in error_handler
>> yield
>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
>> 1572, in update_entry
>> self.conn.modify_s(str(entry.dn), modlist)
>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
>> 600, in modify_s
>> return self.modify_ext_s(dn,modlist,None,None)
>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
>> 573, in modify_ext_s
>> resp_type, resp_data, resp_msgid, resp_ctrls =
>> self.result3(msgid,all=1,timeout=self.timeout)
>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
>> 714, in result3
>> resp_ctrl_classes=resp_ctrl_classes
>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
>> 721, in result4
>> ldap_result = self._ldap_call(self._l.result
>> 4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
>> 294, in _ldap_call
>> result = func(*args,**kwargs)
>> ldap.SERVER_DOWN: {'desc': "Can't contact LDAP server"}
>>
>> During handling of the above exception, another exception occurred:
>>
>> Traceback (most recent call last):
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
>> line 506, in start_creation
>> run_step(full_msg, method)
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
>> line 496, in run_step
>> method()
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>> line 977, in __setup_s4u2proxy
>> __add_principal('ipa-http-delegation', 'HTTP', self)
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>> line 973, in __add_principal
>> api.Backend.ldap2.update_entry(entry)
>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
>> 1572, in update_entry
>> self.conn.modify_s(str(entry.dn), modlist)
>> File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
>> self.gen.throw(type, value, traceback)
>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
>> 1028, in error_handler
>> error=info)
>> ipalib.errors.NetworkError: cannot connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>
>> 2018-05-02T00:22:45Z DEBUG [error] NetworkError: cannot connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>> 2018-05-02T00:22:45Z DEBUG Destroyed connection context.ldap2_2991558640
>> 2018-05-02T00:22:45Z DEBUG Backing up system configuration file
>> '/etc/ipa/default.conf'
>> 2018-05-02T00:22:45Z DEBUG Saving Index File to
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2018-05-02T00:22:45Z DEBUG File
"/usr/lib/python3.6/site-packages/ipapython/admintool.py",
>> line 174, in execute
>> return_value = self.run()
>> File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py",
>> line 319, in run
>> cfgr.run()
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 364, in run
>> self.execute()
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 388, in execute
>> for _nothing in self._executor():
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 430, in __runner
>> exc_handler(exc_info)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 459, in _handle_execute_exception
>> self._handle_exception(exc_info)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 449, in _handle_exception
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>> raise value
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 420, in __runner
>> step()
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 417, in <lambda>
>> step = lambda: next(self.__gen)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>> line 81, in run_generator_with_yield_from
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>> raise value
>> File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>> line 59, in run_generator_with_yield_from
>> value = gen.send(prev_value)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 654, in _configure
>> next(executor)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 430, in __runner
>> exc_handler(exc_info)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 459, in _handle_execute_exception
>> self._handle_exception(exc_info)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 517, in _handle_exception
>> self.__parent._handle_exception(exc_info)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 449, in _handle_exception
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>> raise value
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 514, in _handle_exception
>> super(ComponentBase, self)._handle_exception(exc_info)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 449, in _handle_exception
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>> raise value
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 420, in __runner
>> step()
>> File "/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line 417, in <lambda>
>> step = lambda: next(self.__gen)
>> File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>> line 81, in run_generator_with_yield_from
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>> raise value
>> File "/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>> line 59, in run_generator_with_yield_from
>> value = gen.send(prev_value)
>> File
"/usr/lib/python3.6/site-packages/ipapython/install/common.py",
>> line 66, in _install
>> for unused in self._installer(self.parent):
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py",
>> line 622, in main
>> replica_install(self)
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
>> line 388, in decorated
>> func(installer)
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
>> line 1407, in install
>> pkcs12_info=dirsrv_pkcs12_info)
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
>> line 110, in install_replica_ds
>> setup_pkinit=not options.no_pkinit,
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>> line 419, in create_replica
>> self.start_creation(runtime=30)
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
>> line 506, in start_creation
>> run_step(full_msg, method)
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
>> line 496, in run_step
>> method()
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>> line 977, in __setup_s4u2proxy
>> __add_principal('ipa-http-delegation', 'HTTP', self)
>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>> line 973, in __add_principal
>> api.Backend.ldap2.update_entry(entry)
>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
>> 1572, in update_entry
>> self.conn.modify_s(str(entry.dn), modlist)
>> File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
>> self.gen.throw(type, value, traceback)
>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py", line
>> 1028, in error_handler
>> error=info)
>>
>> 2018-05-02T00:22:45Z DEBUG The ipa-replica-install command failed,
>> exception: NetworkError: cannot connect to 'ldapi://%2Fvar%2Frun%2Fslapd-
>> COMPANY-INTERNAL.socket':
>> 2018-05-02T00:22:45Z ERROR cannot connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>> 2018-05-02T00:22:45Z ERROR The ipa-replica-install command failed. See
>> /var/log/ipareplica-install.log for more information
>>
>> Maybe it just is trying to soon and it hasn't fully started yet? Because
>> now I see that the previous step ALSO shows the URL encoded socket path,
>> and it worked fine - but it looks like the previous step used ldapmodify,
>> but the failing step was accessing LDAP from Python? Perhaps it works fine
>> one way and not the other ?
>>
>
> I won't rule it out but I doubt it. Is ns-slapd still running at this
> point? This could happen, for example, if ns-slapd crashed after enabling
> the automember plugin. The DS error log might have something to say as well.
>
> rob
>
>
>> On Wed, May 2, 2018 at 3:29 PM, Rob Crittenden <rcritten(a)redhat.com
>> <mailto:rcritten@redhat.com>> wrote:
>>
>> Jonathan Vaughn via FreeIPA-users wrote:
>>
>> Yes, I know, not recommended etc, low performance. I'm not going
>> to run the CA on it. I just want to have a backup LDAP/Kerberos
>> server.
>>
>> Right now I'm just trying to test things out. I've got a master
>> and a replica (so you could say two masters I suppose) running
>> in Virtualbox VMs, and I'm trying to set up a 3rd replica on a
>> Pi. All are Fedors 27. I had to downgrade httpd due to
>> https://pagure.io/freeipa/issue/7493
>> <https://pagure.io/freeipa/issue/7493> to even set up the first
>> VM replica, but this issue is separate.
>>
>> Currently, the problem is it can't connect to it's own LDAP
>> instance due to some kind of error ... ipa-replica-install
>> worked fine on the x86_64 VM but on the armv71 Pi 3B when it
>> tries to connect to LDAPI instead of
>> using 'ldapi:///var/run//slapd-COMPANY-INTERNAL.socket' it
>> uses 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket'.
>>
>> So it seems there is yet another ARM (or non-x86_64) bug ...
>> similar to the problem with httpd and passing the KRB5CCNAME
>> properly https://pagure.io/freeipa/issue/7337
>> <https://pagure.io/freeipa/issue/7337>
>>
>> Any ideas on where to look to patch in a fix to this so it uses
>> the correct filename? The socket file is there ... and (at the
>> time it tries) LDAP is running.
>>
>>
>> What makes you think the ldapi URI is the problem?
>>
>> Can you share the logs?
>>
>> rob
>>
>>
>>
>
7:10 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Could this be part of the problem now?
[root@ipa-12 master]# ipa-replica-manage list ipa-12.company.internal
ipa-11.company.internal: replica
ipa-13.company.internal: replica
[root@ipa-12 master]# ipa-replica-manage list ipa-13.company.internal
ipa-11.company.internal: replica
ipa-12.company.internal: replica
[root@ipa-12 master]# ipa-replica-manage list ipa-11.company.internal
Cannot find ipa-11.company.internal in public server list
I can't use ipa-replica-manage to remove the broken links to ipa-11, says I
must use `ipa topologysegment-*`. As far as I can tell, the segments don't
exist to remove for ipa11 since it's only half set up ...
On Mon, May 7, 2018 at 6:40 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
Still trying to figure this out. It looks like slapd is dying, I
thought
it was still running for some reason.
slapd is dying to segfault. strace of it happening doesn't seem to reveal
much:
18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
18:32:41.544907 (+ 0.001195) getpid() = 16014
18:32:41.545269 (+ 0.000329) fstat64(32, {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0
18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) = 6
18:32:41.546603 (+ 0.000818) close(32) = 0
18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
0644) = 0
18:32:41.547741 (+ 0.000676) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=548030641}) = 0
18:32:41.548324 (+ 0.000587) write(2, "[07/May/2018:18:32:41.548030641
"..., 134) = 134
18:32:41.551096 (+ 0.002840) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=551287555}) = 0
18:32:41.551568 (+ 0.000406) write(4, "[07/May/2018:18:32:41.551287555
"..., 134) = 134
18:32:41.552360 (+ 0.000811) fsync(4) = 0
18:32:41.558499 (+ 0.006170) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=558678099}) = 0
18:32:41.558901 (+ 0.000350) write(2, "[07/May/2018:18:32:41.558678099
"..., 136) = 136
18:32:41.561537 (+ 0.002680) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=561718659}) = 0
18:32:41.562357 (+ 0.000793) write(4, "[07/May/2018:18:32:41.561718659
"..., 136) = 136
18:32:41.563293 (+ 0.001148) fsync(4) = 0
18:32:41.566928 (+ 0.003452) getpid() = 16014
18:32:41.567712 (+ 0.000752) socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC,
0) = 32
18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET, SO_SNDBUF,
[163840], [4]) = 0
18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET,
SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0
18:32:41.570064 (+ 0.000513) sendmsg(32, {msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"}, msg_namelen=21,
msg_iov=[{iov_base="READY=1\nSTATUS=slapd started: Re"..., iov_len=69}],
msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 69
18:32:41.570845 (+ 0.000789) close(32) = 0
18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8, FUTEX_WAIT, 16016,
NULL) = ?
18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
It doesn't die immediately after hitting that final futex call - it's just
over 20s later. There's still plenty of free memory at the time.
# free -m
total used free shared buff/cache
available
Mem: 974 222 156 0 594
679
Swap: 2535 24 2511
The last entries in the error log is that it is listening on socket /
port. No error logged for the segfault.
I did see that during previous setup steps (before it dies) there is
apparently an error setting up replication:
[07/May/2018:18:17:46.245041807 -0500] - ERR - NSMMReplicationPlugin -
acquire_replica - agmt="cn=meToipa-12.company.internal" (ipa-12:5):
Unable to acquire replica: permission denied. The bind dn "" does not have
permission to supply replication updates to the replica. Will retry later.
Possibly, the failure to configure replication is leading to the segfault?
Any ideas why it would not have permission / what to check to ensure it has
permission ?
On Thu, May 3, 2018 at 4:02 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
> It's still running.
>
> Here's the error log from slapd during that run:
>
> [01/May/2018:19:22:24.567650453 -0500] - ERR - NSMMReplicationPlugin -
> multimaster_extop_StartNSDS50ReplicationRequest - conn=42 op=5
> replica="dc=company,dc=internal": Unable to acquire replica: error:
> permission denied
> [01/May/2018:19:22:24.613143629 -0500] - ERR - NSMMReplicationPlugin -
> multimaster_extop_StartNSDS50ReplicationRequest - conn=43 op=5
> replica="dc=company,dc=internal": Unable to acquire replica: error:
> permission denied
> [01/May/2018:19:22:31.445153171 -0500] - INFO - NSMMReplicationPlugin -
> repl5_tot_run - Beginning total update of replica
> "agmt="cn=meToipa-11.company.internal" (ipa-11:389)".
> [01/May/2018:19:22:31.672485209 -0500] - NOTICE - NSMMReplicationPlugin
> - replica_subentry_check - Need to create replication keep alive entry
> <cn=repl keep alive 7,dc=company,dc=internal>
> [01/May/2018:19:22:31.674922201 -0500] - INFO - NSMMReplicationPlugin -
> replica_subentry_create - add dn: cn=repl keep alive
> 7,dc=company,dc=internal
> objectclass: top
> objectclass: ldapsubentry
> objectclass: extensibleObject
> cn: repl keep alive 7
> [01/May/2018:19:22:36.125109819 -0500] - ERR - NSMMReplicationPlugin -
> multimaster_extop_StartNSDS50ReplicationRequest - conn=44 op=5
> replica="dc=company,dc=internal": Unable to acquire replica: error:
> permission denied
> [01/May/2018:19:22:36.598178749 -0500] - INFO - NSMMReplicationPlugin -
> repl5_tot_run - Finished total update of replica
> "agmt="cn=meToipa-11.company.internal" (ipa-11:389)". Sent 503
entries.
> [01/May/2018:19:27:40.482836683 -0500] - ERR - NSMMReplicationPlugin -
> bind_and_check_pwp - agmt="cn=meToipa-11.company.internal" (ipa-11:389)
> - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
> LDAP server) ()
>
> ipa-11 is the Pi I'm trying to set up replication on. Looks like there
> was a problem setting up replication due to permissions? But the other
> replica started up just fine ...
>
> On Thu, May 3, 2018 at 3:13 PM, Rob Crittenden <rcritten(a)redhat.com>
> wrote:
>
>> Jonathan Vaughn wrote:
>>
>>> Here's the output from ipa-replica-install :
>>>
>>> # ipa-replica-install
>>> WARNING: conflicting time&date synchronization service 'chronyd'
will
>>> be disabled in favor of ntpd
>>>
>>> Password for admin(a)COMPANY.INTERNAL:
>>> Run connection check to master
>>> Connection check OK
>>> Configuring NTP daemon (ntpd)
>>> [1/4]: stopping ntpd
>>> [2/4]: writing configuration
>>> [3/4]: configuring ntpd to start on boot
>>> [4/4]: starting ntpd
>>> Done configuring NTP daemon (ntpd).
>>> Configuring directory server (dirsrv). Estimated time: 30 seconds
>>> [1/41]: creating directory server instance
>>> [2/41]: enabling ldapi
>>> [3/41]: configure autobind for root
>>> [4/41]: stopping directory server
>>> [5/41]: updating configuration in dse.ldif
>>> [6/41]: starting directory server
>>> [7/41]: adding default schema
>>> [8/41]: enabling memberof plugin
>>> [9/41]: enabling winsync plugin
>>> [10/41]: configuring replication version plugin
>>> [11/41]: enabling IPA enrollment plugin
>>> [12/41]: configuring uniqueness plugin
>>> [13/41]: configuring uuid plugin
>>> [14/41]: configuring modrdn plugin
>>> [15/41]: configuring DNS plugin
>>> [16/41]: enabling entryUSN plugin
>>> [17/41]: configuring lockout plugin
>>> [18/41]: configuring topology plugin
>>> [19/41]: creating indices
>>> [20/41]: enabling referential integrity plugin
>>> [21/41]: configuring certmap.conf
>>> [22/41]: configure new location for managed entries
>>> [23/41]: configure dirsrv ccache
>>> [24/41]: enabling SASL mapping fallback
>>> [25/41]: restarting directory server
>>> [26/41]: creating DS keytab
>>> [27/41]: ignore time skew for initial replication
>>> [28/41]: setting up initial replication
>>> Starting replication, please wait until this has completed.
>>> Update in progress, 11 seconds elapsed
>>> Update succeeded
>>>
>>> [29/41]: prevent time skew after initial replication
>>> [30/41]: adding sasl mappings to the directory
>>> [31/41]: updating schema
>>> [32/41]: setting Auto Member configuration
>>> [33/41]: enabling S4U2Proxy delegation
>>> [error] NetworkError: cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>> Your system may be partly configured.
>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>>>
>>> ipapython.admintool: ERROR cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>> ipapython.admintool: ERROR The ipa-replica-install command failed.
>>> See /var/log/ipareplica-install.log for more information
>>>
>>> And here's the /var/log/ipareplica-install.log from just before it
>>> fails to the failure:
>>>
>>> 2018-05-02T00:22:44Z DEBUG [32/41]: setting Auto Member configuration
>>> 2018-05-02T00:22:44Z DEBUG Starting external process
>>> 2018-05-02T00:22:44Z DEBUG args=/usr/bin/ldapmodify -v -f
>>> /tmp/tmpt7bxf4x1 -H ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket
>>> -Y EXTERNAL
>>> 2018-05-02T00:22:45Z DEBUG Process finished, return code=0
>>> 2018-05-02T00:22:45Z DEBUG stdout=add nsslapd-pluginConfigArea:
>>> cn=automember,cn=etc,dc=company,dc=internal
>>> modifying entry "cn=Auto Membership Plugin,cn=plugins,cn=config"
>>> modify complete
>>>
>>>
>>> 2018-05-02T00:22:45Z DEBUG stderr=ldap_initialize(
>>> ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket/??base )
>>> SASL/EXTERNAL authentication started
>>> SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>>> SASL SSF: 0
>>>
>>> 2018-05-02T00:22:45Z DEBUG duration: 0 seconds
>>> 2018-05-02T00:22:45Z DEBUG [33/41]: enabling S4U2Proxy delegation
>>> 2018-05-02T00:22:45Z DEBUG Traceback (most recent call last):
>>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line
>>> 979, in error_handler
>>> yield
>>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line
>>> 1572, in update_entry
>>> self.conn.modify_s(str(entry.dn), modlist)
>>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 600, in modify_s
>>> return self.modify_ext_s(dn,modlist,None,None)
>>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 573, in modify_ext_s
>>> resp_type, resp_data, resp_msgid, resp_ctrls =
>>> self.result3(msgid,all=1,timeout=self.timeout)
>>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 714, in result3
>>> resp_ctrl_classes=resp_ctrl_classes
>>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 721, in result4
>>> ldap_result = self._ldap_call(self._l.result
>>> 4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
>>> File "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 294, in _ldap_call
>>> result = func(*args,**kwargs)
>>> ldap.SERVER_DOWN: {'desc': "Can't contact LDAP
server"}
>>>
>>> During handling of the above exception, another exception occurred:
>>>
>>> Traceback (most recent call last):
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
>>> line 506, in start_creation
>>> run_step(full_msg, method)
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
>>> line 496, in run_step
>>> method()
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>>> line 977, in __setup_s4u2proxy
>>> __add_principal('ipa-http-delegation', 'HTTP', self)
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>>> line 973, in __add_principal
>>> api.Backend.ldap2.update_entry(entry)
>>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line
>>> 1572, in update_entry
>>> self.conn.modify_s(str(entry.dn), modlist)
>>> File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
>>> self.gen.throw(type, value, traceback)
>>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line
>>> 1028, in error_handler
>>> error=info)
>>> ipalib.errors.NetworkError: cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>>
>>> 2018-05-02T00:22:45Z DEBUG [error] NetworkError: cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>> 2018-05-02T00:22:45Z DEBUG Destroyed connection context.ldap2_2991558640
>>> 2018-05-02T00:22:45Z DEBUG Backing up system configuration file
>>> '/etc/ipa/default.conf'
>>> 2018-05-02T00:22:45Z DEBUG Saving Index File to
>>> '/var/lib/ipa/sysrestore/sysrestore.index'
>>> 2018-05-02T00:22:45Z DEBUG File
"/usr/lib/python3.6/site-packages/ipapython/admintool.py",
>>> line 174, in execute
>>> return_value = self.run()
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/cli.py",
>>> line 319, in run
>>> cfgr.run()
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 364, in run
>>> self.execute()
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 388, in execute
>>> for _nothing in self._executor():
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 430, in __runner
>>> exc_handler(exc_info)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 459, in _handle_execute_exception
>>> self._handle_exception(exc_info)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 449, in _handle_exception
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>>> raise value
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 420, in __runner
>>> step()
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 417, in <lambda>
>>> step = lambda: next(self.__gen)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>>> line 81, in run_generator_with_yield_from
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>>> raise value
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>>> line 59, in run_generator_with_yield_from
>>> value = gen.send(prev_value)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 654, in _configure
>>> next(executor)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 430, in __runner
>>> exc_handler(exc_info)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 459, in _handle_execute_exception
>>> self._handle_exception(exc_info)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 517, in _handle_exception
>>> self.__parent._handle_exception(exc_info)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 449, in _handle_exception
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>>> raise value
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 514, in _handle_exception
>>> super(ComponentBase, self)._handle_exception(exc_info)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 449, in _handle_exception
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>>> raise value
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 420, in __runner
>>> step()
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line 417, in <lambda>
>>> step = lambda: next(self.__gen)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>>> line 81, in run_generator_with_yield_from
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py", line 693, in
reraise
>>> raise value
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>>> line 59, in run_generator_with_yield_from
>>> value = gen.send(prev_value)
>>> File
"/usr/lib/python3.6/site-packages/ipapython/install/common.py",
>>> line 66, in _install
>>> for unused in self._installer(self.parent):
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py",
>>> line 622, in main
>>> replica_install(self)
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
>>> line 388, in decorated
>>> func(installer)
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
>>> line 1407, in install
>>> pkcs12_info=dirsrv_pkcs12_info)
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
>>> line 110, in install_replica_ds
>>> setup_pkinit=not options.no_pkinit,
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>>> line 419, in create_replica
>>> self.start_creation(runtime=30)
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
>>> line 506, in start_creation
>>> run_step(full_msg, method)
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
>>> line 496, in run_step
>>> method()
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>>> line 977, in __setup_s4u2proxy
>>> __add_principal('ipa-http-delegation', 'HTTP', self)
>>> File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
>>> line 973, in __add_principal
>>> api.Backend.ldap2.update_entry(entry)
>>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line
>>> 1572, in update_entry
>>> self.conn.modify_s(str(entry.dn), modlist)
>>> File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
>>> self.gen.throw(type, value, traceback)
>>> File "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line
>>> 1028, in error_handler
>>> error=info)
>>>
>>> 2018-05-02T00:22:45Z DEBUG The ipa-replica-install command failed,
>>> exception: NetworkError: cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-
>>> COMPANY-INTERNAL.socket':
>>> 2018-05-02T00:22:45Z ERROR cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>> 2018-05-02T00:22:45Z ERROR The ipa-replica-install command failed. See
>>> /var/log/ipareplica-install.log for more information
>>>
>>> Maybe it just is trying to soon and it hasn't fully started yet?
>>> Because now I see that the previous step ALSO shows the URL encoded socket
>>> path, and it worked fine - but it looks like the previous step used
>>> ldapmodify, but the failing step was accessing LDAP from Python? Perhaps it
>>> works fine one way and not the other ?
>>>
>>
>> I won't rule it out but I doubt it. Is ns-slapd still running at this
>> point? This could happen, for example, if ns-slapd crashed after enabling
>> the automember plugin. The DS error log might have something to say as well.
>>
>> rob
>>
>>
>>> On Wed, May 2, 2018 at 3:29 PM, Rob Crittenden <rcritten(a)redhat.com
>>> <mailto:rcritten@redhat.com>> wrote:
>>>
>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>
>>> Yes, I know, not recommended etc, low performance. I'm not going
>>> to run the CA on it. I just want to have a backup LDAP/Kerberos
>>> server.
>>>
>>> Right now I'm just trying to test things out. I've got a
master
>>> and a replica (so you could say two masters I suppose) running
>>> in Virtualbox VMs, and I'm trying to set up a 3rd replica on a
>>> Pi. All are Fedors 27. I had to downgrade httpd due to
>>> https://pagure.io/freeipa/issue/7493
>>> <https://pagure.io/freeipa/issue/7493> to even set up the
first
>>> VM replica, but this issue is separate.
>>>
>>> Currently, the problem is it can't connect to it's own LDAP
>>> instance due to some kind of error ... ipa-replica-install
>>> worked fine on the x86_64 VM but on the armv71 Pi 3B when it
>>> tries to connect to LDAPI instead of
>>> using 'ldapi:///var/run//slapd-COMPANY-INTERNAL.socket' it
>>> uses 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket'.
>>>
>>> So it seems there is yet another ARM (or non-x86_64) bug ...
>>> similar to the problem with httpd and passing the KRB5CCNAME
>>> properly https://pagure.io/freeipa/issue/7337
>>> <https://pagure.io/freeipa/issue/7337>
>>>
>>> Any ideas on where to look to patch in a fix to this so it uses
>>> the correct filename? The socket file is there ... and (at the
>>> time it tries) LDAP is running.
>>>
>>>
>>> What makes you think the ldapi URI is the problem?
>>>
>>> Can you share the logs?
>>>
>>> rob
>>>
>>>
>>>
>>
>
7:52 a.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Jonathan Vaughn via FreeIPA-users wrote:
Still trying to figure this out. It looks like slapd is dying, I
thought
it was still running for some reason.
slapd is dying to segfault. strace of it happening doesn't seem to
reveal much:
A stack trace would very much help trying to track down the cause.
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
rob
18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", O_WRONLY|O_CREAT|O_TRUNC,
0666) = 32
18:32:41.544907 (+ 0.001195) getpid() = 16014
18:32:41.545269 (+ 0.000329) fstat64(32, {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0
18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) = 6
18:32:41.546603 (+ 0.000818) close(32) = 0
18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", 0644) = 0
18:32:41.547741 (+ 0.000676) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=548030641}) = 0
18:32:41.548324 (+ 0.000587) write(2,
"[07/May/2018:18:32:41.548030641 "..., 134) = 134
18:32:41.551096 (+ 0.002840) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=551287555}) = 0
18:32:41.551568 (+ 0.000406) write(4,
"[07/May/2018:18:32:41.551287555 "..., 134) = 134
18:32:41.552360 (+ 0.000811) fsync(4) = 0
18:32:41.558499 (+ 0.006170) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=558678099}) = 0
18:32:41.558901 (+ 0.000350) write(2,
"[07/May/2018:18:32:41.558678099 "..., 136) = 136
18:32:41.561537 (+ 0.002680) clock_gettime(CLOCK_REALTIME,
{tv_sec=1525735961, tv_nsec=561718659}) = 0
18:32:41.562357 (+ 0.000793) write(4,
"[07/May/2018:18:32:41.561718659 "..., 136) = 136
18:32:41.563293 (+ 0.001148) fsync(4) = 0
18:32:41.566928 (+ 0.003452) getpid() = 16014
18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET, SO_SNDBUF,
[163840], [4]) = 0
18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET,
SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0
18:32:41.570064 (+ 0.000513) sendmsg(32,
{msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL) = 69
18:32:41.570845 (+ 0.000789) close(32) = 0
18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8, FUTEX_WAIT, 16016,
NULL) = ?
18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
It doesn't die immediately after hitting that final futex call - it's
just over 20s later. There's still plenty of free memory at the time.
# free -m
total used free shared buff/cache
available
Mem: 974 222 156 0 594
679
Swap: 2535 24 2511
The last entries in the error log is that it is listening on socket /
port. No error logged for the segfault.
I did see that during previous setup steps (before it dies) there is
apparently an error setting up replication:
[07/May/2018:18:17:46.245041807 -0500] - ERR - NSMMReplicationPlugin -
acquire_replica - agmt="cn=meToipa-12.company.internal" (ipa-12:5):
Unable to acquire replica: permission denied. The bind dn "" does not
have permission to supply replication updates to the replica. Will retry
later.
Possibly, the failure to configure replication is leading to the
segfault? Any ideas why it would not have permission / what to check to
ensure it has permission ?
On Thu, May 3, 2018 at 4:02 PM, Jonathan Vaughn <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>> wrote:
It's still running.
Here's the error log from slapd during that run:
[01/May/2018:19:22:24.567650453 -0500] - ERR - NSMMReplicationPlugin
- multimaster_extop_StartNSDS50ReplicationRequest - conn=42 op=5
replica="dc=company,dc=internal": Unable to acquire replica: error:
permission denied
[01/May/2018:19:22:24.613143629 -0500] - ERR - NSMMReplicationPlugin
- multimaster_extop_StartNSDS50ReplicationRequest - conn=43 op=5
replica="dc=company,dc=internal": Unable to acquire replica: error:
permission denied
[01/May/2018:19:22:31.445153171 -0500] - INFO -
NSMMReplicationPlugin - repl5_tot_run - Beginning total update of
replica "agmt="cn=meToipa-11.company.internal" (ipa-11:389)".
[01/May/2018:19:22:31.672485209 -0500] - NOTICE -
NSMMReplicationPlugin - replica_subentry_check - Need to create
replication keep alive entry <cn=repl keep alive
7,dc=company,dc=internal>
[01/May/2018:19:22:31.674922201 -0500] - INFO -
NSMMReplicationPlugin - replica_subentry_create - add dn: cn=repl
keep alive 7,dc=company,dc=internal
objectclass: top
objectclass: ldapsubentry
objectclass: extensibleObject
cn: repl keep alive 7
[01/May/2018:19:22:36.125109819 -0500] - ERR - NSMMReplicationPlugin
- multimaster_extop_StartNSDS50ReplicationRequest - conn=44 op=5
replica="dc=company,dc=internal": Unable to acquire replica: error:
permission denied
[01/May/2018:19:22:36.598178749 -0500] - INFO -
NSMMReplicationPlugin - repl5_tot_run - Finished total update of
replica "agmt="cn=meToipa-11.company.internal" (ipa-11:389)".
Sent
503 entries.
[01/May/2018:19:27:40.482836683 -0500] - ERR - NSMMReplicationPlugin
- bind_and_check_pwp - agmt="cn=meToipa-11.company.internal"
(ipa-11:389) - Replication bind with GSSAPI auth failed: LDAP error
-1 (Can't contact LDAP server) ()
ipa-11 is the Pi I'm trying to set up replication on. Looks like
there was a problem setting up replication due to permissions? But
the other replica started up just fine ...
On Thu, May 3, 2018 at 3:13 PM, Rob Crittenden <rcritten(a)redhat.com
<mailto:rcritten@redhat.com>> wrote:
Jonathan Vaughn wrote:
Here's the output from ipa-replica-install :
# ipa-replica-install
WARNING: conflicting time&date synchronization service
'chronyd' will
be disabled in favor of ntpd
Password for admin(a)COMPANY.INTERNAL:
Run connection check to master
Connection check OK
Configuring NTP daemon (ntpd)
[1/4]: stopping ntpd
[2/4]: writing configuration
[3/4]: configuring ntpd to start on boot
[4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30
seconds
[1/41]: creating directory server instance
[2/41]: enabling ldapi
[3/41]: configure autobind for root
[4/41]: stopping directory server
[5/41]: updating configuration in dse.ldif
[6/41]: starting directory server
[7/41]: adding default schema
[8/41]: enabling memberof plugin
[9/41]: enabling winsync plugin
[10/41]: configuring replication version plugin
[11/41]: enabling IPA enrollment plugin
[12/41]: configuring uniqueness plugin
[13/41]: configuring uuid plugin
[14/41]: configuring modrdn plugin
[15/41]: configuring DNS plugin
[16/41]: enabling entryUSN plugin
[17/41]: configuring lockout plugin
[18/41]: configuring topology plugin
[19/41]: creating indices
[20/41]: enabling referential integrity plugin
[21/41]: configuring certmap.conf
[22/41]: configure new location for managed entries
[23/41]: configure dirsrv ccache
[24/41]: enabling SASL mapping fallback
[25/41]: restarting directory server
[26/41]: creating DS keytab
[27/41]: ignore time skew for initial replication
[28/41]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 11 seconds elapsed
Update succeeded
[29/41]: prevent time skew after initial replication
[30/41]: adding sasl mappings to the directory
[31/41]: updating schema
[32/41]: setting Auto Member configuration
[33/41]: enabling S4U2Proxy delegation
[error] NetworkError: cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
ipapython.admintool: ERROR The ipa-replica-install
command failed. See /var/log/ipareplica-install.log for more
information
And here's the /var/log/ipareplica-install.log from just
before it fails to the failure:
2018-05-02T00:22:44Z DEBUG [32/41]: setting Auto Member
configuration
2018-05-02T00:22:44Z DEBUG Starting external process
2018-05-02T00:22:44Z DEBUG args=/usr/bin/ldapmodify -v -f
/tmp/tmpt7bxf4x1 -H
ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket -Y EXTERNAL
2018-05-02T00:22:45Z DEBUG Process finished, return code=0
2018-05-02T00:22:45Z DEBUG stdout=add nsslapd-pluginConfigArea:
cn=automember,cn=etc,dc=company,dc=internal
modifying entry "cn=Auto Membership Plugin,cn=plugins,cn=config"
modify complete
2018-05-02T00:22:45Z DEBUG stderr=ldap_initialize(
ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket/??base )
SASL/EXTERNAL authentication started
SASL username:
gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
2018-05-02T00:22:45Z DEBUG duration: 0 seconds
2018-05-02T00:22:45Z DEBUG [33/41]: enabling S4U2Proxy
delegation
2018-05-02T00:22:45Z DEBUG Traceback (most recent call last):
File
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line 979, in error_handler
yield
File
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line 1572, in update_entry
self.conn.modify_s(str(entry.dn), modlist)
File
"/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
600, in modify_s
return self.modify_ext_s(dn,modlist,None,None)
File
"/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
573, in modify_ext_s
resp_type, resp_data, resp_msgid, resp_ctrls =
self.result3(msgid,all=1,timeout=self.timeout)
File
"/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
714, in result3
resp_ctrl_classes=resp_ctrl_classes
File
"/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
721, in result4
ldap_result =
self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File
"/usr/lib/python3.6/site-packages/ldap/ldapobject.py", line
294, in _ldap_call
result = func(*args,**kwargs)
ldap.SERVER_DOWN: {'desc': "Can't contact LDAP
server"}
During handling of the above exception, another exception
occurred:
Traceback (most recent call last):
File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
line 506, in start_creation
run_step(full_msg, method)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
line 496, in run_step
method()
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
line 977, in __setup_s4u2proxy
__add_principal('ipa-http-delegation', 'HTTP', self)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
line 973, in __add_principal
api.Backend.ldap2.update_entry(entry)
File
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line 1572, in update_entry
self.conn.modify_s(str(entry.dn), modlist)
File "/usr/lib/python3.6/contextlib.py", line 99, in
__exit__
self.gen.throw(type, value, traceback)
File
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line 1028, in error_handler
error=info)
ipalib.errors.NetworkError: cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
2018-05-02T00:22:45Z DEBUG [error] NetworkError: cannot
connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
2018-05-02T00:22:45Z DEBUG Destroyed connection
context.ldap2_2991558640
2018-05-02T00:22:45Z DEBUG Backing up system configuration
file '/etc/ipa/default.conf'
2018-05-02T00:22:45Z DEBUG Saving Index File to
'/var/lib/ipa/sysrestore/sysrestore.index'
2018-05-02T00:22:45Z DEBUG File
"/usr/lib/python3.6/site-packages/ipapython/admintool.py",
line 174, in execute
return_value = self.run()
File
"/usr/lib/python3.6/site-packages/ipapython/install/cli.py",
line 319, in run
cfgr.run()
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
364, in run
self.execute()
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
388, in execute
for _nothing in self._executor():
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
430, in __runner
exc_handler(exc_info)
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
459, in _handle_execute_exception
self._handle_exception(exc_info)
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
449, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line
693, in reraise
raise value
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
420, in __runner
step()
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
417, in <lambda>
step = lambda: next(self.__gen)
File
"/usr/lib/python3.6/site-packages/ipapython/install/util.py", line
81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line
693, in reraise
raise value
File
"/usr/lib/python3.6/site-packages/ipapython/install/util.py", line
59, in run_generator_with_yield_from
value = gen.send(prev_value)
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
654, in _configure
next(executor)
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
430, in __runner
exc_handler(exc_info)
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
459, in _handle_execute_exception
self._handle_exception(exc_info)
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
517, in _handle_exception
self.__parent._handle_exception(exc_info)
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
449, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line
693, in reraise
raise value
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
514, in _handle_exception
super(ComponentBase, self)._handle_exception(exc_info)
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
449, in _handle_exception
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line
693, in reraise
raise value
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
420, in __runner
step()
File
"/usr/lib/python3.6/site-packages/ipapython/install/core.py", line
417, in <lambda>
step = lambda: next(self.__gen)
File
"/usr/lib/python3.6/site-packages/ipapython/install/util.py", line
81, in run_generator_with_yield_from
six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line
693, in reraise
raise value
File
"/usr/lib/python3.6/site-packages/ipapython/install/util.py", line
59, in run_generator_with_yield_from
value = gen.send(prev_value)
File
"/usr/lib/python3.6/site-packages/ipapython/install/common.py",
line 66, in _install
for unused in self._installer(self.parent):
File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py",
line 622, in main
replica_install(self)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
line 388, in decorated
func(installer)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
line 1407, in install
pkcs12_info=dirsrv_pkcs12_info)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py",
line 110, in install_replica_ds
setup_pkinit=not options.no_pkinit,
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
line 419, in create_replica
self.start_creation(runtime=30)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
line 506, in start_creation
run_step(full_msg, method)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/service.py",
line 496, in run_step
method()
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
line 977, in __setup_s4u2proxy
__add_principal('ipa-http-delegation', 'HTTP', self)
File
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstance.py",
line 973, in __add_principal
api.Backend.ldap2.update_entry(entry)
File
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line 1572, in update_entry
self.conn.modify_s(str(entry.dn), modlist)
File "/usr/lib/python3.6/contextlib.py", line 99, in
__exit__
self.gen.throw(type, value, traceback)
File
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
line 1028, in error_handler
error=info)
2018-05-02T00:22:45Z DEBUG The ipa-replica-install command
failed, exception: NetworkError: cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
2018-05-02T00:22:45Z ERROR cannot connect to
'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
2018-05-02T00:22:45Z ERROR The ipa-replica-install command
failed. See /var/log/ipareplica-install.log for more information
Maybe it just is trying to soon and it hasn't fully started
yet? Because now I see that the previous step ALSO shows the
URL encoded socket path, and it worked fine - but it looks
like the previous step used ldapmodify, but the failing step
was accessing LDAP from Python? Perhaps it works fine one
way and not the other ?
I won't rule it out but I doubt it. Is ns-slapd still running at
this point? This could happen, for example, if ns-slapd crashed
after enabling the automember plugin. The DS error log might
have something to say as well.
rob
On Wed, May 2, 2018 at 3:29 PM, Rob Crittenden
<rcritten(a)redhat.com <mailto:rcritten@redhat.com>
<mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>
wrote:
Jonathan Vaughn via FreeIPA-users wrote:
Yes, I know, not recommended etc, low performance.
I'm not going
to run the CA on it. I just want to have a backup
LDAP/Kerberos
server.
Right now I'm just trying to test things out. I've
got a master
and a replica (so you could say two masters I
suppose) running
in Virtualbox VMs, and I'm trying to set up a 3rd
replica on a
Pi. All are Fedors 27. I had to downgrade httpd due to
https://pagure.io/freeipa/issue/7493
<https://pagure.io/freeipa/issue/7493>
<https://pagure.io/freeipa/issue/7493
<https://pagure.io/freeipa/issue/7493>> to even set up the first
VM replica, but this issue is separate.
Currently, the problem is it can't connect to it's
own LDAP
instance due to some kind of error ...
ipa-replica-install
worked fine on the x86_64 VM but on the armv71 Pi
3B when it
tries to connect to LDAPI instead of
using 'ldapi:///var/run//slapd-COMPANY-INTERNAL.socket' it
uses 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket'.
So it seems there is yet another ARM (or
non-x86_64) bug ...
similar to the problem with httpd and passing the
KRB5CCNAME
properly https://pagure.io/freeipa/issue/7337
<https://pagure.io/freeipa/issue/7337>
<https://pagure.io/freeipa/issue/7337
<https://pagure.io/freeipa/issue/7337>>
Any ideas on where to look to patch in a fix to
this so it uses
the correct filename? The socket file is there ...
and (at the
time it tries) LDAP is running.
What makes you think the ldapi URI is the problem?
Can you share the logs?
rob
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
4:19 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Here's a stacktrace per that FAQ entry
GNU gdb (GDB) Fedora 8.0.1-36.fc27
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute
it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "armv7hl-redhat-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>;.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
/usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
done.
[New LWP 17245]
[New LWP 17210]
[New LWP 17208]
[New LWP 17211]
[New LWP 17221]
[New LWP 17214]
[New LWP 17239]
[New LWP 17222]
[New LWP 17215]
[New LWP 17223]
[New LWP 17224]
[New LWP 17225]
[New LWP 17226]
[New LWP 17260]
[New LWP 17261]
[New LWP 17253]
[New LWP 17262]
[New LWP 17252]
[New LWP 17263]
[New LWP 17251]
[New LWP 17264]
[New LWP 17250]
[New LWP 17267]
[New LWP 17249]
[New LWP 17268]
[New LWP 17248]
[New LWP 17247]
[New LWP 17246]
[New LWP 17244]
[New LWP 17243]
[New LWP 17240]
[New LWP 17241]
[New LWP 17238]
[New LWP 17229]
[New LWP 17228]
[New LWP 17227]
[New LWP 17216]
[New LWP 17212]
[New LWP 17269]
[New LWP 17266]
[New LWP 17265]
[New LWP 17259]
[New LWP 17258]
[New LWP 17213]
[New LWP 17217]
[New LWP 17218]
[New LWP 17219]
[New LWP 17220]
[New LWP 17230]
[New LWP 17231]
[New LWP 17232]
[New LWP 17233]
[New LWP 17234]
[New LWP 17254]
[New LWP 17255]
[New LWP 17256]
[New LWP 17257]
[New LWP 17236]
[New LWP 17237]
[New LWP 17242]
Missing separate debuginfo for /usr/lib/dirsrv/libnunc-stans.so.0
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/3a/9153b2860989f51a2522ce96d034a229c1438a.debug
Missing separate debuginfo for /usr/lib/dirsrv/libslapd.so.0
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/cf/da5da98d7b486884dd54fcb480f90b55ca7380.debug
Missing separate debuginfo for /usr/lib/dirsrv/libsds.so.0
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/51/33bfed2a83bbc28037162723be9726217d1736.debug
Missing separate debuginfo for /usr/lib/dirsrv/libldaputil.so.0
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/f7/6e7e6e283afb9708689bacf06c45e91898800d.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Missing separate debuginfo for /usr/lib/dirsrv/libns-dshttpd-1.3.7.10.so
Try: dnf --enablerepo='*debug*' install
/usr/lib/debug/.build-id/5e/998d123be26e436e64e5e3e5c7684c55ebba0e.debug
Core was generated by `/usr/sbin/ns-slapd -D
/etc/dirsrv/slapd-COMPANY-INTERNAL -i /var/run/dirsrv/s'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0xb67bbf2e in strlen () from /lib/libc.so.6
[Current thread is 1 (Thread 0x9e13c280 (LWP 17245))]
Thread 60 (Thread 0x9f9a6280 (LWP 17242)):
#0 0xb69a1320 in pthread_cond_timedwait@(a)GLIBC_2.4 () at
/lib/libpthread.so.0
#1 0xb6a11f60 in pt_TimedWait () at /lib/libnspr4.so
#2 0xb6a12272 in PR_WaitCondVar () at /lib/libnspr4.so
#3 0xb3547fcc in protocol_sleep (prp=0x195d8c0, duration=300000) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:1222
#4 0xb354c378 in repl5_inc_run (prp=0x195d8c0) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:782
rc = <optimized out prp_priv = 0xb6fe4a50
<__stack_chk_guard replica = 0x0
cons_schema_csn = 0x2bf6540
ruv = 0x0
num_changes_sent = <optimized out
use_busy_backoff_timer = <optimized out
next_fire_time = <optimized out now = <optimized out busywaittime = 0
pausetime = 0
loops = <optimized out wait_change_timer_set = 1
current_state = 2
next_state = 2
done = 0
e1 = <optimized out #5 0xb3551a60 in prot_thread_main
(arg=0x143c8a0) at
ldap/servers/plugins/replication/repl5_protocol.c:255
rp = 0x143c8a0
done = 0
agmt = 0x131e900
#6 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#7 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#8 0xb6818da8 in None () at /lib/libc.so.6
Thread 59 (Thread 0xa21ab280 (LWP 17237)):
#0 0xb6811c14 in select () at /lib/libc.so.6
#1 0xb6f55694 in DS_Sleep () at /usr/lib/dirsrv/libslapd.so.0
#2 0xb362eccc in checkpoint_threadmain (param=0x1331340) at
ldap/servers/slapd/back-ldbm/dblayer.c:4323
interval = 0
rval = <optimized out priv = <optimized out li = 0x1331340
debug_checkpointing = 0
home_dir = <optimized out
list = 0x0
listp = <optimized out penv = 0xa21a9c04
checkpoint_expire = {tv_sec = 949530, tv_nsec = 734434832}
compactdb_expire = {tv_sec = 3541470, tv_nsec = 734432905}
compactdb_interval_update = <optimized out
checkpoint_interval_update = <optimized out
compactdb_interval = <optimized out
checkpoint_interval = <optimized out
txn = {back_txn_txn = 0x0}
#3 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#5 0xb6818da8 in None () at /lib/libc.so.6
Thread 58 (Thread 0xa29ac280 (LWP 17236)):
#0 0xb6811c14 in select () at /lib/libc.so.6
#1 0xb6f55694 in DS_Sleep () at /usr/lib/dirsrv/libslapd.so.0
#2 0xb362b0cc in deadlock_threadmain (param=<optimized out>) at
ldap/servers/slapd/back-ldbm/dblayer.c:4101
rval = <optimized out priv = 0x1330460
interval = 100
#3 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#5 0xb6818da8 in None () at /lib/libc.so.6
Thread 57 (Thread 0x980df280 (LWP 17257)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeb10,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeb10
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 56 (Thread 0x988e0280 (LWP 17256)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeae0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeae0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 55 (Thread 0x990e1280 (LWP 17255)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aebd0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aebd0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 54 (Thread 0x998e2280 (LWP 17254)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeba0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeba0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 53 (Thread 0xa71b5280 (LWP 17234)):
#0 0xb6818ed8 in epoll_pwait () at /lib/libc.so.6
#1 0xb6819188 in epoll_wait () at /lib/libc.so.6
#2 0xb68de154 in epoll_dispatch () at /lib/libevent-2.0.so.5
#3 0xb68c698c in event_base_loop () at /lib/libevent-2.0.so.5
#4 0xb6fa2568 in ns_event_fw_loop () at /usr/lib/dirsrv/libnunc-stans.so.0
#5 0xb6fa113c in event_loop_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#6 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#7 0xb6818da8 in None () at /lib/libc.so.6
Thread 52 (Thread 0xa79b6280 (LWP 17233)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 51 (Thread 0xa81b7280 (LWP 17232)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 50 (Thread 0xa89b8280 (LWP 17231)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 49 (Thread 0xa91b9280 (LWP 17230)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 48 (Thread 0xae1c3280 (LWP 17220)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 47 (Thread 0xae9c4280 (LWP 17219)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 46 (Thread 0xaf1c5280 (LWP 17218)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 45 (Thread 0xaf9c6280 (LWP 17217)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 44 (Thread 0xb19ca280 (LWP 17213)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 43 (Thread 0x978de280 (LWP 17258)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeb40,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeb40
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 42 (Thread 0x970dd280 (LWP 17259)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeb70,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeb70
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 41 (Thread 0x940d7280 (LWP 17265)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aedb0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aedb0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 40 (Thread 0x938d6280 (LWP 17266)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aed50,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aed50
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 39 (Thread 0x920d3280 (LWP 17269)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aed80,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aed80
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 38 (Thread 0xb21cb280 (LWP 17212)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 37 (Thread 0xb01c7280 (LWP 17216)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 36 (Thread 0xaa9bc280 (LWP 17227)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 35 (Thread 0xaa1bb280 (LWP 17228)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 34 (Thread 0xa99ba280 (LWP 17229)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 33 (Thread 0xa19aa280 (LWP 17238)):
#0 0xb6811c14 in select () at /lib/libc.so.6
#1 0xb6f55694 in DS_Sleep () at /usr/lib/dirsrv/libslapd.so.0
#2 0xb362b340 in trickle_threadmain (param=<optimized out>) at
ldap/servers/slapd/back-ldbm/dblayer.c:4538
interval = 250
rval = <optimized out priv = 0x1330460
debug_checkpointing = 0
#3 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#5 0xb6818da8 in None () at /lib/libc.so.6
Thread 32 (Thread 0xa01a7280 (LWP 17241)):
#0 0xb69a1320 in pthread_cond_timedwait@(a)GLIBC_2.4 () at
/lib/libpthread.so.0
#1 0xb6a11f60 in pt_TimedWait () at /lib/libnspr4.so
#2 0xb6a12272 in PR_WaitCondVar () at /lib/libnspr4.so
#3 0xb353218c in _cl5TrimMain (param=<optimized out>) at
ldap/servers/plugins/replication/cl5_api.c:3287
timePrev = <optimized out
timeCompactPrev = <optimized out
timeNow = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 31 (Thread 0xa09a8280 (LWP 17240)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0xb6f41c14 in slapi_wait_condvar () at /usr/lib/dirsrv/libslapd.so.0
#3 0xb37d92fc in cos_cache_wait_on_change (arg=<optimized out>) at
ldap/servers/plugins/cos/cos_cache.c:404
#4 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 30 (Thread 0x9f151280 (LWP 17243)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0xb6f41c14 in slapi_wait_condvar () at /usr/lib/dirsrv/libslapd.so.0
#3 0xb3415718 in roles_cache_wait_on_change (arg=0x14f9200) at
ldap/servers/plugins/roles/roles_cache.c:387
roles_def = 0x14f9200
#4 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 29 (Thread 0x9e93d280 (LWP 17244)):
#0 0xb69a1320 in pthread_cond_timedwait@(a)GLIBC_2.4 () at
/lib/libpthread.so.0
#1 0xb6a11f60 in pt_TimedWait () at /lib/libnspr4.so
#2 0xb6a12272 in PR_WaitCondVar () at /lib/libnspr4.so
#3 0x00503d34 in housecleaning (cur_time=<optimized out>) at
ldap/servers/slapd/house.c:46
interval = 30000
#4 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 28 (Thread 0x9d8ea280 (LWP 17246)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17ae960,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17ae960
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 27 (Thread 0x9d0e9280 (LWP 17247)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17ae990,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17ae990
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 26 (Thread 0x9c8e8280 (LWP 17248)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17ae9f0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17ae9f0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 25 (Thread 0x928d4280 (LWP 17268)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aecf0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aecf0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 24 (Thread 0x9c0e7280 (LWP 17249)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aea20,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aea20
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 23 (Thread 0x930d5280 (LWP 17267)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aed20,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aed20
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 22 (Thread 0x9b8e6280 (LWP 17250)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17ae9c0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17ae9c0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 21 (Thread 0x948d8280 (LWP 17264)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aecc0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aecc0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 20 (Thread 0x9b0e5280 (LWP 17251)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aea50,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aea50
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 19 (Thread 0x950d9280 (LWP 17263)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aec30,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aec30
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 18 (Thread 0x9a8e4280 (LWP 17252)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aea80,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aea80
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 17 (Thread 0x958da280 (LWP 17262)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aec90,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aec90
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 16 (Thread 0x9a0e3280 (LWP 17253)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeab0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeab0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 15 (Thread 0x960db280 (LWP 17261)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aec60,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aec60
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 14 (Thread 0x968dc280 (LWP 17260)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aec00,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aec00
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 13 (Thread 0xab1bd280 (LWP 17226)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 12 (Thread 0xab9be280 (LWP 17225)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 11 (Thread 0xac1bf280 (LWP 17224)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 10 (Thread 0xac9c0280 (LWP 17223)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 9 (Thread 0xb09c8280 (LWP 17215)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 8 (Thread 0xad1c1280 (LWP 17222)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 7 (Thread 0xa11a9280 (LWP 17239)):
#0 0xb6811c14 in select () at /lib/libc.so.6
#1 0xb6f55694 in DS_Sleep () at /usr/lib/dirsrv/libslapd.so.0
#2 0xb3625b2c in perf_threadmain (param=<optimized out>) at
ldap/servers/slapd/back-ldbm/dblayer.c:3599
priv = 0x1330460
li = <optimized out #3 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#5 0xb6818da8 in None () at /lib/libc.so.6
Thread 6 (Thread 0xb11c9280 (LWP 17214)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 5 (Thread 0xad9c2280 (LWP 17221)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 4 (Thread 0xb29cc280 (LWP 17211)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 3 (Thread 0xb5fb70c0 (LWP 17208)):
#0 0xb699b414 in pthread_join () at /lib/libpthread.so.0
#1 0xb6fa2230 in ns_thrpool_wait () at /usr/lib/dirsrv/libnunc-stans.so.0
#2 0x00500680 in slapd_daemon (ports=<optimized out>, tp=<optimized out>)
at ldap/servers/slapd/daemon.c:1081
n_tcps = 0x14c2f40
s_tcps = 0x0
i_unix = 0x14c2ec8
fdesp = <optimized out num_poll = 3
pr_timeout = <optimized out
threads = <optimized out in_referral_mode =
<optimized out #3 0x004f1bf8 in main (argc=0,
argv=0xbec1fcc4) at
ldap/servers/slapd/main.c:1200
return_value = <optimized out
mcfg = {extraname = 0x0, slapd_exemode = 1, n_port = 389, i_port =
1, s_port = 636, myname = 0x11f2eb0 "ns-slapd", ldif_file = 0x0, ldif_files
= 0, cmd_line_instance_name = 0x0, cmd_line_instance_names = 0x0,
skip_db_protect_check = 0, db2ldif_include = 0x0, db2ldif_exclude = 0x0,
ldif2db_removedupvals = 1, ldif2db_noattrindexes = 0, db2index_attrs = 0x0,
ldif_printkey = 5, archive_name = 0x0, db2ldif_dump_replica = 0,
db2ldif_dump_uniqueid = 1, ldif2db_generate_uniqueid = 1,
ldif2db_namespaceid = 0x0, importexport_encrypt = 0, upgradedb_flags = 0,
upgradednformat_dryrun = 0, is_quiet = 0, dbverify_verbose = 0,
dbverify_dbdir = 0x0}
slapdFrontendConfig = <optimized out
ports_info = {n_port = 389, s_port = 636, n_listenaddr = 0x14c2ef0,
s_listenaddr = 0x14c2f98, n_socket = 0x14c2f40, i_listenaddr = 0x14c2e08,
i_port = 1, i_socket = 0x14c2ec8, s_socket = 0x0}
tp = 0x12a3f00
m = <optimized out notify = <optimized
out
Thread 2 (Thread 0xb31cd280 (LWP 17210)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in worker_thread_func () at
/usr/lib/dirsrv/libnunc-stans.so.0
#2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#3 0xb6818da8 in None () at /lib/libc.so.6
Thread 1 (Thread 0x9e13c280 (LWP 17245)):
#0 0xb67bbf2e in strlen () at /lib/libc.so.6
#1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
#2 0x00000000 in None ()
On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden <rcritten(a)redhat.com> wrote:
> Jonathan Vaughn via FreeIPA-users wrote:
>> Still trying to figure this out. It looks like slapd
is dying, I thought
>> it was still running for some reason.
> >> slapd is dying to segfault. strace of it happening
doesn't seem to reveal
>> much:
> > A stack trace would very much
help trying to track down the cause.
>
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#
> debugging-crashes
> rob
>> 18:32:41.543717 (+
0.000801) openat(AT_FDCWD,
>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
O_WRONLY|O_CREAT|O_TRUNC,
>> 0666) = 32
>> 18:32:41.544907 (+ 0.001195) getpid() = 16014
>> 18:32:41.545269 (+ 0.000329) fstat64(32, {st_mode=S_IFREG|0644,
>> st_size=0, ...}) = 0
>> 18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) = 6
>> 18:32:41.546603 (+ 0.000818) close(32) = 0
>> 18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>> 0644) = 0
>> 18:32:41.547741 (+ 0.000676) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>> 18:32:41.548324 (+ 0.000587) write(2, "[07/May/2018:18:32:41.548030641
>> "..., 134) = 134
>> 18:32:41.551096 (+ 0.002840) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>> 18:32:41.551568 (+ 0.000406) write(4, "[07/May/2018:18:32:41.551287555
>> "..., 134) = 134
>> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
>> 18:32:41.558499 (+ 0.006170) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>> 18:32:41.558901 (+ 0.000350) write(2, "[07/May/2018:18:32:41.558678099
>> "..., 136) = 136
>> 18:32:41.561537 (+ 0.002680) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>> 18:32:41.562357 (+ 0.000793) write(4, "[07/May/2018:18:32:41.561718659
>> "..., 136) = 136
>> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
>> 18:32:41.566928 (+ 0.003452) getpid() = 16014
>> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC,
>> 0) = 32
>> 18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET, SO_SNDBUF,
>> [163840], [4]) = 0
>> 18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET,
>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
>> 18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET, SO_SNDBUF,
>> [8388608], 4) = 0
>> 18:32:41.570064 (+ 0.000513) sendmsg(32,
>> {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
>> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
>> MSG_NOSIGNAL) = 69
>> 18:32:41.570845 (+ 0.000789) close(32) = 0
>> 18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8, FUTEX_WAIT, 16016,
>> NULL) = ?
>> 18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
> >> It doesn't die immediately after hitting that
final futex call - it's
>> just over 20s later. There's still plenty of free memory at the time.
> >> # free -m
>> total used free shared buff/cache
>> available
>> Mem: 974 222 156 0 594
>> 679
>> Swap: 2535 24 2511
> >> The last entries in the error log is that it is
listening on socket /
>> port. No error logged for the segfault.
> >> I did see that during previous setup steps (before it
dies) there is
>> apparently an error setting up replication:
> >> [07/May/2018:18:17:46.245041807 -0500] - ERR -
NSMMReplicationPlugin -
>> acquire_replica - agmt="cn=meToipa-12.company.internal" (ipa-12:5):
>> Unable to acquire replica: permission denied. The bind dn "" does not
have
>> permission to supply replication updates to the replica. Will retry later.
> >> Possibly, the failure to configure replication is
leading to the
>> segfault? Any ideas why it would not have permission / what to check to
>> ensure it has permission ?
> > >> On Thu, May 3, 2018 at 4:02
PM, Jonathan Vaughn <jonathan(a)creatuity.com
>> <mailto:jonathan@creatuity.com>> wrote:
> >> It's still running.
> >> Here's the error log from slapd during that
run:
> >> [01/May/2018:19:22:24.567650453 -0500] - ERR -
NSMMReplicationPlugin
>> - multimaster_extop_StartNSDS50ReplicationRequest - conn=42 op=5
>> replica="dc=company,dc=internal": Unable to acquire replica:
error:
>> permission denied
>> [01/May/2018:19:22:24.613143629 -0500] - ERR - NSMMReplicationPlugin
>> - multimaster_extop_StartNSDS50ReplicationRequest - conn=43 op=5
>> replica="dc=company,dc=internal": Unable to acquire replica:
error:
>> permission denied
>> [01/May/2018:19:22:31.445153171 -0500] - INFO -
>> NSMMReplicationPlugin - repl5_tot_run - Beginning total update of
>> replica "agmt="cn=meToipa-11.company.internal"
(ipa-11:389)".
>> [01/May/2018:19:22:31.672485209 -0500] - NOTICE -
>> NSMMReplicationPlugin - replica_subentry_check - Need to create
>> replication keep alive entry <cn=repl keep alive
>> 7,dc=company,dc=internal >>
[01/May/2018:19:22:31.674922201 -0500] - INFO -
>> NSMMReplicationPlugin - replica_subentry_create - add dn: cn=repl
>> keep alive 7,dc=company,dc=internal
>> objectclass: top
>> objectclass: ldapsubentry
>> objectclass: extensibleObject
>> cn: repl keep alive 7
>> [01/May/2018:19:22:36.125109819 -0500] - ERR - NSMMReplicationPlugin
>> - multimaster_extop_StartNSDS50ReplicationRequest - conn=44 op=5
>> replica="dc=company,dc=internal": Unable to acquire replica:
error:
>> permission denied
>> [01/May/2018:19:22:36.598178749 -0500] - INFO -
>> NSMMReplicationPlugin - repl5_tot_run - Finished total update of
>> replica "agmt="cn=meToipa-11.company.internal"
(ipa-11:389)". Sent
>> 503 entries.
>> [01/May/2018:19:27:40.482836683 -0500] - ERR - NSMMReplicationPlugin
>> - bind_and_check_pwp - agmt="cn=meToipa-11.company.internal"
>> (ipa-11:389) - Replication bind with GSSAPI auth failed: LDAP error
>> -1 (Can't contact LDAP server) ()
> >> ipa-11 is the Pi I'm trying to set up
replication on. Looks like
>> there was a problem setting up replication due to permissions? But
>> the other replica started up just fine ...
> >> On Thu, May 3, 2018 at 3:13 PM, Rob Crittenden
<rcritten(a)redhat.com
>> <mailto:rcritten@redhat.com>> wrote:
> >> Jonathan Vaughn wrote:
> >> Here's the output from
ipa-replica-install :
> >> # ipa-replica-install
>> WARNING: conflicting time&date synchronization service
>> 'chronyd' will
>> be disabled in favor of ntpd
> >> Password for admin(a)COMPANY.INTERNAL:
>> Run connection check to master
>> Connection check OK
>> Configuring NTP daemon (ntpd)
>> [1/4]: stopping ntpd
>> [2/4]: writing configuration
>> [3/4]: configuring ntpd to start on boot
>> [4/4]: starting ntpd
>> Done configuring NTP daemon (ntpd).
>> Configuring directory server (dirsrv). Estimated time: 30
>> seconds
>> [1/41]: creating directory server instance
>> [2/41]: enabling ldapi
>> [3/41]: configure autobind for root
>> [4/41]: stopping directory server
>> [5/41]: updating configuration in dse.ldif
>> [6/41]: starting directory server
>> [7/41]: adding default schema
>> [8/41]: enabling memberof plugin
>> [9/41]: enabling winsync plugin
>> [10/41]: configuring replication version plugin
>> [11/41]: enabling IPA enrollment plugin
>> [12/41]: configuring uniqueness plugin
>> [13/41]: configuring uuid plugin
>> [14/41]: configuring modrdn plugin
>> [15/41]: configuring DNS plugin
>> [16/41]: enabling entryUSN plugin
>> [17/41]: configuring lockout plugin
>> [18/41]: configuring topology plugin
>> [19/41]: creating indices
>> [20/41]: enabling referential integrity plugin
>> [21/41]: configuring certmap.conf
>> [22/41]: configure new location for managed entries
>> [23/41]: configure dirsrv ccache
>> [24/41]: enabling SASL mapping fallback
>> [25/41]: restarting directory server
>> [26/41]: creating DS keytab
>> [27/41]: ignore time skew for initial replication
>> [28/41]: setting up initial replication
>> Starting replication, please wait until this has completed.
>> Update in progress, 11 seconds elapsed
>> Update succeeded
> >> [29/41]: prevent time skew after
initial replication
>> [30/41]: adding sasl mappings to the directory
>> [31/41]: updating schema
>> [32/41]: setting Auto Member configuration
>> [33/41]: enabling S4U2Proxy delegation
>> [error] NetworkError: cannot connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>> Your system may be partly configured.
>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
> >> ipapython.admintool: ERROR cannot
connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>> ipapython.admintool: ERROR The ipa-replica-install
>> command failed. See /var/log/ipareplica-install.log for more
>> information
> >> And here's the
/var/log/ipareplica-install.log from just
>> before it fails to the failure:
> >> 2018-05-02T00:22:44Z DEBUG [32/41]:
setting Auto Member
>> configuration
>> 2018-05-02T00:22:44Z DEBUG Starting external process
>> 2018-05-02T00:22:44Z DEBUG args=/usr/bin/ldapmodify -v -f
>> /tmp/tmpt7bxf4x1 -H
>> ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket -Y
>> EXTERNAL
>> 2018-05-02T00:22:45Z DEBUG Process finished, return code=0
>> 2018-05-02T00:22:45Z DEBUG stdout=add
>> nsslapd-pluginConfigArea:
>> cn=automember,cn=etc,dc=company,dc=internal
>> modifying entry "cn=Auto Membership
>> Plugin,cn=plugins,cn=config"
>> modify complete
> > >>
2018-05-02T00:22:45Z DEBUG stderr=ldap_initialize(
>> ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket/??base )
>> SASL/EXTERNAL authentication started
>> SASL username:
>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>> SASL SSF: 0
> >> 2018-05-02T00:22:45Z DEBUG duration: 0
seconds
>> 2018-05-02T00:22:45Z DEBUG [33/41]: enabling S4U2Proxy
>> delegation
>> 2018-05-02T00:22:45Z DEBUG Traceback (most recent call last):
>> File
>> "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>> line 979, in error_handler
>> yield
>> File
>> "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>> line 1572, in update_entry
>> self.conn.modify_s(str(entry.dn), modlist)
>> File
>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>> 600, in modify_s
>> return self.modify_ext_s(dn,modlist,None,None)
>> File
>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>> 573, in modify_ext_s
>> resp_type, resp_data, resp_msgid, resp_ctrls =
>> self.result3(msgid,all=1,timeout=self.timeout)
>> File
>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>> 714, in result3
>> resp_ctrl_classes=resp_ctrl_classes
>> File
>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>> 721, in result4
>> ldap_result =
>> self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,
>> add_intermediates,add_extop)
>> File
>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>> 294, in _ldap_call
>> result = func(*args,**kwargs)
>> ldap.SERVER_DOWN: {'desc': "Can't contact LDAP
server"}
> >> During handling of the above exception,
another exception
>> occurred:
> >> Traceback (most recent call last):
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/service.
>> py",
>> line 506, in start_creation
>> run_step(full_msg, method)
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/service.
>> py",
>> line 496, in run_step
>> method()
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>> ce.py",
>> line 977, in __setup_s4u2proxy
>> __add_principal('ipa-http-delegation', 'HTTP',
self)
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>> ce.py",
>> line 973, in __add_principal
>> api.Backend.ldap2.update_entry(entry)
>> File
>> "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>> line 1572, in update_entry
>> self.conn.modify_s(str(entry.dn), modlist)
>> File "/usr/lib/python3.6/contextlib.py", line 99, in
>> __exit__
>> self.gen.throw(type, value, traceback)
>> File
>> "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>> line 1028, in error_handler
>> error=info)
>> ipalib.errors.NetworkError: cannot connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
> >> 2018-05-02T00:22:45Z DEBUG [error]
NetworkError: cannot
>> connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>> 2018-05-02T00:22:45Z DEBUG Destroyed connection
>> context.ldap2_2991558640
>> 2018-05-02T00:22:45Z DEBUG Backing up system configuration
>> file '/etc/ipa/default.conf'
>> 2018-05-02T00:22:45Z DEBUG Saving Index File to
>> '/var/lib/ipa/sysrestore/sysrestore.index'
>> 2018-05-02T00:22:45Z DEBUG File
>> "/usr/lib/python3.6/site-packages/ipapython/admintool.py",
>> line 174, in execute
>> return_value = self.run()
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/cli.py",
>> line 319, in run
>> cfgr.run()
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 364, in run
>> self.execute()
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 388, in execute
>> for _nothing in self._executor():
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 430, in __runner
>> exc_handler(exc_info)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 459, in _handle_execute_exception
>> self._handle_exception(exc_info)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 449, in _handle_exception
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line
>> 693, in reraise
>> raise value
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 420, in __runner
>> step()
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 417, in <lambda >>
step = lambda: next(self.__gen)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>> line
>> 81, in run_generator_with_yield_from
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line
>> 693, in reraise
>> raise value
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>> line
>> 59, in run_generator_with_yield_from
>> value = gen.send(prev_value)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 654, in _configure
>> next(executor)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 430, in __runner
>> exc_handler(exc_info)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 459, in _handle_execute_exception
>> self._handle_exception(exc_info)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 517, in _handle_exception
>> self.__parent._handle_exception(exc_info)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 449, in _handle_exception
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line
>> 693, in reraise
>> raise value
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 514, in _handle_exception
>> super(ComponentBase, self)._handle_exception(exc_info)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 449, in _handle_exception
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line
>> 693, in reraise
>> raise value
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 420, in __runner
>> step()
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>> line
>> 417, in <lambda >>
step = lambda: next(self.__gen)
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>> line
>> 81, in run_generator_with_yield_from
>> six.reraise(*exc_info)
>> File "/usr/lib/python3.6/site-packages/six.py", line
>> 693, in reraise
>> raise value
>> File
>>
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>> line
>> 59, in run_generator_with_yield_from
>> value = gen.send(prev_value)
>> File
>> "/usr/lib/python3.6/site-packages/ipapython/install/common.
>> py",
>> line 66, in _install
>> for unused in self._installer(self.parent):
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/server/_
>> _init__.py",
>> line 622, in main
>> replica_install(self)
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/server/
>> replicainstall.py",
>> line 388, in decorated
>> func(installer)
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/server/
>> replicainstall.py",
>> line 1407, in install
>> pkcs12_info=dirsrv_pkcs12_info)
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/server/
>> replicainstall.py",
>> line 110, in install_replica_ds
>> setup_pkinit=not options.no_pkinit,
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>> ce.py",
>> line 419, in create_replica
>> self.start_creation(runtime=30)
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/service.
>> py",
>> line 506, in start_creation
>> run_step(full_msg, method)
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/service.
>> py",
>> line 496, in run_step
>> method()
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>> ce.py",
>> line 977, in __setup_s4u2proxy
>> __add_principal('ipa-http-delegation', 'HTTP',
self)
>> File
>> "/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>> ce.py",
>> line 973, in __add_principal
>> api.Backend.ldap2.update_entry(entry)
>> File
>> "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>> line 1572, in update_entry
>> self.conn.modify_s(str(entry.dn), modlist)
>> File "/usr/lib/python3.6/contextlib.py", line 99, in
>> __exit__
>> self.gen.throw(type, value, traceback)
>> File
>> "/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>> line 1028, in error_handler
>> error=info)
> >> 2018-05-02T00:22:45Z DEBUG The
ipa-replica-install command
>> failed, exception: NetworkError: cannot connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>> 2018-05-02T00:22:45Z ERROR cannot connect to
>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>> 2018-05-02T00:22:45Z ERROR The ipa-replica-install command
>> failed. See /var/log/ipareplica-install.log for more
>> information
> >> Maybe it just is trying to soon and it
hasn't fully started
>> yet? Because now I see that the previous step ALSO shows the
>> URL encoded socket path, and it worked fine - but it looks
>> like the previous step used ldapmodify, but the failing step
>> was accessing LDAP from Python? Perhaps it works fine one
>> way and not the other ?
> > >> I won't rule it
out but I doubt it. Is ns-slapd still running at
>> this point? This could happen, for example, if ns-slapd crashed
>> after enabling the automember plugin. The DS error log might
>> have something to say as well.
> >> rob
> > >> On Wed, May 2,
2018 at 3:29 PM, Rob Crittenden
>> <rcritten(a)redhat.com <mailto:rcritten@redhat.com >> <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>> > >> wrote:
> >> Jonathan Vaughn via FreeIPA-users
wrote:
> >> Yes, I know, not recommended etc,
low performance.
>> I'm not going
>> to run the CA on it. I just want to have a backup
>> LDAP/Kerberos
>> server.
> >> Right now I'm just trying to
test things out. I've
>> got a master
>> and a replica (so you could say two masters I
>> suppose) running
>> in Virtualbox VMs, and I'm trying to set up a 3rd
>> replica on a
>> Pi. All are Fedors 27. I had to downgrade httpd due
>> to
>> https://pagure.io/freeipa/issue/7493
>> <https://pagure.io/freeipa/issue/7493 >>
<https://pagure.io/freeipa/issue/7493
>> <https://pagure.io/freeipa/issue/7493>> to even set up the
>> first
>> VM replica, but this issue is separate.
> >> Currently, the problem is it
can't connect to it's
>> own LDAP
>> instance due to some kind of error ...
>> ipa-replica-install
>> worked fine on the x86_64 VM but on the armv71 Pi
>> 3B when it
>> tries to connect to LDAPI instead of
>> using
'ldapi:///var/run//slapd-COMPANY-INTERNAL.socket'
>> it
>> uses 'ldapi://%2Fvar%2Frun%2Fs
>> lapd-COMPANY-INTERNAL.socket'.
> >> So it seems there is yet another
ARM (or
>> non-x86_64) bug ...
>> similar to the problem with httpd and passing the
>> KRB5CCNAME
>> properly https://pagure.io/freeipa/issue/7337
>> <https://pagure.io/freeipa/issue/7337 >>
<https://pagure.io/freeipa/issue/7337
>> <https://pagure.io/freeipa/issue/7337> > >> Any
ideas on where to look to patch in a fix to
>> this so it uses
>> the correct filename? The socket file is there ...
>> and (at the
>> time it tries) LDAP is running.
> > >> What makes
you think the ldapi URI is the problem?
> >> Can you share the logs?
> >> rob
> > > > > > > >> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedo
>> rahosted.org
> >
6:22 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Not sure if it makes a difference... I was looking into this again and
realized I had a bunch of messages from gdb telling me to install more
debuginfo. I've done that now, here it is again freshly run through gdb
GNU gdb (GDB) Fedora 8.0.1-36.fc27
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute
it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "armv7hl-redhat-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>;.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
/usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
done.
[New LWP 17245]
[New LWP 17210]
[New LWP 17208]
[New LWP 17211]
[New LWP 17221]
[New LWP 17214]
[New LWP 17239]
[New LWP 17222]
[New LWP 17215]
[New LWP 17223]
[New LWP 17224]
[New LWP 17225]
[New LWP 17226]
[New LWP 17260]
[New LWP 17261]
[New LWP 17253]
[New LWP 17262]
[New LWP 17252]
[New LWP 17263]
[New LWP 17251]
[New LWP 17264]
[New LWP 17250]
[New LWP 17267]
[New LWP 17249]
[New LWP 17268]
[New LWP 17248]
[New LWP 17247]
[New LWP 17246]
[New LWP 17244]
[New LWP 17243]
[New LWP 17240]
[New LWP 17241]
[New LWP 17238]
[New LWP 17229]
[New LWP 17228]
[New LWP 17227]
[New LWP 17216]
[New LWP 17212]
[New LWP 17269]
[New LWP 17266]
[New LWP 17265]
[New LWP 17259]
[New LWP 17258]
[New LWP 17213]
[New LWP 17217]
[New LWP 17218]
[New LWP 17219]
[New LWP 17220]
[New LWP 17230]
[New LWP 17231]
[New LWP 17232]
[New LWP 17233]
[New LWP 17234]
[New LWP 17254]
[New LWP 17255]
[New LWP 17256]
[New LWP 17257]
[New LWP 17236]
[New LWP 17237]
[New LWP 17242]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `/usr/sbin/ns-slapd -D
/etc/dirsrv/slapd-COMPANY-INTERNAL -i /var/run/dirsrv/s'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0xb67bbf2e in strlen () from /lib/libc.so.6
[Current thread is 1 (Thread 0x9e13c280 (LWP 17245))]
Thread 60 (Thread 0x9f9a6280 (LWP 17242)):
#0 0xb69a1320 in pthread_cond_timedwait@(a)GLIBC_2.4 () at
/lib/libpthread.so.0
#1 0xb6a11f60 in pt_TimedWait () at /lib/libnspr4.so
#2 0xb6a12272 in PR_WaitCondVar () at /lib/libnspr4.so
#3 0xb3547fcc in protocol_sleep (prp=0x195d8c0, duration=300000) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:1222
#4 0xb354c378 in repl5_inc_run (prp=0x195d8c0) at
ldap/servers/plugins/replication/repl5_inc_protocol.c:782
rc = <optimized out prp_priv = 0xb6fe4a50
<__stack_chk_guard replica = 0x0
cons_schema_csn = 0x2bf6540
ruv = 0x0
num_changes_sent = <optimized out
use_busy_backoff_timer = <optimized out
next_fire_time = <optimized out now = <optimized out busywaittime = 0
pausetime = 0
loops = <optimized out wait_change_timer_set = 1
current_state = 2
next_state = 2
done = 0
e1 = <optimized out #5 0xb3551a60 in prot_thread_main
(arg=0x143c8a0) at
ldap/servers/plugins/replication/repl5_protocol.c:255
rp = 0x143c8a0
done = 0
agmt = 0x131e900
#6 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#7 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#8 0xb6818da8 in None () at /lib/libc.so.6
Thread 59 (Thread 0xa21ab280 (LWP 17237)):
#0 0xb6811c14 in select () at /lib/libc.so.6
#1 0xb6f55694 in DS_Sleep (ticks=ticks@entry=0) at
ldap/servers/slapd/util.c:1086
mSecs = <optimized out tm = {tv_sec = 1, tv_usec =
53017}
#2 0xb362eccc in checkpoint_threadmain (param=0x1331340) at
ldap/servers/slapd/back-ldbm/dblayer.c:4323
interval = 0
rval = <optimized out priv = <optimized out li = 0x1331340
debug_checkpointing = 0
home_dir = <optimized out
list = 0x0
listp = <optimized out penv = 0xa21a9c04
checkpoint_expire = {tv_sec = 949530, tv_nsec = 734434832}
compactdb_expire = {tv_sec = 3541470, tv_nsec = 734432905}
compactdb_interval_update = <optimized out
checkpoint_interval_update = <optimized out
compactdb_interval = <optimized out
checkpoint_interval = <optimized out
txn = {back_txn_txn = 0x0}
#3 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#5 0xb6818da8 in None () at /lib/libc.so.6
Thread 58 (Thread 0xa29ac280 (LWP 17236)):
#0 0xb6811c14 in select () at /lib/libc.so.6
#1 0xb6f55694 in DS_Sleep (ticks=ticks@entry=100) at
ldap/servers/slapd/util.c:1086
mSecs = <optimized out tm = {tv_sec = 0, tv_usec =
61426}
#2 0xb362b0cc in deadlock_threadmain (param=<optimized out>) at
ldap/servers/slapd/back-ldbm/dblayer.c:4101
rval = <optimized out priv = 0x1330460
interval = 100
#3 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#5 0xb6818da8 in None () at /lib/libc.so.6
Thread 57 (Thread 0x980df280 (LWP 17257)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeb10,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeb10
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 56 (Thread 0x988e0280 (LWP 17256)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeae0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeae0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 55 (Thread 0x990e1280 (LWP 17255)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aebd0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aebd0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 54 (Thread 0x998e2280 (LWP 17254)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeba0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeba0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 53 (Thread 0xa71b5280 (LWP 17234)):
#0 0xb6818ed8 in epoll_pwait () at /lib/libc.so.6
#1 0xb6819188 in epoll_wait () at /lib/libc.so.6
#2 0xb68de154 in epoll_dispatch () at /lib/libevent-2.0.so.5
#3 0xb68c698c in event_base_loop () at /lib/libevent-2.0.so.5
#4 0xb6fa2568 in ns_event_fw_loop (ns_event_fw_ctx=<optimized out>) at
src/nunc-stans/ns/ns_event_fw_event.c:308
rc = <optimized out #5 0xb6fa113c in
event_loop_thread_func (arg=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:581
tp = 0x12a3f00
#6 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#7 0xb6818da8 in None () at /lib/libc.so.6
Thread 52 (Thread 0xa79b6280 (LWP 17233)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 51 (Thread 0xa81b7280 (LWP 17232)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 50 (Thread 0xa89b8280 (LWP 17231)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 49 (Thread 0xa91b9280 (LWP 17230)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 48 (Thread 0xae1c3280 (LWP 17220)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 47 (Thread 0xae9c4280 (LWP 17219)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 46 (Thread 0xaf1c5280 (LWP 17218)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 45 (Thread 0xaf9c6280 (LWP 17217)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 44 (Thread 0xb19ca280 (LWP 17213)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 43 (Thread 0x978de280 (LWP 17258)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeb40,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeb40
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 42 (Thread 0x970dd280 (LWP 17259)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeb70,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeb70
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 41 (Thread 0x940d7280 (LWP 17265)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aedb0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aedb0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 40 (Thread 0x938d6280 (LWP 17266)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aed50,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aed50
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 39 (Thread 0x920d3280 (LWP 17269)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aed80,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aed80
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 38 (Thread 0xb21cb280 (LWP 17212)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 37 (Thread 0xb01c7280 (LWP 17216)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 36 (Thread 0xaa9bc280 (LWP 17227)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 35 (Thread 0xaa1bb280 (LWP 17228)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 34 (Thread 0xa99ba280 (LWP 17229)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 33 (Thread 0xa19aa280 (LWP 17238)):
#0 0xb6811c14 in select () at /lib/libc.so.6
#1 0xb6f55694 in DS_Sleep (ticks=ticks@entry=250) at
ldap/servers/slapd/util.c:1086
mSecs = <optimized out tm = {tv_sec = 0, tv_usec =
76792}
#2 0xb362b340 in trickle_threadmain (param=<optimized out>) at
ldap/servers/slapd/back-ldbm/dblayer.c:4538
interval = 250
rval = <optimized out priv = 0x1330460
debug_checkpointing = 0
#3 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#5 0xb6818da8 in None () at /lib/libc.so.6
Thread 32 (Thread 0xa01a7280 (LWP 17241)):
#0 0xb69a1320 in pthread_cond_timedwait@(a)GLIBC_2.4 () at
/lib/libpthread.so.0
#1 0xb6a11f60 in pt_TimedWait () at /lib/libnspr4.so
#2 0xb6a12272 in PR_WaitCondVar () at /lib/libnspr4.so
#3 0xb353218c in _cl5TrimMain (param=<optimized out>) at
ldap/servers/plugins/replication/cl5_api.c:3287
timePrev = <optimized out
timeCompactPrev = <optimized out
timeNow = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 31 (Thread 0xa09a8280 (LWP 17240)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0xb6f41c14 in slapi_wait_condvar (cvar=0x18f6900, timeout=0x0) at
ldap/servers/slapd/slapi2nspr.c:149
prit = <optimized out #3 0xb37d92fc in
cos_cache_wait_on_change (arg=<optimized out>) at
ldap/servers/plugins/cos/cos_cache.c:404
#4 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 30 (Thread 0x9f151280 (LWP 17243)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0xb6f41c14 in slapi_wait_condvar (cvar=0x14f9340,
timeout=timeout@entry=0x0)
at ldap/servers/slapd/slapi2nspr.c:149
prit = <optimized out #3 0xb3415718 in
roles_cache_wait_on_change (arg=0x14f9200) at
ldap/servers/plugins/roles/roles_cache.c:387
roles_def = 0x14f9200
#4 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 29 (Thread 0x9e93d280 (LWP 17244)):
#0 0xb69a1320 in pthread_cond_timedwait@(a)GLIBC_2.4 () at
/lib/libpthread.so.0
#1 0xb6a11f60 in pt_TimedWait () at /lib/libnspr4.so
#2 0xb6a12272 in PR_WaitCondVar () at /lib/libnspr4.so
#3 0x00503d34 in housecleaning (cur_time=<optimized out>) at
ldap/servers/slapd/house.c:46
interval = 30000
#4 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 28 (Thread 0x9d8ea280 (LWP 17246)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17ae960,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17ae960
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 27 (Thread 0x9d0e9280 (LWP 17247)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17ae990,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17ae990
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 26 (Thread 0x9c8e8280 (LWP 17248)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17ae9f0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17ae9f0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 25 (Thread 0x928d4280 (LWP 17268)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aecf0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aecf0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 24 (Thread 0x9c0e7280 (LWP 17249)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aea20,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aea20
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 23 (Thread 0x930d5280 (LWP 17267)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aed20,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aed20
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 22 (Thread 0x9b8e6280 (LWP 17250)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17ae9c0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17ae9c0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 21 (Thread 0x948d8280 (LWP 17264)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aecc0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aecc0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 20 (Thread 0x9b0e5280 (LWP 17251)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aea50,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aea50
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 19 (Thread 0x950d9280 (LWP 17263)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aec30,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aec30
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 18 (Thread 0x9a8e4280 (LWP 17252)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aea80,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aea80
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 17 (Thread 0x958da280 (LWP 17262)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aec90,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aec90
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 16 (Thread 0x9a0e3280 (LWP 17253)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aeab0,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aeab0
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 15 (Thread 0x960db280 (LWP 17261)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aec60,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aec60
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 14 (Thread 0x968dc280 (LWP 17260)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
#2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work
(pb=pb@entry=0x17aec00,
interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
ret = 0
wqitem = 0x0
op_stack_obj = 0x0
#3 0x004fa6f0 in connection_threadmain () at
ldap/servers/slapd/connection.c:1528
pb_conn = 0x0
is_timedout = 0
curtime = 0
pb = 0x17aec00
conn = 0x0
op = 0x0
tag = 0
need_wakeup = -1224848816
thread_turbo_flag = 0
ret = <optimized out more_data = 0
replication_connection = <optimized out
doshutdown = 0
maxthreads = <optimized out
bypasspollcnt = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 13 (Thread 0xab1bd280 (LWP 17226)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 12 (Thread 0xab9be280 (LWP 17225)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 11 (Thread 0xac1bf280 (LWP 17224)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 10 (Thread 0xac9c0280 (LWP 17223)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 9 (Thread 0xb09c8280 (LWP 17215)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 8 (Thread 0xad1c1280 (LWP 17222)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 7 (Thread 0xa11a9280 (LWP 17239)):
#0 0xb6811c14 in select () at /lib/libc.so.6
#1 0xb6f55694 in DS_Sleep (ticks=<optimized out>) at
ldap/servers/slapd/util.c:1086
mSecs = <optimized out tm = {tv_sec = 0, tv_usec =
555659}
#2 0xb367a6b4 in perfctrs_wait (milliseconds=milliseconds@entry=1000,
priv=<optimized out>, db_env=<optimized out>) at
ldap/servers/slapd/back-ldbm/perfctrs.c:80
interval = <optimized out #3
0xb3625b2c in perf_threadmain (param=<optimized out>) at
ldap/servers/slapd/back-ldbm/dblayer.c:3599
priv = 0x1330460
li = <optimized out #4 0xb6a15d68 in _pt_root () at
/lib/libnspr4.so
#5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#6 0xb6818da8 in None () at /lib/libc.so.6
Thread 6 (Thread 0xb11c9280 (LWP 17214)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 5 (Thread 0xad9c2280 (LWP 17221)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 4 (Thread 0xb29cc280 (LWP 17211)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 3 (Thread 0xb5fb70c0 (LWP 17208)):
#0 0xb699b414 in pthread_join () at /lib/libpthread.so.0
#1 0xb6fa2230 in ns_thrpool_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:1614
rc = <optimized out retval = <optimized
out thr = 0x1430ce0
#2 0x00500680 in slapd_daemon (ports=<optimized out>, tp=<optimized out>)
at ldap/servers/slapd/daemon.c:1081
n_tcps = 0x14c2f40
s_tcps = 0x0
i_unix = 0x14c2ec8
fdesp = <optimized out num_poll = 3
pr_timeout = <optimized out
threads = <optimized out in_referral_mode =
<optimized out #3 0x004f1bf8 in main (argc=0,
argv=0xbec1fcc4) at
ldap/servers/slapd/main.c:1200
return_value = <optimized out
mcfg = {extraname = 0x0, slapd_exemode = 1, n_port = 389, i_port =
1, s_port = 636, myname = 0x11f2eb0 "ns-slapd", ldif_file = 0x0, ldif_files
= 0, cmd_line_instance_name = 0x0, cmd_line_instance_names = 0x0,
skip_db_protect_check = 0, db2ldif_include = 0x0, db2ldif_exclude = 0x0,
ldif2db_removedupvals = 1, ldif2db_noattrindexes = 0, db2index_attrs = 0x0,
ldif_printkey = 5, archive_name = 0x0, db2ldif_dump_replica = 0,
db2ldif_dump_uniqueid = 1, ldif2db_generate_uniqueid = 1,
ldif2db_namespaceid = 0x0, importexport_encrypt = 0, upgradedb_flags = 0,
upgradednformat_dryrun = 0, is_quiet = 0, dbverify_verbose = 0,
dbverify_dbdir = 0x0}
slapdFrontendConfig = <optimized out
ports_info = {n_port = 389, s_port = 636, n_listenaddr = 0x14c2ef0,
s_listenaddr = 0x14c2f98, n_socket = 0x14c2f40, i_listenaddr = 0x14c2e08,
i_port = 1, i_socket = 0x14c2ec8, s_socket = 0x0}
tp = 0x12a3f00
m = <optimized out notify = <optimized
out
Thread 2 (Thread 0xb31cd280 (LWP 17210)):
#0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
#1 0xb6fa14a8 in [IDLE THREAD] work_q_wait (tp=0x12a3f00) at
src/nunc-stans/ns/ns_thrpool.c:344
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#2 0xb6fa14a8 in worker_thread_func (arg=<optimized out>) at
src/nunc-stans/ns/ns_thrpool.c:396
job = 0x0
tp = 0x12a3f00
result = <optimized out is_shutdown = 0
#3 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
#4 0xb6818da8 in None () at /lib/libc.so.6
Thread 1 (Thread 0x9e13c280 (LWP 17245)):
#0 0xb67bbf2e in strlen () at /lib/libc.so.6
#1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
#2 0x00000000 in None ()
On Tue, May 8, 2018 at 4:19 PM, Jonathan Vaughn <jonathan(a)creatuity.com wrote:
> Here's a stacktrace per that FAQ entry
> GNU gdb (GDB) Fedora 8.0.1-36.fc27
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.
> html > This is free software: you are
free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "armv7hl-redhat-linux-gnueabi".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>;.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>;.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
> /usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
> done.
> [New LWP 17245]
> [New LWP 17210]
> [New LWP 17208]
> [New LWP 17211]
> [New LWP 17221]
> [New LWP 17214]
> [New LWP 17239]
> [New LWP 17222]
> [New LWP 17215]
> [New LWP 17223]
> [New LWP 17224]
> [New LWP 17225]
> [New LWP 17226]
> [New LWP 17260]
> [New LWP 17261]
> [New LWP 17253]
> [New LWP 17262]
> [New LWP 17252]
> [New LWP 17263]
> [New LWP 17251]
> [New LWP 17264]
> [New LWP 17250]
> [New LWP 17267]
> [New LWP 17249]
> [New LWP 17268]
> [New LWP 17248]
> [New LWP 17247]
> [New LWP 17246]
> [New LWP 17244]
> [New LWP 17243]
> [New LWP 17240]
> [New LWP 17241]
> [New LWP 17238]
> [New LWP 17229]
> [New LWP 17228]
> [New LWP 17227]
> [New LWP 17216]
> [New LWP 17212]
> [New LWP 17269]
> [New LWP 17266]
> [New LWP 17265]
> [New LWP 17259]
> [New LWP 17258]
> [New LWP 17213]
> [New LWP 17217]
> [New LWP 17218]
> [New LWP 17219]
> [New LWP 17220]
> [New LWP 17230]
> [New LWP 17231]
> [New LWP 17232]
> [New LWP 17233]
> [New LWP 17234]
> [New LWP 17254]
> [New LWP 17255]
> [New LWP 17256]
> [New LWP 17257]
> [New LWP 17236]
> [New LWP 17237]
> [New LWP 17242]
> Missing separate debuginfo for /usr/lib/dirsrv/libnunc-stans.so.0
> Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/3a/
> 9153b2860989f51a2522ce96d034a229c1438a.debug
> Missing separate debuginfo for /usr/lib/dirsrv/libslapd.so.0
> Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/cf/
> da5da98d7b486884dd54fcb480f90b55ca7380.debug
> Missing separate debuginfo for /usr/lib/dirsrv/libsds.so.0
> Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/51/
> 33bfed2a83bbc28037162723be9726217d1736.debug
> Missing separate debuginfo for /usr/lib/dirsrv/libldaputil.so.0
> Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/f7/
> 6e7e6e283afb9708689bacf06c45e91898800d.debug
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/libthread_db.so.1".
> Missing separate debuginfo for /usr/lib/dirsrv/libns-dshttpd-1.3.7.10.so
> Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/5e/
> 998d123be26e436e64e5e3e5c7684c55ebba0e.debug
> Core was generated by `/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-COMPANY-INTERNAL
> -i /var/run/dirsrv/s'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0xb67bbf2e in strlen () from /lib/libc.so.6
> [Current thread is 1 (Thread 0x9e13c280 (LWP 17245))]
> Thread 60 (Thread 0x9f9a6280 (LWP 17242)):
> #0 0xb69a1320 in pthread_cond_timedwait@(a)GLIBC_2.4 () at
> /lib/libpthread.so.0
> #1 0xb6a11f60 in pt_TimedWait () at /lib/libnspr4.so
> #2 0xb6a12272 in PR_WaitCondVar () at /lib/libnspr4.so
> #3 0xb3547fcc in protocol_sleep (prp=0x195d8c0, duration=300000) at
> ldap/servers/plugins/replication/repl5_inc_protocol.c:1222
> #4 0xb354c378 in repl5_inc_run (prp=0x195d8c0) at ldap/servers/plugins/
> replication/repl5_inc_protocol.c:782
> rc = <optimized out >
prp_priv = 0xb6fe4a50 <__stack_chk_guard >
replica = 0x0
> cons_schema_csn = 0x2bf6540
> ruv = 0x0
> num_changes_sent = <optimized out >
use_busy_backoff_timer = <optimized out >
next_fire_time = <optimized out >
now = <optimized out > busywaittime = 0
> pausetime = 0
> loops = <optimized out >
wait_change_timer_set = 1
> current_state = 2
> next_state = 2
> done = 0
> e1 = <optimized out > #5
0xb3551a60 in prot_thread_main (arg=0x143c8a0) at ldap/servers/plugins/
> replication/repl5_protocol.c:255
> rp = 0x143c8a0
> done = 0
> agmt = 0x131e900
> #6 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #7 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #8 0xb6818da8 in None () at /lib/libc.so.6
> Thread 59 (Thread 0xa21ab280 (LWP 17237)):
> #0 0xb6811c14 in select () at /lib/libc.so.6
> #1 0xb6f55694 in DS_Sleep () at /usr/lib/dirsrv/libslapd.so.0
> #2 0xb362eccc in checkpoint_threadmain (param=0x1331340) at
> ldap/servers/slapd/back-ldbm/dblayer.c:4323
> interval = 0
> rval = <optimized out >
priv = <optimized out > li = 0x1331340
> debug_checkpointing = 0
> home_dir = <optimized out >
list = 0x0
> listp = <optimized out >
penv = 0xa21a9c04
> checkpoint_expire = {tv_sec = 949530, tv_nsec = 734434832}
> compactdb_expire = {tv_sec = 3541470, tv_nsec = 734432905}
> compactdb_interval_update = <optimized out >
checkpoint_interval_update = <optimized out >
compactdb_interval = <optimized out >
checkpoint_interval = <optimized out >
txn = {back_txn_txn = 0x0}
> #3 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #5 0xb6818da8 in None () at /lib/libc.so.6
> Thread 58 (Thread 0xa29ac280 (LWP 17236)):
> #0 0xb6811c14 in select () at /lib/libc.so.6
> #1 0xb6f55694 in DS_Sleep () at /usr/lib/dirsrv/libslapd.so.0
> #2 0xb362b0cc in deadlock_threadmain (param=<optimized out>) at
> ldap/servers/slapd/back-ldbm/dblayer.c:4101
> rval = <optimized out >
priv = 0x1330460
> interval = 100
> #3 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #5 0xb6818da8 in None () at /lib/libc.so.6
> Thread 57 (Thread 0x980df280 (LWP 17257)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aeb10,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aeb10
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 56 (Thread 0x988e0280 (LWP 17256)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aeae0,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aeae0
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 55 (Thread 0x990e1280 (LWP 17255)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aebd0,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aebd0
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 54 (Thread 0x998e2280 (LWP 17254)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aeba0,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aeba0
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 53 (Thread 0xa71b5280 (LWP 17234)):
> #0 0xb6818ed8 in epoll_pwait () at /lib/libc.so.6
> #1 0xb6819188 in epoll_wait () at /lib/libc.so.6
> #2 0xb68de154 in epoll_dispatch () at /lib/libevent-2.0.so.5
> #3 0xb68c698c in event_base_loop () at /lib/libevent-2.0.so.5
> #4 0xb6fa2568 in ns_event_fw_loop () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #5 0xb6fa113c in event_loop_thread_func () at
> /usr/lib/dirsrv/libnunc-stans.so.0
> #6 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #7 0xb6818da8 in None () at /lib/libc.so.6
> Thread 52 (Thread 0xa79b6280 (LWP 17233)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 51 (Thread 0xa81b7280 (LWP 17232)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 50 (Thread 0xa89b8280 (LWP 17231)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 49 (Thread 0xa91b9280 (LWP 17230)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 48 (Thread 0xae1c3280 (LWP 17220)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 47 (Thread 0xae9c4280 (LWP 17219)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 46 (Thread 0xaf1c5280 (LWP 17218)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 45 (Thread 0xaf9c6280 (LWP 17217)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 44 (Thread 0xb19ca280 (LWP 17213)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 43 (Thread 0x978de280 (LWP 17258)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aeb40,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aeb40
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 42 (Thread 0x970dd280 (LWP 17259)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aeb70,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aeb70
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 41 (Thread 0x940d7280 (LWP 17265)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aedb0,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aedb0
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 40 (Thread 0x938d6280 (LWP 17266)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aed50,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aed50
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 39 (Thread 0x920d3280 (LWP 17269)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aed80,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aed80
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 38 (Thread 0xb21cb280 (LWP 17212)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 37 (Thread 0xb01c7280 (LWP 17216)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 36 (Thread 0xaa9bc280 (LWP 17227)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 35 (Thread 0xaa1bb280 (LWP 17228)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 34 (Thread 0xa99ba280 (LWP 17229)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 33 (Thread 0xa19aa280 (LWP 17238)):
> #0 0xb6811c14 in select () at /lib/libc.so.6
> #1 0xb6f55694 in DS_Sleep () at /usr/lib/dirsrv/libslapd.so.0
> #2 0xb362b340 in trickle_threadmain (param=<optimized out>) at
> ldap/servers/slapd/back-ldbm/dblayer.c:4538
> interval = 250
> rval = <optimized out >
priv = 0x1330460
> debug_checkpointing = 0
> #3 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #5 0xb6818da8 in None () at /lib/libc.so.6
> Thread 32 (Thread 0xa01a7280 (LWP 17241)):
> #0 0xb69a1320 in pthread_cond_timedwait@(a)GLIBC_2.4 () at
> /lib/libpthread.so.0
> #1 0xb6a11f60 in pt_TimedWait () at /lib/libnspr4.so
> #2 0xb6a12272 in PR_WaitCondVar () at /lib/libnspr4.so
> #3 0xb353218c in _cl5TrimMain (param=<optimized out>) at
> ldap/servers/plugins/replication/cl5_api.c:3287
> timePrev = <optimized out >
timeCompactPrev = <optimized out >
timeNow = <optimized out > #4 0xb6a15d68 in _pt_root ()
at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 31 (Thread 0xa09a8280 (LWP 17240)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0xb6f41c14 in slapi_wait_condvar () at /usr/lib/dirsrv/libslapd.so.0
> #3 0xb37d92fc in cos_cache_wait_on_change (arg=<optimized out>) at
> ldap/servers/plugins/cos/cos_cache.c:404
> #4 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 30 (Thread 0x9f151280 (LWP 17243)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0xb6f41c14 in slapi_wait_condvar () at /usr/lib/dirsrv/libslapd.so.0
> #3 0xb3415718 in roles_cache_wait_on_change (arg=0x14f9200) at
> ldap/servers/plugins/roles/roles_cache.c:387
> roles_def = 0x14f9200
> #4 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 29 (Thread 0x9e93d280 (LWP 17244)):
> #0 0xb69a1320 in pthread_cond_timedwait@(a)GLIBC_2.4 () at
> /lib/libpthread.so.0
> #1 0xb6a11f60 in pt_TimedWait () at /lib/libnspr4.so
> #2 0xb6a12272 in PR_WaitCondVar () at /lib/libnspr4.so
> #3 0x00503d34 in housecleaning (cur_time=<optimized out>) at
> ldap/servers/slapd/house.c:46
> interval = 30000
> #4 0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 28 (Thread 0x9d8ea280 (LWP 17246)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17ae960,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17ae960
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 27 (Thread 0x9d0e9280 (LWP 17247)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17ae990,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17ae990
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 26 (Thread 0x9c8e8280 (LWP 17248)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17ae9f0,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17ae9f0
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 25 (Thread 0x928d4280 (LWP 17268)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aecf0,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aecf0
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 24 (Thread 0x9c0e7280 (LWP 17249)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aea20,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aea20
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 23 (Thread 0x930d5280 (LWP 17267)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aed20,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aed20
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 22 (Thread 0x9b8e6280 (LWP 17250)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17ae9c0,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17ae9c0
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 21 (Thread 0x948d8280 (LWP 17264)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aecc0,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aecc0
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 20 (Thread 0x9b0e5280 (LWP 17251)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aea50,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aea50
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 19 (Thread 0x950d9280 (LWP 17263)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aec30,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aec30
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 18 (Thread 0x9a8e4280 (LWP 17252)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aea80,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aea80
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 17 (Thread 0x958da280 (LWP 17262)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aec90,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aec90
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 16 (Thread 0x9a0e3280 (LWP 17253)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aeab0,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aeab0
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 15 (Thread 0x960db280 (LWP 17261)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aec60,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aec60
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 14 (Thread 0x968dc280 (LWP 17260)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6a1229a in PR_WaitCondVar () at /lib/libnspr4.so
> #2 0x004f9130 in [IDLE THREAD] connection_wait_for_new_work (pb=pb@entry=0x17aec00,
> interval=interval@entry=4294967295) at ldap/servers/slapd/connection.c:962
> ret = 0
> wqitem = 0x0
> op_stack_obj = 0x0
> #3 0x004fa6f0 in connection_threadmain () at
> ldap/servers/slapd/connection.c:1528
> pb_conn = 0x0
> is_timedout = 0
> curtime = 0
> pb = 0x17aec00
> conn = 0x0
> op = 0x0
> tag = 0
> need_wakeup = -1224848816
> thread_turbo_flag = 0
> ret = <optimized out >
more_data = 0
> replication_connection = <optimized out >
doshutdown = 0
> maxthreads = <optimized out >
bypasspollcnt = <optimized out > #4
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #5 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #6 0xb6818da8 in None () at /lib/libc.so.6
> Thread 13 (Thread 0xab1bd280 (LWP 17226)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 12 (Thread 0xab9be280 (LWP 17225)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 11 (Thread 0xac1bf280 (LWP 17224)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 10 (Thread 0xac9c0280 (LWP 17223)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 9 (Thread 0xb09c8280 (LWP 17215)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 8 (Thread 0xad1c1280 (LWP 17222)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 7 (Thread 0xa11a9280 (LWP 17239)):
> #0 0xb6811c14 in select () at /lib/libc.so.6
> #1 0xb6f55694 in DS_Sleep () at /usr/lib/dirsrv/libslapd.so.0
> #2 0xb3625b2c in perf_threadmain (param=<optimized out>) at
> ldap/servers/slapd/back-ldbm/dblayer.c:3599
> priv = 0x1330460
> li = <optimized out > #3
0xb6a15d68 in _pt_root () at /lib/libnspr4.so
> #4 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #5 0xb6818da8 in None () at /lib/libc.so.6
> Thread 6 (Thread 0xb11c9280 (LWP 17214)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 5 (Thread 0xad9c2280 (LWP 17221)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 4 (Thread 0xb29cc280 (LWP 17211)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 3 (Thread 0xb5fb70c0 (LWP 17208)):
> #0 0xb699b414 in pthread_join () at /lib/libpthread.so.0
> #1 0xb6fa2230 in ns_thrpool_wait () at /usr/lib/dirsrv/libnunc-stans.so.0
> #2 0x00500680 in slapd_daemon (ports=<optimized out>, tp=<optimized
out>)
> at ldap/servers/slapd/daemon.c:1081
> n_tcps = 0x14c2f40
> s_tcps = 0x0
> i_unix = 0x14c2ec8
> fdesp = <optimized out >
num_poll = 3
> pr_timeout = <optimized out >
threads = <optimized out > in_referral_mode =
<optimized out > #3 0x004f1bf8 in main (argc=0,
argv=0xbec1fcc4) at
> ldap/servers/slapd/main.c:1200
> return_value = <optimized out >
mcfg = {extraname = 0x0, slapd_exemode = 1, n_port = 389, i_port =
> 1, s_port = 636, myname = 0x11f2eb0 "ns-slapd", ldif_file = 0x0,
ldif_files
> = 0, cmd_line_instance_name = 0x0, cmd_line_instance_names = 0x0,
> skip_db_protect_check = 0, db2ldif_include = 0x0, db2ldif_exclude = 0x0,
> ldif2db_removedupvals = 1, ldif2db_noattrindexes = 0, db2index_attrs = 0x0,
> ldif_printkey = 5, archive_name = 0x0, db2ldif_dump_replica = 0,
> db2ldif_dump_uniqueid = 1, ldif2db_generate_uniqueid = 1,
> ldif2db_namespaceid = 0x0, importexport_encrypt = 0, upgradedb_flags = 0,
> upgradednformat_dryrun = 0, is_quiet = 0, dbverify_verbose = 0,
> dbverify_dbdir = 0x0}
> slapdFrontendConfig = <optimized out >
ports_info = {n_port = 389, s_port = 636, n_listenaddr =
> 0x14c2ef0, s_listenaddr = 0x14c2f98, n_socket = 0x14c2f40, i_listenaddr =
> 0x14c2e08, i_port = 1, i_socket = 0x14c2ec8, s_socket = 0x0}
> tp = 0x12a3f00
> m = <optimized out > notify = <optimized
out > Thread 2 (Thread 0xb31cd280
(LWP 17210)):
> #0 0xb69a0e60 in pthread_cond_wait@(a)GLIBC_2.4 () at /lib/libpthread.so.0
> #1 0xb6fa14a8 in worker_thread_func () at /usr/lib/dirsrv/libnunc-stans.
> so.0
> #2 0xb6999ee8 in start_thread () at /lib/libpthread.so.0
> #3 0xb6818da8 in None () at /lib/libc.so.6
> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
> #2 0x00000000 in None ()
> On Tue, May 8, 2018 at 7:52 AM,
Rob Crittenden <rcritten(a)redhat.com >
wrote:
>> Jonathan Vaughn via FreeIPA-users wrote:
> >>> Still trying to figure this out. It looks like
slapd is dying, I thought
>>> it was still running for some reason.
>> >>> slapd is dying to
segfault. strace of it happening doesn't seem to
>>> reveal much:
>> > >>
A stack trace would very much help trying to track down the cause.
> >>
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>> ebugging-crashes
> >> rob
> > >>> 18:32:41.543717 (+
0.000801) openat(AT_FDCWD,
>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
O_WRONLY|O_CREAT|O_TRUNC,
>>> 0666) = 32
>>> 18:32:41.544907 (+ 0.001195) getpid() = 16014
>>> 18:32:41.545269 (+ 0.000329) fstat64(32, {st_mode=S_IFREG|0644,
>>> st_size=0, ...}) = 0
>>> 18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) = 6
>>> 18:32:41.546603 (+ 0.000818) close(32) = 0
>>> 18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> 0644) = 0
>>> 18:32:41.547741 (+ 0.000676) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>>> 18:32:41.548324 (+ 0.000587) write(2,
"[07/May/2018:18:32:41.548030641
>>> "..., 134) = 134
>>> 18:32:41.551096 (+ 0.002840) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>>> 18:32:41.551568 (+ 0.000406) write(4,
"[07/May/2018:18:32:41.551287555
>>> "..., 134) = 134
>>> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
>>> 18:32:41.558499 (+ 0.006170) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>>> 18:32:41.558901 (+ 0.000350) write(2,
"[07/May/2018:18:32:41.558678099
>>> "..., 136) = 136
>>> 18:32:41.561537 (+ 0.002680) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>>> 18:32:41.562357 (+ 0.000793) write(4,
"[07/May/2018:18:32:41.561718659
>>> "..., 136) = 136
>>> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
>>> 18:32:41.566928 (+ 0.003452) getpid() = 16014
>>> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>> 18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET, SO_SNDBUF,
>>> [163840], [4]) = 0
>>> 18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET,
>>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
>>> 18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET, SO_SNDBUF,
>>> [8388608], 4) = 0
>>> 18:32:41.570064 (+ 0.000513) sendmsg(32,
>>> {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
>>> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
>>> MSG_NOSIGNAL) = 69
>>> 18:32:41.570845 (+ 0.000789) close(32) = 0
>>> 18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8, FUTEX_WAIT, 16016,
>>> NULL) = ?
>>> 18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
>> >>> It doesn't die
immediately after hitting that final futex call - it's
>>> just over 20s later. There's still plenty of free memory at the time.
>> >>> # free -m
>>> total used free shared buff/cache
>>> available
>>> Mem: 974 222 156 0 594
>>> 679
>>> Swap: 2535 24 2511
>> >>> The last entries in the
error log is that it is listening on socket /
>>> port. No error logged for the segfault.
>> >>> I did see that during
previous setup steps (before it dies) there is
>>> apparently an error setting up replication:
>> >>>
[07/May/2018:18:17:46.245041807 -0500] - ERR - NSMMReplicationPlugin -
>>> acquire_replica - agmt="cn=meToipa-12.company.internal"
(ipa-12:5):
>>> Unable to acquire replica: permission denied. The bind dn "" does
not have
>>> permission to supply replication updates to the replica. Will retry later.
>> >>> Possibly, the failure
to configure replication is leading to the
>>> segfault? Any ideas why it would not have permission / what to check to
>>> ensure it has permission ?
>> >> >>> On Thu, May 3, 2018 at 4:02 PM, Jonathan Vaughn
<jonathan(a)creatuity.com
>>> <mailto:jonathan@creatuity.com>> wrote:
>> >>> It's still
running.
>> >>> Here's the
error log from slapd during that run:
>> >>>
[01/May/2018:19:22:24.567650453 -0500] - ERR - NSMMReplicationPlugin
>>> - multimaster_extop_StartNSDS50ReplicationRequest - conn=42 op=5
>>> replica="dc=company,dc=internal": Unable to acquire replica:
error:
>>> permission denied
>>> [01/May/2018:19:22:24.613143629 -0500] - ERR - NSMMReplicationPlugin
>>> - multimaster_extop_StartNSDS50ReplicationRequest - conn=43 op=5
>>> replica="dc=company,dc=internal": Unable to acquire replica:
error:
>>> permission denied
>>> [01/May/2018:19:22:31.445153171 -0500] - INFO -
>>> NSMMReplicationPlugin - repl5_tot_run - Beginning total update of
>>> replica "agmt="cn=meToipa-11.company.internal"
(ipa-11:389)".
>>> [01/May/2018:19:22:31.672485209 -0500] - NOTICE -
>>> NSMMReplicationPlugin - replica_subentry_check - Need to create
>>> replication keep alive entry <cn=repl keep alive
>>> 7,dc=company,dc=internal
>>> [01/May/2018:19:22:31.674922201 -0500] - INFO -
>>> NSMMReplicationPlugin - replica_subentry_create - add dn: cn=repl
>>> keep alive 7,dc=company,dc=internal
>>> objectclass: top
>>> objectclass: ldapsubentry
>>> objectclass: extensibleObject
>>> cn: repl keep alive 7
>>> [01/May/2018:19:22:36.125109819 -0500] - ERR - NSMMReplicationPlugin
>>> - multimaster_extop_StartNSDS50ReplicationRequest - conn=44 op=5
>>> replica="dc=company,dc=internal": Unable to acquire replica:
error:
>>> permission denied
>>> [01/May/2018:19:22:36.598178749 -0500] - INFO -
>>> NSMMReplicationPlugin - repl5_tot_run - Finished total update of
>>> replica "agmt="cn=meToipa-11.company.internal"
(ipa-11:389)". Sent
>>> 503 entries.
>>> [01/May/2018:19:27:40.482836683 -0500] - ERR - NSMMReplicationPlugin
>>> - bind_and_check_pwp - agmt="cn=meToipa-11.company.internal"
>>> (ipa-11:389) - Replication bind with GSSAPI auth failed: LDAP error
>>> -1 (Can't contact LDAP server) ()
>> >>> ipa-11 is the Pi
I'm trying to set up replication on. Looks like
>>> there was a problem setting up replication due to permissions? But
>>> the other replica started up just fine ...
>> >>> On Thu, May 3, 2018
at 3:13 PM, Rob Crittenden <rcritten(a)redhat.com
>>> <mailto:rcritten@redhat.com>> wrote:
>> >>> Jonathan Vaughn
wrote:
>> >>> Here's
the output from ipa-replica-install :
>> >>> #
ipa-replica-install
>>> WARNING: conflicting time&date synchronization service
>>> 'chronyd' will
>>> be disabled in favor of ntpd
>> >>> Password
for admin(a)COMPANY.INTERNAL:
>>> Run connection check to master
>>> Connection check OK
>>> Configuring NTP daemon (ntpd)
>>> [1/4]: stopping ntpd
>>> [2/4]: writing configuration
>>> [3/4]: configuring ntpd to start on boot
>>> [4/4]: starting ntpd
>>> Done configuring NTP daemon (ntpd).
>>> Configuring directory server (dirsrv). Estimated time: 30
>>> seconds
>>> [1/41]: creating directory server instance
>>> [2/41]: enabling ldapi
>>> [3/41]: configure autobind for root
>>> [4/41]: stopping directory server
>>> [5/41]: updating configuration in dse.ldif
>>> [6/41]: starting directory server
>>> [7/41]: adding default schema
>>> [8/41]: enabling memberof plugin
>>> [9/41]: enabling winsync plugin
>>> [10/41]: configuring replication version plugin
>>> [11/41]: enabling IPA enrollment plugin
>>> [12/41]: configuring uniqueness plugin
>>> [13/41]: configuring uuid plugin
>>> [14/41]: configuring modrdn plugin
>>> [15/41]: configuring DNS plugin
>>> [16/41]: enabling entryUSN plugin
>>> [17/41]: configuring lockout plugin
>>> [18/41]: configuring topology plugin
>>> [19/41]: creating indices
>>> [20/41]: enabling referential integrity plugin
>>> [21/41]: configuring certmap.conf
>>> [22/41]: configure new location for managed entries
>>> [23/41]: configure dirsrv ccache
>>> [24/41]: enabling SASL mapping fallback
>>> [25/41]: restarting directory server
>>> [26/41]: creating DS keytab
>>> [27/41]: ignore time skew for initial replication
>>> [28/41]: setting up initial replication
>>> Starting replication, please wait until this has completed.
>>> Update in progress, 11 seconds elapsed
>>> Update succeeded
>> >>>
[29/41]: prevent time skew after initial replication
>>> [30/41]: adding sasl mappings to the directory
>>> [31/41]: updating schema
>>> [32/41]: setting Auto Member configuration
>>> [33/41]: enabling S4U2Proxy delegation
>>> [error] NetworkError: cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>> Your system may be partly configured.
>>> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>> >>>
ipapython.admintool: ERROR cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>> ipapython.admintool: ERROR The ipa-replica-install
>>> command failed. See /var/log/ipareplica-install.log for more
>>> information
>> >>> And
here's the /var/log/ipareplica-install.log from just
>>> before it fails to the failure:
>> >>>
2018-05-02T00:22:44Z DEBUG [32/41]: setting Auto Member
>>> configuration
>>> 2018-05-02T00:22:44Z DEBUG Starting external process
>>> 2018-05-02T00:22:44Z DEBUG args=/usr/bin/ldapmodify -v -f
>>> /tmp/tmpt7bxf4x1 -H
>>> ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket -Y
>>> EXTERNAL
>>> 2018-05-02T00:22:45Z DEBUG Process finished, return code=0
>>> 2018-05-02T00:22:45Z DEBUG stdout=add
>>> nsslapd-pluginConfigArea:
>>> cn=automember,cn=etc,dc=company,dc=internal
>>> modifying entry "cn=Auto Membership
>>> Plugin,cn=plugins,cn=config"
>>> modify complete
>> >> >>> 2018-05-02T00:22:45Z DEBUG
stderr=ldap_initialize(
>>> ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket/??base
>>> )
>>> SASL/EXTERNAL authentication started
>>> SASL username:
>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>>> SASL SSF: 0
>> >>>
2018-05-02T00:22:45Z DEBUG duration: 0 seconds
>>> 2018-05-02T00:22:45Z DEBUG [33/41]: enabling S4U2Proxy
>>> delegation
>>> 2018-05-02T00:22:45Z DEBUG Traceback (most recent call last):
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>>> line 979, in error_handler
>>> yield
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>>> line 1572, in update_entry
>>> self.conn.modify_s(str(entry.dn), modlist)
>>> File
>>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 600, in modify_s
>>> return self.modify_ext_s(dn,modlist,None,None)
>>> File
>>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 573, in modify_ext_s
>>> resp_type, resp_data, resp_msgid, resp_ctrls =
>>> self.result3(msgid,all=1,timeout=self.timeout)
>>> File
>>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 714, in result3
>>> resp_ctrl_classes=resp_ctrl_classes
>>> File
>>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 721, in result4
>>> ldap_result =
>>> self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,
>>> add_intermediates,add_extop)
>>> File
>>> "/usr/lib/python3.6/site-packages/ldap/ldapobject.py",
line
>>> 294, in _ldap_call
>>> result = func(*args,**kwargs)
>>> ldap.SERVER_DOWN: {'desc': "Can't contact LDAP
server"}
>> >>> During
handling of the above exception, another exception
>>> occurred:
>> >>> Traceback
(most recent call last):
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/service.
>>> py",
>>> line 506, in start_creation
>>> run_step(full_msg, method)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/service.
>>> py",
>>> line 496, in run_step
>>> method()
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>>> ce.py",
>>> line 977, in __setup_s4u2proxy
>>> __add_principal('ipa-http-delegation',
'HTTP', self)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>>> ce.py",
>>> line 973, in __add_principal
>>> api.Backend.ldap2.update_entry(entry)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>>> line 1572, in update_entry
>>> self.conn.modify_s(str(entry.dn), modlist)
>>> File "/usr/lib/python3.6/contextlib.py", line 99,
in
>>> __exit__
>>> self.gen.throw(type, value, traceback)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>>> line 1028, in error_handler
>>> error=info)
>>> ipalib.errors.NetworkError: cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>> >>>
2018-05-02T00:22:45Z DEBUG [error] NetworkError: cannot
>>> connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>> 2018-05-02T00:22:45Z DEBUG Destroyed connection
>>> context.ldap2_2991558640
>>> 2018-05-02T00:22:45Z DEBUG Backing up system configuration
>>> file '/etc/ipa/default.conf'
>>> 2018-05-02T00:22:45Z DEBUG Saving Index File to
>>> '/var/lib/ipa/sysrestore/sysrestore.index'
>>> 2018-05-02T00:22:45Z DEBUG File
>>>
"/usr/lib/python3.6/site-packages/ipapython/admintool.py",
>>> line 174, in execute
>>> return_value = self.run()
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/cli.py",
>>> line 319, in run
>>> cfgr.run()
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 364, in run
>>> self.execute()
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 388, in execute
>>> for _nothing in self._executor():
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 430, in __runner
>>> exc_handler(exc_info)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 459, in _handle_execute_exception
>>> self._handle_exception(exc_info)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 449, in _handle_exception
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py",
line
>>> 693, in reraise
>>> raise value
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 420, in __runner
>>> step()
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 417, in <lambda
>>> step = lambda: next(self.__gen)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>>> line
>>> 81, in run_generator_with_yield_from
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py",
line
>>> 693, in reraise
>>> raise value
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>>> line
>>> 59, in run_generator_with_yield_from
>>> value = gen.send(prev_value)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 654, in _configure
>>> next(executor)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 430, in __runner
>>> exc_handler(exc_info)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 459, in _handle_execute_exception
>>> self._handle_exception(exc_info)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 517, in _handle_exception
>>> self.__parent._handle_exception(exc_info)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 449, in _handle_exception
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py",
line
>>> 693, in reraise
>>> raise value
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 514, in _handle_exception
>>> super(ComponentBase, self)._handle_exception(exc_info)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 449, in _handle_exception
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py",
line
>>> 693, in reraise
>>> raise value
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 420, in __runner
>>> step()
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/core.py",
>>> line
>>> 417, in <lambda
>>> step = lambda: next(self.__gen)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>>> line
>>> 81, in run_generator_with_yield_from
>>> six.reraise(*exc_info)
>>> File "/usr/lib/python3.6/site-packages/six.py",
line
>>> 693, in reraise
>>> raise value
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/util.py",
>>> line
>>> 59, in run_generator_with_yield_from
>>> value = gen.send(prev_value)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/install/common.p
>>> y",
>>> line 66, in _install
>>> for unused in self._installer(self.parent):
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/server/_
>>> _init__.py",
>>> line 622, in main
>>> replica_install(self)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/server/r
>>> eplicainstall.py",
>>> line 388, in decorated
>>> func(installer)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/server/r
>>> eplicainstall.py",
>>> line 1407, in install
>>> pkcs12_info=dirsrv_pkcs12_info)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/server/r
>>> eplicainstall.py",
>>> line 110, in install_replica_ds
>>> setup_pkinit=not options.no_pkinit,
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>>> ce.py",
>>> line 419, in create_replica
>>> self.start_creation(runtime=30)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/service.
>>> py",
>>> line 506, in start_creation
>>> run_step(full_msg, method)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/service.
>>> py",
>>> line 496, in run_step
>>> method()
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>>> ce.py",
>>> line 977, in __setup_s4u2proxy
>>> __add_principal('ipa-http-delegation',
'HTTP', self)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipaserver/install/dsinstan
>>> ce.py",
>>> line 973, in __add_principal
>>> api.Backend.ldap2.update_entry(entry)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>>> line 1572, in update_entry
>>> self.conn.modify_s(str(entry.dn), modlist)
>>> File "/usr/lib/python3.6/contextlib.py", line 99,
in
>>> __exit__
>>> self.gen.throw(type, value, traceback)
>>> File
>>>
"/usr/lib/python3.6/site-packages/ipapython/ipaldap.py",
>>> line 1028, in error_handler
>>> error=info)
>> >>>
2018-05-02T00:22:45Z DEBUG The ipa-replica-install command
>>> failed, exception: NetworkError: cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>> 2018-05-02T00:22:45Z ERROR cannot connect to
>>> 'ldapi://%2Fvar%2Frun%2Fslapd-COMPANY-INTERNAL.socket':
>>> 2018-05-02T00:22:45Z ERROR The ipa-replica-install command
>>> failed. See /var/log/ipareplica-install.log for more
>>> information
>> >>> Maybe it
just is trying to soon and it hasn't fully started
>>> yet? Because now I see that the previous step ALSO shows the
>>> URL encoded socket path, and it worked fine - but it looks
>>> like the previous step used ldapmodify, but the failing step
>>> was accessing LDAP from Python? Perhaps it works fine one
>>> way and not the other ?
>> >> >>> I won't rule it out but I doubt it. Is
ns-slapd still running at
>>> this point? This could happen, for example, if ns-slapd crashed
>>> after enabling the automember plugin. The DS error log might
>>> have something to say as well.
>> >>> rob
>> >> >>> On Wed, May 2, 2018 at 3:29 PM, Rob
Crittenden
>>> <rcritten(a)redhat.com <mailto:rcritten@redhat.com >>> <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>
>> >>> wrote:
>> >>>
Jonathan Vaughn via FreeIPA-users wrote:
>> >>>
Yes, I know, not recommended etc, low performance.
>>> I'm not going
>>> to run the CA on it. I just want to have a backup
>>> LDAP/Kerberos
>>> server.
>> >>>
Right now I'm just trying to test things out. I've
>>> got a master
>>> and a replica (so you could say two masters I
>>> suppose) running
>>> in Virtualbox VMs, and I'm trying to set up a 3rd
>>> replica on a
>>> Pi. All are Fedors 27. I had to downgrade httpd due
>>> to
>>> https://pagure.io/freeipa/issue/7493
>>> <https://pagure.io/freeipa/issue/7493 >>>
<https://pagure.io/freeipa/issue/7493
>>> <https://pagure.io/freeipa/issue/7493>> to even set up
the
>>> first
>>> VM replica, but this issue is separate.
>> >>>
Currently, the problem is it can't connect to it's
>>> own LDAP
>>> instance due to some kind of error ...
>>> ipa-replica-install
>>> worked fine on the x86_64 VM but on the armv71 Pi
>>> 3B when it
>>> tries to connect to LDAPI instead of
>>> using
'ldapi:///var/run//slapd-COMPANY-INTERNAL.socket'
>>> it
>>> uses 'ldapi://%2Fvar%2Frun%2Fs
>>> lapd-COMPANY-INTERNAL.socket'.
>> >>> So
it seems there is yet another ARM (or
>>> non-x86_64) bug ...
>>> similar to the problem with httpd and passing the
>>> KRB5CCNAME
>>> properly https://pagure.io/freeipa/issue/7337
>>> <https://pagure.io/freeipa/issue/7337 >>>
<https://pagure.io/freeipa/issue/7337
>>> <https://pagure.io/freeipa/issue/7337> >>
>>> Any ideas on where to look to patch in a fix to
>>> this so it uses
>>> the correct filename? The socket file is there ...
>>> and (at the
>>> time it tries) LDAP is running.
>> >> >>> What makes you think the ldapi
URI is the problem?
>> >>> Can
you share the logs?
>> >>> rob
>> >> >>
>> >> >>
>> >>>
_______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedo
>>> rahosted.org
>> >> >
10:27 a.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Hi Jonathan,
This is weird as the crashing thread stack looks truncated (did you
copy/paste all of it ?)
Thread 1 (Thread 0x9e13c280 (LWP 17245)):
#0 0xb67bbf2e in strlen () at /lib/libc.so.6
#1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
#2 0x00000000 in None ()
Did you install 389-ds-base-debuginfo ?
How did you get that backtrace ? from a core dumped, pstack ? Can you
attach a debugger before the crash occurs ?
It looks it crashed soon at startup, could it be related to a broken
dse.ldif. It should exists a dse.ldif.OK, is it possibly to try to start
with it ?
best regards
thierry
On 05/12/2018 01:22 AM, Jonathan Vaughn via FreeIPA-users wrote:
Not sure if it makes a difference... I was looking into this again
and
realized I had a bunch of messages from gdb telling me to install more
debuginfo. I've done that now, here it is again freshly run through gdb
GNU gdb (GDB) Fedora 8.0.1-36.fc27
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "armv7hl-redhat-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>;.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
/usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
done.
...
Thread 1 (Thread 0x9e13c280 (LWP 17245)):
#0 0xb67bbf2e in strlen () at /lib/libc.so.6
#1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
#2 0x00000000 in None ()
On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden
<rcritten(a)redhat.com <mailto:rcritten@redhat.com>> wrote:
Jonathan Vaughn via FreeIPA-users wrote:
Still trying to figure this out. It looks like slapd is
dying, I thought it was still running for some reason.
slapd is dying to segfault. strace of it happening doesn't
seem to reveal much:
A stack trace would very much help trying to track down the cause.
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-cras...
rob
18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
18:32:41.544907 (+ 0.001195) getpid() = 16014
18:32:41.545269 (+ 0.000329) fstat64(32,
{st_mode=S_IFREG|0644, st_size=0, ...}) = 0
18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) = 6
18:32:41.546603 (+ 0.000818) close(32) = 0
18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", 0644) = 0
18:32:41.547741 (+ 0.000676)
clock_gettime(CLOCK_REALTIME, {tv_sec=1525735961,
tv_nsec=548030641}) = 0
18:32:41.548324 (+ 0.000587) write(2,
"[07/May/2018:18:32:41.548030641 "..., 134) = 134
18:32:41.551096 (+ 0.002840)
clock_gettime(CLOCK_REALTIME, {tv_sec=1525735961,
tv_nsec=551287555}) = 0
18:32:41.551568 (+ 0.000406) write(4,
"[07/May/2018:18:32:41.551287555 "..., 134) = 134
18:32:41.552360 (+ 0.000811) fsync(4) = 0
18:32:41.558499 (+ 0.006170)
clock_gettime(CLOCK_REALTIME, {tv_sec=1525735961,
tv_nsec=558678099}) = 0
18:32:41.558901 (+ 0.000350) write(2,
"[07/May/2018:18:32:41.558678099 "..., 136) = 136
18:32:41.561537 (+ 0.002680)
clock_gettime(CLOCK_REALTIME, {tv_sec=1525735961,
tv_nsec=561718659}) = 0
18:32:41.562357 (+ 0.000793) write(4,
"[07/May/2018:18:32:41.561718659 "..., 136) = 136
18:32:41.563293 (+ 0.001148) fsync(4) = 0
18:32:41.566928 (+ 0.003452) getpid() = 16014
18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
18:32:41.568628 (+ 0.000912) getsockopt(32,
SOL_SOCKET, SO_SNDBUF, [163840], [4]) = 0
18:32:41.568972 (+ 0.000319) setsockopt(32,
SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM
(Operation not permitted)
18:32:41.569548 (+ 0.000589) setsockopt(32,
SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0
18:32:41.570064 (+ 0.000513) sendmsg(32,
{msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"}, msg_namelen=21,
msg_iov=[{iov_base="READY=1\nSTATUS=slapd started: Re"...,
iov_len=69}], msg_iovlen=1, msg_controllen=0,
msg_flags=0}, MSG_NOSIGNAL) = 69
18:32:41.570845 (+ 0.000789) close(32) = 0
18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8,
FUTEX_WAIT, 16016, NULL) = ?
18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
3:20 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Here's a strace from before it dies. Most of the elapsed time is it waiting
on some futex call it looks like near the end, when it finally "returns"
(from lack of strace output for duration of call I assume it didn't
actually return but SIGSEGV in that call) and strace prints ' = ?' on the
futex it then immediately reports SIGSEGV after. So maybe the problem is
that futex call, which may mean the problem is not directly in 389DS /
FreeIPA itself?
15:13:31.626587 (+ 0.000630) listen(8, 128) = 0 <0.000068>
15:13:31.626857 (+ 0.000235) listen(9, 128) = 0 <0.000048>
15:13:31.627111 (+ 0.000251) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=41120614}) = 0 <0.000085>
15:13:31.627457 (+ 0.000356) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=627560772}) = 0 <0.000043>
15:13:31.631233 (+ 0.003839) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=45286796}) = 0 <0.000077>
15:13:31.631720 (+ 0.000427) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=45661430}) = 0 <0.000042>
15:13:31.631955 (+ 0.000220) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=632049036}) = 0 <0.000047>
15:13:31.635669 (+ 0.003785) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=49725840}) = 0 <0.000146>
15:13:31.636484 (+ 0.000784) write(16, "a", 1) = 1 <0.000118>
15:13:31.636855 (+ 0.000341) sched_yield() = 0 <0.000252>
15:13:31.637322 (+ 0.000470) futex(0x1cb57a0, FUTEX_WAKE_PRIVATE, 1) =
1 <0.000088>
15:13:31.637897 (+ 0.000610) write(16, "a", 1) = 1 <0.000221>
15:13:31.638394 (+ 0.000467) sched_yield() = 0 <0.000047>
15:13:31.638619 (+ 0.000202) futex(0x1cb5710, FUTEX_WAKE_PRIVATE, 1) =
1 <0.000065>
15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", O_WRONLY|O_CREAT|O_TRUNC,
0666) = 33 <0.000831>
15:13:31.640260 (+ 0.001387) getpid() = 32353 <0.000077>
15:13:31.640558 (+ 0.000256) fstat64(33, {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0 <0.000119>
15:13:31.641106 (+ 0.000556) write(33, "32353\n", 6) = 6 <0.000127>
15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
15:13:31.642216 (+ 0.000758)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", 0644) = 0 <0.000152>
15:13:31.642900 (+ 0.000679) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=643020294}) = 0 <0.000056>
15:13:31.643495 (+ 0.000590) write(2, "[14/May/2018:15:13:31.643020294
"..., 134) = 134 <0.002697>
15:13:31.646515 (+ 0.003052) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=646694394}) = 0 <0.000075>
15:13:31.646892 (+ 0.000337) write(4, "[14/May/2018:15:13:31.646694394
"..., 134) = 134 <0.000522>
15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
15:13:31.654425 (+ 0.006617) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=654598946}) = 0 <0.000253>
15:13:31.655137 (+ 0.000717) write(2, "[14/May/2018:15:13:31.654598946
"..., 136) = 136 <0.002427>
15:13:31.658312 (+ 0.003165) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=658486117}) = 0 <0.000251>
15:13:31.659032 (+ 0.000682) write(4, "[14/May/2018:15:13:31.658486117
"..., 136) = 136 <0.000346>
15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
15:13:31.663230 (+ 0.003642) getpid() = 32353 <0.000045>
15:13:31.663732 (+ 0.000454) socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC,
0) = 33 <0.000296>
15:13:31.664760 (+ 0.001048) getsockopt(33, SOL_SOCKET, SO_SNDBUF,
[163840], [4]) = 0 <0.000108>
15:13:31.665141 (+ 0.000386) setsockopt(33, SOL_SOCKET, SO_SNDBUFFORCE,
[8388608], 4) = -1 EPERM (Operation not permitted) <0.000051>
15:13:31.665500 (+ 0.000334) setsockopt(33, SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0 <0.000229>
15:13:31.665973 (+ 0.000468) sendmsg(33, {msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"}, msg_namelen=21,
msg_iov=[{iov_base="READY=1\nSTATUS=slapd started: Re"..., iov_len=69}],
msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 69 <0.000600>
15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
15:13:31.677320 (+ 0.010146) futex(0xb31122e8, FUTEX_WAIT, 32357, NULL)
= ?
15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV (core dumped) +++
On Mon, May 14, 2018 at 10:27 AM, thierry bordaz <tbordaz(a)redhat.com> wrote:
Hi Jonathan,
This is weird as the crashing thread stack looks truncated (did you
copy/paste all of it ?)
Thread 1 (Thread 0x9e13c280 (LWP 17245)):
#0 0xb67bbf2e in strlen () at /lib/libc.so.6
#1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
#2 0x00000000 in None ()
Did you install 389-ds-base-debuginfo ?
How did you get that backtrace ? from a core dumped, pstack ? Can you
attach a debugger before the crash occurs ?
It looks it crashed soon at startup, could it be related to a broken
dse.ldif. It should exists a dse.ldif.OK, is it possibly to try to start
with it ?
best regards
thierry
On 05/12/2018 01:22 AM, Jonathan Vaughn via FreeIPA-users wrote:
Not sure if it makes a difference... I was looking into this again and
realized I had a bunch of messages from gdb telling me to install more
debuginfo. I've done that now, here it is again freshly run through gdb
GNU gdb (GDB) Fedora 8.0.1-36.fc27
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.
html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "armv7hl-redhat-linux-gnueabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>;.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
/usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
done.
...
Thread 1 (Thread 0x9e13c280 (LWP 17245)):
#0 0xb67bbf2e in strlen () at /lib/libc.so.6
#1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
#2 0x00000000 in None ()
On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden <rcritten(a)redhat.com>
> wrote:
>
>> Jonathan Vaughn via FreeIPA-users wrote:
>>
>>> Still trying to figure this out. It looks like slapd is dying, I
>>> thought it was still running for some reason.
>>>
>>> slapd is dying to segfault. strace of it happening doesn't seem to
>>> reveal much:
>>>
>>
>> A stack trace would very much help trying to track down the cause.
>>
>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>> ebugging-crashes
>>
>> rob
>>
>>
>>> 18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
>>> 18:32:41.544907 (+ 0.001195) getpid() = 16014
>>> 18:32:41.545269 (+ 0.000329) fstat64(32, {st_mode=S_IFREG|0644,
>>> st_size=0, ...}) = 0
>>> 18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) = 6
>>> 18:32:41.546603 (+ 0.000818) close(32) = 0
>>> 18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> 0644) = 0
>>> 18:32:41.547741 (+ 0.000676) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>>> 18:32:41.548324 (+ 0.000587) write(2,
"[07/May/2018:18:32:41.548030641
>>> "..., 134) = 134
>>> 18:32:41.551096 (+ 0.002840) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>>> 18:32:41.551568 (+ 0.000406) write(4,
"[07/May/2018:18:32:41.551287555
>>> "..., 134) = 134
>>> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
>>> 18:32:41.558499 (+ 0.006170) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>>> 18:32:41.558901 (+ 0.000350) write(2,
"[07/May/2018:18:32:41.558678099
>>> "..., 136) = 136
>>> 18:32:41.561537 (+ 0.002680) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>>> 18:32:41.562357 (+ 0.000793) write(4,
"[07/May/2018:18:32:41.561718659
>>> "..., 136) = 136
>>> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
>>> 18:32:41.566928 (+ 0.003452) getpid() = 16014
>>> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>> 18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET, SO_SNDBUF,
>>> [163840], [4]) = 0
>>> 18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET,
>>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
>>> 18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET, SO_SNDBUF,
>>> [8388608], 4) = 0
>>> 18:32:41.570064 (+ 0.000513) sendmsg(32,
>>> {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
>>> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
>>> MSG_NOSIGNAL) = 69
>>> 18:32:41.570845 (+ 0.000789) close(32) = 0
>>> 18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8, FUTEX_WAIT, 16016,
>>> NULL) = ?
>>> 18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
>>>
>>>
3:01 a.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Hi Jonathan,
This problem looks new to me and has something specific to your environment.
I think the best approach is to continue to debug on your system if you
have the possibility to do so.
From strace we can see that DS started smoothly (created its pid file
then notified systemd it was running fine). According to the pstack
nunc-stans was running and was able to accept network events even if it
appears it detected no incoming connection.
So the server should be ready to serve for some seconds (more than a
minute), then it crashed with one thread dereferencing likely a wrong
pointer.
Could you attach a debugger when the server is started and wait for the
sigsegv to occur. Then confirm the crashing thread backstack.
If it is confirmed, I am afraid this is a stack corruption and valgrind
could help
(http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...).
best regards
thierry
On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
Here's a strace from before it dies. Most of the elapsed time is
it
waiting on some futex call it looks like near the end, when it finally
"returns" (from lack of strace output for duration of call I assume it
didn't actually return but SIGSEGV in that call) and strace prints ' =
?' on the futex it then immediately reports SIGSEGV after. So maybe
the problem is that futex call, which may mean the problem is not
directly in 389DS / FreeIPA itself?
15:13:31.626587 (+ 0.000630) listen(8, 128) = 0 <0.000068>
15:13:31.626857 (+ 0.000235) listen(9, 128) = 0 <0.000048>
15:13:31.627111 (+ 0.000251) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=41120614}) = 0 <0.000085>
15:13:31.627457 (+ 0.000356) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=627560772}) = 0 <0.000043>
15:13:31.631233 (+ 0.003839) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=45286796}) = 0 <0.000077>
15:13:31.631720 (+ 0.000427) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=45661430}) = 0 <0.000042>
15:13:31.631955 (+ 0.000220) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=632049036}) = 0 <0.000047>
15:13:31.635669 (+ 0.003785) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=49725840}) = 0 <0.000146>
15:13:31.636484 (+ 0.000784) write(16, "a", 1) = 1 <0.000118>
15:13:31.636855 (+ 0.000341) sched_yield() = 0 <0.000252>
15:13:31.637322 (+ 0.000470) futex(0x1cb57a0, FUTEX_WAKE_PRIVATE,
1) = 1 <0.000088>
15:13:31.637897 (+ 0.000610) write(16, "a", 1) = 1 <0.000221>
15:13:31.638394 (+ 0.000467) sched_yield() = 0 <0.000047>
15:13:31.638619 (+ 0.000202) futex(0x1cb5710, FUTEX_WAKE_PRIVATE,
1) = 1 <0.000065>
15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33 <0.000831>
15:13:31.640260 (+ 0.001387) getpid() = 32353 <0.000077>
15:13:31.640558 (+ 0.000256) fstat64(33, {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0 <0.000119>
15:13:31.641106 (+ 0.000556) write(33, "32353\n", 6) = 6 <0.000127>
15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
15:13:31.642216 (+ 0.000758)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", 0644) = 0 <0.000152>
15:13:31.642900 (+ 0.000679) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=643020294}) = 0 <0.000056>
15:13:31.643495 (+ 0.000590) write(2,
"[14/May/2018:15:13:31.643020294 "..., 134) = 134 <0.002697>
15:13:31.646515 (+ 0.003052) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=646694394}) = 0 <0.000075>
15:13:31.646892 (+ 0.000337) write(4,
"[14/May/2018:15:13:31.646694394 "..., 134) = 134 <0.000522>
15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
15:13:31.654425 (+ 0.006617) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=654598946}) = 0 <0.000253>
15:13:31.655137 (+ 0.000717) write(2,
"[14/May/2018:15:13:31.654598946 "..., 136) = 136 <0.002427>
15:13:31.658312 (+ 0.003165) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=658486117}) = 0 <0.000251>
15:13:31.659032 (+ 0.000682) write(4,
"[14/May/2018:15:13:31.658486117 "..., 136) = 136 <0.000346>
15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
15:13:31.663230 (+ 0.003642) getpid() = 32353 <0.000045>
15:13:31.663732 (+ 0.000454) socket(AF_UNIX,
SOCK_DGRAM|SOCK_CLOEXEC, 0) = 33 <0.000296>
15:13:31.664760 (+ 0.001048) getsockopt(33, SOL_SOCKET, SO_SNDBUF,
[163840], [4]) = 0 <0.000108>
15:13:31.665141 (+ 0.000386) setsockopt(33, SOL_SOCKET,
SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
<0.000051>
15:13:31.665500 (+ 0.000334) setsockopt(33, SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0 <0.000229>
15:13:31.665973 (+ 0.000468) sendmsg(33,
{msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL) = 69 <0.000600>
15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
15:13:31.677320 (+ 0.010146) futex(0xb31122e8, FUTEX_WAIT, 32357,
NULL) = ?
15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV (core dumped) +++
On Mon, May 14, 2018 at 10:27 AM, thierry bordaz <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>> wrote:
Hi Jonathan,
This is weird as the crashing thread stack looks truncated (did
you copy/paste all of it ?)
Thread 1 (Thread 0x9e13c280 (LWP 17245)):
#0 0xb67bbf2e in strlen () at /lib/libc.so.6
#1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
#2 0x00000000 in None ()
Did you install 389-ds-base-debuginfo ?
How did you get that backtrace ? from a core dumped, pstack ? Can
you attach a debugger before the crash occurs ?
It looks it crashed soon at startup, could it be related to a
broken dse.ldif. It should exists a dse.ldif.OK, is it possibly to
try to start with it ?
best regards
thierry
On 05/12/2018 01:22 AM, Jonathan Vaughn via FreeIPA-users wrote:
> Not sure if it makes a difference... I was looking into this
> again and realized I had a bunch of messages from gdb telling me
> to install more debuginfo. I've done that now, here it is again
> freshly run through gdb
>
> GNU gdb (GDB) Fedora 8.0.1-36.fc27
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html
<http://gnu.org/licenses/gpl.html>>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "armv7hl-redhat-linux-gnueabi".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/
> <http://www.gnu.org/software/gdb/bugs/>>;.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/
> <http://www.gnu.org/software/gdb/documentation/>>;.
> For help, type "help".
> Type "apropos word" to search for commands related to
"word"...
> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
> /usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
> done.
> ...
>
> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
> #2 0x00000000 in None ()
>
>
>
> On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden
> <rcritten(a)redhat.com <mailto:rcritten@redhat.com>> wrote:
>
> Jonathan Vaughn via FreeIPA-users wrote:
>
> Still trying to figure this out. It looks like slapd
> is dying, I thought it was still running for some reason.
>
> slapd is dying to segfault. strace of it happening
> doesn't seem to reveal much:
>
>
> A stack trace would very much help trying to track down
> the cause.
>
>
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
>
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-cras...
>
> rob
>
>
> 18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
> 18:32:41.544907 (+ 0.001195) getpid() = 16014
> 18:32:41.545269 (+ 0.000329) fstat64(32,
> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> 18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) = 6
> 18:32:41.546603 (+ 0.000818) close(32) = 0
> 18:32:41.547061 (+ 0.000448)
> chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
> 0644) = 0
> 18:32:41.547741 (+ 0.000676)
> clock_gettime(CLOCK_REALTIME, {tv_sec=1525735961,
> tv_nsec=548030641}) = 0
> 18:32:41.548324 (+ 0.000587) write(2,
> "[07/May/2018:18:32:41.548030641 "..., 134) = 134
> 18:32:41.551096 (+ 0.002840)
> clock_gettime(CLOCK_REALTIME, {tv_sec=1525735961,
> tv_nsec=551287555}) = 0
> 18:32:41.551568 (+ 0.000406) write(4,
> "[07/May/2018:18:32:41.551287555 "..., 134) = 134
> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
> 18:32:41.558499 (+ 0.006170)
> clock_gettime(CLOCK_REALTIME, {tv_sec=1525735961,
> tv_nsec=558678099}) = 0
> 18:32:41.558901 (+ 0.000350) write(2,
> "[07/May/2018:18:32:41.558678099 "..., 136) = 136
> 18:32:41.561537 (+ 0.002680)
> clock_gettime(CLOCK_REALTIME, {tv_sec=1525735961,
> tv_nsec=561718659}) = 0
> 18:32:41.562357 (+ 0.000793) write(4,
> "[07/May/2018:18:32:41.561718659 "..., 136) = 136
> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
> 18:32:41.566928 (+ 0.003452) getpid() = 16014
> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
> 18:32:41.568628 (+ 0.000912) getsockopt(32,
> SOL_SOCKET, SO_SNDBUF, [163840], [4]) = 0
> 18:32:41.568972 (+ 0.000319) setsockopt(32,
> SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM
> (Operation not permitted)
> 18:32:41.569548 (+ 0.000589) setsockopt(32,
> SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0
> 18:32:41.570064 (+ 0.000513) sendmsg(32,
> {msg_name={sa_family=AF_UNIX,
> sun_path="/run/systemd/notify"}, msg_namelen=21,
> msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0,
> msg_flags=0}, MSG_NOSIGNAL) = 69
> 18:32:41.570845 (+ 0.000789) close(32) = 0
> 18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8,
> FUTEX_WAIT, 16016, NULL) = ?
> 18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
>
4:29 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Here is a backtrace from live gdb after the segfault. Looks like things
went wrong somewhere during in the replication code ?
Thread 36 "ns-slapd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x9e0bc280 (LWP 4662)]
strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
142 ldrd data1a, data1b, [src]
(gdb) bt
#0 0xb6728f2e in strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
#1 0xb6973b40 in cvt_s (flags=0, prec=<optimized out>, width=0, str=0x4
<error: Cannot access memory at address 0x4>, ss=<optimized out>)
at ../../.././nspr/pr/src/io/prprf.c:374
#2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc, fmt=0xb34fddf2 "",
fmt@entry=0xb3527770 <agmt_set> "\360\357\305\002", ap=...)
at ../../.././nspr/pr/src/io/prprf.c:1018
#3 0xb6973c8c in PR_vsmprintf (fmt=fmt@entry=0xb3527770 <agmt_set>
"\360\357\305\002", ap=..., ap@entry=...) at
../../.././nspr/pr/src/io/prprf.c:1184
#4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0 "%s;%s;%s;%ld;%d;%s")
at ldap/servers/slapd/ch_malloc.c:331
#5 0xb34af5a4 in agmt_update_maxcsn (r=r@entry=0x2c89810, sdn=0x3e2bac0,
sdn@entry=0x10, op=<optimized out>, mods=0x10, mods@entry=0x3eec100,
csn=csn@entry=0x3e30220)
at ldap/servers/plugins/replication/repl5_agmt.c:3036
#6 0xb34bd424 in write_changelog_and_ruv (pb=pb@entry=0x2cb54a0) at
ldap/servers/plugins/replication/repl5_plugins.c:1124
#7 0xb34be7ec in multimaster_be_betxnpostop_add (pb=0x2cb54a0) at
ldap/servers/plugins/replication/repl5_plugins.c:855
#8 0xb34be880 in multimaster_mmr_postop (pb=<optimized out>, flags=560) at
ldap/servers/plugins/replication/repl5_plugins.c:616
#9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop (pb=pb@entry=0x2cb54a0,
e=e@entry=0x0, flags=flags@entry=560) at ldap/servers/slapd/plugin_mmr.c:65
#10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
#11 0xb6e2ce4c in op_shared_add (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:679
#12 0xb6e2d35c in add_internal_pb (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:407
#13 0xb6e2e0f8 in slapi_add_internal_pb (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:332
#14 0xb368db50 in ipa_topo_util_segment_write (tconf=tconf@entry=0x2cb5860,
tsegm=tsegm@entry=0x3f2d9a0) at topology_util.c:1251
#15 0xb368e01c in ipa_topo_util_update_agmt_list (conf=0x2cb5860,
repl_segments=<optimized out>) at topology_util.c:696
#16 0xb368891c in ipa_topo_apply_shared_config () at topology_init.c:165
#17 0xb6e4e65c in eq_call_all () at ldap/servers/slapd/eventq.c:278
#18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
ldap/servers/slapd/eventq.c:323
#19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
../../.././nspr/pr/src/pthreads/ptthread.c:201
#20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at pthread_create.c:465
#21 0xb6785da8 in None () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
On Tue, May 15, 2018 at 3:01 AM, thierry bordaz <tbordaz(a)redhat.com> wrote:
Hi Jonathan,
This problem looks new to me and has something specific to your
environment.
I think the best approach is to continue to debug on your system if you
have the possibility to do so.
From strace we can see that DS started smoothly (created its pid file then
notified systemd it was running fine). According to the pstack nunc-stans
was running and was able to accept network events even if it appears it
detected no incoming connection.
So the server should be ready to serve for some seconds (more than a
minute), then it crashed with one thread dereferencing likely a wrong
pointer.
Could you attach a debugger when the server is started and wait for the
sigsegv to occur. Then confirm the crashing thread backstack.
If it is confirmed, I am afraid this is a stack corruption and valgrind
could help (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-
memory-growthinvalid-access-with-valgrind).
best regards
thierry
On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
Here's a strace from before it dies. Most of the elapsed time is it
waiting on some futex call it looks like near the end, when it finally
"returns" (from lack of strace output for duration of call I assume it
didn't actually return but SIGSEGV in that call) and strace prints ' = ?'
on the futex it then immediately reports SIGSEGV after. So maybe the
problem is that futex call, which may mean the problem is not directly in
389DS / FreeIPA itself?
15:13:31.626587 (+ 0.000630) listen(8, 128) = 0 <0.000068>
15:13:31.626857 (+ 0.000235) listen(9, 128) = 0 <0.000048>
15:13:31.627111 (+ 0.000251) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=41120614}) = 0 <0.000085>
15:13:31.627457 (+ 0.000356) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=627560772}) = 0 <0.000043>
15:13:31.631233 (+ 0.003839) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=45286796}) = 0 <0.000077>
15:13:31.631720 (+ 0.000427) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=45661430}) = 0 <0.000042>
15:13:31.631955 (+ 0.000220) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=632049036}) = 0 <0.000047>
15:13:31.635669 (+ 0.003785) clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932, tv_nsec=49725840}) = 0 <0.000146>
15:13:31.636484 (+ 0.000784) write(16, "a", 1) = 1 <0.000118>
15:13:31.636855 (+ 0.000341) sched_yield() = 0 <0.000252>
15:13:31.637322 (+ 0.000470) futex(0x1cb57a0, FUTEX_WAKE_PRIVATE, 1) =
1 <0.000088>
15:13:31.637897 (+ 0.000610) write(16, "a", 1) = 1 <0.000221>
15:13:31.638394 (+ 0.000467) sched_yield() = 0 <0.000047>
15:13:31.638619 (+ 0.000202) futex(0x1cb5710, FUTEX_WAKE_PRIVATE, 1) =
1 <0.000065>
15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33 <0.000831>
15:13:31.640260 (+ 0.001387) getpid() = 32353 <0.000077>
15:13:31.640558 (+ 0.000256) fstat64(33, {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0 <0.000119>
15:13:31.641106 (+ 0.000556) write(33, "32353\n", 6) = 6 <0.000127>
15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
15:13:31.642216 (+ 0.000758)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
0644) = 0 <0.000152>
15:13:31.642900 (+ 0.000679) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=643020294}) = 0 <0.000056>
15:13:31.643495 (+ 0.000590) write(2, "[14/May/2018:15:13:31.643020294
"..., 134) = 134 <0.002697>
15:13:31.646515 (+ 0.003052) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=646694394}) = 0 <0.000075>
15:13:31.646892 (+ 0.000337) write(4, "[14/May/2018:15:13:31.646694394
"..., 134) = 134 <0.000522>
15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
15:13:31.654425 (+ 0.006617) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=654598946}) = 0 <0.000253>
15:13:31.655137 (+ 0.000717) write(2, "[14/May/2018:15:13:31.654598946
"..., 136) = 136 <0.002427>
15:13:31.658312 (+ 0.003165) clock_gettime(CLOCK_REALTIME,
{tv_sec=1526328811, tv_nsec=658486117}) = 0 <0.000251>
15:13:31.659032 (+ 0.000682) write(4, "[14/May/2018:15:13:31.658486117
"..., 136) = 136 <0.000346>
15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
15:13:31.663230 (+ 0.003642) getpid() = 32353 <0.000045>
15:13:31.663732 (+ 0.000454) socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC,
0) = 33 <0.000296>
15:13:31.664760 (+ 0.001048) getsockopt(33, SOL_SOCKET, SO_SNDBUF,
[163840], [4]) = 0 <0.000108>
15:13:31.665141 (+ 0.000386) setsockopt(33, SOL_SOCKET,
SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
<0.000051>
15:13:31.665500 (+ 0.000334) setsockopt(33, SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0 <0.000229>
15:13:31.665973 (+ 0.000468) sendmsg(33, {msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"}, msg_namelen=21,
msg_iov=[{iov_base="READY=1\nSTATUS=slapd started: Re"..., iov_len=69}],
msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 69 <0.000600>
15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
15:13:31.677320 (+ 0.010146) futex(0xb31122e8, FUTEX_WAIT, 32357,
NULL) = ?
15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV (core dumped) +++
On Mon, May 14, 2018 at 10:27 AM, thierry bordaz <tbordaz(a)redhat.com>
wrote:
> Hi Jonathan,
>
> This is weird as the crashing thread stack looks truncated (did you
> copy/paste all of it ?)
>
> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
> #2 0x00000000 in None ()
>
> Did you install 389-ds-base-debuginfo ?
> How did you get that backtrace ? from a core dumped, pstack ? Can you
> attach a debugger before the crash occurs ?
>
> It looks it crashed soon at startup, could it be related to a broken
> dse.ldif. It should exists a dse.ldif.OK, is it possibly to try to start
> with it ?
>
> best regards
> thierry
>
> On 05/12/2018 01:22 AM, Jonathan Vaughn via FreeIPA-users wrote:
>
> Not sure if it makes a difference... I was looking into this again and
> realized I had a bunch of messages from gdb telling me to install more
> debuginfo. I've done that now, here it is again freshly run through gdb
>
> GNU gdb (GDB) Fedora 8.0.1-36.fc27
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.h
> tml>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "armv7hl-redhat-linux-gnueabi".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>;.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>;.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
> /usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
> done.
> ...
>
> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
> #2 0x00000000 in None ()
>
>
>
> On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden <rcritten(a)redhat.com>
>> wrote:
>>
>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>
>>>> Still trying to figure this out. It looks like slapd is dying, I
>>>> thought it was still running for some reason.
>>>>
>>>> slapd is dying to segfault. strace of it happening doesn't seem to
>>>> reveal much:
>>>>
>>>
>>> A stack trace would very much help trying to track down the cause.
>>>
>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>>> ebugging-crashes
>>>
>>> rob
>>>
>>>
>>>> 18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
>>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
>>>> 18:32:41.544907 (+ 0.001195) getpid() = 16014
>>>> 18:32:41.545269 (+ 0.000329) fstat64(32, {st_mode=S_IFREG|0644,
>>>> st_size=0, ...}) = 0
>>>> 18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) = 6
>>>> 18:32:41.546603 (+ 0.000818) close(32) = 0
>>>> 18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> 0644) = 0
>>>> 18:32:41.547741 (+ 0.000676) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>>>> 18:32:41.548324 (+ 0.000587) write(2,
>>>> "[07/May/2018:18:32:41.548030641 "..., 134) = 134
>>>> 18:32:41.551096 (+ 0.002840) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>>>> 18:32:41.551568 (+ 0.000406) write(4,
>>>> "[07/May/2018:18:32:41.551287555 "..., 134) = 134
>>>> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
>>>> 18:32:41.558499 (+ 0.006170) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>>>> 18:32:41.558901 (+ 0.000350) write(2,
>>>> "[07/May/2018:18:32:41.558678099 "..., 136) = 136
>>>> 18:32:41.561537 (+ 0.002680) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>>>> 18:32:41.562357 (+ 0.000793) write(4,
>>>> "[07/May/2018:18:32:41.561718659 "..., 136) = 136
>>>> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
>>>> 18:32:41.566928 (+ 0.003452) getpid() = 16014
>>>> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>>> 18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET, SO_SNDBUF,
>>>> [163840], [4]) = 0
>>>> 18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET,
>>>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
>>>> 18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET, SO_SNDBUF,
>>>> [8388608], 4) = 0
>>>> 18:32:41.570064 (+ 0.000513) sendmsg(32,
>>>> {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
>>>> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>>>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
>>>> MSG_NOSIGNAL) = 69
>>>> 18:32:41.570845 (+ 0.000789) close(32) = 0
>>>> 18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8, FUTEX_WAIT, 16016,
>>>> NULL) = ?
>>>> 18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
>>>>
>>>>
>
5:36 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
This looks really familiar and I thought it was fixed. It should have
been fixed in 1.3.7.10-1 (https://pagure.io/389-ds-base/issue/49618).
In your debug session go "up" into agmt_maxcsn_update() and do:
(gdb) p *agmt
Then send us that output please.
Thanks,
Mark
On 05/15/2018 05:29 PM, Jonathan Vaughn via FreeIPA-users wrote:
Here is a backtrace from live gdb after the segfault. Looks like
things went wrong somewhere during in the replication code ?
Thread 36 "ns-slapd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x9e0bc280 (LWP 4662)]
strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
142 ldrd data1a, data1b, [src]
(gdb) bt
#0 0xb6728f2e in strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
#1 0xb6973b40 in cvt_s (flags=0, prec=<optimized out>, width=0,
str=0x4 <error: Cannot access memory at address 0x4>, ss=<optimized out>)
at ../../.././nspr/pr/src/io/prprf.c:374
#2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc, fmt=0xb34fddf2
"", fmt@entry=0xb3527770 <agmt_set> "\360\357\305\002",
ap=...)
at ../../.././nspr/pr/src/io/prprf.c:1018
#3 0xb6973c8c in PR_vsmprintf (fmt=fmt@entry=0xb3527770 <agmt_set>
"\360\357\305\002", ap=..., ap@entry=...) at
../../.././nspr/pr/src/io/prprf.c:1184
#4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0
"%s;%s;%s;%ld;%d;%s") at ldap/servers/slapd/ch_malloc.c:331
#5 0xb34af5a4 in agmt_update_maxcsn (r=r@entry=0x2c89810,
sdn=0x3e2bac0, sdn@entry=0x10, op=<optimized out>, mods=0x10,
mods@entry=0x3eec100, csn=csn@entry=0x3e30220)
at ldap/servers/plugins/replication/repl5_agmt.c:3036
#6 0xb34bd424 in write_changelog_and_ruv (pb=pb@entry=0x2cb54a0) at
ldap/servers/plugins/replication/repl5_plugins.c:1124
#7 0xb34be7ec in multimaster_be_betxnpostop_add (pb=0x2cb54a0) at
ldap/servers/plugins/replication/repl5_plugins.c:855
#8 0xb34be880 in multimaster_mmr_postop (pb=<optimized out>,
flags=560) at ldap/servers/plugins/replication/repl5_plugins.c:616
#9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop
(pb=pb@entry=0x2cb54a0, e=e@entry=0x0, flags=flags@entry=560) at
ldap/servers/slapd/plugin_mmr.c:65
#10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
#11 0xb6e2ce4c in op_shared_add (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:679
#12 0xb6e2d35c in add_internal_pb (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:407
#13 0xb6e2e0f8 in slapi_add_internal_pb (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:332
#14 0xb368db50 in ipa_topo_util_segment_write
(tconf=tconf@entry=0x2cb5860, tsegm=tsegm@entry=0x3f2d9a0) at
topology_util.c:1251
#15 0xb368e01c in ipa_topo_util_update_agmt_list (conf=0x2cb5860,
repl_segments=<optimized out>) at topology_util.c:696
#16 0xb368891c in ipa_topo_apply_shared_config () at topology_init.c:165
#17 0xb6e4e65c in eq_call_all () at ldap/servers/slapd/eventq.c:278
#18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
ldap/servers/slapd/eventq.c:323
#19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
../../.././nspr/pr/src/pthreads/ptthread.c:201
#20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at pthread_create.c:465
#21 0xb6785da8 in None () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
On Tue, May 15, 2018 at 3:01 AM, thierry bordaz <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>> wrote:
Hi Jonathan,
This problem looks new to me and has something specific to your
environment.
I think the best approach is to continue to debug on your system
if you have the possibility to do so.
From strace we can see that DS started smoothly (created its pid
file then notified systemd it was running fine). According to the
pstack nunc-stans was running and was able to accept network
events even if it appears it detected no incoming connection.
So the server should be ready to serve for some seconds (more than
a minute), then it crashed with one thread dereferencing likely a
wrong pointer.
Could you attach a debugger when the server is started and wait
for the sigsegv to occur. Then confirm the crashing thread backstack.
If it is confirmed, I am afraid this is a stack corruption and
valgrind could help
(http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...>).
best regards
thierry
On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
> Here's a strace from before it dies. Most of the elapsed time is
> it waiting on some futex call it looks like near the end, when it
> finally "returns" (from lack of strace output for duration of
> call I assume it didn't actually return but SIGSEGV in that call)
> and strace prints ' = ?' on the futex it then immediately reports
> SIGSEGV after. So maybe the problem is that futex call, which may
> mean the problem is not directly in 389DS / FreeIPA itself?
>
>
>
> 15:13:31.626587 (+ 0.000630) listen(8, 128) = 0 <0.000068>
> 15:13:31.626857 (+ 0.000235) listen(9, 128) = 0 <0.000048>
> 15:13:31.627111 (+ 0.000251) clock_gettime(CLOCK_MONOTONIC,
> {tv_sec=1464932, tv_nsec=41120614}) = 0 <0.000085>
> 15:13:31.627457 (+ 0.000356) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=627560772}) = 0 <0.000043>
> 15:13:31.631233 (+ 0.003839) clock_gettime(CLOCK_MONOTONIC,
> {tv_sec=1464932, tv_nsec=45286796}) = 0 <0.000077>
> 15:13:31.631720 (+ 0.000427) clock_gettime(CLOCK_MONOTONIC,
> {tv_sec=1464932, tv_nsec=45661430}) = 0 <0.000042>
> 15:13:31.631955 (+ 0.000220) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=632049036}) = 0 <0.000047>
> 15:13:31.635669 (+ 0.003785) clock_gettime(CLOCK_MONOTONIC,
> {tv_sec=1464932, tv_nsec=49725840}) = 0 <0.000146>
> 15:13:31.636484 (+ 0.000784) write(16, "a", 1) = 1
<0.000118>
> 15:13:31.636855 (+ 0.000341) sched_yield() = 0 <0.000252>
> 15:13:31.637322 (+ 0.000470) futex(0x1cb57a0,
> FUTEX_WAKE_PRIVATE, 1) = 1 <0.000088>
> 15:13:31.637897 (+ 0.000610) write(16, "a", 1) = 1
<0.000221>
> 15:13:31.638394 (+ 0.000467) sched_yield() = 0 <0.000047>
> 15:13:31.638619 (+ 0.000202) futex(0x1cb5710,
> FUTEX_WAKE_PRIVATE, 1) = 1 <0.000065>
> 15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33 <0.000831>
> 15:13:31.640260 (+ 0.001387) getpid() = 32353 <0.000077>
> 15:13:31.640558 (+ 0.000256) fstat64(33,
> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000119>
> 15:13:31.641106 (+ 0.000556) write(33, "32353\n", 6) = 6
> <0.000127>
> 15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
> 15:13:31.642216 (+ 0.000758)
> chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", 0644) = 0
> <0.000152>
> 15:13:31.642900 (+ 0.000679) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=643020294}) = 0 <0.000056>
> 15:13:31.643495 (+ 0.000590) write(2,
> "[14/May/2018:15:13:31.643020294 "..., 134) = 134 <0.002697>
> 15:13:31.646515 (+ 0.003052) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=646694394}) = 0 <0.000075>
> 15:13:31.646892 (+ 0.000337) write(4,
> "[14/May/2018:15:13:31.646694394 "..., 134) = 134 <0.000522>
> 15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
> 15:13:31.654425 (+ 0.006617) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=654598946}) = 0 <0.000253>
> 15:13:31.655137 (+ 0.000717) write(2,
> "[14/May/2018:15:13:31.654598946 "..., 136) = 136 <0.002427>
> 15:13:31.658312 (+ 0.003165) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=658486117}) = 0 <0.000251>
> 15:13:31.659032 (+ 0.000682) write(4,
> "[14/May/2018:15:13:31.658486117 "..., 136) = 136 <0.000346>
> 15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
> 15:13:31.663230 (+ 0.003642) getpid() = 32353 <0.000045>
> 15:13:31.663732 (+ 0.000454) socket(AF_UNIX,
> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 33 <0.000296>
> 15:13:31.664760 (+ 0.001048) getsockopt(33, SOL_SOCKET,
> SO_SNDBUF, [163840], [4]) = 0 <0.000108>
> 15:13:31.665141 (+ 0.000386) setsockopt(33, SOL_SOCKET,
> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not
> permitted) <0.000051>
> 15:13:31.665500 (+ 0.000334) setsockopt(33, SOL_SOCKET,
> SO_SNDBUF, [8388608], 4) = 0 <0.000229>
> 15:13:31.665973 (+ 0.000468) sendmsg(33,
> {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd
> started: Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0,
> msg_flags=0}, MSG_NOSIGNAL) = 69 <0.000600>
> 15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
> 15:13:31.677320 (+ 0.010146) futex(0xb31122e8, FUTEX_WAIT,
> 32357, NULL) = ?
> 15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV (core
> dumped) +++
>
> On Mon, May 14, 2018 at 10:27 AM, thierry bordaz
> <tbordaz(a)redhat.com <mailto:tbordaz@redhat.com>> wrote:
>
> Hi Jonathan,
>
> This is weird as the crashing thread stack looks truncated
> (did you copy/paste all of it ?)
>
> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
> #2 0x00000000 in None ()
>
> Did you install 389-ds-base-debuginfo ?
> How did you get that backtrace ? from a core dumped, pstack ?
> Can you attach a debugger before the crash occurs ?
>
> It looks it crashed soon at startup, could it be related to a
> broken dse.ldif. It should exists a dse.ldif.OK, is it
> possibly to try to start with it ?
>
> best regards
> thierry
>
> On 05/12/2018 01:22 AM, Jonathan Vaughn via FreeIPA-users wrote:
>> Not sure if it makes a difference... I was looking into this
>> again and realized I had a bunch of messages from gdb
>> telling me to install more debuginfo. I've done that now,
>> here it is again freshly run through gdb
>>
>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>> Copyright (C) 2017 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html
>> <http://gnu.org/licenses/gpl.html>>
>> This is free software: you are free to change and
>> redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type
>> "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "armv7hl-redhat-linux-gnueabi".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/
>> <http://www.gnu.org/software/gdb/bugs/>>;.
>> Find the GDB manual and other documentation resources online at:
>> <http://www.gnu.org/software/gdb/documentation/
>> <http://www.gnu.org/software/gdb/documentation/>>;.
>> For help, type "help".
>> Type "apropos word" to search for commands related to
"word"...
>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols
>> from
>> /usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>> done.
>> ...
>>
>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>> #2 0x00000000 in None ()
>>
>>
>>
>> On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden
>> <rcritten(a)redhat.com <mailto:rcritten@redhat.com>>
wrote:
>>
>> Jonathan Vaughn via FreeIPA-users wrote:
>>
>> Still trying to figure this out. It looks like
>> slapd is dying, I thought it was still running
>> for some reason.
>>
>> slapd is dying to segfault. strace of it
>> happening doesn't seem to reveal much:
>>
>>
>> A stack trace would very much help trying to track
>> down the cause.
>>
>>
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
>>
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-cras...
>>
>> rob
>>
>>
>> 18:32:41.543717 (+ 0.000801)
>> openat(AT_FDCWD,
>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
>> 18:32:41.544907 (+ 0.001195) getpid() = 16014
>> 18:32:41.545269 (+ 0.000329) fstat64(32,
>> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>> 18:32:41.545799 (+ 0.000536) write(32,
>> "16014\n", 6) = 6
>> 18:32:41.546603 (+ 0.000818) close(32) = 0
>> 18:32:41.547061 (+ 0.000448)
>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>> 0644) = 0
>> 18:32:41.547741 (+ 0.000676)
>> clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>> 18:32:41.548324 (+ 0.000587) write(2,
>> "[07/May/2018:18:32:41.548030641 "..., 134) = 134
>> 18:32:41.551096 (+ 0.002840)
>> clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>> 18:32:41.551568 (+ 0.000406) write(4,
>> "[07/May/2018:18:32:41.551287555 "..., 134) = 134
>> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
>> 18:32:41.558499 (+ 0.006170)
>> clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>> 18:32:41.558901 (+ 0.000350) write(2,
>> "[07/May/2018:18:32:41.558678099 "..., 136) = 136
>> 18:32:41.561537 (+ 0.002680)
>> clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>> 18:32:41.562357 (+ 0.000793) write(4,
>> "[07/May/2018:18:32:41.561718659 "..., 136) = 136
>> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
>> 18:32:41.566928 (+ 0.003452) getpid() = 16014
>> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>> 18:32:41.568628 (+ 0.000912) getsockopt(32,
>> SOL_SOCKET, SO_SNDBUF, [163840], [4]) = 0
>> 18:32:41.568972 (+ 0.000319) setsockopt(32,
>> SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = -1
>> EPERM (Operation not permitted)
>> 18:32:41.569548 (+ 0.000589) setsockopt(32,
>> SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0
>> 18:32:41.570064 (+ 0.000513) sendmsg(32,
>> {msg_name={sa_family=AF_UNIX,
>> sun_path="/run/systemd/notify"}, msg_namelen=21,
>> msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>> started: Re"..., iov_len=69}], msg_iovlen=1,
>> msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 69
>> 18:32:41.570845 (+ 0.000789) close(32) = 0
>> 18:32:41.576358 (+ 0.005575)
>> futex(0xb30ed2e8, FUTEX_WAIT, 16016, NULL) = ?
>> 18:33:01.730774 (+ 20.154428) +++ killed by
>> SIGSEGV +++
>>
>
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
2:38 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
(gdb) up
#1 0xb6926b40 in cvt_s (flags=0, prec=<optimized out>, width=0, str=0x4
<error: Cannot access memory at address 0x4>, ss=<optimized out>)
at ../../.././nspr/pr/src/io/prprf.c:374
374 slen = strlen(str);
(gdb) up
#2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2 "", fmt@entry=0xb34da770
<agmt_set> "\360\317p\002", ap=...) at
../../.././nspr/pr/src/io/prprf.c:1018
1018 rv = cvt_s(ss, u.s, width, prec, flags);
(gdb) up
#3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002", ap=..., ap@entry=...) at
../../.././nspr/pr/src/io/prprf.c:1184
1184 rv = dosprintf(&ss, fmt, ap);
(gdb) up
#4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0 "%s;%s;%s;%ld;%d;%s")
at ldap/servers/slapd/ch_malloc.c:331
331 p = PR_vsmprintf(fmt, ap);
(gdb) up
#5 0xb34625a4 in agmt_update_maxcsn (r=r@entry=0x2737810, sdn=0x38d9a40,
sdn@entry=0x10, op=<optimized out>, mods=0x10, mods@entry=0x27d0100,
csn=csn@entry=0x38de350)
at ldap/servers/plugins/replication/repl5_agmt.c:3036
3036 agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s", slapi_sdn_get_dn(agmt->replarea),
(gdb) p *agmt
$1 = {hostname = 0x2757be0 "ipa-12.company.internal", port = 389,
transport_flags = 0, binddn = 0x26ed650 "", creds = 0x26ed7a0, bindmethod =
3, replarea = 0x2757480,
frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
frac_attr_total_defined = 1, schedule = 0x22dd0c0, auto_initialize = 502,
dn = 0x2756d00, rdn = 0x2756c20,
long_name = 0x22dd100 "agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)", protocol = 0x2220930, changecounters = 0x20cb180,
num_changecounters = 0,
max_changecounters = 256, last_update_start_time = 1526499214,
last_update_end_time = 1526499214,
last_update_status = "Error (0) Replica acquired successfully:
Incremental update succeeded", '\000' <repeats 954 times>,
update_in_progress = 0, is_enabled = 1,
last_init_start_time = 0, last_init_end_time = 0, last_init_status =
'\000' <repeats 1023 times>, lock = 0x2741740, consumerRUV = 0x2772e50,
consumerSchemaCSN = 0x39da520, consumerRID = 4, tmpConsumerRID = 0,
timeout = 120, stop_in_progress = 0, busywaittime = 0, pausetime = 0, priv
= 0x0,
attrs_to_strip = 0x2757ba0, agreement_type = 0, protocol_timeout =
0x26ed5f0, maxcsn = 0x0, flowControlWindow = 1000, flowControlPause = 2000,
ignoreMissingChange = 0,
attr_lock = 0x2757c20, WaitForAsyncResults = 100}
On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds <mreynolds(a)redhat.com> wrote:
This looks really familiar and I thought it was fixed. It should
have
been fixed in 1.3.7.10-1 (https://pagure.io/389-ds-base/issue/49618).
In your debug session go "up" into agmt_maxcsn_update() and do:
(gdb) p *agmt
Then send us that output please.
Thanks,
Mark
On 05/15/2018 05:29 PM, Jonathan Vaughn via FreeIPA-users wrote:
Here is a backtrace from live gdb after the segfault. Looks like things
went wrong somewhere during in the replication code ?
Thread 36 "ns-slapd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x9e0bc280 (LWP 4662)]
strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
142 ldrd data1a, data1b, [src]
(gdb) bt
#0 0xb6728f2e in strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
#1 0xb6973b40 in cvt_s (flags=0, prec=<optimized out>, width=0, str=0x4
<error: Cannot access memory at address 0x4>, ss=<optimized out>)
at ../../.././nspr/pr/src/io/prprf.c:374
#2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc, fmt=0xb34fddf2 "",
fmt@entry=0xb3527770 <agmt_set> "\360\357\305\002", ap=...)
at ../../.././nspr/pr/src/io/prprf.c:1018
#3 0xb6973c8c in PR_vsmprintf (fmt=fmt@entry=0xb3527770 <agmt_set>
"\360\357\305\002", ap=..., ap@entry=...) at ../../.././nspr/pr/src/io/
prprf.c:1184
#4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0 "%s;%s;%s;%ld;%d;%s")
at ldap/servers/slapd/ch_malloc.c:331
#5 0xb34af5a4 in agmt_update_maxcsn (r=r@entry=0x2c89810, sdn=0x3e2bac0,
sdn@entry=0x10, op=<optimized out>, mods=0x10, mods@entry=0x3eec100,
csn=csn@entry=0x3e30220)
at ldap/servers/plugins/replication/repl5_agmt.c:3036
#6 0xb34bd424 in write_changelog_and_ruv (pb=pb@entry=0x2cb54a0) at
ldap/servers/plugins/replication/repl5_plugins.c:1124
#7 0xb34be7ec in multimaster_be_betxnpostop_add (pb=0x2cb54a0) at
ldap/servers/plugins/replication/repl5_plugins.c:855
#8 0xb34be880 in multimaster_mmr_postop (pb=<optimized out>, flags=560)
at ldap/servers/plugins/replication/repl5_plugins.c:616
#9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop (pb=pb@entry=0x2cb54a0,
e=e@entry=0x0, flags=flags@entry=560) at ldap/servers/slapd/plugin_mmr.
c:65
#10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
#11 0xb6e2ce4c in op_shared_add (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:679
#12 0xb6e2d35c in add_internal_pb (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:407
#13 0xb6e2e0f8 in slapi_add_internal_pb (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:332
#14 0xb368db50 in ipa_topo_util_segment_write (tconf=tconf@entry=0x2cb5860,
tsegm=tsegm@entry=0x3f2d9a0) at topology_util.c:1251
#15 0xb368e01c in ipa_topo_util_update_agmt_list (conf=0x2cb5860,
repl_segments=<optimized out>) at topology_util.c:696
#16 0xb368891c in ipa_topo_apply_shared_config () at topology_init.c:165
#17 0xb6e4e65c in eq_call_all () at ldap/servers/slapd/eventq.c:278
#18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
ldap/servers/slapd/eventq.c:323
#19 0xb6982d68 in _pt_root (arg=0x2c8da40) at ../../.././nspr/pr/src/
pthreads/ptthread.c:201
#20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at pthread_create.c:465
#21 0xb6785da8 in None () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
On Tue, May 15, 2018 at 3:01 AM, thierry bordaz <tbordaz(a)redhat.com>
wrote:
> Hi Jonathan,
>
> This problem looks new to me and has something specific to your
> environment.
> I think the best approach is to continue to debug on your system if you
> have the possibility to do so.
>
> From strace we can see that DS started smoothly (created its pid file
> then notified systemd it was running fine). According to the pstack
> nunc-stans was running and was able to accept network events even if it
> appears it detected no incoming connection.
> So the server should be ready to serve for some seconds (more than a
> minute), then it crashed with one thread dereferencing likely a wrong
> pointer.
>
> Could you attach a debugger when the server is started and wait for the
> sigsegv to occur. Then confirm the crashing thread backstack.
> If it is confirmed, I am afraid this is a stack corruption and valgrind
> could help (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-me
> mory-growthinvalid-access-with-valgrind).
>
> best regards
> thierry
>
>
> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
>
> Here's a strace from before it dies. Most of the elapsed time is it
> waiting on some futex call it looks like near the end, when it finally
> "returns" (from lack of strace output for duration of call I assume it
> didn't actually return but SIGSEGV in that call) and strace prints ' =
?'
> on the futex it then immediately reports SIGSEGV after. So maybe the
> problem is that futex call, which may mean the problem is not directly in
> 389DS / FreeIPA itself?
>
>
>
> 15:13:31.626587 (+ 0.000630) listen(8, 128) = 0 <0.000068>
> 15:13:31.626857 (+ 0.000235) listen(9, 128) = 0 <0.000048>
> 15:13:31.627111 (+ 0.000251) clock_gettime(CLOCK_MONOTONIC,
> {tv_sec=1464932, tv_nsec=41120614}) = 0 <0.000085>
> 15:13:31.627457 (+ 0.000356) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=627560772}) = 0 <0.000043>
> 15:13:31.631233 (+ 0.003839) clock_gettime(CLOCK_MONOTONIC,
> {tv_sec=1464932, tv_nsec=45286796}) = 0 <0.000077>
> 15:13:31.631720 (+ 0.000427) clock_gettime(CLOCK_MONOTONIC,
> {tv_sec=1464932, tv_nsec=45661430}) = 0 <0.000042>
> 15:13:31.631955 (+ 0.000220) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=632049036}) = 0 <0.000047>
> 15:13:31.635669 (+ 0.003785) clock_gettime(CLOCK_MONOTONIC,
> {tv_sec=1464932, tv_nsec=49725840}) = 0 <0.000146>
> 15:13:31.636484 (+ 0.000784) write(16, "a", 1) = 1 <0.000118>
> 15:13:31.636855 (+ 0.000341) sched_yield() = 0 <0.000252>
> 15:13:31.637322 (+ 0.000470) futex(0x1cb57a0, FUTEX_WAKE_PRIVATE, 1)
> = 1 <0.000088>
> 15:13:31.637897 (+ 0.000610) write(16, "a", 1) = 1 <0.000221>
> 15:13:31.638394 (+ 0.000467) sched_yield() = 0 <0.000047>
> 15:13:31.638619 (+ 0.000202) futex(0x1cb5710, FUTEX_WAKE_PRIVATE, 1)
> = 1 <0.000065>
> 15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid", O_WRONLY|O_CREAT|O_TRUNC,
> 0666) = 33 <0.000831>
> 15:13:31.640260 (+ 0.001387) getpid() = 32353 <0.000077>
> 15:13:31.640558 (+ 0.000256) fstat64(33, {st_mode=S_IFREG|0644,
> st_size=0, ...}) = 0 <0.000119>
> 15:13:31.641106 (+ 0.000556) write(33, "32353\n", 6) = 6
<0.000127>
> 15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
> 15:13:31.642216 (+ 0.000758)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
> 0644) = 0 <0.000152>
> 15:13:31.642900 (+ 0.000679) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=643020294}) = 0 <0.000056>
> 15:13:31.643495 (+ 0.000590) write(2, "[14/May/2018:15:13:31.643020294
> "..., 134) = 134 <0.002697>
> 15:13:31.646515 (+ 0.003052) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=646694394}) = 0 <0.000075>
> 15:13:31.646892 (+ 0.000337) write(4, "[14/May/2018:15:13:31.646694394
> "..., 134) = 134 <0.000522>
> 15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
> 15:13:31.654425 (+ 0.006617) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=654598946}) = 0 <0.000253>
> 15:13:31.655137 (+ 0.000717) write(2, "[14/May/2018:15:13:31.654598946
> "..., 136) = 136 <0.002427>
> 15:13:31.658312 (+ 0.003165) clock_gettime(CLOCK_REALTIME,
> {tv_sec=1526328811, tv_nsec=658486117}) = 0 <0.000251>
> 15:13:31.659032 (+ 0.000682) write(4, "[14/May/2018:15:13:31.658486117
> "..., 136) = 136 <0.000346>
> 15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
> 15:13:31.663230 (+ 0.003642) getpid() = 32353 <0.000045>
> 15:13:31.663732 (+ 0.000454) socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC,
> 0) = 33 <0.000296>
> 15:13:31.664760 (+ 0.001048) getsockopt(33, SOL_SOCKET, SO_SNDBUF,
> [163840], [4]) = 0 <0.000108>
> 15:13:31.665141 (+ 0.000386) setsockopt(33, SOL_SOCKET,
> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
> <0.000051>
> 15:13:31.665500 (+ 0.000334) setsockopt(33, SOL_SOCKET, SO_SNDBUF,
> [8388608], 4) = 0 <0.000229>
> 15:13:31.665973 (+ 0.000468) sendmsg(33,
> {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
> MSG_NOSIGNAL) = 69 <0.000600>
> 15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
> 15:13:31.677320 (+ 0.010146) futex(0xb31122e8, FUTEX_WAIT, 32357,
> NULL) = ?
> 15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV (core dumped) +++
>
> On Mon, May 14, 2018 at 10:27 AM, thierry bordaz <tbordaz(a)redhat.com>
> wrote:
>
>> Hi Jonathan,
>>
>> This is weird as the crashing thread stack looks truncated (did you
>> copy/paste all of it ?)
>>
>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>> #2 0x00000000 in None ()
>>
>> Did you install 389-ds-base-debuginfo ?
>> How did you get that backtrace ? from a core dumped, pstack ? Can you
>> attach a debugger before the crash occurs ?
>>
>> It looks it crashed soon at startup, could it be related to a broken
>> dse.ldif. It should exists a dse.ldif.OK, is it possibly to try to start
>> with it ?
>>
>> best regards
>> thierry
>>
>> On 05/12/2018 01:22 AM, Jonathan Vaughn via FreeIPA-users wrote:
>>
>> Not sure if it makes a difference... I was looking into this again and
>> realized I had a bunch of messages from gdb telling me to install more
>> debuginfo. I've done that now, here it is again freshly run through gdb
>>
>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>> Copyright (C) 2017 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <
>> http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show
>> copying"
>> and "show warranty" for details.
>> This GDB was configured as "armv7hl-redhat-linux-gnueabi".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>;.
>> Find the GDB manual and other documentation resources online at:
>> <http://www.gnu.org/software/gdb/documentation/>;.
>> For help, type "help".
>> Type "apropos word" to search for commands related to
"word"...
>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
>> /usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>> done.
>> ...
>>
>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>> #2 0x00000000 in None ()
>>
>>
>>
>> On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden <rcritten(a)redhat.com>
>>> wrote:
>>>
>>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>>
>>>>> Still trying to figure this out. It looks like slapd is dying, I
>>>>> thought it was still running for some reason.
>>>>>
>>>>> slapd is dying to segfault. strace of it happening doesn't seem
to
>>>>> reveal much:
>>>>>
>>>>
>>>> A stack trace would very much help trying to track down the cause.
>>>>
>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>>>> ebugging-crashes
>>>>
>>>> rob
>>>>
>>>>
>>>>> 18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
>>>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
>>>>> 18:32:41.544907 (+ 0.001195) getpid() = 16014
>>>>> 18:32:41.545269 (+ 0.000329) fstat64(32, {st_mode=S_IFREG|0644,
>>>>> st_size=0, ...}) = 0
>>>>> 18:32:41.545799 (+ 0.000536) write(32, "16014\n", 6) =
6
>>>>> 18:32:41.546603 (+ 0.000818) close(32) = 0
>>>>> 18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>> 0644) = 0
>>>>> 18:32:41.547741 (+ 0.000676) clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>>>>> 18:32:41.548324 (+ 0.000587) write(2,
>>>>> "[07/May/2018:18:32:41.548030641 "..., 134) = 134
>>>>> 18:32:41.551096 (+ 0.002840) clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>>>>> 18:32:41.551568 (+ 0.000406) write(4,
>>>>> "[07/May/2018:18:32:41.551287555 "..., 134) = 134
>>>>> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
>>>>> 18:32:41.558499 (+ 0.006170) clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>>>>> 18:32:41.558901 (+ 0.000350) write(2,
>>>>> "[07/May/2018:18:32:41.558678099 "..., 136) = 136
>>>>> 18:32:41.561537 (+ 0.002680) clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>>>>> 18:32:41.562357 (+ 0.000793) write(4,
>>>>> "[07/May/2018:18:32:41.561718659 "..., 136) = 136
>>>>> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
>>>>> 18:32:41.566928 (+ 0.003452) getpid() = 16014
>>>>> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
>>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>>>> 18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET,
>>>>> SO_SNDBUF, [163840], [4]) = 0
>>>>> 18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET,
>>>>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
>>>>> 18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET,
>>>>> SO_SNDBUF, [8388608], 4) = 0
>>>>> 18:32:41.570064 (+ 0.000513) sendmsg(32,
>>>>> {msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"},
>>>>> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd
started:
>>>>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0,
msg_flags=0},
>>>>> MSG_NOSIGNAL) = 69
>>>>> 18:32:41.570845 (+ 0.000789) close(32) = 0
>>>>> 18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8, FUTEX_WAIT,
16016,
>>>>> NULL) = ?
>>>>> 18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
>>>>>
>>>>>
>>
>
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
2:43 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
The installed version of 389* is 1.3.7.10-1.fc27 for armv7hl, which appears
to be the latest available version.
On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
(gdb) up
#1 0xb6926b40 in cvt_s (flags=0, prec=<optimized out>, width=0, str=0x4
<error: Cannot access memory at address 0x4>, ss=<optimized out>)
at ../../.././nspr/pr/src/io/prprf.c:374
374 slen = strlen(str);
(gdb) up
#2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2 "", fmt@entry=0xb34da770
<agmt_set> "\360\317p\002", ap=...) at ../../.././nspr/pr/src/io/
prprf.c:1018
1018 rv = cvt_s(ss, u.s, width, prec, flags);
(gdb) up
#3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002", ap=..., ap@entry=...) at ../../.././nspr/pr/src/io/
prprf.c:1184
1184 rv = dosprintf(&ss, fmt, ap);
(gdb) up
#4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0 "%s;%s;%s;%ld;%d;%s")
at ldap/servers/slapd/ch_malloc.c:331
331 p = PR_vsmprintf(fmt, ap);
(gdb) up
#5 0xb34625a4 in agmt_update_maxcsn (r=r@entry=0x2737810, sdn=0x38d9a40,
sdn@entry=0x10, op=<optimized out>, mods=0x10, mods@entry=0x27d0100,
csn=csn@entry=0x38de350)
at ldap/servers/plugins/replication/repl5_agmt.c:3036
3036 agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
slapi_sdn_get_dn(agmt->replarea),
(gdb) p *agmt
$1 = {hostname = 0x2757be0 "ipa-12.company.internal", port = 389,
transport_flags = 0, binddn = 0x26ed650 "", creds = 0x26ed7a0, bindmethod =
3, replarea = 0x2757480,
frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
frac_attr_total_defined = 1, schedule = 0x22dd0c0, auto_initialize = 502,
dn = 0x2756d00, rdn = 0x2756c20,
long_name = 0x22dd100 "agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)", protocol = 0x2220930, changecounters = 0x20cb180,
num_changecounters = 0,
max_changecounters = 256, last_update_start_time = 1526499214,
last_update_end_time = 1526499214,
last_update_status = "Error (0) Replica acquired successfully:
Incremental update succeeded", '\000' <repeats 954 times>,
update_in_progress = 0, is_enabled = 1,
last_init_start_time = 0, last_init_end_time = 0, last_init_status =
'\000' <repeats 1023 times>, lock = 0x2741740, consumerRUV = 0x2772e50,
consumerSchemaCSN = 0x39da520, consumerRID = 4, tmpConsumerRID = 0,
timeout = 120, stop_in_progress = 0, busywaittime = 0, pausetime = 0, priv
= 0x0,
attrs_to_strip = 0x2757ba0, agreement_type = 0, protocol_timeout =
0x26ed5f0, maxcsn = 0x0, flowControlWindow = 1000, flowControlPause = 2000,
ignoreMissingChange = 0,
attr_lock = 0x2757c20, WaitForAsyncResults = 100}
On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds <mreynolds(a)redhat.com>
wrote:
> This looks really familiar and I thought it was fixed. It should have
> been fixed in 1.3.7.10-1 (https://pagure.io/389-ds-base/issue/49618).
> In your debug session go "up" into agmt_maxcsn_update() and do:
>
> (gdb) p *agmt
>
> Then send us that output please.
>
> Thanks,
> Mark
>
>
> On 05/15/2018 05:29 PM, Jonathan Vaughn via FreeIPA-users wrote:
>
> Here is a backtrace from live gdb after the segfault. Looks like things
> went wrong somewhere during in the replication code ?
>
> Thread 36 "ns-slapd" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x9e0bc280 (LWP 4662)]
> strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
> 142 ldrd data1a, data1b, [src]
> (gdb) bt
> #0 0xb6728f2e in strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
> #1 0xb6973b40 in cvt_s (flags=0, prec=<optimized out>, width=0, str=0x4
> <error: Cannot access memory at address 0x4>, ss=<optimized out>)
> at ../../.././nspr/pr/src/io/prprf.c:374
> #2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc, fmt=0xb34fddf2 "",
> fmt@entry=0xb3527770 <agmt_set> "\360\357\305\002", ap=...)
> at ../../.././nspr/pr/src/io/prprf.c:1018
> #3 0xb6973c8c in PR_vsmprintf (fmt=fmt@entry=0xb3527770 <agmt_set>
> "\360\357\305\002", ap=..., ap@entry=...) at
> ../../.././nspr/pr/src/io/prprf.c:1184
> #4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0 "%s;%s;%s;%ld;%d;%s")
> at ldap/servers/slapd/ch_malloc.c:331
> #5 0xb34af5a4 in agmt_update_maxcsn (r=r@entry=0x2c89810,
> sdn=0x3e2bac0, sdn@entry=0x10, op=<optimized out>, mods=0x10,
mods@entry=0x3eec100,
> csn=csn@entry=0x3e30220)
> at ldap/servers/plugins/replication/repl5_agmt.c:3036
> #6 0xb34bd424 in write_changelog_and_ruv (pb=pb@entry=0x2cb54a0) at
> ldap/servers/plugins/replication/repl5_plugins.c:1124
> #7 0xb34be7ec in multimaster_be_betxnpostop_add (pb=0x2cb54a0) at
> ldap/servers/plugins/replication/repl5_plugins.c:855
> #8 0xb34be880 in multimaster_mmr_postop (pb=<optimized out>, flags=560)
> at ldap/servers/plugins/replication/repl5_plugins.c:616
> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop (pb=pb@entry=0x2cb54a0,
> e=e@entry=0x0, flags=flags@entry=560) at ldap/servers/slapd/plugin_mmr.
> c:65
> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
> #11 0xb6e2ce4c in op_shared_add (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:679
> #12 0xb6e2d35c in add_internal_pb (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:407
> #13 0xb6e2e0f8 in slapi_add_internal_pb (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:332
> #14 0xb368db50 in ipa_topo_util_segment_write (tconf=tconf@entry=0x2cb5860,
> tsegm=tsegm@entry=0x3f2d9a0) at topology_util.c:1251
> #15 0xb368e01c in ipa_topo_util_update_agmt_list (conf=0x2cb5860,
> repl_segments=<optimized out>) at topology_util.c:696
> #16 0xb368891c in ipa_topo_apply_shared_config () at topology_init.c:165
> #17 0xb6e4e65c in eq_call_all () at ldap/servers/slapd/eventq.c:278
> #18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
> ldap/servers/slapd/eventq.c:323
> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
> ../../.././nspr/pr/src/pthreads/ptthread.c:201
> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at pthread_create.c:465
> #21 0xb6785da8 in None () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>
>
> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz <tbordaz(a)redhat.com>
> wrote:
>
>> Hi Jonathan,
>>
>> This problem looks new to me and has something specific to your
>> environment.
>> I think the best approach is to continue to debug on your system if you
>> have the possibility to do so.
>>
>> From strace we can see that DS started smoothly (created its pid file
>> then notified systemd it was running fine). According to the pstack
>> nunc-stans was running and was able to accept network events even if it
>> appears it detected no incoming connection.
>> So the server should be ready to serve for some seconds (more than a
>> minute), then it crashed with one thread dereferencing likely a wrong
>> pointer.
>>
>> Could you attach a debugger when the server is started and wait for the
>> sigsegv to occur. Then confirm the crashing thread backstack.
>> If it is confirmed, I am afraid this is a stack corruption and valgrind
>> could help (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-me
>> mory-growthinvalid-access-with-valgrind).
>>
>> best regards
>> thierry
>>
>>
>> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
>>
>> Here's a strace from before it dies. Most of the elapsed time is it
>> waiting on some futex call it looks like near the end, when it finally
>> "returns" (from lack of strace output for duration of call I assume it
>> didn't actually return but SIGSEGV in that call) and strace prints ' =
?'
>> on the futex it then immediately reports SIGSEGV after. So maybe the
>> problem is that futex call, which may mean the problem is not directly in
>> 389DS / FreeIPA itself?
>>
>>
>>
>> 15:13:31.626587 (+ 0.000630) listen(8, 128) = 0 <0.000068>
>> 15:13:31.626857 (+ 0.000235) listen(9, 128) = 0 <0.000048>
>> 15:13:31.627111 (+ 0.000251) clock_gettime(CLOCK_MONOTONIC,
>> {tv_sec=1464932, tv_nsec=41120614}) = 0 <0.000085>
>> 15:13:31.627457 (+ 0.000356) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1526328811, tv_nsec=627560772}) = 0 <0.000043>
>> 15:13:31.631233 (+ 0.003839) clock_gettime(CLOCK_MONOTONIC,
>> {tv_sec=1464932, tv_nsec=45286796}) = 0 <0.000077>
>> 15:13:31.631720 (+ 0.000427) clock_gettime(CLOCK_MONOTONIC,
>> {tv_sec=1464932, tv_nsec=45661430}) = 0 <0.000042>
>> 15:13:31.631955 (+ 0.000220) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1526328811, tv_nsec=632049036}) = 0 <0.000047>
>> 15:13:31.635669 (+ 0.003785) clock_gettime(CLOCK_MONOTONIC,
>> {tv_sec=1464932, tv_nsec=49725840}) = 0 <0.000146>
>> 15:13:31.636484 (+ 0.000784) write(16, "a", 1) = 1
<0.000118>
>> 15:13:31.636855 (+ 0.000341) sched_yield() = 0 <0.000252>
>> 15:13:31.637322 (+ 0.000470) futex(0x1cb57a0, FUTEX_WAKE_PRIVATE, 1)
>> = 1 <0.000088>
>> 15:13:31.637897 (+ 0.000610) write(16, "a", 1) = 1
<0.000221>
>> 15:13:31.638394 (+ 0.000467) sched_yield() = 0 <0.000047>
>> 15:13:31.638619 (+ 0.000202) futex(0x1cb5710, FUTEX_WAKE_PRIVATE, 1)
>> = 1 <0.000065>
>> 15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
O_WRONLY|O_CREAT|O_TRUNC,
>> 0666) = 33 <0.000831>
>> 15:13:31.640260 (+ 0.001387) getpid() = 32353 <0.000077>
>> 15:13:31.640558 (+ 0.000256) fstat64(33, {st_mode=S_IFREG|0644,
>> st_size=0, ...}) = 0 <0.000119>
>> 15:13:31.641106 (+ 0.000556) write(33, "32353\n", 6) = 6
<0.000127>
>> 15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
>> 15:13:31.642216 (+ 0.000758)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>> 0644) = 0 <0.000152>
>> 15:13:31.642900 (+ 0.000679) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1526328811, tv_nsec=643020294}) = 0 <0.000056>
>> 15:13:31.643495 (+ 0.000590) write(2, "[14/May/2018:15:13:31.643020294
>> "..., 134) = 134 <0.002697>
>> 15:13:31.646515 (+ 0.003052) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1526328811, tv_nsec=646694394}) = 0 <0.000075>
>> 15:13:31.646892 (+ 0.000337) write(4, "[14/May/2018:15:13:31.646694394
>> "..., 134) = 134 <0.000522>
>> 15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
>> 15:13:31.654425 (+ 0.006617) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1526328811, tv_nsec=654598946}) = 0 <0.000253>
>> 15:13:31.655137 (+ 0.000717) write(2, "[14/May/2018:15:13:31.654598946
>> "..., 136) = 136 <0.002427>
>> 15:13:31.658312 (+ 0.003165) clock_gettime(CLOCK_REALTIME,
>> {tv_sec=1526328811, tv_nsec=658486117}) = 0 <0.000251>
>> 15:13:31.659032 (+ 0.000682) write(4, "[14/May/2018:15:13:31.658486117
>> "..., 136) = 136 <0.000346>
>> 15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
>> 15:13:31.663230 (+ 0.003642) getpid() = 32353 <0.000045>
>> 15:13:31.663732 (+ 0.000454) socket(AF_UNIX,
>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 33 <0.000296>
>> 15:13:31.664760 (+ 0.001048) getsockopt(33, SOL_SOCKET, SO_SNDBUF,
>> [163840], [4]) = 0 <0.000108>
>> 15:13:31.665141 (+ 0.000386) setsockopt(33, SOL_SOCKET,
>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
>> <0.000051>
>> 15:13:31.665500 (+ 0.000334) setsockopt(33, SOL_SOCKET, SO_SNDBUF,
>> [8388608], 4) = 0 <0.000229>
>> 15:13:31.665973 (+ 0.000468) sendmsg(33,
>> {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
>> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
>> MSG_NOSIGNAL) = 69 <0.000600>
>> 15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
>> 15:13:31.677320 (+ 0.010146) futex(0xb31122e8, FUTEX_WAIT, 32357,
>> NULL) = ?
>> 15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV (core dumped) +++
>>
>> On Mon, May 14, 2018 at 10:27 AM, thierry bordaz <tbordaz(a)redhat.com>
>> wrote:
>>
>>> Hi Jonathan,
>>>
>>> This is weird as the crashing thread stack looks truncated (did you
>>> copy/paste all of it ?)
>>>
>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>>> #2 0x00000000 in None ()
>>>
>>> Did you install 389-ds-base-debuginfo ?
>>> How did you get that backtrace ? from a core dumped, pstack ? Can you
>>> attach a debugger before the crash occurs ?
>>>
>>> It looks it crashed soon at startup, could it be related to a broken
>>> dse.ldif. It should exists a dse.ldif.OK, is it possibly to try to start
>>> with it ?
>>>
>>> best regards
>>> thierry
>>>
>>> On 05/12/2018 01:22 AM, Jonathan Vaughn via FreeIPA-users wrote:
>>>
>>> Not sure if it makes a difference... I was looking into this again and
>>> realized I had a bunch of messages from gdb telling me to install more
>>> debuginfo. I've done that now, here it is again freshly run through gdb
>>>
>>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>>> Copyright (C) 2017 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later <
>>> http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law. Type "show
>>> copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "armv7hl-redhat-linux-gnueabi".
>>> Type "show configuration" for configuration details.
>>> For bug reporting instructions, please see:
>>> <http://www.gnu.org/software/gdb/bugs/>;.
>>> Find the GDB manual and other documentation resources online at:
>>> <http://www.gnu.org/software/gdb/documentation/>;.
>>> For help, type "help".
>>> Type "apropos word" to search for commands related to
"word"...
>>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
>>> /usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>>> done.
>>> ...
>>>
>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>>> #2 0x00000000 in None ()
>>>
>>>
>>>
>>> On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden <rcritten(a)redhat.com>
>>>> wrote:
>>>>
>>>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>>>
>>>>>> Still trying to figure this out. It looks like slapd is dying, I
>>>>>> thought it was still running for some reason.
>>>>>>
>>>>>> slapd is dying to segfault. strace of it happening doesn't
seem to
>>>>>> reveal much:
>>>>>>
>>>>>
>>>>> A stack trace would very much help trying to track down the cause.
>>>>>
>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>>>>> ebugging-crashes
>>>>>
>>>>> rob
>>>>>
>>>>>
>>>>>> 18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
>>>>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
>>>>>> 18:32:41.544907 (+ 0.001195) getpid() = 16014
>>>>>> 18:32:41.545269 (+ 0.000329) fstat64(32,
{st_mode=S_IFREG|0644,
>>>>>> st_size=0, ...}) = 0
>>>>>> 18:32:41.545799 (+ 0.000536) write(32, "16014\n",
6) = 6
>>>>>> 18:32:41.546603 (+ 0.000818) close(32) = 0
>>>>>> 18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>>> 0644) = 0
>>>>>> 18:32:41.547741 (+ 0.000676) clock_gettime(CLOCK_REALTIME,
>>>>>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>>>>>> 18:32:41.548324 (+ 0.000587) write(2,
>>>>>> "[07/May/2018:18:32:41.548030641 "..., 134) = 134
>>>>>> 18:32:41.551096 (+ 0.002840) clock_gettime(CLOCK_REALTIME,
>>>>>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>>>>>> 18:32:41.551568 (+ 0.000406) write(4,
>>>>>> "[07/May/2018:18:32:41.551287555 "..., 134) = 134
>>>>>> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
>>>>>> 18:32:41.558499 (+ 0.006170) clock_gettime(CLOCK_REALTIME,
>>>>>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>>>>>> 18:32:41.558901 (+ 0.000350) write(2,
>>>>>> "[07/May/2018:18:32:41.558678099 "..., 136) = 136
>>>>>> 18:32:41.561537 (+ 0.002680) clock_gettime(CLOCK_REALTIME,
>>>>>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>>>>>> 18:32:41.562357 (+ 0.000793) write(4,
>>>>>> "[07/May/2018:18:32:41.561718659 "..., 136) = 136
>>>>>> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
>>>>>> 18:32:41.566928 (+ 0.003452) getpid() = 16014
>>>>>> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
>>>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>>>>> 18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET,
>>>>>> SO_SNDBUF, [163840], [4]) = 0
>>>>>> 18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET,
>>>>>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not
permitted)
>>>>>> 18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET,
>>>>>> SO_SNDBUF, [8388608], 4) = 0
>>>>>> 18:32:41.570064 (+ 0.000513) sendmsg(32,
>>>>>> {msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"},
>>>>>> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd
started:
>>>>>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0,
msg_flags=0},
>>>>>> MSG_NOSIGNAL) = 69
>>>>>> 18:32:41.570845 (+ 0.000789) close(32) = 0
>>>>>> 18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8, FUTEX_WAIT,
>>>>>> 16016, NULL) = ?
>>>>>> 18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
>>>>>>
>>>>>>
>>>
>>
>>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>
>
>
2:55 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
The installed version of 389* is 1.3.7.10-1.fc27 for armv7hl, which
appears to be the latest available version.
Perhaps something is off with the inttypes on Raspberry. Are you
building this yourself on Raspberry? Can we make code changes and
compile/install them?
Before we do that though, in gdb can you run these commands in the same
gdb frame:
(gdb) p *agmt->replarea
(gdb) p *agmt->rdn
Then do:
# grep -r PRId64 /usr/include/*
# grep -r PRIu16 /usr/include/*
So if you can compile the source, then change this line in
ldap/servers/plugins/replication/repl5_agmt.c:3036, but don't do this
yet until you get me the info I just requested.
From:
agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%" PRId64
";%" PRIu16 ";%s", slapi_sdn_get_dn(agmt->replarea),
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
agmt->port,
agmt->consumerRID, maxcsn);
To:
agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
slapi_sdn_get_dn(agmt->replarea),
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
(long)agmt->port,
(int)agmt->consumerRID, maxcsn);
Thanks,
Mark
On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
<jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
(gdb) up
#1 0xb6926b40 in cvt_s (flags=0, prec=<optimized out>, width=0,
str=0x4 <error: Cannot access memory at address 0x4>,
ss=<optimized out>)
at ../../.././nspr/pr/src/io/prprf.c:374
374 slen = strlen(str);
(gdb) up
#2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2 "",
fmt@entry=0xb34da770 <agmt_set> "\360\317p\002", ap=...) at
../../.././nspr/pr/src/io/prprf.c:1018
1018 rv = cvt_s(ss, u.s, width, prec, flags);
(gdb) up
#3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770
<agmt_set> "\360\317p\002", ap=..., ap@entry=...) at
../../.././nspr/pr/src/io/prprf.c:1184
1184 rv = dosprintf(&ss, fmt, ap);
(gdb) up
#4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0
"%s;%s;%s;%ld;%d;%s") at ldap/servers/slapd/ch_malloc.c:331
331 p = PR_vsmprintf(fmt, ap);
(gdb) up
#5 0xb34625a4 in agmt_update_maxcsn (r=r@entry=0x2737810,
sdn=0x38d9a40, sdn@entry=0x10, op=<optimized out>, mods=0x10,
mods@entry=0x27d0100, csn=csn@entry=0x38de350)
at ldap/servers/plugins/replication/repl5_agmt.c:3036
3036 agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
slapi_sdn_get_dn(agmt->replarea),
(gdb) p *agmt
$1 = {hostname = 0x2757be0 "ipa-12.company.internal", port = 389,
transport_flags = 0, binddn = 0x26ed650 "", creds = 0x26ed7a0,
bindmethod = 3, replarea = 0x2757480,
frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
frac_attr_total_defined = 1, schedule = 0x22dd0c0, auto_initialize
= 502, dn = 0x2756d00, rdn = 0x2756c20,
long_name = 0x22dd100 "agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)", protocol = 0x2220930, changecounters = 0x20cb180,
num_changecounters = 0,
max_changecounters = 256, last_update_start_time = 1526499214,
last_update_end_time = 1526499214,
last_update_status = "Error (0) Replica acquired successfully:
Incremental update succeeded", '\000' <repeats 954 times>,
update_in_progress = 0, is_enabled = 1,
last_init_start_time = 0, last_init_end_time = 0,
last_init_status = '\000' <repeats 1023 times>, lock = 0x2741740,
consumerRUV = 0x2772e50,
consumerSchemaCSN = 0x39da520, consumerRID = 4, tmpConsumerRID =
0, timeout = 120, stop_in_progress = 0, busywaittime = 0,
pausetime = 0, priv = 0x0,
attrs_to_strip = 0x2757ba0, agreement_type = 0, protocol_timeout
= 0x26ed5f0, maxcsn = 0x0, flowControlWindow = 1000,
flowControlPause = 2000, ignoreMissingChange = 0,
attr_lock = 0x2757c20, WaitForAsyncResults = 100}
On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
<mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>> wrote:
This looks really familiar and I thought it was fixed. It
should have been fixed in 1.3.7.10-1
(https://pagure.io/389-ds-base/issue/49618
<https://pagure.io/389-ds-base/issue/49618>). In your debug
session go "up" into agmt_maxcsn_update() and do:
(gdb) p *agmt
Then send us that output please.
Thanks,
Mark
On 05/15/2018 05:29 PM, Jonathan Vaughn via FreeIPA-users wrote:
> Here is a backtrace from live gdb after the segfault. Looks
> like things went wrong somewhere during in the replication
> code ?
>
> Thread 36 "ns-slapd" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x9e0bc280 (LWP 4662)]
> strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
> 142 ldrd data1a, data1b, [src]
> (gdb) bt
> #0 0xb6728f2e in strlen () at
> ../sysdeps/arm/armv6t2/strlen.S:142
> #1 0xb6973b40 in cvt_s (flags=0, prec=<optimized out>,
> width=0, str=0x4 <error: Cannot access memory at address
> 0x4>, ss=<optimized out>)
> at ../../.././nspr/pr/src/io/prprf.c:374
> #2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc,
> fmt=0xb34fddf2 "", fmt@entry=0xb3527770 <agmt_set>
> "\360\357\305\002", ap=...)
> at ../../.././nspr/pr/src/io/prprf.c:1018
> #3 0xb6973c8c in PR_vsmprintf (fmt=fmt@entry=0xb3527770
> <agmt_set> "\360\357\305\002", ap=..., ap@entry=...) at
> ../../.././nspr/pr/src/io/prprf.c:1184
> #4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0
> "%s;%s;%s;%ld;%d;%s") at ldap/servers/slapd/ch_malloc.c:331
> #5 0xb34af5a4 in agmt_update_maxcsn (r=r@entry=0x2c89810,
> sdn=0x3e2bac0, sdn@entry=0x10, op=<optimized out>, mods=0x10,
> mods@entry=0x3eec100, csn=csn@entry=0x3e30220)
> at ldap/servers/plugins/replication/repl5_agmt.c:3036
> #6 0xb34bd424 in write_changelog_and_ruv
> (pb=pb@entry=0x2cb54a0) at
> ldap/servers/plugins/replication/repl5_plugins.c:1124
> #7 0xb34be7ec in multimaster_be_betxnpostop_add
> (pb=0x2cb54a0) at
> ldap/servers/plugins/replication/repl5_plugins.c:855
> #8 0xb34be880 in multimaster_mmr_postop (pb=<optimized out>,
> flags=560) at
> ldap/servers/plugins/replication/repl5_plugins.c:616
> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop
> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0, flags=flags@entry=560)
> at ldap/servers/slapd/plugin_mmr.c:65
> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
> #11 0xb6e2ce4c in op_shared_add (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:679
> #12 0xb6e2d35c in add_internal_pb (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:407
> #13 0xb6e2e0f8 in slapi_add_internal_pb
> (pb=pb@entry=0x2cb54a0) at ldap/servers/slapd/add.c:332
> #14 0xb368db50 in ipa_topo_util_segment_write
> (tconf=tconf@entry=0x2cb5860, tsegm=tsegm@entry=0x3f2d9a0) at
> topology_util.c:1251
> #15 0xb368e01c in ipa_topo_util_update_agmt_list
> (conf=0x2cb5860, repl_segments=<optimized out>) at
> topology_util.c:696
> #16 0xb368891c in ipa_topo_apply_shared_config () at
> topology_init.c:165
> #17 0xb6e4e65c in eq_call_all () at
> ldap/servers/slapd/eventq.c:278
> #18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
> ldap/servers/slapd/eventq.c:323
> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
> ../../.././nspr/pr/src/pthreads/ptthread.c:201
> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at
> pthread_create.c:465
> #21 0xb6785da8 in None () at
> ../sysdeps/unix/sysv/linux/arm/clone.S:73
>
>
> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz
> <tbordaz(a)redhat.com <mailto:tbordaz@redhat.com>> wrote:
>
> Hi Jonathan,
>
> This problem looks new to me and has something specific
> to your environment.
> I think the best approach is to continue to debug on your
> system if you have the possibility to do so.
>
> From strace we can see that DS started smoothly (created
> its pid file then notified systemd it was running fine).
> According to the pstack nunc-stans was running and was
> able to accept network events even if it appears it
> detected no incoming connection.
> So the server should be ready to serve for some seconds
> (more than a minute), then it crashed with one thread
> dereferencing likely a wrong pointer.
>
> Could you attach a debugger when the server is started
> and wait for the sigsegv to occur. Then confirm the
> crashing thread backstack.
> If it is confirmed, I am afraid this is a stack
> corruption and valgrind could help
>
(http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...
>
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...>).
>
> best regards
> thierry
>
>
> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
>> Here's a strace from before it dies. Most of the elapsed
>> time is it waiting on some futex call it looks like near
>> the end, when it finally "returns" (from lack of strace
>> output for duration of call I assume it didn't actually
>> return but SIGSEGV in that call) and strace prints ' =
>> ?' on the futex it then immediately reports SIGSEGV
>> after. So maybe the problem is that futex call, which
>> may mean the problem is not directly in 389DS / FreeIPA
>> itself?
>>
>>
>>
>> 15:13:31.626587 (+ 0.000630) listen(8, 128) = 0
>> <0.000068>
>> 15:13:31.626857 (+ 0.000235) listen(9, 128) = 0
>> <0.000048>
>> 15:13:31.627111 (+ 0.000251)
>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>> tv_nsec=41120614}) = 0 <0.000085>
>> 15:13:31.627457 (+ 0.000356)
>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>> tv_nsec=627560772}) = 0 <0.000043>
>> 15:13:31.631233 (+ 0.003839)
>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>> tv_nsec=45286796}) = 0 <0.000077>
>> 15:13:31.631720 (+ 0.000427)
>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>> tv_nsec=45661430}) = 0 <0.000042>
>> 15:13:31.631955 (+ 0.000220)
>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>> tv_nsec=632049036}) = 0 <0.000047>
>> 15:13:31.635669 (+ 0.003785)
>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>> tv_nsec=49725840}) = 0 <0.000146>
>> 15:13:31.636484 (+ 0.000784) write(16, "a", 1) = 1
>> <0.000118>
>> 15:13:31.636855 (+ 0.000341) sched_yield() = 0
>> <0.000252>
>> 15:13:31.637322 (+ 0.000470) futex(0x1cb57a0,
>> FUTEX_WAKE_PRIVATE, 1) = 1 <0.000088>
>> 15:13:31.637897 (+ 0.000610) write(16, "a", 1) = 1
>> <0.000221>
>> 15:13:31.638394 (+ 0.000467) sched_yield() = 0
>> <0.000047>
>> 15:13:31.638619 (+ 0.000202) futex(0x1cb5710,
>> FUTEX_WAKE_PRIVATE, 1) = 1 <0.000065>
>> 15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33 <0.000831>
>> 15:13:31.640260 (+ 0.001387) getpid() = 32353 <0.000077>
>> 15:13:31.640558 (+ 0.000256) fstat64(33,
>> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000119>
>> 15:13:31.641106 (+ 0.000556) write(33, "32353\n", 6)
>> = 6 <0.000127>
>> 15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
>> 15:13:31.642216 (+ 0.000758)
>> chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>> 0644) = 0 <0.000152>
>> 15:13:31.642900 (+ 0.000679)
>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>> tv_nsec=643020294}) = 0 <0.000056>
>> 15:13:31.643495 (+ 0.000590) write(2,
>> "[14/May/2018:15:13:31.643020294 "..., 134) = 134
<0.002697>
>> 15:13:31.646515 (+ 0.003052)
>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>> tv_nsec=646694394}) = 0 <0.000075>
>> 15:13:31.646892 (+ 0.000337) write(4,
>> "[14/May/2018:15:13:31.646694394 "..., 134) = 134
<0.000522>
>> 15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
>> 15:13:31.654425 (+ 0.006617)
>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>> tv_nsec=654598946}) = 0 <0.000253>
>> 15:13:31.655137 (+ 0.000717) write(2,
>> "[14/May/2018:15:13:31.654598946 "..., 136) = 136
<0.002427>
>> 15:13:31.658312 (+ 0.003165)
>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>> tv_nsec=658486117}) = 0 <0.000251>
>> 15:13:31.659032 (+ 0.000682) write(4,
>> "[14/May/2018:15:13:31.658486117 "..., 136) = 136
<0.000346>
>> 15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
>> 15:13:31.663230 (+ 0.003642) getpid() = 32353 <0.000045>
>> 15:13:31.663732 (+ 0.000454) socket(AF_UNIX,
>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 33 <0.000296>
>> 15:13:31.664760 (+ 0.001048) getsockopt(33,
>> SOL_SOCKET, SO_SNDBUF, [163840], [4]) = 0 <0.000108>
>> 15:13:31.665141 (+ 0.000386) setsockopt(33,
>> SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM
>> (Operation not permitted) <0.000051>
>> 15:13:31.665500 (+ 0.000334) setsockopt(33,
>> SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0 <0.000229>
>> 15:13:31.665973 (+ 0.000468) sendmsg(33,
>> {msg_name={sa_family=AF_UNIX,
>> sun_path="/run/systemd/notify"}, msg_namelen=21,
>> msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0,
>> msg_flags=0}, MSG_NOSIGNAL) = 69 <0.000600>
>> 15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
>> 15:13:31.677320 (+ 0.010146) futex(0xb31122e8,
>> FUTEX_WAIT, 32357, NULL) = ?
>> 15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV
>> (core dumped) +++
>>
>> On Mon, May 14, 2018 at 10:27 AM, thierry bordaz
>> <tbordaz(a)redhat.com <mailto:tbordaz@redhat.com>> wrote:
>>
>> Hi Jonathan,
>>
>> This is weird as the crashing thread stack looks
>> truncated (did you copy/paste all of it ?)
>>
>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>> #2 0x00000000 in None ()
>>
>> Did you install 389-ds-base-debuginfo ?
>> How did you get that backtrace ? from a core dumped,
>> pstack ? Can you attach a debugger before the crash
>> occurs ?
>>
>> It looks it crashed soon at startup, could it be
>> related to a broken dse.ldif. It should exists a
>> dse.ldif.OK, is it possibly to try to start with it ?
>>
>> best regards
>> thierry
>>
>> On 05/12/2018 01:22 AM, Jonathan Vaughn via
>> FreeIPA-users wrote:
>>> Not sure if it makes a difference... I was looking
>>> into this again and realized I had a bunch of
>>> messages from gdb telling me to install more
>>> debuginfo. I've done that now, here it is again
>>> freshly run through gdb
>>>
>>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>>> Copyright (C) 2017 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later
>>> <http://gnu.org/licenses/gpl.html
>>> <http://gnu.org/licenses/gpl.html>>
>>> This is free software: you are free to change and
>>> redistribute it.
>>> There is NO WARRANTY, to the extent permitted by
>>> law. Type "show copying"
>>> and "show warranty" for details.
>>> This GDB was configured as
>>> "armv7hl-redhat-linux-gnueabi".
>>> Type "show configuration" for configuration
details.
>>> For bug reporting instructions, please see:
>>> <http://www.gnu.org/software/gdb/bugs/
>>> <http://www.gnu.org/software/gdb/bugs/>>;.
>>> Find the GDB manual and other documentation
>>> resources online at:
>>> <http://www.gnu.org/software/gdb/documentation/
>>> <http://www.gnu.org/software/gdb/documentation/>>;.
>>> For help, type "help".
>>> Type "apropos word" to search for commands related
>>> to "word"...
>>> Reading symbols from /usr/sbin/ns-slapd...Reading
>>> symbols from
>>>
/usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>>> done.
>>> ...
>>>
>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>>> #2 0x00000000 in None ()
>>>
>>>
>>>
>>> On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden
>>> <rcritten(a)redhat.com
>>> <mailto:rcritten@redhat.com>> wrote:
>>>
>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>
>>> Still trying to figure this out. It
>>> looks like slapd is dying, I thought it
>>> was still running for some reason.
>>>
>>> slapd is dying to segfault. strace of
>>> it happening doesn't seem to reveal much:
>>>
>>>
>>> A stack trace would very much help trying
>>> to track down the cause.
>>>
>>>
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
>>>
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-cras...
>>>
>>> rob
>>>
>>>
>>> 18:32:41.543717 (+ 0.000801)
>>> openat(AT_FDCWD,
>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
>>> 18:32:41.544907 (+ 0.001195)
>>> getpid() = 16014
>>> 18:32:41.545269 (+ 0.000329)
>>> fstat64(32, {st_mode=S_IFREG|0644,
>>> st_size=0, ...}) = 0
>>> 18:32:41.545799 (+ 0.000536)
>>> write(32, "16014\n", 6) = 6
>>> 18:32:41.546603 (+ 0.000818)
>>> close(32) = 0
>>> 18:32:41.547061 (+ 0.000448)
>>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> 0644) = 0
>>> 18:32:41.547741 (+ 0.000676)
>>> clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>>> 18:32:41.548324 (+ 0.000587)
>>> write(2,
>>> "[07/May/2018:18:32:41.548030641 "...,
>>> 134) = 134
>>> 18:32:41.551096 (+ 0.002840)
>>> clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>>> 18:32:41.551568 (+ 0.000406)
>>> write(4,
>>> "[07/May/2018:18:32:41.551287555 "...,
>>> 134) = 134
>>> 18:32:41.552360 (+ 0.000811)
>>> fsync(4) = 0
>>> 18:32:41.558499 (+ 0.006170)
>>> clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>>> 18:32:41.558901 (+ 0.000350)
>>> write(2,
>>> "[07/May/2018:18:32:41.558678099 "...,
>>> 136) = 136
>>> 18:32:41.561537 (+ 0.002680)
>>> clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>>> 18:32:41.562357 (+ 0.000793)
>>> write(4,
>>> "[07/May/2018:18:32:41.561718659 "...,
>>> 136) = 136
>>> 18:32:41.563293 (+ 0.001148)
>>> fsync(4) = 0
>>> 18:32:41.566928 (+ 0.003452)
>>> getpid() = 16014
>>> 18:32:41.567712 (+ 0.000752)
>>> socket(AF_UNIX,
>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>> 18:32:41.568628 (+ 0.000912)
>>> getsockopt(32, SOL_SOCKET, SO_SNDBUF,
>>> [163840], [4]) = 0
>>> 18:32:41.568972 (+ 0.000319)
>>> setsockopt(32, SOL_SOCKET,
>>> SO_SNDBUFFORCE, [8388608], 4) = -1
>>> EPERM (Operation not permitted)
>>> 18:32:41.569548 (+ 0.000589)
>>> setsockopt(32, SOL_SOCKET, SO_SNDBUF,
>>> [8388608], 4) = 0
>>> 18:32:41.570064 (+ 0.000513)
>>> sendmsg(32,
>>> {msg_name={sa_family=AF_UNIX,
>>> sun_path="/run/systemd/notify"},
>>> msg_namelen=21,
>>> msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>> started: Re"..., iov_len=69}],
>>> msg_iovlen=1, msg_controllen=0,
>>> msg_flags=0}, MSG_NOSIGNAL) = 69
>>> 18:32:41.570845 (+ 0.000789)
>>> close(32) = 0
>>> 18:32:41.576358 (+ 0.005575)
>>> futex(0xb30ed2e8, FUTEX_WAIT, 16016,
>>> NULL) = ?
>>> 18:33:01.730774 (+ 20.154428) +++
>>> killed by SIGSEGV +++
>>>
>>
>>
>
>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> <mailto:freeipa-users-leave@lists.fedorahosted.org>
3:03 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
I've been just using the packages from Fedora. I can build it potentially
but I don't have a cross build environment set up at the moment. From
experience I'd want to do that first because building anything on the Pi
usually takes ages.
I'd been "redacting" the hostnames but I'll stop bothering since it
looks
like we're getting far enough into the weeds now that the difference in
string lengths after "redacting" might actually be a red herring.
(gdb) p *agmt
$1 = {hostname = 0x1ef9be0 "ipa-12.creatuity.internal", port = 389,
transport_flags = 0, binddn = 0x1e8f650 "", creds = 0x1e8f7a0, bindmethod =
3, replarea = 0x1ef9480,
frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
frac_attr_total_defined = 1, schedule = 0x1a7f0c0, auto_initialize = 502,
dn = 0x1ef8d00, rdn = 0x1ef8c20,
long_name = 0x1a7f100 "agmt=\"cn=meToipa-12.creatuity.internal\"
(ipa-12:5)", protocol = 0x19c2930, changecounters = 0x186d180,
num_changecounters = 0,
max_changecounters = 256, last_update_start_time = 1526500697,
last_update_end_time = 1526500697,
last_update_status = "Error (0) Replica acquired successfully:
Incremental update succeeded", '\000' <repeats 954 times>,
update_in_progress = 0, is_enabled = 1,
last_init_start_time = 0, last_init_end_time = 0, last_init_status =
'\000' <repeats 1023 times>, lock = 0x1ee3740, consumerRUV = 0x1f14e50,
consumerSchemaCSN = 0x317c520, consumerRID = 4, tmpConsumerRID = 0,
timeout = 120, stop_in_progress = 0, busywaittime = 0, pausetime = 0, priv
= 0x0,
attrs_to_strip = 0x1ef9ba0, agreement_type = 0, protocol_timeout =
0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000, flowControlPause = 2000,
ignoreMissingChange = 0,
attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
(gdb) p *agmt->replarea
$2 = {flag = 15 '\017', udn = 0x1efce80 "dc=creatuity,dc=internal", dn
=
0x1ef9460 "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
"dc=creatuity,dc=internal", ndn_len = 24}
(gdb) p *agmt->rdn
$3 = {flag = 0 '\000', rdn = 0x19c2840
"cn=meToipa-12.creatuity.internal",
rdns = 0x0, butcheredupto = -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
[root@ipa-11 ~]# grep -r PRId64 /usr/include/*
/usr/include/inttypes.h:# define PRId64 __PRI64_PREFIX "d"
[root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
/usr/include/inttypes.h:# define PRIu16 "u"
On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds <mreynolds(a)redhat.com> wrote:
On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
The installed version of 389* is 1.3.7.10-1.fc27 for armv7hl, which
appears to be the latest available version.
Perhaps something is off with the inttypes on Raspberry. Are you building
this yourself on Raspberry? Can we make code changes and compile/install
them?
Before we do that though, in gdb can you run these commands in the same
gdb frame:
(gdb) p *agmt->replarea
(gdb) p *agmt->rdn
Then do:
# grep -r PRId64 /usr/include/*
# grep -r PRIu16 /usr/include/*
So if you can compile the source, then change this line in
ldap/servers/plugins/replication/repl5_agmt.c:3036, but don't do this yet
until you get me the info I just requested.
From:
agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%" PRId64
";%"
PRIu16 ";%s", slapi_sdn_get_dn(agmt->replarea),
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
agmt->port,
agmt->consumerRID, maxcsn);
To:
agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
slapi_sdn_get_dn(agmt->replarea),
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
(long)agmt->port,
(int)agmt->consumerRID, maxcsn);
Thanks,
Mark
On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
> (gdb) up
> #1 0xb6926b40 in cvt_s (flags=0, prec=<optimized out>, width=0, str=0x4
> <error: Cannot access memory at address 0x4>, ss=<optimized out>)
> at ../../.././nspr/pr/src/io/prprf.c:374
> 374 slen = strlen(str);
> (gdb) up
> #2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2 "",
fmt@entry=0xb34da770
> <agmt_set> "\360\317p\002", ap=...) at
../../.././nspr/pr/src/io/prpr
> f.c:1018
> 1018 rv = cvt_s(ss, u.s, width, prec, flags);
> (gdb) up
> #3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770 <agmt_set>
> "\360\317p\002", ap=..., ap@entry=...) at ../../.././nspr/pr/src/io/prpr
> f.c:1184
> 1184 rv = dosprintf(&ss, fmt, ap);
> (gdb) up
> #4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0 "%s;%s;%s;%ld;%d;%s")
> at ldap/servers/slapd/ch_malloc.c:331
> 331 p = PR_vsmprintf(fmt, ap);
> (gdb) up
> #5 0xb34625a4 in agmt_update_maxcsn (r=r@entry=0x2737810,
> sdn=0x38d9a40, sdn@entry=0x10, op=<optimized out>, mods=0x10,
mods@entry=0x27d0100,
> csn=csn@entry=0x38de350)
> at ldap/servers/plugins/replication/repl5_agmt.c:3036
> 3036 agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> slapi_sdn_get_dn(agmt->replarea),
> (gdb) p *agmt
> $1 = {hostname = 0x2757be0 "ipa-12.company.internal", port = 389,
> transport_flags = 0, binddn = 0x26ed650 "", creds = 0x26ed7a0, bindmethod
=
> 3, replarea = 0x2757480,
> frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
> frac_attr_total_defined = 1, schedule = 0x22dd0c0, auto_initialize = 502,
> dn = 0x2756d00, rdn = 0x2756c20,
> long_name = 0x22dd100 "agmt=\"cn=meToipa-12.company.internal\"
> (ipa-12:5)", protocol = 0x2220930, changecounters = 0x20cb180,
> num_changecounters = 0,
> max_changecounters = 256, last_update_start_time = 1526499214,
> last_update_end_time = 1526499214,
> last_update_status = "Error (0) Replica acquired successfully:
> Incremental update succeeded", '\000' <repeats 954 times>,
> update_in_progress = 0, is_enabled = 1,
> last_init_start_time = 0, last_init_end_time = 0, last_init_status =
> '\000' <repeats 1023 times>, lock = 0x2741740, consumerRUV =
0x2772e50,
> consumerSchemaCSN = 0x39da520, consumerRID = 4, tmpConsumerRID = 0,
> timeout = 120, stop_in_progress = 0, busywaittime = 0, pausetime = 0, priv
> = 0x0,
> attrs_to_strip = 0x2757ba0, agreement_type = 0, protocol_timeout =
> 0x26ed5f0, maxcsn = 0x0, flowControlWindow = 1000, flowControlPause = 2000,
> ignoreMissingChange = 0,
> attr_lock = 0x2757c20, WaitForAsyncResults = 100}
>
>
> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds <mreynolds(a)redhat.com>
> wrote:
>
>> This looks really familiar and I thought it was fixed. It should have
>> been fixed in 1.3.7.10-1 (https://pagure.io/389-ds-base/issue/49618).
>> In your debug session go "up" into agmt_maxcsn_update() and do:
>>
>> (gdb) p *agmt
>>
>> Then send us that output please.
>>
>> Thanks,
>> Mark
>>
>>
>> On 05/15/2018 05:29 PM, Jonathan Vaughn via FreeIPA-users wrote:
>>
>> Here is a backtrace from live gdb after the segfault. Looks like things
>> went wrong somewhere during in the replication code ?
>>
>> Thread 36 "ns-slapd" received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x9e0bc280 (LWP 4662)]
>> strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
>> 142 ldrd data1a, data1b, [src]
>> (gdb) bt
>> #0 0xb6728f2e in strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
>> #1 0xb6973b40 in cvt_s (flags=0, prec=<optimized out>, width=0, str=0x4
>> <error: Cannot access memory at address 0x4>, ss=<optimized out>)
>> at ../../.././nspr/pr/src/io/prprf.c:374
>> #2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc, fmt=0xb34fddf2
"",
>> fmt@entry=0xb3527770 <agmt_set> "\360\357\305\002", ap=...)
>> at ../../.././nspr/pr/src/io/prprf.c:1018
>> #3 0xb6973c8c in PR_vsmprintf (fmt=fmt@entry=0xb3527770 <agmt_set>
>> "\360\357\305\002", ap=..., ap@entry=...) at
>> ../../.././nspr/pr/src/io/prprf.c:1184
>> #4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0
>> "%s;%s;%s;%ld;%d;%s") at ldap/servers/slapd/ch_malloc.c:331
>> #5 0xb34af5a4 in agmt_update_maxcsn (r=r@entry=0x2c89810,
>> sdn=0x3e2bac0, sdn@entry=0x10, op=<optimized out>, mods=0x10,
mods@entry=0x3eec100,
>> csn=csn@entry=0x3e30220)
>> at ldap/servers/plugins/replication/repl5_agmt.c:3036
>> #6 0xb34bd424 in write_changelog_and_ruv (pb=pb@entry=0x2cb54a0) at
>> ldap/servers/plugins/replication/repl5_plugins.c:1124
>> #7 0xb34be7ec in multimaster_be_betxnpostop_add (pb=0x2cb54a0) at
>> ldap/servers/plugins/replication/repl5_plugins.c:855
>> #8 0xb34be880 in multimaster_mmr_postop (pb=<optimized out>, flags=560)
>> at ldap/servers/plugins/replication/repl5_plugins.c:616
>> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop (pb=pb@entry=0x2cb54a0,
>> e=e@entry=0x0, flags=flags@entry=560) at ldap/servers/slapd/plugin_mmr.
>> c:65
>> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
>> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>> #11 0xb6e2ce4c in op_shared_add (pb=pb@entry=0x2cb54a0) at
>> ldap/servers/slapd/add.c:679
>> #12 0xb6e2d35c in add_internal_pb (pb=pb@entry=0x2cb54a0) at
>> ldap/servers/slapd/add.c:407
>> #13 0xb6e2e0f8 in slapi_add_internal_pb (pb=pb@entry=0x2cb54a0) at
>> ldap/servers/slapd/add.c:332
>> #14 0xb368db50 in ipa_topo_util_segment_write (tconf=tconf@entry=0x2cb5860,
>> tsegm=tsegm@entry=0x3f2d9a0) at topology_util.c:1251
>> #15 0xb368e01c in ipa_topo_util_update_agmt_list (conf=0x2cb5860,
>> repl_segments=<optimized out>) at topology_util.c:696
>> #16 0xb368891c in ipa_topo_apply_shared_config () at topology_init.c:165
>> #17 0xb6e4e65c in eq_call_all () at ldap/servers/slapd/eventq.c:278
>> #18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
>> ldap/servers/slapd/eventq.c:323
>> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
>> ../../.././nspr/pr/src/pthreads/ptthread.c:201
>> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at pthread_create.c:465
>> #21 0xb6785da8 in None () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>>
>>
>> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz <tbordaz(a)redhat.com>
>> wrote:
>>
>>> Hi Jonathan,
>>>
>>> This problem looks new to me and has something specific to your
>>> environment.
>>> I think the best approach is to continue to debug on your system if you
>>> have the possibility to do so.
>>>
>>> From strace we can see that DS started smoothly (created its pid file
>>> then notified systemd it was running fine). According to the pstack
>>> nunc-stans was running and was able to accept network events even if it
>>> appears it detected no incoming connection.
>>> So the server should be ready to serve for some seconds (more than a
>>> minute), then it crashed with one thread dereferencing likely a wrong
>>> pointer.
>>>
>>> Could you attach a debugger when the server is started and wait for the
>>> sigsegv to occur. Then confirm the crashing thread backstack.
>>> If it is confirmed, I am afraid this is a stack corruption and valgrind
>>> could help (http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-me
>>> mory-growthinvalid-access-with-valgrind).
>>>
>>> best regards
>>> thierry
>>>
>>>
>>> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
>>>
>>> Here's a strace from before it dies. Most of the elapsed time is it
>>> waiting on some futex call it looks like near the end, when it finally
>>> "returns" (from lack of strace output for duration of call I assume
it
>>> didn't actually return but SIGSEGV in that call) and strace prints '
= ?'
>>> on the futex it then immediately reports SIGSEGV after. So maybe the
>>> problem is that futex call, which may mean the problem is not directly in
>>> 389DS / FreeIPA itself?
>>>
>>>
>>>
>>> 15:13:31.626587 (+ 0.000630) listen(8, 128) = 0 <0.000068>
>>> 15:13:31.626857 (+ 0.000235) listen(9, 128) = 0 <0.000048>
>>> 15:13:31.627111 (+ 0.000251) clock_gettime(CLOCK_MONOTONIC,
>>> {tv_sec=1464932, tv_nsec=41120614}) = 0 <0.000085>
>>> 15:13:31.627457 (+ 0.000356) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1526328811, tv_nsec=627560772}) = 0 <0.000043>
>>> 15:13:31.631233 (+ 0.003839) clock_gettime(CLOCK_MONOTONIC,
>>> {tv_sec=1464932, tv_nsec=45286796}) = 0 <0.000077>
>>> 15:13:31.631720 (+ 0.000427) clock_gettime(CLOCK_MONOTONIC,
>>> {tv_sec=1464932, tv_nsec=45661430}) = 0 <0.000042>
>>> 15:13:31.631955 (+ 0.000220) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1526328811, tv_nsec=632049036}) = 0 <0.000047>
>>> 15:13:31.635669 (+ 0.003785) clock_gettime(CLOCK_MONOTONIC,
>>> {tv_sec=1464932, tv_nsec=49725840}) = 0 <0.000146>
>>> 15:13:31.636484 (+ 0.000784) write(16, "a", 1) = 1
<0.000118>
>>> 15:13:31.636855 (+ 0.000341) sched_yield() = 0 <0.000252>
>>> 15:13:31.637322 (+ 0.000470) futex(0x1cb57a0, FUTEX_WAKE_PRIVATE,
>>> 1) = 1 <0.000088>
>>> 15:13:31.637897 (+ 0.000610) write(16, "a", 1) = 1
<0.000221>
>>> 15:13:31.638394 (+ 0.000467) sched_yield() = 0 <0.000047>
>>> 15:13:31.638619 (+ 0.000202) futex(0x1cb5710, FUTEX_WAKE_PRIVATE,
>>> 1) = 1 <0.000065>
>>> 15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33 <0.000831>
>>> 15:13:31.640260 (+ 0.001387) getpid() = 32353 <0.000077>
>>> 15:13:31.640558 (+ 0.000256) fstat64(33, {st_mode=S_IFREG|0644,
>>> st_size=0, ...}) = 0 <0.000119>
>>> 15:13:31.641106 (+ 0.000556) write(33, "32353\n", 6) = 6
<0.000127>
>>> 15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
>>> 15:13:31.642216 (+ 0.000758)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> 0644) = 0 <0.000152>
>>> 15:13:31.642900 (+ 0.000679) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1526328811, tv_nsec=643020294}) = 0 <0.000056>
>>> 15:13:31.643495 (+ 0.000590) write(2,
"[14/May/2018:15:13:31.643020294
>>> "..., 134) = 134 <0.002697>
>>> 15:13:31.646515 (+ 0.003052) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1526328811, tv_nsec=646694394}) = 0 <0.000075>
>>> 15:13:31.646892 (+ 0.000337) write(4,
"[14/May/2018:15:13:31.646694394
>>> "..., 134) = 134 <0.000522>
>>> 15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
>>> 15:13:31.654425 (+ 0.006617) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1526328811, tv_nsec=654598946}) = 0 <0.000253>
>>> 15:13:31.655137 (+ 0.000717) write(2,
"[14/May/2018:15:13:31.654598946
>>> "..., 136) = 136 <0.002427>
>>> 15:13:31.658312 (+ 0.003165) clock_gettime(CLOCK_REALTIME,
>>> {tv_sec=1526328811, tv_nsec=658486117}) = 0 <0.000251>
>>> 15:13:31.659032 (+ 0.000682) write(4,
"[14/May/2018:15:13:31.658486117
>>> "..., 136) = 136 <0.000346>
>>> 15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
>>> 15:13:31.663230 (+ 0.003642) getpid() = 32353 <0.000045>
>>> 15:13:31.663732 (+ 0.000454) socket(AF_UNIX,
>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 33 <0.000296>
>>> 15:13:31.664760 (+ 0.001048) getsockopt(33, SOL_SOCKET, SO_SNDBUF,
>>> [163840], [4]) = 0 <0.000108>
>>> 15:13:31.665141 (+ 0.000386) setsockopt(33, SOL_SOCKET,
>>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
>>> <0.000051>
>>> 15:13:31.665500 (+ 0.000334) setsockopt(33, SOL_SOCKET, SO_SNDBUF,
>>> [8388608], 4) = 0 <0.000229>
>>> 15:13:31.665973 (+ 0.000468) sendmsg(33,
>>> {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
>>> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
>>> MSG_NOSIGNAL) = 69 <0.000600>
>>> 15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
>>> 15:13:31.677320 (+ 0.010146) futex(0xb31122e8, FUTEX_WAIT, 32357,
>>> NULL) = ?
>>> 15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV (core dumped) +++
>>>
>>> On Mon, May 14, 2018 at 10:27 AM, thierry bordaz <tbordaz(a)redhat.com>
>>> wrote:
>>>
>>>> Hi Jonathan,
>>>>
>>>> This is weird as the crashing thread stack looks truncated (did you
>>>> copy/paste all of it ?)
>>>>
>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>>>> #2 0x00000000 in None ()
>>>>
>>>> Did you install 389-ds-base-debuginfo ?
>>>> How did you get that backtrace ? from a core dumped, pstack ? Can you
>>>> attach a debugger before the crash occurs ?
>>>>
>>>> It looks it crashed soon at startup, could it be related to a broken
>>>> dse.ldif. It should exists a dse.ldif.OK, is it possibly to try to start
>>>> with it ?
>>>>
>>>> best regards
>>>> thierry
>>>>
>>>> On 05/12/2018 01:22 AM, Jonathan Vaughn via FreeIPA-users wrote:
>>>>
>>>> Not sure if it makes a difference... I was looking into this again and
>>>> realized I had a bunch of messages from gdb telling me to install more
>>>> debuginfo. I've done that now, here it is again freshly run through
gdb
>>>>
>>>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>>>> Copyright (C) 2017 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later <
>>>> http://gnu.org/licenses/gpl.html>
>>>> This is free software: you are free to change and redistribute it.
>>>> There is NO WARRANTY, to the extent permitted by law. Type "show
>>>> copying"
>>>> and "show warranty" for details.
>>>> This GDB was configured as "armv7hl-redhat-linux-gnueabi".
>>>> Type "show configuration" for configuration details.
>>>> For bug reporting instructions, please see:
>>>> <http://www.gnu.org/software/gdb/bugs/>;.
>>>> Find the GDB manual and other documentation resources online at:
>>>> <http://www.gnu.org/software/gdb/documentation/>;.
>>>> For help, type "help".
>>>> Type "apropos word" to search for commands related to
"word"...
>>>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
>>>> /usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>>>> done.
>>>> ...
>>>>
>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>>>> #2 0x00000000 in None ()
>>>>
>>>>
>>>>
>>>> On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden
<rcritten(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>>>>
>>>>>>> Still trying to figure this out. It looks like slapd is
dying, I
>>>>>>> thought it was still running for some reason.
>>>>>>>
>>>>>>> slapd is dying to segfault. strace of it happening
doesn't seem to
>>>>>>> reveal much:
>>>>>>>
>>>>>>
>>>>>> A stack trace would very much help trying to track down the
cause.
>>>>>>
>>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>>>>>> ebugging-crashes
>>>>>>
>>>>>> rob
>>>>>>
>>>>>>
>>>>>>> 18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
>>>>>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
>>>>>>> 18:32:41.544907 (+ 0.001195) getpid() = 16014
>>>>>>> 18:32:41.545269 (+ 0.000329) fstat64(32,
{st_mode=S_IFREG|0644,
>>>>>>> st_size=0, ...}) = 0
>>>>>>> 18:32:41.545799 (+ 0.000536) write(32,
"16014\n", 6) = 6
>>>>>>> 18:32:41.546603 (+ 0.000818) close(32) = 0
>>>>>>> 18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>>>> 0644) = 0
>>>>>>> 18:32:41.547741 (+ 0.000676)
clock_gettime(CLOCK_REALTIME,
>>>>>>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>>>>>>> 18:32:41.548324 (+ 0.000587) write(2,
>>>>>>> "[07/May/2018:18:32:41.548030641 "..., 134) = 134
>>>>>>> 18:32:41.551096 (+ 0.002840)
clock_gettime(CLOCK_REALTIME,
>>>>>>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>>>>>>> 18:32:41.551568 (+ 0.000406) write(4,
>>>>>>> "[07/May/2018:18:32:41.551287555 "..., 134) = 134
>>>>>>> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
>>>>>>> 18:32:41.558499 (+ 0.006170)
clock_gettime(CLOCK_REALTIME,
>>>>>>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>>>>>>> 18:32:41.558901 (+ 0.000350) write(2,
>>>>>>> "[07/May/2018:18:32:41.558678099 "..., 136) = 136
>>>>>>> 18:32:41.561537 (+ 0.002680)
clock_gettime(CLOCK_REALTIME,
>>>>>>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>>>>>>> 18:32:41.562357 (+ 0.000793) write(4,
>>>>>>> "[07/May/2018:18:32:41.561718659 "..., 136) = 136
>>>>>>> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
>>>>>>> 18:32:41.566928 (+ 0.003452) getpid() = 16014
>>>>>>> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
>>>>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>>>>>> 18:32:41.568628 (+ 0.000912) getsockopt(32, SOL_SOCKET,
>>>>>>> SO_SNDBUF, [163840], [4]) = 0
>>>>>>> 18:32:41.568972 (+ 0.000319) setsockopt(32, SOL_SOCKET,
>>>>>>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not
permitted)
>>>>>>> 18:32:41.569548 (+ 0.000589) setsockopt(32, SOL_SOCKET,
>>>>>>> SO_SNDBUF, [8388608], 4) = 0
>>>>>>> 18:32:41.570064 (+ 0.000513) sendmsg(32,
>>>>>>> {msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"},
>>>>>>> msg_namelen=21,
msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>>>>>>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0,
msg_flags=0},
>>>>>>> MSG_NOSIGNAL) = 69
>>>>>>> 18:32:41.570845 (+ 0.000789) close(32) = 0
>>>>>>> 18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8,
FUTEX_WAIT,
>>>>>>> 16016, NULL) = ?
>>>>>>> 18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV +++
>>>>>>>
>>>>>>>
>>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>
>>
>>
>
5:07 a.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
I've been just using the packages from Fedora. I can build it
potentially but I don't have a cross build environment set up at the
moment. From experience I'd want to do that first because building
anything on the Pi usually takes ages.
I'd been "redacting" the hostnames but I'll stop bothering since it
looks like we're getting far enough into the weeds now that the
difference in string lengths after "redacting" might actually be a red
herring.
(gdb) p *agmt
$1 = {hostname = 0x1ef9be0 "ipa-12.creatuity.internal", port = 389,
transport_flags = 0, binddn = 0x1e8f650 "", creds = 0x1e8f7a0,
bindmethod = 3, replarea = 0x1ef9480,
frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
frac_attr_total_defined = 1, schedule = 0x1a7f0c0, auto_initialize =
502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
long_name = 0x1a7f100 "agmt=\"cn=meToipa-12.creatuity.internal\"
(ipa-12:5)", protocol = 0x19c2930, changecounters = 0x186d180,
num_changecounters = 0,
max_changecounters = 256, last_update_start_time = 1526500697,
last_update_end_time = 1526500697,
last_update_status = "Error (0) Replica acquired successfully:
Incremental update succeeded", '\000' <repeats 954 times>,
update_in_progress = 0, is_enabled = 1,
last_init_start_time = 0, last_init_end_time = 0, last_init_status =
'\000' <repeats 1023 times>, lock = 0x1ee3740, consumerRUV = 0x1f14e50,
consumerSchemaCSN = 0x317c520, consumerRID = 4, tmpConsumerRID = 0,
timeout = 120, stop_in_progress = 0, busywaittime = 0, pausetime = 0,
priv = 0x0,
attrs_to_strip = 0x1ef9ba0, agreement_type = 0, protocol_timeout =
0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000, flowControlPause =
2000, ignoreMissingChange = 0,
attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
(gdb) p *agmt->replarea
$2 = {flag = 15 '\017', udn = 0x1efce80 "dc=creatuity,dc=internal", dn
= 0x1ef9460 "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
"dc=creatuity,dc=internal", ndn_len = 24}
(gdb) p *agmt->rdn
$3 = {flag = 0 '\000', rdn = 0x19c2840
"cn=meToipa-12.creatuity.internal", rdns = 0x0, butcheredupto = -1,
nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
[root@ipa-11 ~]# grep -r PRId64 /usr/include/*
/usr/include/inttypes.h:# define PRId64 __PRI64_PREFIX "d"
[root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
/usr/include/inttypes.h:# define PRIu16 "u"
On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>> wrote:
On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
> The installed version of 389* is 1.3.7.10-1.fc27 for armv7hl,
> which appears to be the latest available version.
Perhaps something is off with the inttypes on Raspberry. Are you
building this yourself on Raspberry? Can we make code changes and
compile/install them?
Before we do that though, in gdb can you run these commands in the
same gdb frame:
(gdb) p *agmt->replarea
(gdb) p *agmt->rdn
Then do:
# grep -r PRId64 /usr/include/*
# grep -r PRIu16 /usr/include/*
So if you can compile the source, then change this line in
ldap/servers/plugins/replication/repl5_agmt.c:3036, but don't do
this yet until you get me the info I just requested.
From:
agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%"
PRId64 ";%" PRIu16 ";%s", slapi_sdn_get_dn(agmt->replarea),
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
agmt->hostname,
agmt->port, agmt->consumerRID, maxcsn);
To:
agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
slapi_sdn_get_dn(agmt->replarea),
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
agmt->hostname,
(long)agmt->port, (int)agmt->consumerRID, maxcsn);
Thanks,
Mark
>
> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
The agreement structure looks valid to me. it should not lead to a crash.
What looks weird to me is the order of the arguments of cvt_s.
It is called: rv = cvt_s(ss, u.s, width, prec, flags);
But the crashing thread shows the opposite order: flags, prec,width, str, ss
The others frame do not show this change of order.
Also 'str=4' that would be a meaningful value for 'prec=4'.
From debug perspective I only imagine disassemble the two last frames
to confirm parameters.
What are the nspr rpms ?
> (gdb) up
> #1 0xb6926b40 in cvt_s (flags=0, prec=<optimized out>,
> width=0, str=0x4 <error: Cannot access memory at address
> 0x4>, ss=<optimized out>)
> at ../../.././nspr/pr/src/io/prprf.c:374
> 374 slen = strlen(str);
> (gdb) up
> #2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2 "",
> fmt@entry=0xb34da770 <agmt_set> "\360\317p\002", ap=...) at
> ../../.././nspr/pr/src/io/prprf.c:1018
> 1018 rv = cvt_s(ss, u.s, width, prec, flags);
> (gdb) up
> #3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770
> <agmt_set> "\360\317p\002", ap=..., ap@entry=...) at
> ../../.././nspr/pr/src/io/prprf.c:1184
> 1184 rv = dosprintf(&ss, fmt, ap);
> (gdb) up
> #4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0
> "%s;%s;%s;%ld;%d;%s") at ldap/servers/slapd/ch_malloc.c:331
> 331 p = PR_vsmprintf(fmt, ap);
> (gdb) up
> #5 0xb34625a4 in agmt_update_maxcsn (r=r@entry=0x2737810,
> sdn=0x38d9a40, sdn@entry=0x10, op=<optimized out>, mods=0x10,
> mods@entry=0x27d0100, csn=csn@entry=0x38de350)
> at ldap/servers/plugins/replication/repl5_agmt.c:3036
> 3036 agmt->maxcsn =
> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> slapi_sdn_get_dn(agmt->replarea),
> (gdb) p *agmt
> $1 = {hostname = 0x2757be0 "ipa-12.company.internal", port =
> 389, transport_flags = 0, binddn = 0x26ed650 "", creds =
> 0x26ed7a0, bindmethod = 3, replarea = 0x2757480,
> frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
> frac_attr_total_defined = 1, schedule = 0x22dd0c0,
> auto_initialize = 502, dn = 0x2756d00, rdn = 0x2756c20,
> long_name = 0x22dd100
> "agmt=\"cn=meToipa-12.company.internal\" (ipa-12:5)",
> protocol = 0x2220930, changecounters = 0x20cb180,
> num_changecounters = 0,
> max_changecounters = 256, last_update_start_time =
> 1526499214, last_update_end_time = 1526499214,
> last_update_status = "Error (0) Replica acquired
> successfully: Incremental update succeeded", '\000' <repeats
> 954 times>, update_in_progress = 0, is_enabled = 1,
> last_init_start_time = 0, last_init_end_time = 0,
> last_init_status = '\000' <repeats 1023 times>, lock =
> 0x2741740, consumerRUV = 0x2772e50,
> consumerSchemaCSN = 0x39da520, consumerRID = 4,
> tmpConsumerRID = 0, timeout = 120, stop_in_progress = 0,
> busywaittime = 0, pausetime = 0, priv = 0x0,
> attrs_to_strip = 0x2757ba0, agreement_type = 0,
> protocol_timeout = 0x26ed5f0, maxcsn = 0x0, flowControlWindow
> = 1000, flowControlPause = 2000, ignoreMissingChange = 0,
> attr_lock = 0x2757c20, WaitForAsyncResults = 100}
>
>
> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>> wrote:
>
> This looks really familiar and I thought it was fixed.
> It should have been fixed in 1.3.7.10-1
> (https://pagure.io/389-ds-base/issue/49618
> <https://pagure.io/389-ds-base/issue/49618>). In your
> debug session go "up" into agmt_maxcsn_update() and do:
>
> (gdb) p *agmt
>
> Then send us that output please.
>
> Thanks,
> Mark
>
>
> On 05/15/2018 05:29 PM, Jonathan Vaughn via FreeIPA-users
> wrote:
>> Here is a backtrace from live gdb after the segfault.
>> Looks like things went wrong somewhere during in the
>> replication code ?
>>
>> Thread 36 "ns-slapd" received signal SIGSEGV,
>> Segmentation fault.
>> [Switching to Thread 0x9e0bc280 (LWP 4662)]
>> strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
>> 142 ldrd data1a, data1b, [src]
>> (gdb) bt
>> #0 0xb6728f2e in strlen () at
>> ../sysdeps/arm/armv6t2/strlen.S:142
>> #1 0xb6973b40 in cvt_s (flags=0, prec=<optimized out>,
>> width=0, str=0x4 <error: Cannot access memory at address
>> 0x4>, ss=<optimized out>)
>> at ../../.././nspr/pr/src/io/prprf.c:374
>> #2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc,
>> fmt=0xb34fddf2 "", fmt@entry=0xb3527770 <agmt_set>
>> "\360\357\305\002", ap=...)
>> at ../../.././nspr/pr/src/io/prprf.c:1018
>> #3 0xb6973c8c in PR_vsmprintf (fmt=fmt@entry=0xb3527770
>> <agmt_set> "\360\357\305\002", ap=..., ap@entry=...)
at
>> ../../.././nspr/pr/src/io/prprf.c:1184
>> #4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0
>> "%s;%s;%s;%ld;%d;%s") at
ldap/servers/slapd/ch_malloc.c:331
>> #5 0xb34af5a4 in agmt_update_maxcsn
>> (r=r@entry=0x2c89810, sdn=0x3e2bac0, sdn@entry=0x10,
>> op=<optimized out>, mods=0x10, mods@entry=0x3eec100,
>> csn=csn@entry=0x3e30220)
>> at ldap/servers/plugins/replication/repl5_agmt.c:3036
>> #6 0xb34bd424 in write_changelog_and_ruv
>> (pb=pb@entry=0x2cb54a0) at
>> ldap/servers/plugins/replication/repl5_plugins.c:1124
>> #7 0xb34be7ec in multimaster_be_betxnpostop_add
>> (pb=0x2cb54a0) at
>> ldap/servers/plugins/replication/repl5_plugins.c:855
>> #8 0xb34be880 in multimaster_mmr_postop (pb=<optimized
>> out>, flags=560) at
>> ldap/servers/plugins/replication/repl5_plugins.c:616
>> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop
>> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0,
>> flags=flags@entry=560) at ldap/servers/slapd/plugin_mmr.c:65
>> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
>> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>> #11 0xb6e2ce4c in op_shared_add (pb=pb@entry=0x2cb54a0)
>> at ldap/servers/slapd/add.c:679
>> #12 0xb6e2d35c in add_internal_pb
>> (pb=pb@entry=0x2cb54a0) at ldap/servers/slapd/add.c:407
>> #13 0xb6e2e0f8 in slapi_add_internal_pb
>> (pb=pb@entry=0x2cb54a0) at ldap/servers/slapd/add.c:332
>> #14 0xb368db50 in ipa_topo_util_segment_write
>> (tconf=tconf@entry=0x2cb5860,
>> tsegm=tsegm@entry=0x3f2d9a0) at topology_util.c:1251
>> #15 0xb368e01c in ipa_topo_util_update_agmt_list
>> (conf=0x2cb5860, repl_segments=<optimized out>) at
>> topology_util.c:696
>> #16 0xb368891c in ipa_topo_apply_shared_config () at
>> topology_init.c:165
>> #17 0xb6e4e65c in eq_call_all () at
>> ldap/servers/slapd/eventq.c:278
>> #18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
>> ldap/servers/slapd/eventq.c:323
>> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
>> ../../.././nspr/pr/src/pthreads/ptthread.c:201
>> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at
>> pthread_create.c:465
>> #21 0xb6785da8 in None () at
>> ../sysdeps/unix/sysv/linux/arm/clone.S:73
>>
>>
>> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz
>> <tbordaz(a)redhat.com <mailto:tbordaz@redhat.com>> wrote:
>>
>> Hi Jonathan,
>>
>> This problem looks new to me and has something
>> specific to your environment.
>> I think the best approach is to continue to debug on
>> your system if you have the possibility to do so.
>>
>> From strace we can see that DS started smoothly
>> (created its pid file then notified systemd it was
>> running fine). According to the pstack nunc-stans
>> was running and was able to accept network events
>> even if it appears it detected no incoming connection.
>> So the server should be ready to serve for some
>> seconds (more than a minute), then it crashed with
>> one thread dereferencing likely a wrong pointer.
>>
>> Could you attach a debugger when the server is
>> started and wait for the sigsegv to occur. Then
>> confirm the crashing thread backstack.
>> If it is confirmed, I am afraid this is a stack
>> corruption and valgrind could help
>>
(http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...
>>
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...>).
>>
>> best regards
>> thierry
>>
>>
>> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
>>> Here's a strace from before it dies. Most of the
>>> elapsed time is it waiting on some futex call it
>>> looks like near the end, when it finally "returns"
>>> (from lack of strace output for duration of call I
>>> assume it didn't actually return but SIGSEGV in
>>> that call) and strace prints ' = ?' on the futex it
>>> then immediately reports SIGSEGV after. So maybe
>>> the problem is that futex call, which may mean the
>>> problem is not directly in 389DS / FreeIPA itself?
>>>
>>>
>>>
>>> 15:13:31.626587 (+ 0.000630) listen(8, 128) = 0
>>> <0.000068>
>>> 15:13:31.626857 (+ 0.000235) listen(9, 128) = 0
>>> <0.000048>
>>> 15:13:31.627111 (+ 0.000251)
>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>> tv_nsec=41120614}) = 0 <0.000085>
>>> 15:13:31.627457 (+ 0.000356)
>>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>>> tv_nsec=627560772}) = 0 <0.000043>
>>> 15:13:31.631233 (+ 0.003839)
>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>> tv_nsec=45286796}) = 0 <0.000077>
>>> 15:13:31.631720 (+ 0.000427)
>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>> tv_nsec=45661430}) = 0 <0.000042>
>>> 15:13:31.631955 (+ 0.000220)
>>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>>> tv_nsec=632049036}) = 0 <0.000047>
>>> 15:13:31.635669 (+ 0.003785)
>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>> tv_nsec=49725840}) = 0 <0.000146>
>>> 15:13:31.636484 (+ 0.000784) write(16, "a", 1) =
1
>>> <0.000118>
>>> 15:13:31.636855 (+ 0.000341) sched_yield() = 0
>>> <0.000252>
>>> 15:13:31.637322 (+ 0.000470) futex(0x1cb57a0,
>>> FUTEX_WAKE_PRIVATE, 1) = 1 <0.000088>
>>> 15:13:31.637897 (+ 0.000610) write(16, "a", 1) =
1
>>> <0.000221>
>>> 15:13:31.638394 (+ 0.000467) sched_yield() = 0
>>> <0.000047>
>>> 15:13:31.638619 (+ 0.000202) futex(0x1cb5710,
>>> FUTEX_WAKE_PRIVATE, 1) = 1 <0.000065>
>>> 15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33 <0.000831>
>>> 15:13:31.640260 (+ 0.001387) getpid() = 32353
>>> <0.000077>
>>> 15:13:31.640558 (+ 0.000256) fstat64(33,
>>> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000119>
>>> 15:13:31.641106 (+ 0.000556) write(33, "32353\n",
>>> 6) = 6 <0.000127>
>>> 15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
>>> 15:13:31.642216 (+ 0.000758)
>>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> 0644) = 0 <0.000152>
>>> 15:13:31.642900 (+ 0.000679)
>>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>>> tv_nsec=643020294}) = 0 <0.000056>
>>> 15:13:31.643495 (+ 0.000590) write(2,
>>> "[14/May/2018:15:13:31.643020294 "..., 134) = 134
>>> <0.002697>
>>> 15:13:31.646515 (+ 0.003052)
>>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>>> tv_nsec=646694394}) = 0 <0.000075>
>>> 15:13:31.646892 (+ 0.000337) write(4,
>>> "[14/May/2018:15:13:31.646694394 "..., 134) = 134
>>> <0.000522>
>>> 15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
>>> 15:13:31.654425 (+ 0.006617)
>>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>>> tv_nsec=654598946}) = 0 <0.000253>
>>> 15:13:31.655137 (+ 0.000717) write(2,
>>> "[14/May/2018:15:13:31.654598946 "..., 136) = 136
>>> <0.002427>
>>> 15:13:31.658312 (+ 0.003165)
>>> clock_gettime(CLOCK_REALTIME, {tv_sec=1526328811,
>>> tv_nsec=658486117}) = 0 <0.000251>
>>> 15:13:31.659032 (+ 0.000682) write(4,
>>> "[14/May/2018:15:13:31.658486117 "..., 136) = 136
>>> <0.000346>
>>> 15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
>>> 15:13:31.663230 (+ 0.003642) getpid() = 32353
>>> <0.000045>
>>> 15:13:31.663732 (+ 0.000454) socket(AF_UNIX,
>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 33 <0.000296>
>>> 15:13:31.664760 (+ 0.001048) getsockopt(33,
>>> SOL_SOCKET, SO_SNDBUF, [163840], [4]) = 0 <0.000108>
>>> 15:13:31.665141 (+ 0.000386) setsockopt(33,
>>> SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = -1
>>> EPERM (Operation not permitted) <0.000051>
>>> 15:13:31.665500 (+ 0.000334) setsockopt(33,
>>> SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0 <0.000229>
>>> 15:13:31.665973 (+ 0.000468) sendmsg(33,
>>> {msg_name={sa_family=AF_UNIX,
>>> sun_path="/run/systemd/notify"}, msg_namelen=21,
>>> msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>>> Re"..., iov_len=69}], msg_iovlen=1,
>>> msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 69
>>> <0.000600>
>>> 15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
>>> 15:13:31.677320 (+ 0.010146) futex(0xb31122e8,
>>> FUTEX_WAIT, 32357, NULL) = ?
>>> 15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV
>>> (core dumped) +++
>>>
>>> On Mon, May 14, 2018 at 10:27 AM, thierry bordaz
>>> <tbordaz(a)redhat.com <mailto:tbordaz@redhat.com>>
wrote:
>>>
>>> Hi Jonathan,
>>>
>>> This is weird as the crashing thread stack
>>> looks truncated (did you copy/paste all of it ?)
>>>
>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>>> #2 0x00000000 in None ()
>>>
>>> Did you install 389-ds-base-debuginfo ?
>>> How did you get that backtrace ? from a core
>>> dumped, pstack ? Can you attach a debugger
>>> before the crash occurs ?
>>>
>>> It looks it crashed soon at startup, could it
>>> be related to a broken dse.ldif. It should
>>> exists a dse.ldif.OK, is it possibly to try to
>>> start with it ?
>>>
>>> best regards
>>> thierry
>>>
>>> On 05/12/2018 01:22 AM, Jonathan Vaughn via
>>> FreeIPA-users wrote:
>>>> Not sure if it makes a difference... I was
>>>> looking into this again and realized I had a
>>>> bunch of messages from gdb telling me to
>>>> install more debuginfo. I've done that now,
>>>> here it is again freshly run through gdb
>>>>
>>>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>>>> Copyright (C) 2017 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later
>>>> <http://gnu.org/licenses/gpl.html
>>>> <http://gnu.org/licenses/gpl.html>>
>>>> This is free software: you are free to change
>>>> and redistribute it.
>>>> There is NO WARRANTY, to the extent permitted
>>>> by law. Type "show copying"
>>>> and "show warranty" for details.
>>>> This GDB was configured as
>>>> "armv7hl-redhat-linux-gnueabi".
>>>> Type "show configuration" for
configuration
>>>> details.
>>>> For bug reporting instructions, please see:
>>>> <http://www.gnu.org/software/gdb/bugs/
>>>> <http://www.gnu.org/software/gdb/bugs/>>;.
>>>> Find the GDB manual and other documentation
>>>> resources online at:
>>>> <http://www.gnu.org/software/gdb/documentation/
>>>>
<http://www.gnu.org/software/gdb/documentation/>>;.
>>>> For help, type "help".
>>>> Type "apropos word" to search for commands
>>>> related to "word"...
>>>> Reading symbols from
>>>> /usr/sbin/ns-slapd...Reading symbols from
>>>>
/usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>>>> done.
>>>> ...
>>>>
>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>>>> #2 0x00000000 in None ()
>>>>
>>>>
>>>>
>>>> On Tue, May 8, 2018 at 7:52 AM, Rob
>>>> Crittenden <rcritten(a)redhat.com
>>>> <mailto:rcritten@redhat.com>> wrote:
>>>>
>>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>>
>>>> Still trying to figure this out.
>>>> It looks like slapd is dying, I
>>>> thought it was still running for
>>>> some reason.
>>>>
>>>> slapd is dying to segfault. strace
>>>> of it happening doesn't seem to
>>>> reveal much:
>>>>
>>>>
>>>> A stack trace would very much help
>>>> trying to track down the cause.
>>>>
>>>>
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
>>>>
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-cras...
>>>>
>>>> rob
>>>>
>>>>
>>>> 18:32:41.543717 (+ 0.000801)
>>>> openat(AT_FDCWD,
>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
>>>> 18:32:41.544907 (+ 0.001195)
>>>> getpid() = 16014
>>>> 18:32:41.545269 (+ 0.000329)
>>>> fstat64(32, {st_mode=S_IFREG|0644,
>>>> st_size=0, ...}) = 0
>>>> 18:32:41.545799 (+ 0.000536)
>>>> write(32, "16014\n", 6) = 6
>>>> 18:32:41.546603 (+ 0.000818)
>>>> close(32) = 0
>>>> 18:32:41.547061 (+ 0.000448)
>>>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> 0644) = 0
>>>> 18:32:41.547741 (+ 0.000676)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1525735961,
>>>> tv_nsec=548030641}) = 0
>>>> 18:32:41.548324 (+ 0.000587)
>>>> write(2,
>>>> "[07/May/2018:18:32:41.548030641
>>>> "..., 134) = 134
>>>> 18:32:41.551096 (+ 0.002840)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1525735961,
>>>> tv_nsec=551287555}) = 0
>>>> 18:32:41.551568 (+ 0.000406)
>>>> write(4,
>>>> "[07/May/2018:18:32:41.551287555
>>>> "..., 134) = 134
>>>> 18:32:41.552360 (+ 0.000811)
>>>> fsync(4) = 0
>>>> 18:32:41.558499 (+ 0.006170)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1525735961,
>>>> tv_nsec=558678099}) = 0
>>>> 18:32:41.558901 (+ 0.000350)
>>>> write(2,
>>>> "[07/May/2018:18:32:41.558678099
>>>> "..., 136) = 136
>>>> 18:32:41.561537 (+ 0.002680)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1525735961,
>>>> tv_nsec=561718659}) = 0
>>>> 18:32:41.562357 (+ 0.000793)
>>>> write(4,
>>>> "[07/May/2018:18:32:41.561718659
>>>> "..., 136) = 136
>>>> 18:32:41.563293 (+ 0.001148)
>>>> fsync(4) = 0
>>>> 18:32:41.566928 (+ 0.003452)
>>>> getpid() = 16014
>>>> 18:32:41.567712 (+ 0.000752)
>>>> socket(AF_UNIX,
>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>>> 18:32:41.568628 (+ 0.000912)
>>>> getsockopt(32, SOL_SOCKET,
>>>> SO_SNDBUF, [163840], [4]) = 0
>>>> 18:32:41.568972 (+ 0.000319)
>>>> setsockopt(32, SOL_SOCKET,
>>>> SO_SNDBUFFORCE, [8388608], 4) = -1
>>>> EPERM (Operation not permitted)
>>>> 18:32:41.569548 (+ 0.000589)
>>>> setsockopt(32, SOL_SOCKET,
>>>> SO_SNDBUF, [8388608], 4) = 0
>>>> 18:32:41.570064 (+ 0.000513)
>>>> sendmsg(32,
>>>> {msg_name={sa_family=AF_UNIX,
>>>>
sun_path="/run/systemd/notify"},
>>>> msg_namelen=21,
>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>> started: Re"..., iov_len=69}],
>>>> msg_iovlen=1, msg_controllen=0,
>>>> msg_flags=0}, MSG_NOSIGNAL) = 69
>>>> 18:32:41.570845 (+ 0.000789)
>>>> close(32) = 0
>>>> 18:32:41.576358 (+ 0.005575)
>>>> futex(0xb30ed2e8, FUTEX_WAIT,
>>>> 16016, NULL) = ?
>>>> 18:33:01.730774 (+ 20.154428)
>>>> +++ killed by SIGSEGV +++
>>>>
>>>
>>>
>>
>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list --freeipa-users(a)lists.fedorahosted.org
>> <mailto:freeipa-users@lists.fedorahosted.org>
>> To unsubscribe send an email
tofreeipa-users-leave(a)lists.fedorahosted.org
>> <mailto:freeipa-users-leave@lists.fedorahosted.org>
>
>
>
4:07 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Oops, hit reply instead of reply-all
NSPR RPMs
# yum list installed nspr*
Installed Packages
nspr.armv7hl
4.19.0-1.fc27
@updates
nspr-debuginfo.armv7hl
4.19.0-1.fc27
@updates-debuginfo
nspr-debugsource.armv7hl
4.19.0-1.fc27
@updates-debuginfo
nspr-devel.armv7hl
4.19.0-1.fc27
@updates
As for building it with the changes you proposed previously - should I not
bother? I realized last night that the last and only time I built anything
*on* a Pi was a Pi Zero and that's why it was so terribly slow. This Pi 3
ain't fast but it's fast enough to build stuff in far less than half a day
(unlike the Zero)...
On Thu, May 17, 2018 at 5:07 AM, thierry bordaz <tbordaz(a)redhat.com> wrote:
On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
I've been just using the packages from Fedora. I can build it potentially
but I don't have a cross build environment set up at the moment. From
experience I'd want to do that first because building anything on the Pi
usually takes ages.
I'd been "redacting" the hostnames but I'll stop bothering since it
looks
like we're getting far enough into the weeds now that the difference in
string lengths after "redacting" might actually be a red herring.
(gdb) p *agmt
$1 = {hostname = 0x1ef9be0 "ipa-12.creatuity.internal", port = 389,
transport_flags = 0, binddn = 0x1e8f650 "", creds = 0x1e8f7a0, bindmethod =
3, replarea = 0x1ef9480,
frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
frac_attr_total_defined = 1, schedule = 0x1a7f0c0, auto_initialize = 502,
dn = 0x1ef8d00, rdn = 0x1ef8c20,
long_name = 0x1a7f100 "agmt=\"cn=meToipa-12.creatuity.internal\"
(ipa-12:5)", protocol = 0x19c2930, changecounters = 0x186d180,
num_changecounters = 0,
max_changecounters = 256, last_update_start_time = 1526500697,
last_update_end_time = 1526500697,
last_update_status = "Error (0) Replica acquired successfully:
Incremental update succeeded", '\000' <repeats 954 times>,
update_in_progress = 0, is_enabled = 1,
last_init_start_time = 0, last_init_end_time = 0, last_init_status =
'\000' <repeats 1023 times>, lock = 0x1ee3740, consumerRUV = 0x1f14e50,
consumerSchemaCSN = 0x317c520, consumerRID = 4, tmpConsumerRID = 0,
timeout = 120, stop_in_progress = 0, busywaittime = 0, pausetime = 0, priv
= 0x0,
attrs_to_strip = 0x1ef9ba0, agreement_type = 0, protocol_timeout =
0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000, flowControlPause = 2000,
ignoreMissingChange = 0,
attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
(gdb) p *agmt->replarea
$2 = {flag = 15 '\017', udn = 0x1efce80 "dc=creatuity,dc=internal", dn
=
0x1ef9460 "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
"dc=creatuity,dc=internal", ndn_len = 24}
(gdb) p *agmt->rdn
$3 = {flag = 0 '\000', rdn = 0x19c2840
"cn=meToipa-12.creatuity.internal",
rdns = 0x0, butcheredupto = -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
[root@ipa-11 ~]# grep -r PRId64 /usr/include/*
/usr/include/inttypes.h:# define PRId64 __PRI64_PREFIX "d"
[root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
/usr/include/inttypes.h:# define PRIu16 "u"
On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds <mreynolds(a)redhat.com>
wrote:
>
>
> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>
> The installed version of 389* is 1.3.7.10-1.fc27 for armv7hl, which
> appears to be the latest available version.
>
>
> Perhaps something is off with the inttypes on Raspberry. Are you
> building this yourself on Raspberry? Can we make code changes and
> compile/install them?
>
> Before we do that though, in gdb can you run these commands in the same
> gdb frame:
>
> (gdb) p *agmt->replarea
> (gdb) p *agmt->rdn
>
>
> Then do:
>
> # grep -r PRId64 /usr/include/*
> # grep -r PRIu16 /usr/include/*
>
>
> So if you can compile the source, then change this line in
> ldap/servers/plugins/replication/repl5_agmt.c:3036, but don't do this
> yet until you get me the info I just requested.
>
> From:
>
> agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%" PRId64
";%"
> PRIu16 ";%s", slapi_sdn_get_dn(agmt->replarea),
>
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
> agmt->port,
> agmt->consumerRID, maxcsn);
>
> To:
>
> agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> slapi_sdn_get_dn(agmt->replarea),
>
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
> (long)agmt->port,
> (int)agmt->consumerRID, maxcsn);
>
>
> Thanks,
> Mark
>
>
> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
> wrote:
>
>
The agreement structure looks valid to me. it should not lead to a crash.
What looks weird to me is the order of the arguments of cvt_s.
It is called: rv = cvt_s(ss, u.s, width, prec, flags);
But the crashing thread shows the opposite order: flags, prec,width, str,
ss
The others frame do not show this change of order.
Also 'str=4' that would be a meaningful value for 'prec=4'.
From debug perspective I only imagine disassemble the two last frames to
confirm parameters.
What are the nspr rpms ?
(gdb) up
>> #1 0xb6926b40 in cvt_s (flags=0, prec=<optimized out>, width=0, str=0x4
>> <error: Cannot access memory at address 0x4>, ss=<optimized out>)
>> at ../../.././nspr/pr/src/io/prprf.c:374
>> 374 slen = strlen(str);
>> (gdb) up
>> #2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2 "",
fmt@entry=0xb34da770
>> <agmt_set> "\360\317p\002", ap=...) at
../../.././nspr/pr/src/io/prpr
>> f.c:1018
>> 1018 rv = cvt_s(ss, u.s, width, prec, flags);
>> (gdb) up
>> #3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770 <agmt_set>
>> "\360\317p\002", ap=..., ap@entry=...) at
../../.././nspr/pr/src/io/prpr
>> f.c:1184
>> 1184 rv = dosprintf(&ss, fmt, ap);
>> (gdb) up
>> #4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0
>> "%s;%s;%s;%ld;%d;%s") at ldap/servers/slapd/ch_malloc.c:331
>> 331 p = PR_vsmprintf(fmt, ap);
>> (gdb) up
>> #5 0xb34625a4 in agmt_update_maxcsn (r=r@entry=0x2737810,
>> sdn=0x38d9a40, sdn@entry=0x10, op=<optimized out>, mods=0x10,
mods@entry=0x27d0100,
>> csn=csn@entry=0x38de350)
>> at ldap/servers/plugins/replication/repl5_agmt.c:3036
>> 3036 agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>> slapi_sdn_get_dn(agmt->replarea),
>> (gdb) p *agmt
>> $1 = {hostname = 0x2757be0 "ipa-12.company.internal", port = 389,
>> transport_flags = 0, binddn = 0x26ed650 "", creds = 0x26ed7a0,
bindmethod =
>> 3, replarea = 0x2757480,
>> frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
>> frac_attr_total_defined = 1, schedule = 0x22dd0c0, auto_initialize = 502,
>> dn = 0x2756d00, rdn = 0x2756c20,
>> long_name = 0x22dd100 "agmt=\"cn=meToipa-12.company.internal\"
>> (ipa-12:5)", protocol = 0x2220930, changecounters = 0x20cb180,
>> num_changecounters = 0,
>> max_changecounters = 256, last_update_start_time = 1526499214,
>> last_update_end_time = 1526499214,
>> last_update_status = "Error (0) Replica acquired successfully:
>> Incremental update succeeded", '\000' <repeats 954 times>,
>> update_in_progress = 0, is_enabled = 1,
>> last_init_start_time = 0, last_init_end_time = 0, last_init_status =
>> '\000' <repeats 1023 times>, lock = 0x2741740, consumerRUV =
0x2772e50,
>> consumerSchemaCSN = 0x39da520, consumerRID = 4, tmpConsumerRID = 0,
>> timeout = 120, stop_in_progress = 0, busywaittime = 0, pausetime = 0, priv
>> = 0x0,
>> attrs_to_strip = 0x2757ba0, agreement_type = 0, protocol_timeout =
>> 0x26ed5f0, maxcsn = 0x0, flowControlWindow = 1000, flowControlPause = 2000,
>> ignoreMissingChange = 0,
>> attr_lock = 0x2757c20, WaitForAsyncResults = 100}
>>
>>
>> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds <mreynolds(a)redhat.com>
>> wrote:
>>
>>> This looks really familiar and I thought it was fixed. It should have
>>> been fixed in 1.3.7.10-1 (https://pagure.io/389-ds-base/issue/49618).
>>> In your debug session go "up" into agmt_maxcsn_update() and do:
>>>
>>> (gdb) p *agmt
>>>
>>> Then send us that output please.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>> On 05/15/2018 05:29 PM, Jonathan Vaughn via FreeIPA-users wrote:
>>>
>>> Here is a backtrace from live gdb after the segfault. Looks like things
>>> went wrong somewhere during in the replication code ?
>>>
>>> Thread 36 "ns-slapd" received signal SIGSEGV, Segmentation fault.
>>> [Switching to Thread 0x9e0bc280 (LWP 4662)]
>>> strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
>>> 142 ldrd data1a, data1b, [src]
>>> (gdb) bt
>>> #0 0xb6728f2e in strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
>>> #1 0xb6973b40 in cvt_s (flags=0, prec=<optimized out>, width=0,
>>> str=0x4 <error: Cannot access memory at address 0x4>, ss=<optimized
out>)
>>> at ../../.././nspr/pr/src/io/prprf.c:374
>>> #2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc, fmt=0xb34fddf2
>>> "", fmt@entry=0xb3527770 <agmt_set>
"\360\357\305\002", ap=...)
>>> at ../../.././nspr/pr/src/io/prprf.c:1018
>>> #3 0xb6973c8c in PR_vsmprintf (fmt=fmt@entry=0xb3527770 <agmt_set>
>>> "\360\357\305\002", ap=..., ap@entry=...) at
>>> ../../.././nspr/pr/src/io/prprf.c:1184
>>> #4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0
>>> "%s;%s;%s;%ld;%d;%s") at ldap/servers/slapd/ch_malloc.c:331
>>> #5 0xb34af5a4 in agmt_update_maxcsn (r=r@entry=0x2c89810,
>>> sdn=0x3e2bac0, sdn@entry=0x10, op=<optimized out>, mods=0x10,
>>> mods@entry=0x3eec100, csn=csn@entry=0x3e30220)
>>> at ldap/servers/plugins/replication/repl5_agmt.c:3036
>>> #6 0xb34bd424 in write_changelog_and_ruv (pb=pb@entry=0x2cb54a0) at
>>> ldap/servers/plugins/replication/repl5_plugins.c:1124
>>> #7 0xb34be7ec in multimaster_be_betxnpostop_add (pb=0x2cb54a0) at
>>> ldap/servers/plugins/replication/repl5_plugins.c:855
>>> #8 0xb34be880 in multimaster_mmr_postop (pb=<optimized out>,
>>> flags=560) at ldap/servers/plugins/replication/repl5_plugins.c:616
>>> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop (pb=pb@entry=0x2cb54a0,
>>> e=e@entry=0x0, flags=flags@entry=560) at ldap/servers/slapd/plugin_mmr.
>>> c:65
>>> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
>>> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>>> #11 0xb6e2ce4c in op_shared_add (pb=pb@entry=0x2cb54a0) at
>>> ldap/servers/slapd/add.c:679
>>> #12 0xb6e2d35c in add_internal_pb (pb=pb@entry=0x2cb54a0) at
>>> ldap/servers/slapd/add.c:407
>>> #13 0xb6e2e0f8 in slapi_add_internal_pb (pb=pb@entry=0x2cb54a0) at
>>> ldap/servers/slapd/add.c:332
>>> #14 0xb368db50 in ipa_topo_util_segment_write (tconf=tconf@entry=0x2cb5860,
>>> tsegm=tsegm@entry=0x3f2d9a0) at topology_util.c:1251
>>> #15 0xb368e01c in ipa_topo_util_update_agmt_list (conf=0x2cb5860,
>>> repl_segments=<optimized out>) at topology_util.c:696
>>> #16 0xb368891c in ipa_topo_apply_shared_config () at topology_init.c:165
>>> #17 0xb6e4e65c in eq_call_all () at ldap/servers/slapd/eventq.c:278
>>> #18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
>>> ldap/servers/slapd/eventq.c:323
>>> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
>>> ../../.././nspr/pr/src/pthreads/ptthread.c:201
>>> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at pthread_create.c:465
>>> #21 0xb6785da8 in None () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>>>
>>>
>>> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz <tbordaz(a)redhat.com>
>>> wrote:
>>>
>>>> Hi Jonathan,
>>>>
>>>> This problem looks new to me and has something specific to your
>>>> environment.
>>>> I think the best approach is to continue to debug on your system if
>>>> you have the possibility to do so.
>>>>
>>>> From strace we can see that DS started smoothly (created its pid file
>>>> then notified systemd it was running fine). According to the pstack
>>>> nunc-stans was running and was able to accept network events even if it
>>>> appears it detected no incoming connection.
>>>> So the server should be ready to serve for some seconds (more than a
>>>> minute), then it crashed with one thread dereferencing likely a wrong
>>>> pointer.
>>>>
>>>> Could you attach a debugger when the server is started and wait for
>>>> the sigsegv to occur. Then confirm the crashing thread backstack.
>>>> If it is confirmed, I am afraid this is a stack corruption and
>>>> valgrind could help (http://www.port389.org/docs/3
>>>> 89ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>>>> ).
>>>>
>>>> best regards
>>>> thierry
>>>>
>>>>
>>>> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
>>>>
>>>> Here's a strace from before it dies. Most of the elapsed time is it
>>>> waiting on some futex call it looks like near the end, when it finally
>>>> "returns" (from lack of strace output for duration of call I
assume it
>>>> didn't actually return but SIGSEGV in that call) and strace prints
' = ?'
>>>> on the futex it then immediately reports SIGSEGV after. So maybe the
>>>> problem is that futex call, which may mean the problem is not directly
in
>>>> 389DS / FreeIPA itself?
>>>>
>>>>
>>>>
>>>> 15:13:31.626587 (+ 0.000630) listen(8, 128) = 0 <0.000068>
>>>> 15:13:31.626857 (+ 0.000235) listen(9, 128) = 0 <0.000048>
>>>> 15:13:31.627111 (+ 0.000251) clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932, tv_nsec=41120614}) = 0 <0.000085>
>>>> 15:13:31.627457 (+ 0.000356) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=627560772}) = 0 <0.000043>
>>>> 15:13:31.631233 (+ 0.003839) clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932, tv_nsec=45286796}) = 0 <0.000077>
>>>> 15:13:31.631720 (+ 0.000427) clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932, tv_nsec=45661430}) = 0 <0.000042>
>>>> 15:13:31.631955 (+ 0.000220) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=632049036}) = 0 <0.000047>
>>>> 15:13:31.635669 (+ 0.003785) clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932, tv_nsec=49725840}) = 0 <0.000146>
>>>> 15:13:31.636484 (+ 0.000784) write(16, "a", 1) = 1
<0.000118>
>>>> 15:13:31.636855 (+ 0.000341) sched_yield() = 0 <0.000252>
>>>> 15:13:31.637322 (+ 0.000470) futex(0x1cb57a0, FUTEX_WAKE_PRIVATE,
>>>> 1) = 1 <0.000088>
>>>> 15:13:31.637897 (+ 0.000610) write(16, "a", 1) = 1
<0.000221>
>>>> 15:13:31.638394 (+ 0.000467) sched_yield() = 0 <0.000047>
>>>> 15:13:31.638619 (+ 0.000202) futex(0x1cb5710, FUTEX_WAKE_PRIVATE,
>>>> 1) = 1 <0.000065>
>>>> 15:13:31.638908 (+ 0.000298) openat(AT_FDCWD,
>>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33 <0.000831>
>>>> 15:13:31.640260 (+ 0.001387) getpid() = 32353 <0.000077>
>>>> 15:13:31.640558 (+ 0.000256) fstat64(33, {st_mode=S_IFREG|0644,
>>>> st_size=0, ...}) = 0 <0.000119>
>>>> 15:13:31.641106 (+ 0.000556) write(33, "32353\n", 6) = 6
<0.000127>
>>>> 15:13:31.641472 (+ 0.000362) close(33) = 0 <0.000519>
>>>> 15:13:31.642216 (+ 0.000758)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> 0644) = 0 <0.000152>
>>>> 15:13:31.642900 (+ 0.000679) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=643020294}) = 0 <0.000056>
>>>> 15:13:31.643495 (+ 0.000590) write(2,
>>>> "[14/May/2018:15:13:31.643020294 "..., 134) = 134
<0.002697>
>>>> 15:13:31.646515 (+ 0.003052) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=646694394}) = 0 <0.000075>
>>>> 15:13:31.646892 (+ 0.000337) write(4,
>>>> "[14/May/2018:15:13:31.646694394 "..., 134) = 134
<0.000522>
>>>> 15:13:31.647841 (+ 0.000973) fsync(4) = 0 <0.005967>
>>>> 15:13:31.654425 (+ 0.006617) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=654598946}) = 0 <0.000253>
>>>> 15:13:31.655137 (+ 0.000717) write(2,
>>>> "[14/May/2018:15:13:31.654598946 "..., 136) = 136
<0.002427>
>>>> 15:13:31.658312 (+ 0.003165) clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=658486117}) = 0 <0.000251>
>>>> 15:13:31.659032 (+ 0.000682) write(4,
>>>> "[14/May/2018:15:13:31.658486117 "..., 136) = 136
<0.000346>
>>>> 15:13:31.659623 (+ 0.000595) fsync(4) = 0 <0.003311>
>>>> 15:13:31.663230 (+ 0.003642) getpid() = 32353 <0.000045>
>>>> 15:13:31.663732 (+ 0.000454) socket(AF_UNIX,
>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 33 <0.000296>
>>>> 15:13:31.664760 (+ 0.001048) getsockopt(33, SOL_SOCKET, SO_SNDBUF,
>>>> [163840], [4]) = 0 <0.000108>
>>>> 15:13:31.665141 (+ 0.000386) setsockopt(33, SOL_SOCKET,
>>>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not permitted)
>>>> <0.000051>
>>>> 15:13:31.665500 (+ 0.000334) setsockopt(33, SOL_SOCKET, SO_SNDBUF,
>>>> [8388608], 4) = 0 <0.000229>
>>>> 15:13:31.665973 (+ 0.000468) sendmsg(33,
>>>> {msg_name={sa_family=AF_UNIX, sun_path="/run/systemd/notify"},
>>>> msg_namelen=21, msg_iov=[{iov_base="READY=1\nSTATUS=slapd started:
>>>> Re"..., iov_len=69}], msg_iovlen=1, msg_controllen=0, msg_flags=0},
>>>> MSG_NOSIGNAL) = 69 <0.000600>
>>>> 15:13:31.667270 (+ 0.001334) close(33) = 0 <0.003140>
>>>> 15:13:31.677320 (+ 0.010146) futex(0xb31122e8, FUTEX_WAIT, 32357,
>>>> NULL) = ?
>>>> 15:14:27.661809 (+ 55.984448) +++ killed by SIGSEGV (core dumped)
>>>> +++
>>>>
>>>> On Mon, May 14, 2018 at 10:27 AM, thierry bordaz
<tbordaz(a)redhat.com>
>>>> wrote:
>>>>
>>>>> Hi Jonathan,
>>>>>
>>>>> This is weird as the crashing thread stack looks truncated (did you
>>>>> copy/paste all of it ?)
>>>>>
>>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>>>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>>>>> #2 0x00000000 in None ()
>>>>>
>>>>> Did you install 389-ds-base-debuginfo ?
>>>>> How did you get that backtrace ? from a core dumped, pstack ? Can
you
>>>>> attach a debugger before the crash occurs ?
>>>>>
>>>>> It looks it crashed soon at startup, could it be related to a broken
>>>>> dse.ldif. It should exists a dse.ldif.OK, is it possibly to try to
start
>>>>> with it ?
>>>>>
>>>>> best regards
>>>>> thierry
>>>>>
>>>>> On 05/12/2018 01:22 AM, Jonathan Vaughn via FreeIPA-users wrote:
>>>>>
>>>>> Not sure if it makes a difference... I was looking into this again
>>>>> and realized I had a bunch of messages from gdb telling me to install
more
>>>>> debuginfo. I've done that now, here it is again freshly run
through gdb
>>>>>
>>>>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>>>>> Copyright (C) 2017 Free Software Foundation, Inc.
>>>>> License GPLv3+: GNU GPL version 3 or later <
>>>>> http://gnu.org/licenses/gpl.html>
>>>>> This is free software: you are free to change and redistribute it.
>>>>> There is NO WARRANTY, to the extent permitted by law. Type
"show
>>>>> copying"
>>>>> and "show warranty" for details.
>>>>> This GDB was configured as "armv7hl-redhat-linux-gnueabi".
>>>>> Type "show configuration" for configuration details.
>>>>> For bug reporting instructions, please see:
>>>>> <http://www.gnu.org/software/gdb/bugs/>;.
>>>>> Find the GDB manual and other documentation resources online at:
>>>>> <http://www.gnu.org/software/gdb/documentation/>;.
>>>>> For help, type "help".
>>>>> Type "apropos word" to search for commands related to
"word"...
>>>>> Reading symbols from /usr/sbin/ns-slapd...Reading symbols from
>>>>> /usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>>>>> done.
>>>>> ...
>>>>>
>>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>>>> #1 0xb6a06b40 in dosprintf () at /lib/libnspr4.so
>>>>> #2 0x00000000 in None ()
>>>>>
>>>>>
>>>>>
>>>>> On Tue, May 8, 2018 at 7:52 AM, Rob Crittenden
<rcritten(a)redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>>>>>
>>>>>>>> Still trying to figure this out. It looks like slapd is
dying, I
>>>>>>>> thought it was still running for some reason.
>>>>>>>>
>>>>>>>> slapd is dying to segfault. strace of it happening
doesn't seem to
>>>>>>>> reveal much:
>>>>>>>>
>>>>>>>
>>>>>>> A stack trace would very much help trying to track down the
cause.
>>>>>>>
>>>>>>> http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#d
>>>>>>> ebugging-crashes
>>>>>>>
>>>>>>> rob
>>>>>>>
>>>>>>>
>>>>>>>> 18:32:41.543717 (+ 0.000801) openat(AT_FDCWD,
>>>>>>>> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 32
>>>>>>>> 18:32:41.544907 (+ 0.001195) getpid() = 16014
>>>>>>>> 18:32:41.545269 (+ 0.000329) fstat64(32,
>>>>>>>> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>>>>>>>> 18:32:41.545799 (+ 0.000536) write(32,
"16014\n", 6) = 6
>>>>>>>> 18:32:41.546603 (+ 0.000818) close(32) = 0
>>>>>>>> 18:32:41.547061 (+ 0.000448)
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>>>>> 0644) = 0
>>>>>>>> 18:32:41.547741 (+ 0.000676)
clock_gettime(CLOCK_REALTIME,
>>>>>>>> {tv_sec=1525735961, tv_nsec=548030641}) = 0
>>>>>>>> 18:32:41.548324 (+ 0.000587) write(2,
>>>>>>>> "[07/May/2018:18:32:41.548030641 "..., 134) =
134
>>>>>>>> 18:32:41.551096 (+ 0.002840)
clock_gettime(CLOCK_REALTIME,
>>>>>>>> {tv_sec=1525735961, tv_nsec=551287555}) = 0
>>>>>>>> 18:32:41.551568 (+ 0.000406) write(4,
>>>>>>>> "[07/May/2018:18:32:41.551287555 "..., 134) =
134
>>>>>>>> 18:32:41.552360 (+ 0.000811) fsync(4) = 0
>>>>>>>> 18:32:41.558499 (+ 0.006170)
clock_gettime(CLOCK_REALTIME,
>>>>>>>> {tv_sec=1525735961, tv_nsec=558678099}) = 0
>>>>>>>> 18:32:41.558901 (+ 0.000350) write(2,
>>>>>>>> "[07/May/2018:18:32:41.558678099 "..., 136) =
136
>>>>>>>> 18:32:41.561537 (+ 0.002680)
clock_gettime(CLOCK_REALTIME,
>>>>>>>> {tv_sec=1525735961, tv_nsec=561718659}) = 0
>>>>>>>> 18:32:41.562357 (+ 0.000793) write(4,
>>>>>>>> "[07/May/2018:18:32:41.561718659 "..., 136) =
136
>>>>>>>> 18:32:41.563293 (+ 0.001148) fsync(4) = 0
>>>>>>>> 18:32:41.566928 (+ 0.003452) getpid() = 16014
>>>>>>>> 18:32:41.567712 (+ 0.000752) socket(AF_UNIX,
>>>>>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>>>>>>> 18:32:41.568628 (+ 0.000912) getsockopt(32,
SOL_SOCKET,
>>>>>>>> SO_SNDBUF, [163840], [4]) = 0
>>>>>>>> 18:32:41.568972 (+ 0.000319) setsockopt(32,
SOL_SOCKET,
>>>>>>>> SO_SNDBUFFORCE, [8388608], 4) = -1 EPERM (Operation not
permitted)
>>>>>>>> 18:32:41.569548 (+ 0.000589) setsockopt(32,
SOL_SOCKET,
>>>>>>>> SO_SNDBUF, [8388608], 4) = 0
>>>>>>>> 18:32:41.570064 (+ 0.000513) sendmsg(32,
>>>>>>>> {msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"},
>>>>>>>> msg_namelen=21,
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>>>>>> started: Re"..., iov_len=69}], msg_iovlen=1,
msg_controllen=0,
>>>>>>>> msg_flags=0}, MSG_NOSIGNAL) = 69
>>>>>>>> 18:32:41.570845 (+ 0.000789) close(32) = 0
>>>>>>>> 18:32:41.576358 (+ 0.005575) futex(0xb30ed2e8,
FUTEX_WAIT,
>>>>>>>> 16016, NULL) = ?
>>>>>>>> 18:33:01.730774 (+ 20.154428) +++ killed by SIGSEGV
+++
>>>>>>>>
>>>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>
>>>
>>>
>>
>
>
4:11 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Jonathan Vaughn via FreeIPA-users wrote:
Oops, hit reply instead of reply-all
NSPR RPMs
# yum list installed nspr*
Installed Packages
nspr.armv7hl
4.19.0-1.fc27
@updates
nspr-debuginfo.armv7hl
4.19.0-1.fc27
@updates-debuginfo
nspr-debugsource.armv7hl
4.19.0-1.fc27
@updates-debuginfo
nspr-devel.armv7hl
4.19.0-1.fc27
@updates
As for building it with the changes you proposed previously - should I
not bother? I realized last night that the last and only time I built
anything *on* a Pi was a Pi Zero and that's why it was so terribly slow.
This Pi 3 ain't fast but it's fast enough to build stuff in far less
than half a day (unlike the Zero)...
If you are handy with rpms you could build it in copr.
rob
On Thu, May 17, 2018 at 5:07 AM, thierry bordaz <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>> wrote:
On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
> I've been just using the packages from Fedora. I can build it
> potentially but I don't have a cross build environment set up at
> the moment. From experience I'd want to do that first because
> building anything on the Pi usually takes ages.
>
> I'd been "redacting" the hostnames but I'll stop bothering
since
> it looks like we're getting far enough into the weeds now that the
> difference in string lengths after "redacting" might actually be a
> red herring.
>
> (gdb) p *agmt
> $1 = {hostname = 0x1ef9be0 "ipa-12.creatuity.internal", port =
> 389, transport_flags = 0, binddn = 0x1e8f650 "", creds =
> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
> frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
> frac_attr_total_defined = 1, schedule = 0x1a7f0c0, auto_initialize
> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
> long_name = 0x1a7f100
"agmt=\"cn=meToipa-12.creatuity.internal\"
> (ipa-12:5)", protocol = 0x19c2930, changecounters = 0x186d180,
> num_changecounters = 0,
> max_changecounters = 256, last_update_start_time = 1526500697,
> last_update_end_time = 1526500697,
> last_update_status = "Error (0) Replica acquired successfully:
> Incremental update succeeded", '\000' <repeats 954 times>,
> update_in_progress = 0, is_enabled = 1,
> last_init_start_time = 0, last_init_end_time = 0,
> last_init_status = '\000' <repeats 1023 times>, lock = 0x1ee3740,
> consumerRUV = 0x1f14e50,
> consumerSchemaCSN = 0x317c520, consumerRID = 4, tmpConsumerRID =
> 0, timeout = 120, stop_in_progress = 0, busywaittime = 0,
> pausetime = 0, priv = 0x0,
> attrs_to_strip = 0x1ef9ba0, agreement_type = 0, protocol_timeout
> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000,
> flowControlPause = 2000, ignoreMissingChange = 0,
> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
> (gdb) p *agmt->replarea
> $2 = {flag = 15 '\017', udn = 0x1efce80
> "dc=creatuity,dc=internal", dn = 0x1ef9460
> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
> "dc=creatuity,dc=internal", ndn_len = 24}
> (gdb) p *agmt->rdn
> $3 = {flag = 0 '\000', rdn = 0x19c2840
> "cn=meToipa-12.creatuity.internal", rdns = 0x0, butcheredupto =
> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
>
> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
> /usr/include/inttypes.h:# define PRId64 __PRI64_PREFIX "d"
> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
> /usr/include/inttypes.h:# define PRIu16 "u"
>
>
>
> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>> wrote:
>
>
>
> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>> The installed version of 389* is 1.3.7.10-1.fc27 for armv7hl,
>> which appears to be the latest available version.
>
> Perhaps something is off with the inttypes on Raspberry. Are
> you building this yourself on Raspberry? Can we make code
> changes and compile/install them?
>
> Before we do that though, in gdb can you run these commands in
> the same gdb frame:
>
> (gdb) p *agmt->replarea
> (gdb) p *agmt->rdn
>
>
> Then do:
>
> # grep -r PRId64 /usr/include/*
> # grep -r PRIu16 /usr/include/*
>
>
> So if you can compile the source, then change this line in
> ldap/servers/plugins/replication/repl5_agmt.c:3036, but don't
> do this yet until you get me the info I just requested.
>
> From:
>
> agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%"
> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replarea),
>
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
> agmt->hostname,
> agmt->port,
> agmt->consumerRID, maxcsn);
>
> To:
>
> agmt->maxcsn =
> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> slapi_sdn_get_dn(agmt->replarea),
>
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
> agmt->hostname,
>
> (long)agmt->port, (int)agmt->consumerRID, maxcsn);
>
>
> Thanks,
> Mark
>
>>
>> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
>> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>>
wrote:
>
The agreement structure looks valid to me. it should not lead to a
crash.
What looks weird to me is the order of the arguments of cvt_s.
It is called: rv = cvt_s(ss, u.s, width, prec, flags);
But the crashing thread shows the opposite order: flags, prec,width,
str, ss
The others frame do not show this change of order.
Also 'str=4' that would be a meaningful value for 'prec=4'.
From debug perspective I only imagine disassemble the two last
frames to confirm parameters.
What are the nspr rpms ?
>> (gdb) up
>> #1 0xb6926b40 in cvt_s (flags=0, prec=<optimized out>,
>> width=0, str=0x4 <error: Cannot access memory at address
>> 0x4>, ss=<optimized out>)
>> at ../../.././nspr/pr/src/io/prprf.c:374
>> 374 slen = strlen(str);
>> (gdb) up
>> #2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2 "",
>> fmt@entry=0xb34da770 <agmt_set> "\360\317p\002",
ap=...)
>> at ../../.././nspr/pr/src/io/prprf.c:1018
>> 1018 rv = cvt_s(ss, u.s, width, prec, flags);
>> (gdb) up
>> #3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770
>> <agmt_set> "\360\317p\002", ap=..., ap@entry=...) at
>> ../../.././nspr/pr/src/io/prprf.c:1184
>> 1184 rv = dosprintf(&ss, fmt, ap);
>> (gdb) up
>> #4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0
>> "%s;%s;%s;%ld;%d;%s") at
ldap/servers/slapd/ch_malloc.c:331
>> 331 p = PR_vsmprintf(fmt, ap);
>> (gdb) up
>> #5 0xb34625a4 in agmt_update_maxcsn
>> (r=r@entry=0x2737810, sdn=0x38d9a40, sdn@entry=0x10,
>> op=<optimized out>, mods=0x10, mods@entry=0x27d0100,
>> csn=csn@entry=0x38de350)
>> at ldap/servers/plugins/replication/repl5_agmt.c:3036
>> 3036 agmt->maxcsn =
>> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>> slapi_sdn_get_dn(agmt->replarea),
>> (gdb) p *agmt
>> $1 = {hostname = 0x2757be0 "ipa-12.company.internal",
>> port = 389, transport_flags = 0, binddn = 0x26ed650 "",
>> creds = 0x26ed7a0, bindmethod = 3, replarea = 0x2757480,
>> frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
>> frac_attr_total_defined = 1, schedule = 0x22dd0c0,
>> auto_initialize = 502, dn = 0x2756d00, rdn = 0x2756c20,
>> long_name = 0x22dd100
>> "agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)",
>> protocol = 0x2220930, changecounters = 0x20cb180,
>> num_changecounters = 0,
>> max_changecounters = 256, last_update_start_time =
>> 1526499214, last_update_end_time = 1526499214,
>> last_update_status = "Error (0) Replica acquired
>> successfully: Incremental update succeeded", '\000'
>> <repeats 954 times>, update_in_progress = 0, is_enabled = 1,
>> last_init_start_time = 0, last_init_end_time = 0,
>> last_init_status = '\000' <repeats 1023 times>, lock =
>> 0x2741740, consumerRUV = 0x2772e50,
>> consumerSchemaCSN = 0x39da520, consumerRID = 4,
>> tmpConsumerRID = 0, timeout = 120, stop_in_progress = 0,
>> busywaittime = 0, pausetime = 0, priv = 0x0,
>> attrs_to_strip = 0x2757ba0, agreement_type = 0,
>> protocol_timeout = 0x26ed5f0, maxcsn = 0x0,
>> flowControlWindow = 1000, flowControlPause = 2000,
>> ignoreMissingChange = 0,
>> attr_lock = 0x2757c20, WaitForAsyncResults = 100}
>>
>>
>> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
>> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>>
wrote:
>>
>> This looks really familiar and I thought it was
>> fixed. It should have been fixed in 1.3.7.10-1
>> (https://pagure.io/389-ds-base/issue/49618
>> <https://pagure.io/389-ds-base/issue/49618>). In
>> your debug session go "up" into agmt_maxcsn_update()
>> and do:
>>
>> (gdb) p *agmt
>>
>> Then send us that output please.
>>
>> Thanks,
>> Mark
>>
>>
>> On 05/15/2018 05:29 PM, Jonathan Vaughn via
>> FreeIPA-users wrote:
>>> Here is a backtrace from live gdb after the
>>> segfault. Looks like things went wrong somewhere
>>> during in the replication code ?
>>>
>>> Thread 36 "ns-slapd" received signal SIGSEGV,
>>> Segmentation fault.
>>> [Switching to Thread 0x9e0bc280 (LWP 4662)]
>>> strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
>>> 142 ldrd data1a, data1b, [src]
>>> (gdb) bt
>>> #0 0xb6728f2e in strlen () at
>>> ../sysdeps/arm/armv6t2/strlen.S:142
>>> #1 0xb6973b40 in cvt_s (flags=0, prec=<optimized
>>> out>, width=0, str=0x4 <error: Cannot access memory
>>> at address 0x4>, ss=<optimized out>)
>>> at ../../.././nspr/pr/src/io/prprf.c:374
>>> #2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc,
>>> fmt=0xb34fddf2 "", fmt@entry=0xb3527770
<agmt_set>
>>> "\360\357\305\002", ap=...)
>>> at ../../.././nspr/pr/src/io/prprf.c:1018
>>> #3 0xb6973c8c in PR_vsmprintf
>>> (fmt=fmt@entry=0xb3527770 <agmt_set>
>>> "\360\357\305\002", ap=..., ap@entry=...) at
>>> ../../.././nspr/pr/src/io/prprf.c:1184
>>> #4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0
>>> "%s;%s;%s;%ld;%d;%s") at
>>> ldap/servers/slapd/ch_malloc.c:331
>>> #5 0xb34af5a4 in agmt_update_maxcsn
>>> (r=r@entry=0x2c89810, sdn=0x3e2bac0, sdn@entry=0x10,
>>> op=<optimized out>, mods=0x10, mods@entry=0x3eec100,
>>> csn=csn@entry=0x3e30220)
>>> at
>>> ldap/servers/plugins/replication/repl5_agmt.c:3036
>>> #6 0xb34bd424 in write_changelog_and_ruv
>>> (pb=pb@entry=0x2cb54a0) at
>>> ldap/servers/plugins/replication/repl5_plugins.c:1124
>>> #7 0xb34be7ec in multimaster_be_betxnpostop_add
>>> (pb=0x2cb54a0) at
>>> ldap/servers/plugins/replication/repl5_plugins.c:855
>>> #8 0xb34be880 in multimaster_mmr_postop
>>> (pb=<optimized out>, flags=560) at
>>> ldap/servers/plugins/replication/repl5_plugins.c:616
>>> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop
>>> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0,
>>> flags=flags@entry=560) at
>>> ldap/servers/slapd/plugin_mmr.c:65
>>> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
>>> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>>> #11 0xb6e2ce4c in op_shared_add
>>> (pb=pb@entry=0x2cb54a0) at ldap/servers/slapd/add.c:679
>>> #12 0xb6e2d35c in add_internal_pb
>>> (pb=pb@entry=0x2cb54a0) at ldap/servers/slapd/add.c:407
>>> #13 0xb6e2e0f8 in slapi_add_internal_pb
>>> (pb=pb@entry=0x2cb54a0) at ldap/servers/slapd/add.c:332
>>> #14 0xb368db50 in ipa_topo_util_segment_write
>>> (tconf=tconf@entry=0x2cb5860,
>>> tsegm=tsegm@entry=0x3f2d9a0) at topology_util.c:1251
>>> #15 0xb368e01c in ipa_topo_util_update_agmt_list
>>> (conf=0x2cb5860, repl_segments=<optimized out>) at
>>> topology_util.c:696
>>> #16 0xb368891c in ipa_topo_apply_shared_config () at
>>> topology_init.c:165
>>> #17 0xb6e4e65c in eq_call_all () at
>>> ldap/servers/slapd/eventq.c:278
>>> #18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
>>> ldap/servers/slapd/eventq.c:323
>>> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
>>> ../../.././nspr/pr/src/pthreads/ptthread.c:201
>>> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at
>>> pthread_create.c:465
>>> #21 0xb6785da8 in None () at
>>> ../sysdeps/unix/sysv/linux/arm/clone.S:73
>>>
>>>
>>> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz
>>> <tbordaz(a)redhat.com <mailto:tbordaz@redhat.com>>
wrote:
>>>
>>> Hi Jonathan,
>>>
>>> This problem looks new to me and has something
>>> specific to your environment.
>>> I think the best approach is to continue to
>>> debug on your system if you have the possibility
>>> to do so.
>>>
>>> From strace we can see that DS started smoothly
>>> (created its pid file then notified systemd it
>>> was running fine). According to the pstack
>>> nunc-stans was running and was able to accept
>>> network events even if it appears it detected no
>>> incoming connection.
>>> So the server should be ready to serve for some
>>> seconds (more than a minute), then it crashed
>>> with one thread dereferencing likely a wrong
>>> pointer.
>>>
>>> Could you attach a debugger when the server is
>>> started and wait for the sigsegv to occur. Then
>>> confirm the crashing thread backstack.
>>> If it is confirmed, I am afraid this is a stack
>>> corruption and valgrind could help
>>>
(http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...
>>>
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...>).
>>>
>>> best regards
>>> thierry
>>>
>>>
>>> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
>>>> Here's a strace from before it dies. Most of
>>>> the elapsed time is it waiting on some futex
>>>> call it looks like near the end, when it
>>>> finally "returns" (from lack of strace
output
>>>> for duration of call I assume it didn't
>>>> actually return but SIGSEGV in that call) and
>>>> strace prints ' = ?' on the futex it then
>>>> immediately reports SIGSEGV after. So maybe the
>>>> problem is that futex call, which may mean the
>>>> problem is not directly in 389DS / FreeIPA itself?
>>>>
>>>>
>>>>
>>>> 15:13:31.626587 (+ 0.000630) listen(8, 128)
>>>> = 0 <0.000068>
>>>> 15:13:31.626857 (+ 0.000235) listen(9, 128)
>>>> = 0 <0.000048>
>>>> 15:13:31.627111 (+ 0.000251)
>>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>>> tv_nsec=41120614}) = 0 <0.000085>
>>>> 15:13:31.627457 (+ 0.000356)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=627560772}) = 0
>>>> <0.000043>
>>>> 15:13:31.631233 (+ 0.003839)
>>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>>> tv_nsec=45286796}) = 0 <0.000077>
>>>> 15:13:31.631720 (+ 0.000427)
>>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>>> tv_nsec=45661430}) = 0 <0.000042>
>>>> 15:13:31.631955 (+ 0.000220)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=632049036}) = 0
>>>> <0.000047>
>>>> 15:13:31.635669 (+ 0.003785)
>>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>>> tv_nsec=49725840}) = 0 <0.000146>
>>>> 15:13:31.636484 (+ 0.000784) write(16,
"a",
>>>> 1) = 1 <0.000118>
>>>> 15:13:31.636855 (+ 0.000341) sched_yield()
>>>> = 0 <0.000252>
>>>> 15:13:31.637322 (+ 0.000470)
>>>> futex(0x1cb57a0, FUTEX_WAKE_PRIVATE, 1) = 1
>>>> <0.000088>
>>>> 15:13:31.637897 (+ 0.000610) write(16,
"a",
>>>> 1) = 1 <0.000221>
>>>> 15:13:31.638394 (+ 0.000467) sched_yield()
>>>> = 0 <0.000047>
>>>> 15:13:31.638619 (+ 0.000202)
>>>> futex(0x1cb5710, FUTEX_WAKE_PRIVATE, 1) = 1
>>>> <0.000065>
>>>> 15:13:31.638908 (+ 0.000298)
>>>> openat(AT_FDCWD,
>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
<0.000831>
>>>> 15:13:31.640260 (+ 0.001387) getpid() =
>>>> 32353 <0.000077>
>>>> 15:13:31.640558 (+ 0.000256) fstat64(33,
>>>> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>>>> <0.000119>
>>>> 15:13:31.641106 (+ 0.000556) write(33,
>>>> "32353\n", 6) = 6 <0.000127>
>>>> 15:13:31.641472 (+ 0.000362) close(33) = 0
>>>> <0.000519>
>>>> 15:13:31.642216 (+ 0.000758)
>>>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> 0644) = 0 <0.000152>
>>>> 15:13:31.642900 (+ 0.000679)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=643020294}) = 0
>>>> <0.000056>
>>>> 15:13:31.643495 (+ 0.000590) write(2,
>>>> "[14/May/2018:15:13:31.643020294 "..., 134)
=
>>>> 134 <0.002697>
>>>> 15:13:31.646515 (+ 0.003052)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=646694394}) = 0
>>>> <0.000075>
>>>> 15:13:31.646892 (+ 0.000337) write(4,
>>>> "[14/May/2018:15:13:31.646694394 "..., 134)
=
>>>> 134 <0.000522>
>>>> 15:13:31.647841 (+ 0.000973) fsync(4) = 0
>>>> <0.005967>
>>>> 15:13:31.654425 (+ 0.006617)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=654598946}) = 0
>>>> <0.000253>
>>>> 15:13:31.655137 (+ 0.000717) write(2,
>>>> "[14/May/2018:15:13:31.654598946 "..., 136)
=
>>>> 136 <0.002427>
>>>> 15:13:31.658312 (+ 0.003165)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=658486117}) = 0
>>>> <0.000251>
>>>> 15:13:31.659032 (+ 0.000682) write(4,
>>>> "[14/May/2018:15:13:31.658486117 "..., 136)
=
>>>> 136 <0.000346>
>>>> 15:13:31.659623 (+ 0.000595) fsync(4) = 0
>>>> <0.003311>
>>>> 15:13:31.663230 (+ 0.003642) getpid() =
>>>> 32353 <0.000045>
>>>> 15:13:31.663732 (+ 0.000454)
>>>> socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) =
>>>> 33 <0.000296>
>>>> 15:13:31.664760 (+ 0.001048) getsockopt(33,
>>>> SOL_SOCKET, SO_SNDBUF, [163840], [4]) = 0
>>>> <0.000108>
>>>> 15:13:31.665141 (+ 0.000386) setsockopt(33,
>>>> SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = -1
>>>> EPERM (Operation not permitted) <0.000051>
>>>> 15:13:31.665500 (+ 0.000334) setsockopt(33,
>>>> SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0
<0.000229>
>>>> 15:13:31.665973 (+ 0.000468) sendmsg(33,
>>>> {msg_name={sa_family=AF_UNIX,
>>>> sun_path="/run/systemd/notify"},
>>>> msg_namelen=21,
>>>> msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>> started: Re"..., iov_len=69}], msg_iovlen=1,
>>>> msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) =
>>>> 69 <0.000600>
>>>> 15:13:31.667270 (+ 0.001334) close(33) = 0
>>>> <0.003140>
>>>> 15:13:31.677320 (+ 0.010146)
>>>> futex(0xb31122e8, FUTEX_WAIT, 32357, NULL) = ?
>>>> 15:14:27.661809 (+ 55.984448) +++ killed by
>>>> SIGSEGV (core dumped) +++
>>>>
>>>> On Mon, May 14, 2018 at 10:27 AM, thierry
>>>> bordaz <tbordaz(a)redhat.com
>>>> <mailto:tbordaz@redhat.com>> wrote:
>>>>
>>>> Hi Jonathan,
>>>>
>>>> This is weird as the crashing thread stack
>>>> looks truncated (did you copy/paste all of
>>>> it ?)
>>>>
>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>>> #1 0xb6a06b40 in dosprintf () at
>>>> /lib/libnspr4.so
>>>> #2 0x00000000 in None ()
>>>>
>>>> Did you install 389-ds-base-debuginfo ?
>>>> How did you get that backtrace ? from a
>>>> core dumped, pstack ? Can you attach a
>>>> debugger before the crash occurs ?
>>>>
>>>> It looks it crashed soon at startup, could
>>>> it be related to a broken dse.ldif. It
>>>> should exists a dse.ldif.OK, is it possibly
>>>> to try to start with it ?
>>>>
>>>> best regards
>>>> thierry
>>>>
>>>> On 05/12/2018 01:22 AM, Jonathan Vaughn via
>>>> FreeIPA-users wrote:
>>>>> Not sure if it makes a difference... I was
>>>>> looking into this again and realized I had
>>>>> a bunch of messages from gdb telling me to
>>>>> install more debuginfo. I've done that
>>>>> now, here it is again freshly run through
>>>>> gdb
>>>>>
>>>>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>>>>> Copyright (C) 2017 Free Software
>>>>> Foundation, Inc.
>>>>> License GPLv3+: GNU GPL version 3 or later
>>>>> <http://gnu.org/licenses/gpl.html
>>>>> <http://gnu.org/licenses/gpl.html>>
>>>>> This is free software: you are free to
>>>>> change and redistribute it.
>>>>> There is NO WARRANTY, to the extent
>>>>> permitted by law. Type "show
copying"
>>>>> and "show warranty" for details.
>>>>> This GDB was configured as
>>>>> "armv7hl-redhat-linux-gnueabi".
>>>>> Type "show configuration" for
>>>>> configuration details.
>>>>> For bug reporting instructions, please see:
>>>>> <http://www.gnu.org/software/gdb/bugs/
>>>>>
<http://www.gnu.org/software/gdb/bugs/>>;.
>>>>> Find the GDB manual and other
>>>>> documentation resources online at:
>>>>>
<http://www.gnu.org/software/gdb/documentation/
>>>>>
<http://www.gnu.org/software/gdb/documentation/>>;.
>>>>> For help, type "help".
>>>>> Type "apropos word" to search for
commands
>>>>> related to "word"...
>>>>> Reading symbols from
>>>>> /usr/sbin/ns-slapd...Reading symbols from
>>>>>
/usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>>>>> done.
>>>>> ...
>>>>>
>>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>>> #0 0xb67bbf2e in strlen () at
/lib/libc.so.6
>>>>> #1 0xb6a06b40 in dosprintf () at
>>>>> /lib/libnspr4.so
>>>>> #2 0x00000000 in None ()
>>>>>
>>>>>
>>>>>
>>>>> On Tue, May 8, 2018 at 7:52 AM, Rob
>>>>> Crittenden <rcritten(a)redhat.com
>>>>> <mailto:rcritten@redhat.com>>
wrote:
>>>>>
>>>>> Jonathan Vaughn via FreeIPA-users
>>>>> wrote:
>>>>>
>>>>> Still trying to figure this
>>>>> out. It looks like slapd is
>>>>> dying, I thought it was still
>>>>> running for some reason.
>>>>>
>>>>> slapd is dying to segfault.
>>>>> strace of it happening
doesn't
>>>>> seem to reveal much:
>>>>>
>>>>>
>>>>> A stack trace would very much help
>>>>> trying to track down the cause.
>>>>>
>>>>>
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
>>>>>
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-cras...
>>>>>
>>>>> rob
>>>>>
>>>>>
>>>>> 18:32:41.543717 (+
>>>>> 0.000801) openat(AT_FDCWD,
>>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>> O_WRONLY|O_CREAT|O_TRUNC,
>>>>> 0666) = 32
>>>>> 18:32:41.544907 (+
>>>>> 0.001195) getpid() = 16014
>>>>> 18:32:41.545269 (+
>>>>> 0.000329) fstat64(32,
>>>>> {st_mode=S_IFREG|0644,
>>>>> st_size=0, ...}) = 0
>>>>> 18:32:41.545799 (+
>>>>> 0.000536) write(32,
>>>>> "16014\n", 6) = 6
>>>>> 18:32:41.546603 (+
>>>>> 0.000818) close(32) = 0
>>>>> 18:32:41.547061 (+
>>>>> 0.000448)
>>>>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>> 0644) = 0
>>>>> 18:32:41.547741 (+
>>>>> 0.000676)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961,
>>>>> tv_nsec=548030641}) = 0
>>>>> 18:32:41.548324 (+
>>>>> 0.000587) write(2,
>>>>>
"[07/May/2018:18:32:41.548030641
>>>>> "..., 134) = 134
>>>>> 18:32:41.551096 (+
>>>>> 0.002840)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961,
>>>>> tv_nsec=551287555}) = 0
>>>>> 18:32:41.551568 (+
>>>>> 0.000406) write(4,
>>>>>
"[07/May/2018:18:32:41.551287555
>>>>> "..., 134) = 134
>>>>> 18:32:41.552360 (+
>>>>> 0.000811) fsync(4) = 0
>>>>> 18:32:41.558499 (+
>>>>> 0.006170)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961,
>>>>> tv_nsec=558678099}) = 0
>>>>> 18:32:41.558901 (+
>>>>> 0.000350) write(2,
>>>>>
"[07/May/2018:18:32:41.558678099
>>>>> "..., 136) = 136
>>>>> 18:32:41.561537 (+
>>>>> 0.002680)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961,
>>>>> tv_nsec=561718659}) = 0
>>>>> 18:32:41.562357 (+
>>>>> 0.000793) write(4,
>>>>>
"[07/May/2018:18:32:41.561718659
>>>>> "..., 136) = 136
>>>>> 18:32:41.563293 (+
>>>>> 0.001148) fsync(4) = 0
>>>>> 18:32:41.566928 (+
>>>>> 0.003452) getpid() = 16014
>>>>> 18:32:41.567712 (+
>>>>> 0.000752) socket(AF_UNIX,
>>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>>>> 18:32:41.568628 (+
>>>>> 0.000912) getsockopt(32,
>>>>> SOL_SOCKET, SO_SNDBUF,
>>>>> [163840], [4]) = 0
>>>>> 18:32:41.568972 (+
>>>>> 0.000319) setsockopt(32,
>>>>> SOL_SOCKET, SO_SNDBUFFORCE,
>>>>> [8388608], 4) = -1 EPERM
>>>>> (Operation not permitted)
>>>>> 18:32:41.569548 (+
>>>>> 0.000589) setsockopt(32,
>>>>> SOL_SOCKET, SO_SNDBUF,
>>>>> [8388608], 4) = 0
>>>>> 18:32:41.570064 (+
>>>>> 0.000513) sendmsg(32,
>>>>> {msg_name={sa_family=AF_UNIX,
>>>>>
sun_path="/run/systemd/notify"},
>>>>> msg_namelen=21,
>>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>>> started: Re"...,
iov_len=69}],
>>>>> msg_iovlen=1,
>>>>> msg_controllen=0,
>>>>> msg_flags=0}, MSG_NOSIGNAL) = 69
>>>>> 18:32:41.570845 (+
>>>>> 0.000789) close(32) = 0
>>>>> 18:32:41.576358 (+
>>>>> 0.005575) futex(0xb30ed2e8,
>>>>> FUTEX_WAIT, 16016, NULL) = ?
>>>>> 18:33:01.730774 (+
>>>>> 20.154428) +++ killed by
>>>>> SIGSEGV +++
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>> <mailto:freeipa-users@lists.fedorahosted.org>
>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
>>> <mailto:freeipa-users-leave@lists.fedorahosted.org>
>>
>>
>>
>
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorah...
4:16 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
I've never used COPR. I've dabbled with RPMs in the past but that was...
CentOS 6 I think, and I wasn't making source code changes so much as just
copying and pasting SRPMs from another RPM platform to build for CentOS,
using the regular rpmbuild stuff.
I did actually try just copying the configure line (manually expanding the
various vars) from the SRPM and just doing an old fashioned make, but the
first try bailed shy of an hour at linking with some error about not having
-fPIC specified, so I restarted it with -fPIC to see if it would at least
build. So if I should still try the code changes, if that build finishes I
can try it (I was planning to just copy the respective binaries on top of
the installed ones), but if it turned out the suggested code changes were
pointless then there's no point waiting for it to finish building.
On Thu, May 17, 2018 at 4:11 PM, Rob Crittenden <rcritten(a)redhat.com> wrote:
Jonathan Vaughn via FreeIPA-users wrote:
> Oops, hit reply instead of reply-all
>
> NSPR RPMs
>
> # yum list installed nspr*
> Installed Packages
> nspr.armv7hl
> 4.19.0-1.fc27
> @updates
> nspr-debuginfo.armv7hl
> 4.19.0-1.fc27
> @updates-debuginfo
> nspr-debugsource.armv7hl
> 4.19.0-1.fc27
> @updates-debuginfo
> nspr-devel.armv7hl
> 4.19.0-1.fc27
> @updates
>
> As for building it with the changes you proposed previously - should I
> not bother? I realized last night that the last and only time I built
> anything *on* a Pi was a Pi Zero and that's why it was so terribly slow.
> This Pi 3 ain't fast but it's fast enough to build stuff in far less
> than half a day (unlike the Zero)...
If you are handy with rpms you could build it in copr.
rob
>
>
> On Thu, May 17, 2018 at 5:07 AM, thierry bordaz <tbordaz(a)redhat.com
> <mailto:tbordaz@redhat.com>> wrote:
>
>
>
> On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
>> I've been just using the packages from Fedora. I can build it
>> potentially but I don't have a cross build environment set up at
>> the moment. From experience I'd want to do that first because
>> building anything on the Pi usually takes ages.
>>
>> I'd been "redacting" the hostnames but I'll stop bothering
since
>> it looks like we're getting far enough into the weeds now that the
>> difference in string lengths after "redacting" might actually be
a
>> red herring.
>>
>> (gdb) p *agmt
>> $1 = {hostname = 0x1ef9be0 "ipa-12.creatuity.internal", port =
>> 389, transport_flags = 0, binddn = 0x1e8f650 "", creds =
>> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
>> frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
>> frac_attr_total_defined = 1, schedule = 0x1a7f0c0, auto_initialize
>> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
>> long_name = 0x1a7f100
"agmt=\"cn=meToipa-12.creatuity.internal\"
>> (ipa-12:5)", protocol = 0x19c2930, changecounters = 0x186d180,
>> num_changecounters = 0,
>> max_changecounters = 256, last_update_start_time = 1526500697,
>> last_update_end_time = 1526500697,
>> last_update_status = "Error (0) Replica acquired successfully:
>> Incremental update succeeded", '\000' <repeats 954
times>,
>> update_in_progress = 0, is_enabled = 1,
>> last_init_start_time = 0, last_init_end_time = 0,
>> last_init_status = '\000' <repeats 1023 times>, lock =
0x1ee3740,
>> consumerRUV = 0x1f14e50,
>> consumerSchemaCSN = 0x317c520, consumerRID = 4, tmpConsumerRID =
>> 0, timeout = 120, stop_in_progress = 0, busywaittime = 0,
>> pausetime = 0, priv = 0x0,
>> attrs_to_strip = 0x1ef9ba0, agreement_type = 0, protocol_timeout
>> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000,
>> flowControlPause = 2000, ignoreMissingChange = 0,
>> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
>> (gdb) p *agmt->replarea
>> $2 = {flag = 15 '\017', udn = 0x1efce80
>> "dc=creatuity,dc=internal", dn = 0x1ef9460
>> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
>> "dc=creatuity,dc=internal", ndn_len = 24}
>> (gdb) p *agmt->rdn
>> $3 = {flag = 0 '\000', rdn = 0x19c2840
>> "cn=meToipa-12.creatuity.internal", rdns = 0x0, butcheredupto =
>> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
>>
>> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
>> /usr/include/inttypes.h:# define PRId64 __PRI64_PREFIX
"d"
>> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
>> /usr/include/inttypes.h:# define PRIu16 "u"
>>
>>
>>
>> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
>> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>> wrote:
>>
>>
>>
>> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>>> The installed version of 389* is 1.3.7.10-1.fc27 for armv7hl,
>>> which appears to be the latest available version.
>>
>> Perhaps something is off with the inttypes on Raspberry. Are
>> you building this yourself on Raspberry? Can we make code
>> changes and compile/install them?
>>
>> Before we do that though, in gdb can you run these commands in
>> the same gdb frame:
>>
>> (gdb) p *agmt->replarea
>> (gdb) p *agmt->rdn
>>
>>
>> Then do:
>>
>> # grep -r PRId64 /usr/include/*
>> # grep -r PRIu16 /usr/include/*
>>
>>
>> So if you can compile the source, then change this line in
>> ldap/servers/plugins/replication/repl5_agmt.c:3036, but don't
>> do this yet until you get me the info I just requested.
>>
>> From:
>>
>> agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%"
>> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replarea),
>>
>> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>> agmt->hostname,
>> agmt->port,
>> agmt->consumerRID, maxcsn);
>>
>> To:
>>
>> agmt->maxcsn =
>> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>> slapi_sdn_get_dn(agmt->replarea),
>>
>> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>> agmt->hostname,
>>
>> (long)agmt->port, (int)agmt->consumerRID, maxcsn);
>>
>>
>> Thanks,
>> Mark
>>
>>>
>>> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
>>> <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>>
wrote:
>>
>
> The agreement structure looks valid to me. it should not lead to a
> crash.
>
> What looks weird to me is the order of the arguments of cvt_s.
> It is called: rv = cvt_s(ss, u.s, width, prec, flags);
> But the crashing thread shows the opposite order: flags, prec,width,
> str, ss
>
> The others frame do not show this change of order.
> Also 'str=4' that would be a meaningful value for 'prec=4'.
>
> From debug perspective I only imagine disassemble the two last
> frames to confirm parameters.
> What are the nspr rpms ?
>
>
>
>
>>> (gdb) up
>>> #1 0xb6926b40 in cvt_s (flags=0, prec=<optimized out>,
>>> width=0, str=0x4 <error: Cannot access memory at address
>>> 0x4>, ss=<optimized out>)
>>> at ../../.././nspr/pr/src/io/prprf.c:374
>>> 374 slen = strlen(str);
>>> (gdb) up
>>> #2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2
"",
>>> fmt@entry=0xb34da770 <agmt_set> "\360\317p\002",
ap=...)
>>> at ../../.././nspr/pr/src/io/prprf.c:1018
>>> 1018 rv = cvt_s(ss, u.s, width, prec,
flags);
>>> (gdb) up
>>> #3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770
>>> <agmt_set> "\360\317p\002", ap=...,
ap@entry=...) at
>>> ../../.././nspr/pr/src/io/prprf.c:1184
>>> 1184 rv = dosprintf(&ss, fmt, ap);
>>> (gdb) up
>>> #4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0
>>> "%s;%s;%s;%ld;%d;%s") at
ldap/servers/slapd/ch_malloc.
c:331
>>> 331 p = PR_vsmprintf(fmt, ap);
>>> (gdb) up
>>> #5 0xb34625a4 in agmt_update_maxcsn
>>> (r=r@entry=0x2737810, sdn=0x38d9a40, sdn@entry=0x10,
>>> op=<optimized out>, mods=0x10, mods@entry=0x27d0100,
>>> csn=csn@entry=0x38de350)
>>> at ldap/servers/plugins/replication/repl5_agmt.c:3036
>>> 3036 agmt->maxcsn =
>>> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>> slapi_sdn_get_dn(agmt->replarea),
>>> (gdb) p *agmt
>>> $1 = {hostname = 0x2757be0 "ipa-12.company.internal",
>>> port = 389, transport_flags = 0, binddn = 0x26ed650
"",
>>> creds = 0x26ed7a0, bindmethod = 3, replarea = 0x2757480,
>>> frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
>>> frac_attr_total_defined = 1, schedule = 0x22dd0c0,
>>> auto_initialize = 502, dn = 0x2756d00, rdn = 0x2756c20,
>>> long_name = 0x22dd100
>>> "agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)",
>>> protocol = 0x2220930, changecounters = 0x20cb180,
>>> num_changecounters = 0,
>>> max_changecounters = 256, last_update_start_time =
>>> 1526499214, last_update_end_time = 1526499214,
>>> last_update_status = "Error (0) Replica acquired
>>> successfully: Incremental update succeeded",
'\000'
>>> <repeats 954 times>, update_in_progress = 0, is_enabled =
1,
>>> last_init_start_time = 0, last_init_end_time = 0,
>>> last_init_status = '\000' <repeats 1023 times>,
lock =
>>> 0x2741740, consumerRUV = 0x2772e50,
>>> consumerSchemaCSN = 0x39da520, consumerRID = 4,
>>> tmpConsumerRID = 0, timeout = 120, stop_in_progress = 0,
>>> busywaittime = 0, pausetime = 0, priv = 0x0,
>>> attrs_to_strip = 0x2757ba0, agreement_type = 0,
>>> protocol_timeout = 0x26ed5f0, maxcsn = 0x0,
>>> flowControlWindow = 1000, flowControlPause = 2000,
>>> ignoreMissingChange = 0,
>>> attr_lock = 0x2757c20, WaitForAsyncResults = 100}
>>>
>>>
>>> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
>>> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>>
wrote:
>>>
>>> This looks really familiar and I thought it was
>>> fixed. It should have been fixed in 1.3.7.10-1
>>> (https://pagure.io/389-ds-base/issue/49618
>>> <https://pagure.io/389-ds-base/issue/49618>). In
>>> your debug session go "up" into
agmt_maxcsn_update()
>>> and do:
>>>
>>> (gdb) p *agmt
>>>
>>> Then send us that output please.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>> On 05/15/2018 05:29 PM, Jonathan Vaughn via
>>> FreeIPA-users wrote:
>>>> Here is a backtrace from live gdb after the
>>>> segfault. Looks like things went wrong somewhere
>>>> during in the replication code ?
>>>>
>>>> Thread 36 "ns-slapd" received signal SIGSEGV,
>>>> Segmentation fault.
>>>> [Switching to Thread 0x9e0bc280 (LWP 4662)]
>>>> strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
>>>> 142 ldrd data1a, data1b, [src]
>>>> (gdb) bt
>>>> #0 0xb6728f2e in strlen () at
>>>> ../sysdeps/arm/armv6t2/strlen.S:142
>>>> #1 0xb6973b40 in cvt_s (flags=0, prec=<optimized
>>>> out>, width=0, str=0x4 <error: Cannot access
memory
>>>> at address 0x4>, ss=<optimized out>)
>>>> at ../../.././nspr/pr/src/io/prprf.c:374
>>>> #2 0xb6973b40 in dosprintf (ss=ss@entry=0x9e0bb4bc,
>>>> fmt=0xb34fddf2 "", fmt@entry=0xb3527770
<agmt_set>
>>>> "\360\357\305\002", ap=...)
>>>> at ../../.././nspr/pr/src/io/prprf.c:1018
>>>> #3 0xb6973c8c in PR_vsmprintf
>>>> (fmt=fmt@entry=0xb3527770 <agmt_set>
>>>> "\360\357\305\002", ap=..., ap@entry=...) at
>>>> ../../.././nspr/pr/src/io/prprf.c:1184
>>>> #4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0
>>>> "%s;%s;%s;%ld;%d;%s") at
>>>> ldap/servers/slapd/ch_malloc.c:331
>>>> #5 0xb34af5a4 in agmt_update_maxcsn
>>>> (r=r@entry=0x2c89810, sdn=0x3e2bac0, sdn@entry=0x10,
>>>> op=<optimized out>, mods=0x10,
mods@entry=0x3eec100,
>>>> csn=csn@entry=0x3e30220)
>>>> at
>>>> ldap/servers/plugins/replication/repl5_agmt.c:3036
>>>> #6 0xb34bd424 in write_changelog_and_ruv
>>>> (pb=pb@entry=0x2cb54a0) at
>>>> ldap/servers/plugins/replication/repl5_plugins.c:1124
>>>> #7 0xb34be7ec in multimaster_be_betxnpostop_add
>>>> (pb=0x2cb54a0) at
>>>> ldap/servers/plugins/replication/repl5_plugins.c:855
>>>> #8 0xb34be880 in multimaster_mmr_postop
>>>> (pb=<optimized out>, flags=560) at
>>>> ldap/servers/plugins/replication/repl5_plugins.c:616
>>>> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop
>>>> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0,
>>>> flags=flags@entry=560) at
>>>> ldap/servers/slapd/plugin_mmr.c:65
>>>> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
>>>> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>>>> #11 0xb6e2ce4c in op_shared_add
>>>> (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:679
>>>> #12 0xb6e2d35c in add_internal_pb
>>>> (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:407
>>>> #13 0xb6e2e0f8 in slapi_add_internal_pb
>>>> (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:332
>>>> #14 0xb368db50 in ipa_topo_util_segment_write
>>>> (tconf=tconf@entry=0x2cb5860,
>>>> tsegm=tsegm@entry=0x3f2d9a0) at topology_util.c:1251
>>>> #15 0xb368e01c in ipa_topo_util_update_agmt_list
>>>> (conf=0x2cb5860, repl_segments=<optimized out>)
at
>>>> topology_util.c:696
>>>> #16 0xb368891c in ipa_topo_apply_shared_config () at
>>>> topology_init.c:165
>>>> #17 0xb6e4e65c in eq_call_all () at
>>>> ldap/servers/slapd/eventq.c:278
>>>> #18 0xb6e4e65c in eq_loop (arg=<optimized out>)
at
>>>> ldap/servers/slapd/eventq.c:323
>>>> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
>>>> ../../.././nspr/pr/src/pthreads/ptthread.c:201
>>>> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at
>>>> pthread_create.c:465
>>>> #21 0xb6785da8 in None () at
>>>> ../sysdeps/unix/sysv/linux/arm/clone.S:73
>>>>
>>>>
>>>> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz
>>>> <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>>
wrote:
>>>>
>>>> Hi Jonathan,
>>>>
>>>> This problem looks new to me and has something
>>>> specific to your environment.
>>>> I think the best approach is to continue to
>>>> debug on your system if you have the possibility
>>>> to do so.
>>>>
>>>> From strace we can see that DS started smoothly
>>>> (created its pid file then notified systemd it
>>>> was running fine). According to the pstack
>>>> nunc-stans was running and was able to accept
>>>> network events even if it appears it detected no
>>>> incoming connection.
>>>> So the server should be ready to serve for some
>>>> seconds (more than a minute), then it crashed
>>>> with one thread dereferencing likely a wrong
>>>> pointer.
>>>>
>>>> Could you attach a debugger when the server is
>>>> started and wait for the sigsegv to occur. Then
>>>> confirm the crashing thread backstack.
>>>> If it is confirmed, I am afraid this is a stack
>>>> corruption and valgrind could help
>>>> (http://www.port389.org/docs/
389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>>>> <http://www.port389.org/docs/
389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind>).
>>>>
>>>> best regards
>>>> thierry
>>>>
>>>>
>>>> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
>>>>> Here's a strace from before it dies. Most
of
>>>>> the elapsed time is it waiting on some futex
>>>>> call it looks like near the end, when it
>>>>> finally "returns" (from lack of strace
output
>>>>> for duration of call I assume it didn't
>>>>> actually return but SIGSEGV in that call) and
>>>>> strace prints ' = ?' on the futex it
then
>>>>> immediately reports SIGSEGV after. So maybe the
>>>>> problem is that futex call, which may mean the
>>>>> problem is not directly in 389DS / FreeIPA
itself?
>>>>>
>>>>>
>>>>>
>>>>> 15:13:31.626587 (+ 0.000630) listen(8, 128)
>>>>> = 0 <0.000068>
>>>>> 15:13:31.626857 (+ 0.000235) listen(9, 128)
>>>>> = 0 <0.000048>
>>>>> 15:13:31.627111 (+ 0.000251)
>>>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>>>> tv_nsec=41120614}) = 0 <0.000085>
>>>>> 15:13:31.627457 (+ 0.000356)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811, tv_nsec=627560772}) = 0
>>>>> <0.000043>
>>>>> 15:13:31.631233 (+ 0.003839)
>>>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>>>> tv_nsec=45286796}) = 0 <0.000077>
>>>>> 15:13:31.631720 (+ 0.000427)
>>>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>>>> tv_nsec=45661430}) = 0 <0.000042>
>>>>> 15:13:31.631955 (+ 0.000220)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811, tv_nsec=632049036}) = 0
>>>>> <0.000047>
>>>>> 15:13:31.635669 (+ 0.003785)
>>>>> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
>>>>> tv_nsec=49725840}) = 0 <0.000146>
>>>>> 15:13:31.636484 (+ 0.000784) write(16,
"a",
>>>>> 1) = 1 <0.000118>
>>>>> 15:13:31.636855 (+ 0.000341) sched_yield()
>>>>> = 0 <0.000252>
>>>>> 15:13:31.637322 (+ 0.000470)
>>>>> futex(0x1cb57a0, FUTEX_WAKE_PRIVATE, 1) = 1
>>>>> <0.000088>
>>>>> 15:13:31.637897 (+ 0.000610) write(16,
"a",
>>>>> 1) = 1 <0.000221>
>>>>> 15:13:31.638394 (+ 0.000467) sched_yield()
>>>>> = 0 <0.000047>
>>>>> 15:13:31.638619 (+ 0.000202)
>>>>> futex(0x1cb5710, FUTEX_WAKE_PRIVATE, 1) = 1
>>>>> <0.000065>
>>>>> 15:13:31.638908 (+ 0.000298)
>>>>> openat(AT_FDCWD,
>>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
<0.000831>
>>>>> 15:13:31.640260 (+ 0.001387) getpid() =
>>>>> 32353 <0.000077>
>>>>> 15:13:31.640558 (+ 0.000256) fstat64(33,
>>>>> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>>>>> <0.000119>
>>>>> 15:13:31.641106 (+ 0.000556) write(33,
>>>>> "32353\n", 6) = 6 <0.000127>
>>>>> 15:13:31.641472 (+ 0.000362) close(33) = 0
>>>>> <0.000519>
>>>>> 15:13:31.642216 (+ 0.000758)
>>>>> chmod("/var/run/dirsrv/slapd-
COMPANY-INTERNAL.pid",
>>>>> 0644) = 0 <0.000152>
>>>>> 15:13:31.642900 (+ 0.000679)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811, tv_nsec=643020294}) = 0
>>>>> <0.000056>
>>>>> 15:13:31.643495 (+ 0.000590) write(2,
>>>>> "[14/May/2018:15:13:31.643020294 "...,
134) =
>>>>> 134 <0.002697>
>>>>> 15:13:31.646515 (+ 0.003052)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811, tv_nsec=646694394}) = 0
>>>>> <0.000075>
>>>>> 15:13:31.646892 (+ 0.000337) write(4,
>>>>> "[14/May/2018:15:13:31.646694394 "...,
134) =
>>>>> 134 <0.000522>
>>>>> 15:13:31.647841 (+ 0.000973) fsync(4) = 0
>>>>> <0.005967>
>>>>> 15:13:31.654425 (+ 0.006617)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811, tv_nsec=654598946}) = 0
>>>>> <0.000253>
>>>>> 15:13:31.655137 (+ 0.000717) write(2,
>>>>> "[14/May/2018:15:13:31.654598946 "...,
136) =
>>>>> 136 <0.002427>
>>>>> 15:13:31.658312 (+ 0.003165)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811, tv_nsec=658486117}) = 0
>>>>> <0.000251>
>>>>> 15:13:31.659032 (+ 0.000682) write(4,
>>>>> "[14/May/2018:15:13:31.658486117 "...,
136) =
>>>>> 136 <0.000346>
>>>>> 15:13:31.659623 (+ 0.000595) fsync(4) = 0
>>>>> <0.003311>
>>>>> 15:13:31.663230 (+ 0.003642) getpid() =
>>>>> 32353 <0.000045>
>>>>> 15:13:31.663732 (+ 0.000454)
>>>>> socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) =
>>>>> 33 <0.000296>
>>>>> 15:13:31.664760 (+ 0.001048) getsockopt(33,
>>>>> SOL_SOCKET, SO_SNDBUF, [163840], [4]) = 0
>>>>> <0.000108>
>>>>> 15:13:31.665141 (+ 0.000386) setsockopt(33,
>>>>> SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = -1
>>>>> EPERM (Operation not permitted)
<0.000051>
>>>>> 15:13:31.665500 (+ 0.000334) setsockopt(33,
>>>>> SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0
<0.000229>
>>>>> 15:13:31.665973 (+ 0.000468) sendmsg(33,
>>>>> {msg_name={sa_family=AF_UNIX,
>>>>> sun_path="/run/systemd/notify"},
>>>>> msg_namelen=21,
>>>>> msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>>> started: Re"..., iov_len=69}],
msg_iovlen=1,
>>>>> msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) =
>>>>> 69 <0.000600>
>>>>> 15:13:31.667270 (+ 0.001334) close(33) = 0
>>>>> <0.003140>
>>>>> 15:13:31.677320 (+ 0.010146)
>>>>> futex(0xb31122e8, FUTEX_WAIT, 32357, NULL) = ?
>>>>> 15:14:27.661809 (+ 55.984448) +++ killed by
>>>>> SIGSEGV (core dumped) +++
>>>>>
>>>>> On Mon, May 14, 2018 at 10:27 AM, thierry
>>>>> bordaz <tbordaz(a)redhat.com
>>>>> <mailto:tbordaz@redhat.com>> wrote:
>>>>>
>>>>> Hi Jonathan,
>>>>>
>>>>> This is weird as the crashing thread stack
>>>>> looks truncated (did you copy/paste all of
>>>>> it ?)
>>>>>
>>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>>> #0 0xb67bbf2e in strlen () at
/lib/libc.so.6
>>>>> #1 0xb6a06b40 in dosprintf () at
>>>>> /lib/libnspr4.so
>>>>> #2 0x00000000 in None ()
>>>>>
>>>>> Did you install 389-ds-base-debuginfo ?
>>>>> How did you get that backtrace ? from a
>>>>> core dumped, pstack ? Can you attach a
>>>>> debugger before the crash occurs ?
>>>>>
>>>>> It looks it crashed soon at startup, could
>>>>> it be related to a broken dse.ldif. It
>>>>> should exists a dse.ldif.OK, is it possibly
>>>>> to try to start with it ?
>>>>>
>>>>> best regards
>>>>> thierry
>>>>>
>>>>> On 05/12/2018 01:22 AM, Jonathan Vaughn via
>>>>> FreeIPA-users wrote:
>>>>>> Not sure if it makes a difference... I
was
>>>>>> looking into this again and realized I
had
>>>>>> a bunch of messages from gdb telling me
to
>>>>>> install more debuginfo. I've done
that
>>>>>> now, here it is again freshly run
through
>>>>>> gdb
>>>>>>
>>>>>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>>>>>> Copyright (C) 2017 Free Software
>>>>>> Foundation, Inc.
>>>>>> License GPLv3+: GNU GPL version 3 or
later
>>>>>> <http://gnu.org/licenses/gpl.html
>>>>>>
<http://gnu.org/licenses/gpl.html>>
>>>>>> This is free software: you are free to
>>>>>> change and redistribute it.
>>>>>> There is NO WARRANTY, to the extent
>>>>>> permitted by law. Type "show
copying"
>>>>>> and "show warranty" for
details.
>>>>>> This GDB was configured as
>>>>>>
"armv7hl-redhat-linux-gnueabi".
>>>>>> Type "show configuration" for
>>>>>> configuration details.
>>>>>> For bug reporting instructions, please
see:
>>>>>>
<http://www.gnu.org/software/gdb/bugs/
>>>>>>
<http://www.gnu.org/software/gdb/bugs/>>;.
>>>>>> Find the GDB manual and other
>>>>>> documentation resources online at:
>>>>>> <http://www.gnu.org/software/
gdb/documentation/
>>>>>> <http://www.gnu.org/software/
gdb/documentation/>>.
>>>>>> For help, type "help".
>>>>>> Type "apropos word" to search
for commands
>>>>>> related to "word"...
>>>>>> Reading symbols from
>>>>>> /usr/sbin/ns-slapd...Reading symbols
from
>>>>>> /usr/lib/debug/usr/sbin/ns-
slapd-1.3.7.10-1.fc27.arm.debug...done.
>>>>>> done.
>>>>>> ...
>>>>>>
>>>>>> Thread 1 (Thread 0x9e13c280 (LWP
17245)):
>>>>>> #0 0xb67bbf2e in strlen () at
/lib/libc.so.6
>>>>>> #1 0xb6a06b40 in dosprintf () at
>>>>>> /lib/libnspr4.so
>>>>>> #2 0x00000000 in None ()
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 8, 2018 at 7:52 AM, Rob
>>>>>> Crittenden <rcritten(a)redhat.com
>>>>>>
<mailto:rcritten@redhat.com>> wrote:
>>>>>>
>>>>>> Jonathan Vaughn via
FreeIPA-users
>>>>>> wrote:
>>>>>>
>>>>>> Still trying to figure this
>>>>>> out. It looks like slapd is
>>>>>> dying, I thought it was
still
>>>>>> running for some reason.
>>>>>>
>>>>>> slapd is dying to segfault.
>>>>>> strace of it happening
doesn't
>>>>>> seem to reveal much:
>>>>>>
>>>>>>
>>>>>> A stack trace would very much
help
>>>>>> trying to track down the cause.
>>>>>>
>>>>>> http://directory.
fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
>>>>>> <http://directory.
fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes>
>>>>>>
>>>>>> rob
>>>>>>
>>>>>>
>>>>>> 18:32:41.543717 (+
>>>>>> 0.000801) openat(AT_FDCWD,
>>>>>>
"/var/run/dirsrv/slapd-
COMPANY-INTERNAL.pid",
>>>>>> O_WRONLY|O_CREAT|O_TRUNC,
>>>>>> 0666) = 32
>>>>>> 18:32:41.544907 (+
>>>>>> 0.001195) getpid() = 16014
>>>>>> 18:32:41.545269 (+
>>>>>> 0.000329) fstat64(32,
>>>>>> {st_mode=S_IFREG|0644,
>>>>>> st_size=0, ...}) = 0
>>>>>> 18:32:41.545799 (+
>>>>>> 0.000536) write(32,
>>>>>> "16014\n", 6) = 6
>>>>>> 18:32:41.546603 (+
>>>>>> 0.000818) close(32) = 0
>>>>>> 18:32:41.547061 (+
>>>>>> 0.000448)
>>>>>>
chmod("/var/run/dirsrv/slapd-
COMPANY-INTERNAL.pid",
>>>>>> 0644) = 0
>>>>>> 18:32:41.547741 (+
>>>>>> 0.000676)
>>>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> {tv_sec=1525735961,
>>>>>> tv_nsec=548030641}) = 0
>>>>>> 18:32:41.548324 (+
>>>>>> 0.000587) write(2,
>>>>>>
"[07/May/2018:18:32:41.
548030641
>>>>>> "..., 134) = 134
>>>>>> 18:32:41.551096 (+
>>>>>> 0.002840)
>>>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> {tv_sec=1525735961,
>>>>>> tv_nsec=551287555}) = 0
>>>>>> 18:32:41.551568 (+
>>>>>> 0.000406) write(4,
>>>>>>
"[07/May/2018:18:32:41.
551287555
>>>>>> "..., 134) = 134
>>>>>> 18:32:41.552360 (+
>>>>>> 0.000811) fsync(4) = 0
>>>>>> 18:32:41.558499 (+
>>>>>> 0.006170)
>>>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> {tv_sec=1525735961,
>>>>>> tv_nsec=558678099}) = 0
>>>>>> 18:32:41.558901 (+
>>>>>> 0.000350) write(2,
>>>>>>
"[07/May/2018:18:32:41.
558678099
>>>>>> "..., 136) = 136
>>>>>> 18:32:41.561537 (+
>>>>>> 0.002680)
>>>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> {tv_sec=1525735961,
>>>>>> tv_nsec=561718659}) = 0
>>>>>> 18:32:41.562357 (+
>>>>>> 0.000793) write(4,
>>>>>>
"[07/May/2018:18:32:41.
561718659
>>>>>> "..., 136) = 136
>>>>>> 18:32:41.563293 (+
>>>>>> 0.001148) fsync(4) = 0
>>>>>> 18:32:41.566928 (+
>>>>>> 0.003452) getpid() = 16014
>>>>>> 18:32:41.567712 (+
>>>>>> 0.000752) socket(AF_UNIX,
>>>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0)
= 32
>>>>>> 18:32:41.568628 (+
>>>>>> 0.000912) getsockopt(32,
>>>>>> SOL_SOCKET, SO_SNDBUF,
>>>>>> [163840], [4]) = 0
>>>>>> 18:32:41.568972 (+
>>>>>> 0.000319) setsockopt(32,
>>>>>> SOL_SOCKET, SO_SNDBUFFORCE,
>>>>>> [8388608], 4) = -1 EPERM
>>>>>> (Operation not permitted)
>>>>>> 18:32:41.569548 (+
>>>>>> 0.000589) setsockopt(32,
>>>>>> SOL_SOCKET, SO_SNDBUF,
>>>>>> [8388608], 4) = 0
>>>>>> 18:32:41.570064 (+
>>>>>> 0.000513) sendmsg(32,
>>>>>>
{msg_name={sa_family=AF_UNIX,
>>>>>>
sun_path="/run/systemd/notify"
},
>>>>>> msg_namelen=21,
>>>>>>
msg_iov=[{iov_base="READY=1\
nSTATUS=slapd
>>>>>> started: Re"...,
iov_len=69}],
>>>>>> msg_iovlen=1,
>>>>>> msg_controllen=0,
>>>>>> msg_flags=0}, MSG_NOSIGNAL)
= 69
>>>>>> 18:32:41.570845 (+
>>>>>> 0.000789) close(32) = 0
>>>>>> 18:32:41.576358 (+
>>>>>> 0.005575)
futex(0xb30ed2e8,
>>>>>> FUTEX_WAIT, 16016, NULL) =
?
>>>>>> 18:33:01.730774 (+
>>>>>> 20.154428) +++ killed by
>>>>>> SIGSEGV +++
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users@lists.
fedorahosted.org
>>>> <mailto:freeipa-users@lists.fedorahosted.org>
>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
>>>>
<mailto:freeipa-users-leave@lists.fedorahosted.org>
>>>
>>>
>>>
>>
>>
>
>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.
fedorahosted.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/freeipa-
users(a)lists.fedorahosted.org/message/NNFFJQTAJ2MG52U23V6QYOOAFL722JJL/
>
6:16 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Welp, I'm still getting the -fPIC error ... so I may need to figure out how
to do it with COPR or something.
On Thu, May 17, 2018 at 4:16 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
I've never used COPR. I've dabbled with RPMs in the past but
that was...
CentOS 6 I think, and I wasn't making source code changes so much as just
copying and pasting SRPMs from another RPM platform to build for CentOS,
using the regular rpmbuild stuff.
I did actually try just copying the configure line (manually expanding the
various vars) from the SRPM and just doing an old fashioned make, but the
first try bailed shy of an hour at linking with some error about not having
-fPIC specified, so I restarted it with -fPIC to see if it would at least
build. So if I should still try the code changes, if that build finishes I
can try it (I was planning to just copy the respective binaries on top of
the installed ones), but if it turned out the suggested code changes were
pointless then there's no point waiting for it to finish building.
On Thu, May 17, 2018 at 4:11 PM, Rob Crittenden <rcritten(a)redhat.com>
wrote:
> Jonathan Vaughn via FreeIPA-users wrote:
> > Oops, hit reply instead of reply-all
> >
> > NSPR RPMs
> >
> > # yum list installed nspr*
> > Installed Packages
> > nspr.armv7hl
> > 4.19.0-1.fc27
> > @updates
> > nspr-debuginfo.armv7hl
> > 4.19.0-1.fc27
> > @updates-debuginfo
> > nspr-debugsource.armv7hl
> > 4.19.0-1.fc27
> > @updates-debuginfo
> > nspr-devel.armv7hl
> > 4.19.0-1.fc27
> > @updates
> >
> > As for building it with the changes you proposed previously - should I
> > not bother? I realized last night that the last and only time I built
> > anything *on* a Pi was a Pi Zero and that's why it was so terribly slow.
> > This Pi 3 ain't fast but it's fast enough to build stuff in far less
> > than half a day (unlike the Zero)...
>
> If you are handy with rpms you could build it in copr.
>
> rob
>
> >
> >
> > On Thu, May 17, 2018 at 5:07 AM, thierry bordaz <tbordaz(a)redhat.com
> > <mailto:tbordaz@redhat.com>> wrote:
> >
> >
> >
> > On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
> >> I've been just using the packages from Fedora. I can build it
> >> potentially but I don't have a cross build environment set up at
> >> the moment. From experience I'd want to do that first because
> >> building anything on the Pi usually takes ages.
> >>
> >> I'd been "redacting" the hostnames but I'll stop
bothering since
> >> it looks like we're getting far enough into the weeds now that the
> >> difference in string lengths after "redacting" might actually
be a
> >> red herring.
> >>
> >> (gdb) p *agmt
> >> $1 = {hostname = 0x1ef9be0 "ipa-12.creatuity.internal", port
=
> >> 389, transport_flags = 0, binddn = 0x1e8f650 "", creds =
> >> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
> >> frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
> >> frac_attr_total_defined = 1, schedule = 0x1a7f0c0, auto_initialize
> >> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
> >> long_name = 0x1a7f100
"agmt=\"cn=meToipa-12.creatuity.internal\"
> >> (ipa-12:5)", protocol = 0x19c2930, changecounters = 0x186d180,
> >> num_changecounters = 0,
> >> max_changecounters = 256, last_update_start_time = 1526500697,
> >> last_update_end_time = 1526500697,
> >> last_update_status = "Error (0) Replica acquired successfully:
> >> Incremental update succeeded", '\000' <repeats 954
times>,
> >> update_in_progress = 0, is_enabled = 1,
> >> last_init_start_time = 0, last_init_end_time = 0,
> >> last_init_status = '\000' <repeats 1023 times>, lock =
0x1ee3740,
> >> consumerRUV = 0x1f14e50,
> >> consumerSchemaCSN = 0x317c520, consumerRID = 4, tmpConsumerRID =
> >> 0, timeout = 120, stop_in_progress = 0, busywaittime = 0,
> >> pausetime = 0, priv = 0x0,
> >> attrs_to_strip = 0x1ef9ba0, agreement_type = 0, protocol_timeout
> >> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000,
> >> flowControlPause = 2000, ignoreMissingChange = 0,
> >> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
> >> (gdb) p *agmt->replarea
> >> $2 = {flag = 15 '\017', udn = 0x1efce80
> >> "dc=creatuity,dc=internal", dn = 0x1ef9460
> >> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
> >> "dc=creatuity,dc=internal", ndn_len = 24}
> >> (gdb) p *agmt->rdn
> >> $3 = {flag = 0 '\000', rdn = 0x19c2840
> >> "cn=meToipa-12.creatuity.internal", rdns = 0x0, butcheredupto
=
> >> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
> >>
> >> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
> >> /usr/include/inttypes.h:# define PRId64 __PRI64_PREFIX
"d"
> >> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
> >> /usr/include/inttypes.h:# define PRIu16 "u"
> >>
> >>
> >>
> >> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
> >> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>> wrote:
> >>
> >>
> >>
> >> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
> >>> The installed version of 389* is 1.3.7.10-1.fc27 for armv7hl,
> >>> which appears to be the latest available version.
> >>
> >> Perhaps something is off with the inttypes on Raspberry. Are
> >> you building this yourself on Raspberry? Can we make code
> >> changes and compile/install them?
> >>
> >> Before we do that though, in gdb can you run these commands in
> >> the same gdb frame:
> >>
> >> (gdb) p *agmt->replarea
> >> (gdb) p *agmt->rdn
> >>
> >>
> >> Then do:
> >>
> >> # grep -r PRId64 /usr/include/*
> >> # grep -r PRIu16 /usr/include/*
> >>
> >>
> >> So if you can compile the source, then change this line in
> >> ldap/servers/plugins/replication/repl5_agmt.c:3036, but don't
> >> do this yet until you get me the info I just requested.
> >>
> >> From:
> >>
> >> agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%"
> >> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replarea),
> >>
> >> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
> >> agmt->hostname,
> >> agmt->port,
> >> agmt->consumerRID, maxcsn);
> >>
> >> To:
> >>
> >> agmt->maxcsn =
> >> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> >> slapi_sdn_get_dn(agmt->replarea),
> >>
> >> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
> >> agmt->hostname,
> >>
> >> (long)agmt->port, (int)agmt->consumerRID, maxcsn);
> >>
> >>
> >> Thanks,
> >> Mark
> >>
> >>>
> >>> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
> >>> <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>>
> wrote:
> >>
> >
> > The agreement structure looks valid to me. it should not lead to a
> > crash.
> >
> > What looks weird to me is the order of the arguments of cvt_s.
> > It is called: rv = cvt_s(ss, u.s, width, prec, flags);
> > But the crashing thread shows the opposite order: flags, prec,width,
> > str, ss
> >
> > The others frame do not show this change of order.
> > Also 'str=4' that would be a meaningful value for 'prec=4'.
> >
> > From debug perspective I only imagine disassemble the two last
> > frames to confirm parameters.
> > What are the nspr rpms ?
> >
> >
> >
> >
> >>> (gdb) up
> >>> #1 0xb6926b40 in cvt_s (flags=0, prec=<optimized
out>,
> >>> width=0, str=0x4 <error: Cannot access memory at address
> >>> 0x4>, ss=<optimized out>)
> >>> at ../../.././nspr/pr/src/io/prprf.c:374
> >>> 374 slen = strlen(str);
> >>> (gdb) up
> >>> #2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2
"",
> >>> fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002", ap=...)
> >>> at ../../.././nspr/pr/src/io/prprf.c:1018
> >>> 1018 rv = cvt_s(ss, u.s, width, prec,
> flags);
> >>> (gdb) up
> >>> #3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770
> >>> <agmt_set> "\360\317p\002", ap=...,
ap@entry=...) at
> >>> ../../.././nspr/pr/src/io/prprf.c:1184
> >>> 1184 rv = dosprintf(&ss, fmt, ap);
> >>> (gdb) up
> >>> #4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0
> >>> "%s;%s;%s;%ld;%d;%s") at
ldap/servers/slapd/ch_malloc.c
> :331
> >>> 331 p = PR_vsmprintf(fmt, ap);
> >>> (gdb) up
> >>> #5 0xb34625a4 in agmt_update_maxcsn
> >>> (r=r@entry=0x2737810, sdn=0x38d9a40, sdn@entry=0x10,
> >>> op=<optimized out>, mods=0x10, mods@entry=0x27d0100,
> >>> csn=csn@entry=0x38de350)
> >>> at ldap/servers/plugins/replication/repl5_agmt.c:3036
> >>> 3036 agmt->maxcsn =
> >>> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> >>> slapi_sdn_get_dn(agmt->replarea),
> >>> (gdb) p *agmt
> >>> $1 = {hostname = 0x2757be0
"ipa-12.company.internal",
> >>> port = 389, transport_flags = 0, binddn = 0x26ed650
"",
> >>> creds = 0x26ed7a0, bindmethod = 3, replarea = 0x2757480,
> >>> frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
> >>> frac_attr_total_defined = 1, schedule = 0x22dd0c0,
> >>> auto_initialize = 502, dn = 0x2756d00, rdn = 0x2756c20,
> >>> long_name = 0x22dd100
> >>> "agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)",
> >>> protocol = 0x2220930, changecounters = 0x20cb180,
> >>> num_changecounters = 0,
> >>> max_changecounters = 256, last_update_start_time =
> >>> 1526499214, last_update_end_time = 1526499214,
> >>> last_update_status = "Error (0) Replica acquired
> >>> successfully: Incremental update succeeded",
'\000'
> >>> <repeats 954 times>, update_in_progress = 0,
is_enabled =
> 1,
> >>> last_init_start_time = 0, last_init_end_time = 0,
> >>> last_init_status = '\000' <repeats 1023
times>, lock =
> >>> 0x2741740, consumerRUV = 0x2772e50,
> >>> consumerSchemaCSN = 0x39da520, consumerRID = 4,
> >>> tmpConsumerRID = 0, timeout = 120, stop_in_progress = 0,
> >>> busywaittime = 0, pausetime = 0, priv = 0x0,
> >>> attrs_to_strip = 0x2757ba0, agreement_type = 0,
> >>> protocol_timeout = 0x26ed5f0, maxcsn = 0x0,
> >>> flowControlWindow = 1000, flowControlPause = 2000,
> >>> ignoreMissingChange = 0,
> >>> attr_lock = 0x2757c20, WaitForAsyncResults = 100}
> >>>
> >>>
> >>> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
> >>> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>>
> wrote:
> >>>
> >>> This looks really familiar and I thought it was
> >>> fixed. It should have been fixed in 1.3.7.10-1
> >>> (https://pagure.io/389-ds-base/issue/49618
> >>> <https://pagure.io/389-ds-base/issue/49618>).
In
> >>> your debug session go "up" into
agmt_maxcsn_update()
> >>> and do:
> >>>
> >>> (gdb) p *agmt
> >>>
> >>> Then send us that output please.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>> On 05/15/2018 05:29 PM, Jonathan Vaughn via
> >>> FreeIPA-users wrote:
> >>>> Here is a backtrace from live gdb after the
> >>>> segfault. Looks like things went wrong somewhere
> >>>> during in the replication code ?
> >>>>
> >>>> Thread 36 "ns-slapd" received signal
SIGSEGV,
> >>>> Segmentation fault.
> >>>> [Switching to Thread 0x9e0bc280 (LWP 4662)]
> >>>> strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
> >>>> 142 ldrd data1a, data1b, [src]
> >>>> (gdb) bt
> >>>> #0 0xb6728f2e in strlen () at
> >>>> ../sysdeps/arm/armv6t2/strlen.S:142
> >>>> #1 0xb6973b40 in cvt_s (flags=0,
prec=<optimized
> >>>> out>, width=0, str=0x4 <error: Cannot access
memory
> >>>> at address 0x4>, ss=<optimized out>)
> >>>> at ../../.././nspr/pr/src/io/prprf.c:374
> >>>> #2 0xb6973b40 in dosprintf
(ss=ss@entry=0x9e0bb4bc,
> >>>> fmt=0xb34fddf2 "", fmt@entry=0xb3527770
<agmt_set>
> >>>> "\360\357\305\002", ap=...)
> >>>> at ../../.././nspr/pr/src/io/prprf.c:1018
> >>>> #3 0xb6973c8c in PR_vsmprintf
> >>>> (fmt=fmt@entry=0xb3527770 <agmt_set>
> >>>> "\360\357\305\002", ap=..., ap@entry=...)
at
> >>>> ../../.././nspr/pr/src/io/prprf.c:1184
> >>>> #4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0
> >>>> "%s;%s;%s;%ld;%d;%s") at
> >>>> ldap/servers/slapd/ch_malloc.c:331
> >>>> #5 0xb34af5a4 in agmt_update_maxcsn
> >>>> (r=r@entry=0x2c89810, sdn=0x3e2bac0,
sdn@entry=0x10,
> >>>> op=<optimized out>, mods=0x10,
mods@entry=0x3eec100,
> >>>> csn=csn@entry=0x3e30220)
> >>>> at
> >>>> ldap/servers/plugins/replication/repl5_agmt.c:3036
> >>>> #6 0xb34bd424 in write_changelog_and_ruv
> >>>> (pb=pb@entry=0x2cb54a0) at
> >>>> ldap/servers/plugins/replicat
> ion/repl5_plugins.c:1124
> >>>> #7 0xb34be7ec in multimaster_be_betxnpostop_add
> >>>> (pb=0x2cb54a0) at
> >>>>
ldap/servers/plugins/replication/repl5_plugins.c:855
> >>>> #8 0xb34be880 in multimaster_mmr_postop
> >>>> (pb=<optimized out>, flags=560) at
> >>>>
ldap/servers/plugins/replication/repl5_plugins.c:616
> >>>> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop
> >>>> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0,
> >>>> flags=flags@entry=560) at
> >>>> ldap/servers/slapd/plugin_mmr.c:65
> >>>> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
> >>>> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
> >>>> #11 0xb6e2ce4c in op_shared_add
> >>>> (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:679
> >>>> #12 0xb6e2d35c in add_internal_pb
> >>>> (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:407
> >>>> #13 0xb6e2e0f8 in slapi_add_internal_pb
> >>>> (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:332
> >>>> #14 0xb368db50 in ipa_topo_util_segment_write
> >>>> (tconf=tconf@entry=0x2cb5860,
> >>>> tsegm=tsegm@entry=0x3f2d9a0) at
topology_util.c:1251
> >>>> #15 0xb368e01c in ipa_topo_util_update_agmt_list
> >>>> (conf=0x2cb5860, repl_segments=<optimized
out>) at
> >>>> topology_util.c:696
> >>>> #16 0xb368891c in ipa_topo_apply_shared_config ()
at
> >>>> topology_init.c:165
> >>>> #17 0xb6e4e65c in eq_call_all () at
> >>>> ldap/servers/slapd/eventq.c:278
> >>>> #18 0xb6e4e65c in eq_loop (arg=<optimized
out>) at
> >>>> ldap/servers/slapd/eventq.c:323
> >>>> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
> >>>> ../../.././nspr/pr/src/pthreads/ptthread.c:201
> >>>> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at
> >>>> pthread_create.c:465
> >>>> #21 0xb6785da8 in None () at
> >>>> ../sysdeps/unix/sysv/linux/arm/clone.S:73
> >>>>
> >>>>
> >>>> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz
> >>>> <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>>
> wrote:
> >>>>
> >>>> Hi Jonathan,
> >>>>
> >>>> This problem looks new to me and has something
> >>>> specific to your environment.
> >>>> I think the best approach is to continue to
> >>>> debug on your system if you have the
possibility
> >>>> to do so.
> >>>>
> >>>> From strace we can see that DS started smoothly
> >>>> (created its pid file then notified systemd it
> >>>> was running fine). According to the pstack
> >>>> nunc-stans was running and was able to accept
> >>>> network events even if it appears it detected
no
> >>>> incoming connection.
> >>>> So the server should be ready to serve for some
> >>>> seconds (more than a minute), then it crashed
> >>>> with one thread dereferencing likely a wrong
> >>>> pointer.
> >>>>
> >>>> Could you attach a debugger when the server is
> >>>> started and wait for the sigsegv to occur. Then
> >>>> confirm the crashing thread backstack.
> >>>> If it is confirmed, I am afraid this is a stack
> >>>> corruption and valgrind could help
> >>>> (http://www.port389.org/docs/
> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
> >>>> <http://www.port389.org/docs/
> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind>).
> >>>>
> >>>> best regards
> >>>> thierry
> >>>>
> >>>>
> >>>> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
> >>>>> Here's a strace from before it dies.
Most of
> >>>>> the elapsed time is it waiting on some
futex
> >>>>> call it looks like near the end, when it
> >>>>> finally "returns" (from lack of
strace output
> >>>>> for duration of call I assume it didn't
> >>>>> actually return but SIGSEGV in that call)
and
> >>>>> strace prints ' = ?' on the futex it
then
> >>>>> immediately reports SIGSEGV after. So maybe
the
> >>>>> problem is that futex call, which may mean
the
> >>>>> problem is not directly in 389DS / FreeIPA
> itself?
> >>>>>
> >>>>>
> >>>>>
> >>>>> 15:13:31.626587 (+ 0.000630) listen(8,
128)
> >>>>> = 0 <0.000068>
> >>>>> 15:13:31.626857 (+ 0.000235) listen(9,
128)
> >>>>> = 0 <0.000048>
> >>>>> 15:13:31.627111 (+ 0.000251)
> >>>>> clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932,
> >>>>> tv_nsec=41120614}) = 0 <0.000085>
> >>>>> 15:13:31.627457 (+ 0.000356)
> >>>>> clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811, tv_nsec=627560772}) = 0
> >>>>> <0.000043>
> >>>>> 15:13:31.631233 (+ 0.003839)
> >>>>> clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932,
> >>>>> tv_nsec=45286796}) = 0 <0.000077>
> >>>>> 15:13:31.631720 (+ 0.000427)
> >>>>> clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932,
> >>>>> tv_nsec=45661430}) = 0 <0.000042>
> >>>>> 15:13:31.631955 (+ 0.000220)
> >>>>> clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811, tv_nsec=632049036}) = 0
> >>>>> <0.000047>
> >>>>> 15:13:31.635669 (+ 0.003785)
> >>>>> clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932,
> >>>>> tv_nsec=49725840}) = 0 <0.000146>
> >>>>> 15:13:31.636484 (+ 0.000784) write(16,
"a",
> >>>>> 1) = 1 <0.000118>
> >>>>> 15:13:31.636855 (+ 0.000341)
sched_yield()
> >>>>> = 0 <0.000252>
> >>>>> 15:13:31.637322 (+ 0.000470)
> >>>>> futex(0x1cb57a0, FUTEX_WAKE_PRIVATE, 1) = 1
> >>>>> <0.000088>
> >>>>> 15:13:31.637897 (+ 0.000610) write(16,
"a",
> >>>>> 1) = 1 <0.000221>
> >>>>> 15:13:31.638394 (+ 0.000467)
sched_yield()
> >>>>> = 0 <0.000047>
> >>>>> 15:13:31.638619 (+ 0.000202)
> >>>>> futex(0x1cb5710, FUTEX_WAKE_PRIVATE, 1) = 1
> >>>>> <0.000065>
> >>>>> 15:13:31.638908 (+ 0.000298)
> >>>>> openat(AT_FDCWD,
> >>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
> >>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
<0.000831>
> >>>>> 15:13:31.640260 (+ 0.001387) getpid() =
> >>>>> 32353 <0.000077>
> >>>>> 15:13:31.640558 (+ 0.000256)
fstat64(33,
> >>>>> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> >>>>> <0.000119>
> >>>>> 15:13:31.641106 (+ 0.000556) write(33,
> >>>>> "32353\n", 6) = 6
<0.000127>
> >>>>> 15:13:31.641472 (+ 0.000362) close(33) =
0
> >>>>> <0.000519>
> >>>>> 15:13:31.642216 (+ 0.000758)
> >>>>> chmod("/var/run/dirsrv/slapd-
> COMPANY-INTERNAL.pid",
> >>>>> 0644) = 0 <0.000152>
> >>>>> 15:13:31.642900 (+ 0.000679)
> >>>>> clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811, tv_nsec=643020294}) = 0
> >>>>> <0.000056>
> >>>>> 15:13:31.643495 (+ 0.000590) write(2,
> >>>>> "[14/May/2018:15:13:31.643020294
"..., 134) =
> >>>>> 134 <0.002697>
> >>>>> 15:13:31.646515 (+ 0.003052)
> >>>>> clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811, tv_nsec=646694394}) = 0
> >>>>> <0.000075>
> >>>>> 15:13:31.646892 (+ 0.000337) write(4,
> >>>>> "[14/May/2018:15:13:31.646694394
"..., 134) =
> >>>>> 134 <0.000522>
> >>>>> 15:13:31.647841 (+ 0.000973) fsync(4) =
0
> >>>>> <0.005967>
> >>>>> 15:13:31.654425 (+ 0.006617)
> >>>>> clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811, tv_nsec=654598946}) = 0
> >>>>> <0.000253>
> >>>>> 15:13:31.655137 (+ 0.000717) write(2,
> >>>>> "[14/May/2018:15:13:31.654598946
"..., 136) =
> >>>>> 136 <0.002427>
> >>>>> 15:13:31.658312 (+ 0.003165)
> >>>>> clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811, tv_nsec=658486117}) = 0
> >>>>> <0.000251>
> >>>>> 15:13:31.659032 (+ 0.000682) write(4,
> >>>>> "[14/May/2018:15:13:31.658486117
"..., 136) =
> >>>>> 136 <0.000346>
> >>>>> 15:13:31.659623 (+ 0.000595) fsync(4) =
0
> >>>>> <0.003311>
> >>>>> 15:13:31.663230 (+ 0.003642) getpid() =
> >>>>> 32353 <0.000045>
> >>>>> 15:13:31.663732 (+ 0.000454)
> >>>>> socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0)
=
> >>>>> 33 <0.000296>
> >>>>> 15:13:31.664760 (+ 0.001048)
getsockopt(33,
> >>>>> SOL_SOCKET, SO_SNDBUF, [163840], [4]) = 0
> >>>>> <0.000108>
> >>>>> 15:13:31.665141 (+ 0.000386)
setsockopt(33,
> >>>>> SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) =
-1
> >>>>> EPERM (Operation not permitted)
<0.000051>
> >>>>> 15:13:31.665500 (+ 0.000334)
setsockopt(33,
> >>>>> SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0
> <0.000229>
> >>>>> 15:13:31.665973 (+ 0.000468)
sendmsg(33,
> >>>>> {msg_name={sa_family=AF_UNIX,
> >>>>> sun_path="/run/systemd/notify"},
> >>>>> msg_namelen=21,
> >>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
> >>>>> started: Re"..., iov_len=69}],
msg_iovlen=1,
> >>>>> msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL) =
> >>>>> 69 <0.000600>
> >>>>> 15:13:31.667270 (+ 0.001334) close(33) =
0
> >>>>> <0.003140>
> >>>>> 15:13:31.677320 (+ 0.010146)
> >>>>> futex(0xb31122e8, FUTEX_WAIT, 32357, NULL) =
?
> >>>>> 15:14:27.661809 (+ 55.984448) +++ killed
by
> >>>>> SIGSEGV (core dumped) +++
> >>>>>
> >>>>> On Mon, May 14, 2018 at 10:27 AM, thierry
> >>>>> bordaz <tbordaz(a)redhat.com
> >>>>> <mailto:tbordaz@redhat.com>>
wrote:
> >>>>>
> >>>>> Hi Jonathan,
> >>>>>
> >>>>> This is weird as the crashing thread
stack
> >>>>> looks truncated (did you copy/paste all
of
> >>>>> it ?)
> >>>>>
> >>>>> Thread 1 (Thread 0x9e13c280 (LWP
17245)):
> >>>>> #0 0xb67bbf2e in strlen () at
> /lib/libc.so.6
> >>>>> #1 0xb6a06b40 in dosprintf () at
> >>>>> /lib/libnspr4.so
> >>>>> #2 0x00000000 in None ()
> >>>>>
> >>>>> Did you install 389-ds-base-debuginfo ?
> >>>>> How did you get that backtrace ? from a
> >>>>> core dumped, pstack ? Can you attach a
> >>>>> debugger before the crash occurs ?
> >>>>>
> >>>>> It looks it crashed soon at startup,
could
> >>>>> it be related to a broken dse.ldif. It
> >>>>> should exists a dse.ldif.OK, is it
possibly
> >>>>> to try to start with it ?
> >>>>>
> >>>>> best regards
> >>>>> thierry
> >>>>>
> >>>>> On 05/12/2018 01:22 AM, Jonathan Vaughn
via
> >>>>> FreeIPA-users wrote:
> >>>>>> Not sure if it makes a difference...
I was
> >>>>>> looking into this again and realized
I had
> >>>>>> a bunch of messages from gdb telling
me to
> >>>>>> install more debuginfo. I've
done that
> >>>>>> now, here it is again freshly run
through
> >>>>>> gdb
> >>>>>>
> >>>>>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
> >>>>>> Copyright (C) 2017 Free Software
> >>>>>> Foundation, Inc.
> >>>>>> License GPLv3+: GNU GPL version 3 or
later
> >>>>>>
<http://gnu.org/licenses/gpl.html
> >>>>>>
<http://gnu.org/licenses/gpl.html>>
> >>>>>> This is free software: you are free
to
> >>>>>> change and redistribute it.
> >>>>>> There is NO WARRANTY, to the extent
> >>>>>> permitted by law. Type "show
copying"
> >>>>>> and "show warranty" for
details.
> >>>>>> This GDB was configured as
> >>>>>>
"armv7hl-redhat-linux-gnueabi".
> >>>>>> Type "show configuration"
for
> >>>>>> configuration details.
> >>>>>> For bug reporting instructions,
please see:
> >>>>>>
<http://www.gnu.org/software/gdb/bugs/
> >>>>>>
<http://www.gnu.org/software/gdb/bugs/>>;.
> >>>>>> Find the GDB manual and other
> >>>>>> documentation resources online at:
> >>>>>> <http://www.gnu.org/software/
> gdb/documentation/
> >>>>>> <http://www.gnu.org/software/
> gdb/documentation/>>.
> >>>>>> For help, type "help".
> >>>>>> Type "apropos word" to
search for commands
> >>>>>> related to "word"...
> >>>>>> Reading symbols from
> >>>>>> /usr/sbin/ns-slapd...Reading symbols
from
> >>>>>> /usr/lib/debug/usr/sbin/ns-sl
> apd-1.3.7.10-1.fc27.arm.debug...done.
> >>>>>> done.
> >>>>>> ...
> >>>>>>
> >>>>>> Thread 1 (Thread 0x9e13c280 (LWP
17245)):
> >>>>>> #0 0xb67bbf2e in strlen () at
> /lib/libc.so.6
> >>>>>> #1 0xb6a06b40 in dosprintf () at
> >>>>>> /lib/libnspr4.so
> >>>>>> #2 0x00000000 in None ()
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, May 8, 2018 at 7:52 AM,
Rob
> >>>>>> Crittenden
<rcritten(a)redhat.com
> >>>>>>
<mailto:rcritten@redhat.com>> wrote:
> >>>>>>
> >>>>>> Jonathan Vaughn via
FreeIPA-users
> >>>>>> wrote:
> >>>>>>
> >>>>>> Still trying to figure
this
> >>>>>> out. It looks like slapd
is
> >>>>>> dying, I thought it was
still
> >>>>>> running for some
reason.
> >>>>>>
> >>>>>> slapd is dying to
segfault.
> >>>>>> strace of it happening
doesn't
> >>>>>> seem to reveal much:
> >>>>>>
> >>>>>>
> >>>>>> A stack trace would very
much help
> >>>>>> trying to track down the
cause.
> >>>>>>
> >>>>>>
http://directory.fedoraprojec
> t.org/docs/389ds/FAQ/faq.html#debugging-crashes
> >>>>>>
<http://directory.fedoraproje
> ct.org/docs/389ds/FAQ/faq.html#debugging-crashes>
> >>>>>>
> >>>>>> rob
> >>>>>>
> >>>>>>
> >>>>>> 18:32:41.543717 (+
> >>>>>> 0.000801)
openat(AT_FDCWD,
> >>>>>>
"/var/run/dirsrv/slapd-COMPAN
> Y-INTERNAL.pid",
> >>>>>>
O_WRONLY|O_CREAT|O_TRUNC,
> >>>>>> 0666) = 32
> >>>>>> 18:32:41.544907 (+
> >>>>>> 0.001195) getpid() =
16014
> >>>>>> 18:32:41.545269 (+
> >>>>>> 0.000329) fstat64(32,
> >>>>>> {st_mode=S_IFREG|0644,
> >>>>>> st_size=0, ...}) = 0
> >>>>>> 18:32:41.545799 (+
> >>>>>> 0.000536) write(32,
> >>>>>> "16014\n", 6)
= 6
> >>>>>> 18:32:41.546603 (+
> >>>>>> 0.000818) close(32) =
0
> >>>>>> 18:32:41.547061 (+
> >>>>>> 0.000448)
> >>>>>>
chmod("/var/run/dirsrv/slapd-
> COMPANY-INTERNAL.pid",
> >>>>>> 0644) = 0
> >>>>>> 18:32:41.547741 (+
> >>>>>> 0.000676)
> >>>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>>> {tv_sec=1525735961,
> >>>>>> tv_nsec=548030641}) = 0
> >>>>>> 18:32:41.548324 (+
> >>>>>> 0.000587) write(2,
> >>>>>>
"[07/May/2018:18:32:41.548030
> 641
> >>>>>> "..., 134) = 134
> >>>>>> 18:32:41.551096 (+
> >>>>>> 0.002840)
> >>>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>>> {tv_sec=1525735961,
> >>>>>> tv_nsec=551287555}) = 0
> >>>>>> 18:32:41.551568 (+
> >>>>>> 0.000406) write(4,
> >>>>>>
"[07/May/2018:18:32:41.551287
> 555
> >>>>>> "..., 134) = 134
> >>>>>> 18:32:41.552360 (+
> >>>>>> 0.000811) fsync(4) = 0
> >>>>>> 18:32:41.558499 (+
> >>>>>> 0.006170)
> >>>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>>> {tv_sec=1525735961,
> >>>>>> tv_nsec=558678099}) = 0
> >>>>>> 18:32:41.558901 (+
> >>>>>> 0.000350) write(2,
> >>>>>>
"[07/May/2018:18:32:41.558678
> 099
> >>>>>> "..., 136) = 136
> >>>>>> 18:32:41.561537 (+
> >>>>>> 0.002680)
> >>>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>>> {tv_sec=1525735961,
> >>>>>> tv_nsec=561718659}) = 0
> >>>>>> 18:32:41.562357 (+
> >>>>>> 0.000793) write(4,
> >>>>>>
"[07/May/2018:18:32:41.561718
> 659
> >>>>>> "..., 136) = 136
> >>>>>> 18:32:41.563293 (+
> >>>>>> 0.001148) fsync(4) = 0
> >>>>>> 18:32:41.566928 (+
> >>>>>> 0.003452) getpid() =
16014
> >>>>>> 18:32:41.567712 (+
> >>>>>> 0.000752)
socket(AF_UNIX,
> >>>>>> SOCK_DGRAM|SOCK_CLOEXEC,
0) =
> 32
> >>>>>> 18:32:41.568628 (+
> >>>>>> 0.000912)
getsockopt(32,
> >>>>>> SOL_SOCKET, SO_SNDBUF,
> >>>>>> [163840], [4]) = 0
> >>>>>> 18:32:41.568972 (+
> >>>>>> 0.000319)
setsockopt(32,
> >>>>>> SOL_SOCKET,
SO_SNDBUFFORCE,
> >>>>>> [8388608], 4) = -1
EPERM
> >>>>>> (Operation not
permitted)
> >>>>>> 18:32:41.569548 (+
> >>>>>> 0.000589)
setsockopt(32,
> >>>>>> SOL_SOCKET, SO_SNDBUF,
> >>>>>> [8388608], 4) = 0
> >>>>>> 18:32:41.570064 (+
> >>>>>> 0.000513) sendmsg(32,
> >>>>>>
{msg_name={sa_family=AF_UNIX,
> >>>>>>
sun_path="/run/systemd/
> notify"},
> >>>>>> msg_namelen=21,
> >>>>>>
msg_iov=[{iov_base="READY=1\n
> STATUS=slapd
> >>>>>> started: Re"...,
iov_len=69}],
> >>>>>> msg_iovlen=1,
> >>>>>> msg_controllen=0,
> >>>>>> msg_flags=0},
MSG_NOSIGNAL) =
> 69
> >>>>>> 18:32:41.570845 (+
> >>>>>> 0.000789) close(32) =
0
> >>>>>> 18:32:41.576358 (+
> >>>>>> 0.005575)
futex(0xb30ed2e8,
> >>>>>> FUTEX_WAIT, 16016, NULL)
= ?
> >>>>>> 18:33:01.730774 (+
> >>>>>> 20.154428) +++ killed
by
> >>>>>> SIGSEGV +++
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> FreeIPA-users mailing list --
> freeipa-users(a)lists.fedorahosted.org
> >>>> <mailto:freeipa-users@lists.fedorahosted.org>
> >>>> To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>
<mailto:freeipa-users-leave@lists.fedorahosted.org>
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
> >
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to freeipa-users-leave(a)lists.fedo
> rahosted.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.or
> g/archives/list/freeipa-users(a)lists.fedorahosted.org/message
> /NNFFJQTAJ2MG52U23V6QYOOAFL722JJL/
> >
>
>
7:06 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Sorry, missed your build instructions, only saw them just now. I've just
kicked off that build and I'll check it tomorrow once I'm back in.
One note, it appears it's ./autogen.sh not ./autogen.pl ...
On Thu, May 17, 2018 at 6:16 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
Welp, I'm still getting the -fPIC error ... so I may need to
figure out
how to do it with COPR or something.
On Thu, May 17, 2018 at 4:16 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
> I've never used COPR. I've dabbled with RPMs in the past but that was...
> CentOS 6 I think, and I wasn't making source code changes so much as just
> copying and pasting SRPMs from another RPM platform to build for CentOS,
> using the regular rpmbuild stuff.
>
> I did actually try just copying the configure line (manually expanding
> the various vars) from the SRPM and just doing an old fashioned make, but
> the first try bailed shy of an hour at linking with some error about not
> having -fPIC specified, so I restarted it with -fPIC to see if it would at
> least build. So if I should still try the code changes, if that build
> finishes I can try it (I was planning to just copy the respective binaries
> on top of the installed ones), but if it turned out the suggested code
> changes were pointless then there's no point waiting for it to finish
> building.
>
> On Thu, May 17, 2018 at 4:11 PM, Rob Crittenden <rcritten(a)redhat.com>
> wrote:
>
>> Jonathan Vaughn via FreeIPA-users wrote:
>> > Oops, hit reply instead of reply-all
>> >
>> > NSPR RPMs
>> >
>> > # yum list installed nspr*
>> > Installed Packages
>> > nspr.armv7hl
>> > 4.19.0-1.fc27
>>
>> > @updates
>> > nspr-debuginfo.armv7hl
>> > 4.19.0-1.fc27
>>
>> > @updates-debuginfo
>> > nspr-debugsource.armv7hl
>> > 4.19.0-1.fc27
>>
>> > @updates-debuginfo
>> > nspr-devel.armv7hl
>> > 4.19.0-1.fc27
>>
>> > @updates
>> >
>> > As for building it with the changes you proposed previously - should I
>> > not bother? I realized last night that the last and only time I built
>> > anything *on* a Pi was a Pi Zero and that's why it was so terribly
>> slow.
>> > This Pi 3 ain't fast but it's fast enough to build stuff in far
less
>> > than half a day (unlike the Zero)...
>>
>> If you are handy with rpms you could build it in copr.
>>
>> rob
>>
>> >
>> >
>> > On Thu, May 17, 2018 at 5:07 AM, thierry bordaz <tbordaz(a)redhat.com
>> > <mailto:tbordaz@redhat.com>> wrote:
>> >
>> >
>> >
>> > On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
>> >> I've been just using the packages from Fedora. I can build it
>> >> potentially but I don't have a cross build environment set up
at
>> >> the moment. From experience I'd want to do that first because
>> >> building anything on the Pi usually takes ages.
>> >>
>> >> I'd been "redacting" the hostnames but I'll stop
bothering since
>> >> it looks like we're getting far enough into the weeds now that
the
>> >> difference in string lengths after "redacting" might
actually be a
>> >> red herring.
>> >>
>> >> (gdb) p *agmt
>> >> $1 = {hostname = 0x1ef9be0 "ipa-12.creatuity.internal",
port =
>> >> 389, transport_flags = 0, binddn = 0x1e8f650 "", creds =
>> >> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
>> >> frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
>> >> frac_attr_total_defined = 1, schedule = 0x1a7f0c0, auto_initialize
>> >> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
>> >> long_name = 0x1a7f100 "agmt=\"cn=meToipa-12.creatuit
>> y.internal\"
>> >> (ipa-12:5)", protocol = 0x19c2930, changecounters = 0x186d180,
>> >> num_changecounters = 0,
>> >> max_changecounters = 256, last_update_start_time = 1526500697,
>> >> last_update_end_time = 1526500697,
>> >> last_update_status = "Error (0) Replica acquired
successfully:
>> >> Incremental update succeeded", '\000' <repeats 954
times>,
>> >> update_in_progress = 0, is_enabled = 1,
>> >> last_init_start_time = 0, last_init_end_time = 0,
>> >> last_init_status = '\000' <repeats 1023 times>, lock =
0x1ee3740,
>> >> consumerRUV = 0x1f14e50,
>> >> consumerSchemaCSN = 0x317c520, consumerRID = 4, tmpConsumerRID =
>> >> 0, timeout = 120, stop_in_progress = 0, busywaittime = 0,
>> >> pausetime = 0, priv = 0x0,
>> >> attrs_to_strip = 0x1ef9ba0, agreement_type = 0, protocol_timeout
>> >> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000,
>> >> flowControlPause = 2000, ignoreMissingChange = 0,
>> >> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
>> >> (gdb) p *agmt->replarea
>> >> $2 = {flag = 15 '\017', udn = 0x1efce80
>> >> "dc=creatuity,dc=internal", dn = 0x1ef9460
>> >> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
>> >> "dc=creatuity,dc=internal", ndn_len = 24}
>> >> (gdb) p *agmt->rdn
>> >> $3 = {flag = 0 '\000', rdn = 0x19c2840
>> >> "cn=meToipa-12.creatuity.internal", rdns = 0x0,
butcheredupto =
>> >> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
>> >>
>> >> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
>> >> /usr/include/inttypes.h:# define PRId64 __PRI64_PREFIX
"d"
>> >> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
>> >> /usr/include/inttypes.h:# define PRIu16 "u"
>> >>
>> >>
>> >>
>> >> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
>> >> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>>
wrote:
>> >>
>> >>
>> >>
>> >> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>> >>> The installed version of 389* is 1.3.7.10-1.fc27 for
armv7hl,
>> >>> which appears to be the latest available version.
>> >>
>> >> Perhaps something is off with the inttypes on Raspberry. Are
>> >> you building this yourself on Raspberry? Can we make code
>> >> changes and compile/install them?
>> >>
>> >> Before we do that though, in gdb can you run these commands in
>> >> the same gdb frame:
>> >>
>> >> (gdb) p *agmt->replarea
>> >> (gdb) p *agmt->rdn
>> >>
>> >>
>> >> Then do:
>> >>
>> >> # grep -r PRId64 /usr/include/*
>> >> # grep -r PRIu16 /usr/include/*
>> >>
>> >>
>> >> So if you can compile the source, then change this line in
>> >> ldap/servers/plugins/replication/repl5_agmt.c:3036, but
don't
>> >> do this yet until you get me the info I just requested.
>> >>
>> >> From:
>> >>
>> >> agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%"
>> >> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replarea),
>> >>
>> >> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>> >> agmt->hostname,
>> >> agmt->port,
>> >> agmt->consumerRID, maxcsn);
>> >>
>> >> To:
>> >>
>> >> agmt->maxcsn =
>> >> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>> >> slapi_sdn_get_dn(agmt->replarea),
>> >>
>> >> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>> >> agmt->hostname,
>> >>
>> >> (long)agmt->port, (int)agmt->consumerRID, maxcsn);
>> >>
>> >>
>> >> Thanks,
>> >> Mark
>> >>
>> >>>
>> >>> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
>> >>> <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>>
>> wrote:
>> >>
>> >
>> > The agreement structure looks valid to me. it should not lead to a
>> > crash.
>> >
>> > What looks weird to me is the order of the arguments of cvt_s.
>> > It is called: rv = cvt_s(ss, u.s, width, prec, flags);
>> > But the crashing thread shows the opposite order: flags,
>> prec,width,
>> > str, ss
>> >
>> > The others frame do not show this change of order.
>> > Also 'str=4' that would be a meaningful value for
'prec=4'.
>> >
>> > From debug perspective I only imagine disassemble the two last
>> > frames to confirm parameters.
>> > What are the nspr rpms ?
>> >
>> >
>> >
>> >
>> >>> (gdb) up
>> >>> #1 0xb6926b40 in cvt_s (flags=0, prec=<optimized
out>,
>> >>> width=0, str=0x4 <error: Cannot access memory at
address
>> >>> 0x4>, ss=<optimized out>)
>> >>> at ../../.././nspr/pr/src/io/prprf.c:374
>> >>> 374 slen = strlen(str);
>> >>> (gdb) up
>> >>> #2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2
>> "",
>> >>> fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002", ap=...)
>> >>> at ../../.././nspr/pr/src/io/prprf.c:1018
>> >>> 1018 rv = cvt_s(ss, u.s, width, prec,
>> flags);
>> >>> (gdb) up
>> >>> #3 0xb6926c8c in PR_vsmprintf
(fmt=fmt@entry=0xb34da770
>> >>> <agmt_set> "\360\317p\002", ap=...,
ap@entry=...) at
>> >>> ../../.././nspr/pr/src/io/prprf.c:1184
>> >>> 1184 rv = dosprintf(&ss, fmt, ap);
>> >>> (gdb) up
>> >>> #4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0
>> >>> "%s;%s;%s;%ld;%d;%s") at
ldap/servers/slapd/ch_malloc.c
>> :331
>> >>> 331 p = PR_vsmprintf(fmt, ap);
>> >>> (gdb) up
>> >>> #5 0xb34625a4 in agmt_update_maxcsn
>> >>> (r=r@entry=0x2737810, sdn=0x38d9a40, sdn@entry=0x10,
>> >>> op=<optimized out>, mods=0x10,
mods@entry=0x27d0100,
>> >>> csn=csn@entry=0x38de350)
>> >>> at ldap/servers/plugins/replicati
>> on/repl5_agmt.c:3036
>> >>> 3036 agmt->maxcsn =
>> >>> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>> >>> slapi_sdn_get_dn(agmt->replarea),
>> >>> (gdb) p *agmt
>> >>> $1 = {hostname = 0x2757be0
"ipa-12.company.internal",
>> >>> port = 389, transport_flags = 0, binddn = 0x26ed650
"",
>> >>> creds = 0x26ed7a0, bindmethod = 3, replarea =
0x2757480,
>> >>> frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
>> >>> frac_attr_total_defined = 1, schedule = 0x22dd0c0,
>> >>> auto_initialize = 502, dn = 0x2756d00, rdn = 0x2756c20,
>> >>> long_name = 0x22dd100
>> >>> "agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)",
>> >>> protocol = 0x2220930, changecounters = 0x20cb180,
>> >>> num_changecounters = 0,
>> >>> max_changecounters = 256, last_update_start_time =
>> >>> 1526499214, last_update_end_time = 1526499214,
>> >>> last_update_status = "Error (0) Replica acquired
>> >>> successfully: Incremental update succeeded",
'\000'
>> >>> <repeats 954 times>, update_in_progress = 0,
is_enabled
>> = 1,
>> >>> last_init_start_time = 0, last_init_end_time = 0,
>> >>> last_init_status = '\000' <repeats 1023
times>, lock =
>> >>> 0x2741740, consumerRUV = 0x2772e50,
>> >>> consumerSchemaCSN = 0x39da520, consumerRID = 4,
>> >>> tmpConsumerRID = 0, timeout = 120, stop_in_progress =
0,
>> >>> busywaittime = 0, pausetime = 0, priv = 0x0,
>> >>> attrs_to_strip = 0x2757ba0, agreement_type = 0,
>> >>> protocol_timeout = 0x26ed5f0, maxcsn = 0x0,
>> >>> flowControlWindow = 1000, flowControlPause = 2000,
>> >>> ignoreMissingChange = 0,
>> >>> attr_lock = 0x2757c20, WaitForAsyncResults = 100}
>> >>>
>> >>>
>> >>> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
>> >>> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>>
>> wrote:
>> >>>
>> >>> This looks really familiar and I thought it was
>> >>> fixed. It should have been fixed in 1.3.7.10-1
>> >>> (https://pagure.io/389-ds-base/issue/49618
>> >>> <https://pagure.io/389-ds-base/issue/49618>).
In
>> >>> your debug session go "up" into
agmt_maxcsn_update()
>> >>> and do:
>> >>>
>> >>> (gdb) p *agmt
>> >>>
>> >>> Then send us that output please.
>> >>>
>> >>> Thanks,
>> >>> Mark
>> >>>
>> >>>
>> >>> On 05/15/2018 05:29 PM, Jonathan Vaughn via
>> >>> FreeIPA-users wrote:
>> >>>> Here is a backtrace from live gdb after the
>> >>>> segfault. Looks like things went wrong
somewhere
>> >>>> during in the replication code ?
>> >>>>
>> >>>> Thread 36 "ns-slapd" received signal
SIGSEGV,
>> >>>> Segmentation fault.
>> >>>> [Switching to Thread 0x9e0bc280 (LWP 4662)]
>> >>>> strlen () at
../sysdeps/arm/armv6t2/strlen.S:142
>> >>>> 142 ldrd data1a, data1b, [src]
>> >>>> (gdb) bt
>> >>>> #0 0xb6728f2e in strlen () at
>> >>>> ../sysdeps/arm/armv6t2/strlen.S:142
>> >>>> #1 0xb6973b40 in cvt_s (flags=0,
prec=<optimized
>> >>>> out>, width=0, str=0x4 <error: Cannot
access memory
>> >>>> at address 0x4>, ss=<optimized out>)
>> >>>> at ../../.././nspr/pr/src/io/prprf.c:374
>> >>>> #2 0xb6973b40 in dosprintf (ss=ss@entry
>> =0x9e0bb4bc,
>> >>>> fmt=0xb34fddf2 "",
fmt@entry=0xb3527770 <agmt_set>
>> >>>> "\360\357\305\002", ap=...)
>> >>>> at ../../.././nspr/pr/src/io/prprf.c:1018
>> >>>> #3 0xb6973c8c in PR_vsmprintf
>> >>>> (fmt=fmt@entry=0xb3527770 <agmt_set>
>> >>>> "\360\357\305\002", ap=...,
ap@entry=...) at
>> >>>> ../../.././nspr/pr/src/io/prprf.c:1184
>> >>>> #4 0xb6e36d18 in slapi_ch_smprintf
(fmt=0xb34fdde0
>> >>>> "%s;%s;%s;%ld;%d;%s") at
>> >>>> ldap/servers/slapd/ch_malloc.c:331
>> >>>> #5 0xb34af5a4 in agmt_update_maxcsn
>> >>>> (r=r@entry=0x2c89810, sdn=0x3e2bac0, sdn@entry
>> =0x10,
>> >>>> op=<optimized out>, mods=0x10, mods@entry
>> =0x3eec100,
>> >>>> csn=csn@entry=0x3e30220)
>> >>>> at
>> >>>>
ldap/servers/plugins/replication/repl5_agmt.c:3036
>> >>>> #6 0xb34bd424 in write_changelog_and_ruv
>> >>>> (pb=pb@entry=0x2cb54a0) at
>> >>>> ldap/servers/plugins/replicat
>> ion/repl5_plugins.c:1124
>> >>>> #7 0xb34be7ec in
multimaster_be_betxnpostop_add
>> >>>> (pb=0x2cb54a0) at
>> >>>> ldap/servers/plugins/replicat
>> ion/repl5_plugins.c:855
>> >>>> #8 0xb34be880 in multimaster_mmr_postop
>> >>>> (pb=<optimized out>, flags=560) at
>> >>>> ldap/servers/plugins/replicat
>> ion/repl5_plugins.c:616
>> >>>> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop
>> >>>> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0,
>> >>>> flags=flags@entry=560) at
>> >>>> ldap/servers/slapd/plugin_mmr.c:65
>> >>>> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0)
at
>> >>>> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>> >>>> #11 0xb6e2ce4c in op_shared_add
>> >>>> (pb=pb@entry=0x2cb54a0) at
>> ldap/servers/slapd/add.c:679
>> >>>> #12 0xb6e2d35c in add_internal_pb
>> >>>> (pb=pb@entry=0x2cb54a0) at
>> ldap/servers/slapd/add.c:407
>> >>>> #13 0xb6e2e0f8 in slapi_add_internal_pb
>> >>>> (pb=pb@entry=0x2cb54a0) at
>> ldap/servers/slapd/add.c:332
>> >>>> #14 0xb368db50 in ipa_topo_util_segment_write
>> >>>> (tconf=tconf@entry=0x2cb5860,
>> >>>> tsegm=tsegm@entry=0x3f2d9a0) at
>> topology_util.c:1251
>> >>>> #15 0xb368e01c in
ipa_topo_util_update_agmt_list
>> >>>> (conf=0x2cb5860, repl_segments=<optimized
out>) at
>> >>>> topology_util.c:696
>> >>>> #16 0xb368891c in ipa_topo_apply_shared_config
() at
>> >>>> topology_init.c:165
>> >>>> #17 0xb6e4e65c in eq_call_all () at
>> >>>> ldap/servers/slapd/eventq.c:278
>> >>>> #18 0xb6e4e65c in eq_loop (arg=<optimized
out>) at
>> >>>> ldap/servers/slapd/eventq.c:323
>> >>>> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
>> >>>> ../../.././nspr/pr/src/pthreads/ptthread.c:201
>> >>>> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280)
at
>> >>>> pthread_create.c:465
>> >>>> #21 0xb6785da8 in None () at
>> >>>> ../sysdeps/unix/sysv/linux/arm/clone.S:73
>> >>>>
>> >>>>
>> >>>> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz
>> >>>> <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>>
>> wrote:
>> >>>>
>> >>>> Hi Jonathan,
>> >>>>
>> >>>> This problem looks new to me and has
something
>> >>>> specific to your environment.
>> >>>> I think the best approach is to continue to
>> >>>> debug on your system if you have the
possibility
>> >>>> to do so.
>> >>>>
>> >>>> From strace we can see that DS started
smoothly
>> >>>> (created its pid file then notified systemd
it
>> >>>> was running fine). According to the pstack
>> >>>> nunc-stans was running and was able to
accept
>> >>>> network events even if it appears it
detected no
>> >>>> incoming connection.
>> >>>> So the server should be ready to serve for
some
>> >>>> seconds (more than a minute), then it
crashed
>> >>>> with one thread dereferencing likely a
wrong
>> >>>> pointer.
>> >>>>
>> >>>> Could you attach a debugger when the server
is
>> >>>> started and wait for the sigsegv to occur.
Then
>> >>>> confirm the crashing thread backstack.
>> >>>> If it is confirmed, I am afraid this is a
stack
>> >>>> corruption and valgrind could help
>> >>>> (http://www.port389.org/docs/
>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>> >>>> <http://www.port389.org/docs/
>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>> >).
>> >>>>
>> >>>> best regards
>> >>>> thierry
>> >>>>
>> >>>>
>> >>>> On 05/14/2018 10:20 PM, Jonathan Vaughn
wrote:
>> >>>>> Here's a strace from before it dies.
Most of
>> >>>>> the elapsed time is it waiting on some
futex
>> >>>>> call it looks like near the end, when
it
>> >>>>> finally "returns" (from lack
of strace output
>> >>>>> for duration of call I assume it
didn't
>> >>>>> actually return but SIGSEGV in that
call) and
>> >>>>> strace prints ' = ?' on the
futex it then
>> >>>>> immediately reports SIGSEGV after. So
maybe the
>> >>>>> problem is that futex call, which may
mean the
>> >>>>> problem is not directly in 389DS /
FreeIPA
>> itself?
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> 15:13:31.626587 (+ 0.000630)
listen(8, 128)
>> >>>>> = 0 <0.000068>
>> >>>>> 15:13:31.626857 (+ 0.000235)
listen(9, 128)
>> >>>>> = 0 <0.000048>
>> >>>>> 15:13:31.627111 (+ 0.000251)
>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>> {tv_sec=1464932,
>> >>>>> tv_nsec=41120614}) = 0 <0.000085>
>> >>>>> 15:13:31.627457 (+ 0.000356)
>> >>>>> clock_gettime(CLOCK_REALTIME,
>> >>>>> {tv_sec=1526328811, tv_nsec=627560772})
= 0
>> >>>>> <0.000043>
>> >>>>> 15:13:31.631233 (+ 0.003839)
>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>> {tv_sec=1464932,
>> >>>>> tv_nsec=45286796}) = 0 <0.000077>
>> >>>>> 15:13:31.631720 (+ 0.000427)
>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>> {tv_sec=1464932,
>> >>>>> tv_nsec=45661430}) = 0 <0.000042>
>> >>>>> 15:13:31.631955 (+ 0.000220)
>> >>>>> clock_gettime(CLOCK_REALTIME,
>> >>>>> {tv_sec=1526328811, tv_nsec=632049036})
= 0
>> >>>>> <0.000047>
>> >>>>> 15:13:31.635669 (+ 0.003785)
>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>> {tv_sec=1464932,
>> >>>>> tv_nsec=49725840}) = 0 <0.000146>
>> >>>>> 15:13:31.636484 (+ 0.000784)
write(16, "a",
>> >>>>> 1) = 1 <0.000118>
>> >>>>> 15:13:31.636855 (+ 0.000341)
sched_yield()
>> >>>>> = 0 <0.000252>
>> >>>>> 15:13:31.637322 (+ 0.000470)
>> >>>>> futex(0x1cb57a0, FUTEX_WAKE_PRIVATE, 1)
= 1
>> >>>>> <0.000088>
>> >>>>> 15:13:31.637897 (+ 0.000610)
write(16, "a",
>> >>>>> 1) = 1 <0.000221>
>> >>>>> 15:13:31.638394 (+ 0.000467)
sched_yield()
>> >>>>> = 0 <0.000047>
>> >>>>> 15:13:31.638619 (+ 0.000202)
>> >>>>> futex(0x1cb5710, FUTEX_WAKE_PRIVATE, 1)
= 1
>> >>>>> <0.000065>
>> >>>>> 15:13:31.638908 (+ 0.000298)
>> >>>>> openat(AT_FDCWD,
>> >>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>> >>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
<0.000831>
>> >>>>> 15:13:31.640260 (+ 0.001387)
getpid() =
>> >>>>> 32353 <0.000077>
>> >>>>> 15:13:31.640558 (+ 0.000256)
fstat64(33,
>> >>>>> {st_mode=S_IFREG|0644, st_size=0, ...})
= 0
>> >>>>> <0.000119>
>> >>>>> 15:13:31.641106 (+ 0.000556)
write(33,
>> >>>>> "32353\n", 6) = 6
<0.000127>
>> >>>>> 15:13:31.641472 (+ 0.000362)
close(33) = 0
>> >>>>> <0.000519>
>> >>>>> 15:13:31.642216 (+ 0.000758)
>> >>>>> chmod("/var/run/dirsrv/slapd-
>> COMPANY-INTERNAL.pid",
>> >>>>> 0644) = 0 <0.000152>
>> >>>>> 15:13:31.642900 (+ 0.000679)
>> >>>>> clock_gettime(CLOCK_REALTIME,
>> >>>>> {tv_sec=1526328811, tv_nsec=643020294})
= 0
>> >>>>> <0.000056>
>> >>>>> 15:13:31.643495 (+ 0.000590)
write(2,
>> >>>>> "[14/May/2018:15:13:31.643020294
"..., 134) =
>> >>>>> 134 <0.002697>
>> >>>>> 15:13:31.646515 (+ 0.003052)
>> >>>>> clock_gettime(CLOCK_REALTIME,
>> >>>>> {tv_sec=1526328811, tv_nsec=646694394})
= 0
>> >>>>> <0.000075>
>> >>>>> 15:13:31.646892 (+ 0.000337)
write(4,
>> >>>>> "[14/May/2018:15:13:31.646694394
"..., 134) =
>> >>>>> 134 <0.000522>
>> >>>>> 15:13:31.647841 (+ 0.000973)
fsync(4) = 0
>> >>>>> <0.005967>
>> >>>>> 15:13:31.654425 (+ 0.006617)
>> >>>>> clock_gettime(CLOCK_REALTIME,
>> >>>>> {tv_sec=1526328811, tv_nsec=654598946})
= 0
>> >>>>> <0.000253>
>> >>>>> 15:13:31.655137 (+ 0.000717)
write(2,
>> >>>>> "[14/May/2018:15:13:31.654598946
"..., 136) =
>> >>>>> 136 <0.002427>
>> >>>>> 15:13:31.658312 (+ 0.003165)
>> >>>>> clock_gettime(CLOCK_REALTIME,
>> >>>>> {tv_sec=1526328811, tv_nsec=658486117})
= 0
>> >>>>> <0.000251>
>> >>>>> 15:13:31.659032 (+ 0.000682)
write(4,
>> >>>>> "[14/May/2018:15:13:31.658486117
"..., 136) =
>> >>>>> 136 <0.000346>
>> >>>>> 15:13:31.659623 (+ 0.000595)
fsync(4) = 0
>> >>>>> <0.003311>
>> >>>>> 15:13:31.663230 (+ 0.003642)
getpid() =
>> >>>>> 32353 <0.000045>
>> >>>>> 15:13:31.663732 (+ 0.000454)
>> >>>>> socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC,
0) =
>> >>>>> 33 <0.000296>
>> >>>>> 15:13:31.664760 (+ 0.001048)
getsockopt(33,
>> >>>>> SOL_SOCKET, SO_SNDBUF, [163840], [4]) =
0
>> >>>>> <0.000108>
>> >>>>> 15:13:31.665141 (+ 0.000386)
setsockopt(33,
>> >>>>> SOL_SOCKET, SO_SNDBUFFORCE, [8388608],
4) = -1
>> >>>>> EPERM (Operation not permitted)
<0.000051>
>> >>>>> 15:13:31.665500 (+ 0.000334)
setsockopt(33,
>> >>>>> SOL_SOCKET, SO_SNDBUF, [8388608], 4) =
0
>> <0.000229>
>> >>>>> 15:13:31.665973 (+ 0.000468)
sendmsg(33,
>> >>>>> {msg_name={sa_family=AF_UNIX,
>> >>>>>
sun_path="/run/systemd/notify"},
>> >>>>> msg_namelen=21,
>> >>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>> >>>>> started: Re"..., iov_len=69}],
msg_iovlen=1,
>> >>>>> msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL) =
>> >>>>> 69 <0.000600>
>> >>>>> 15:13:31.667270 (+ 0.001334)
close(33) = 0
>> >>>>> <0.003140>
>> >>>>> 15:13:31.677320 (+ 0.010146)
>> >>>>> futex(0xb31122e8, FUTEX_WAIT, 32357,
NULL) = ?
>> >>>>> 15:14:27.661809 (+ 55.984448) +++
killed by
>> >>>>> SIGSEGV (core dumped) +++
>> >>>>>
>> >>>>> On Mon, May 14, 2018 at 10:27 AM,
thierry
>> >>>>> bordaz <tbordaz(a)redhat.com
>> >>>>> <mailto:tbordaz@redhat.com>>
wrote:
>> >>>>>
>> >>>>> Hi Jonathan,
>> >>>>>
>> >>>>> This is weird as the crashing thread
stack
>> >>>>> looks truncated (did you copy/paste
all of
>> >>>>> it ?)
>> >>>>>
>> >>>>> Thread 1 (Thread 0x9e13c280 (LWP
17245)):
>> >>>>> #0 0xb67bbf2e in strlen () at
>> /lib/libc.so.6
>> >>>>> #1 0xb6a06b40 in dosprintf () at
>> >>>>> /lib/libnspr4.so
>> >>>>> #2 0x00000000 in None ()
>> >>>>>
>> >>>>> Did you install
389-ds-base-debuginfo ?
>> >>>>> How did you get that backtrace ?
from a
>> >>>>> core dumped, pstack ? Can you attach
a
>> >>>>> debugger before the crash occurs ?
>> >>>>>
>> >>>>> It looks it crashed soon at startup,
could
>> >>>>> it be related to a broken dse.ldif.
It
>> >>>>> should exists a dse.ldif.OK, is it
possibly
>> >>>>> to try to start with it ?
>> >>>>>
>> >>>>> best regards
>> >>>>> thierry
>> >>>>>
>> >>>>> On 05/12/2018 01:22 AM, Jonathan
Vaughn via
>> >>>>> FreeIPA-users wrote:
>> >>>>>> Not sure if it makes a
difference... I was
>> >>>>>> looking into this again and
realized I had
>> >>>>>> a bunch of messages from gdb
telling me to
>> >>>>>> install more debuginfo. I've
done that
>> >>>>>> now, here it is again freshly
run through
>> >>>>>> gdb
>> >>>>>>
>> >>>>>> GNU gdb (GDB) Fedora
8.0.1-36.fc27
>> >>>>>> Copyright (C) 2017 Free
Software
>> >>>>>> Foundation, Inc.
>> >>>>>> License GPLv3+: GNU GPL version
3 or later
>> >>>>>>
<http://gnu.org/licenses/gpl.html
>> >>>>>>
<http://gnu.org/licenses/gpl.html>>
>> >>>>>> This is free software: you are
free to
>> >>>>>> change and redistribute it.
>> >>>>>> There is NO WARRANTY, to the
extent
>> >>>>>> permitted by law. Type
"show copying"
>> >>>>>> and "show warranty"
for details.
>> >>>>>> This GDB was configured as
>> >>>>>>
"armv7hl-redhat-linux-gnueabi".
>> >>>>>> Type "show
configuration" for
>> >>>>>> configuration details.
>> >>>>>> For bug reporting instructions,
please
>> see:
>> >>>>>>
<http://www.gnu.org/software/gdb/bugs/
>> >>>>>>
<http://www.gnu.org/software/gdb/bugs/>>;.
>> >>>>>> Find the GDB manual and other
>> >>>>>> documentation resources online
at:
>> >>>>>>
<http://www.gnu.org/software/
>> gdb/documentation/
>> >>>>>>
<http://www.gnu.org/software/
>> gdb/documentation/>>.
>> >>>>>> For help, type
"help".
>> >>>>>> Type "apropos word" to
search for commands
>> >>>>>> related to "word"...
>> >>>>>> Reading symbols from
>> >>>>>> /usr/sbin/ns-slapd...Reading
symbols from
>> >>>>>> /usr/lib/debug/usr/sbin/ns-sl
>> apd-1.3.7.10-1.fc27.arm.debug...done.
>> >>>>>> done.
>> >>>>>> ...
>> >>>>>>
>> >>>>>> Thread 1 (Thread 0x9e13c280 (LWP
17245)):
>> >>>>>> #0 0xb67bbf2e in strlen () at
>> /lib/libc.so.6
>> >>>>>> #1 0xb6a06b40 in dosprintf ()
at
>> >>>>>> /lib/libnspr4.so
>> >>>>>> #2 0x00000000 in None ()
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Tue, May 8, 2018 at 7:52
AM, Rob
>> >>>>>> Crittenden
<rcritten(a)redhat.com
>> >>>>>>
<mailto:rcritten@redhat.com>> wrote:
>> >>>>>>
>> >>>>>> Jonathan Vaughn via
FreeIPA-users
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>> Still trying to
figure this
>> >>>>>> out. It looks like
slapd is
>> >>>>>> dying, I thought it
was still
>> >>>>>> running for some
reason.
>> >>>>>>
>> >>>>>> slapd is dying to
segfault.
>> >>>>>> strace of it
happening doesn't
>> >>>>>> seem to reveal
much:
>> >>>>>>
>> >>>>>>
>> >>>>>> A stack trace would very
much help
>> >>>>>> trying to track down the
cause.
>> >>>>>>
>> >>>>>>
http://directory.fedoraprojec
>> t.org/docs/389ds/FAQ/faq.html#debugging-crashes
>> >>>>>>
<http://directory.fedoraproje
>> ct.org/docs/389ds/FAQ/faq.html#debugging-crashes>
>> >>>>>>
>> >>>>>> rob
>> >>>>>>
>> >>>>>>
>> >>>>>> 18:32:41.543717 (+
>> >>>>>> 0.000801)
openat(AT_FDCWD,
>> >>>>>>
"/var/run/dirsrv/slapd-COMPAN
>> Y-INTERNAL.pid",
>> >>>>>>
O_WRONLY|O_CREAT|O_TRUNC,
>> >>>>>> 0666) = 32
>> >>>>>> 18:32:41.544907 (+
>> >>>>>> 0.001195) getpid()
= 16014
>> >>>>>> 18:32:41.545269 (+
>> >>>>>> 0.000329)
fstat64(32,
>> >>>>>>
{st_mode=S_IFREG|0644,
>> >>>>>> st_size=0, ...}) =
0
>> >>>>>> 18:32:41.545799 (+
>> >>>>>> 0.000536)
write(32,
>> >>>>>> "16014\n",
6) = 6
>> >>>>>> 18:32:41.546603 (+
>> >>>>>> 0.000818) close(32)
= 0
>> >>>>>> 18:32:41.547061 (+
>> >>>>>> 0.000448)
>> >>>>>>
chmod("/var/run/dirsrv/slapd-
>> COMPANY-INTERNAL.pid",
>> >>>>>> 0644) = 0
>> >>>>>> 18:32:41.547741 (+
>> >>>>>> 0.000676)
>> >>>>>>
clock_gettime(CLOCK_REALTIME,
>> >>>>>> {tv_sec=1525735961,
>> >>>>>> tv_nsec=548030641})
= 0
>> >>>>>> 18:32:41.548324 (+
>> >>>>>> 0.000587) write(2,
>> >>>>>>
"[07/May/2018:18:32:41.548030
>> 641
>> >>>>>> "..., 134) =
134
>> >>>>>> 18:32:41.551096 (+
>> >>>>>> 0.002840)
>> >>>>>>
clock_gettime(CLOCK_REALTIME,
>> >>>>>> {tv_sec=1525735961,
>> >>>>>> tv_nsec=551287555})
= 0
>> >>>>>> 18:32:41.551568 (+
>> >>>>>> 0.000406) write(4,
>> >>>>>>
"[07/May/2018:18:32:41.551287
>> 555
>> >>>>>> "..., 134) =
134
>> >>>>>> 18:32:41.552360 (+
>> >>>>>> 0.000811) fsync(4)
= 0
>> >>>>>> 18:32:41.558499 (+
>> >>>>>> 0.006170)
>> >>>>>>
clock_gettime(CLOCK_REALTIME,
>> >>>>>> {tv_sec=1525735961,
>> >>>>>> tv_nsec=558678099})
= 0
>> >>>>>> 18:32:41.558901 (+
>> >>>>>> 0.000350) write(2,
>> >>>>>>
"[07/May/2018:18:32:41.558678
>> 099
>> >>>>>> "..., 136) =
136
>> >>>>>> 18:32:41.561537 (+
>> >>>>>> 0.002680)
>> >>>>>>
clock_gettime(CLOCK_REALTIME,
>> >>>>>> {tv_sec=1525735961,
>> >>>>>> tv_nsec=561718659})
= 0
>> >>>>>> 18:32:41.562357 (+
>> >>>>>> 0.000793) write(4,
>> >>>>>>
"[07/May/2018:18:32:41.561718
>> 659
>> >>>>>> "..., 136) =
136
>> >>>>>> 18:32:41.563293 (+
>> >>>>>> 0.001148) fsync(4)
= 0
>> >>>>>> 18:32:41.566928 (+
>> >>>>>> 0.003452) getpid()
= 16014
>> >>>>>> 18:32:41.567712 (+
>> >>>>>> 0.000752)
socket(AF_UNIX,
>> >>>>>>
SOCK_DGRAM|SOCK_CLOEXEC, 0) =
>> 32
>> >>>>>> 18:32:41.568628 (+
>> >>>>>> 0.000912)
getsockopt(32,
>> >>>>>> SOL_SOCKET,
SO_SNDBUF,
>> >>>>>> [163840], [4]) = 0
>> >>>>>> 18:32:41.568972 (+
>> >>>>>> 0.000319)
setsockopt(32,
>> >>>>>> SOL_SOCKET,
SO_SNDBUFFORCE,
>> >>>>>> [8388608], 4) = -1
EPERM
>> >>>>>> (Operation not
permitted)
>> >>>>>> 18:32:41.569548 (+
>> >>>>>> 0.000589)
setsockopt(32,
>> >>>>>> SOL_SOCKET,
SO_SNDBUF,
>> >>>>>> [8388608], 4) = 0
>> >>>>>> 18:32:41.570064 (+
>> >>>>>> 0.000513)
sendmsg(32,
>> >>>>>>
{msg_name={sa_family=AF_UNIX,
>> >>>>>>
sun_path="/run/systemd/notify
>> "},
>> >>>>>> msg_namelen=21,
>> >>>>>>
msg_iov=[{iov_base="READY=1\n
>> STATUS=slapd
>> >>>>>> started:
Re"..., iov_len=69}],
>> >>>>>> msg_iovlen=1,
>> >>>>>> msg_controllen=0,
>> >>>>>> msg_flags=0},
MSG_NOSIGNAL) =
>> 69
>> >>>>>> 18:32:41.570845 (+
>> >>>>>> 0.000789) close(32)
= 0
>> >>>>>> 18:32:41.576358 (+
>> >>>>>> 0.005575)
futex(0xb30ed2e8,
>> >>>>>> FUTEX_WAIT, 16016,
NULL) = ?
>> >>>>>> 18:33:01.730774 (+
>> >>>>>> 20.154428) +++
killed by
>> >>>>>> SIGSEGV +++
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> FreeIPA-users mailing list --
>> freeipa-users(a)lists.fedorahosted.org
>> >>>>
<mailto:freeipa-users@lists.fedorahosted.org>
>> >>>> To unsubscribe send an email to
>> freeipa-users-leave(a)lists.fedorahosted.org
>> >>>>
<mailto:freeipa-users-leave@lists.fedorahosted.org>
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> > To unsubscribe send an email to freeipa-users-leave(a)lists.fedo
>> rahosted.org
>> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> > List Guidelines: https://fedoraproject.org/wiki
>> /Mailing_list_guidelines
>> > List Archives: https://lists.fedoraproject.or
>> g/archives/list/freeipa-users(a)lists.fedorahosted.org/message
>> /NNFFJQTAJ2MG52U23V6QYOOAFL722JJL/
>> >
>>
>>
>
9:09 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
On 05/17/2018 08:06 PM, Jonathan Vaughn wrote:
Sorry, missed your build instructions, only saw them just now.
I've
just kicked off that build and I'll check it tomorrow once I'm back in.
One note, it appears it's ./autogen.sh not ./autogen.pl
<http://autogen.pl> ...
Whoops! Good catch, and good luck...
On Thu, May 17, 2018 at 6:16 PM, Jonathan Vaughn
<jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
Welp, I'm still getting the -fPIC error ... so I may need to
figure out how to do it with COPR or something.
On Thu, May 17, 2018 at 4:16 PM, Jonathan Vaughn
<jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
I've never used COPR. I've dabbled with RPMs in the past but
that was... CentOS 6 I think, and I wasn't making source code
changes so much as just copying and pasting SRPMs from another
RPM platform to build for CentOS, using the regular rpmbuild
stuff.
I did actually try just copying the configure line (manually
expanding the various vars) from the SRPM and just doing an
old fashioned make, but the first try bailed shy of an hour at
linking with some error about not having -fPIC specified, so I
restarted it with -fPIC to see if it would at least build. So
if I should still try the code changes, if that build finishes
I can try it (I was planning to just copy the respective
binaries on top of the installed ones), but if it turned out
the suggested code changes were pointless then there's no
point waiting for it to finish building.
On Thu, May 17, 2018 at 4:11 PM, Rob Crittenden
<rcritten(a)redhat.com <mailto:rcritten@redhat.com>> wrote:
Jonathan Vaughn via FreeIPA-users wrote:
> Oops, hit reply instead of reply-all
>
> NSPR RPMs
>
> # yum list installed nspr*
> Installed Packages
> nspr.armv7hl
> 4.19.0-1.fc27
> @updates
> nspr-debuginfo.armv7hl
> 4.19.0-1.fc27
> @updates-debuginfo
> nspr-debugsource.armv7hl
> 4.19.0-1.fc27
> @updates-debuginfo
> nspr-devel.armv7hl
> 4.19.0-1.fc27
> @updates
>
> As for building it with the changes you proposed
previously - should I
> not bother? I realized last night that the last and only
time I built
> anything *on* a Pi was a Pi Zero and that's why it was
so terribly slow.
> This Pi 3 ain't fast but it's fast enough to build stuff
in far less
> than half a day (unlike the Zero)...
If you are handy with rpms you could build it in copr.
rob
>
>
> On Thu, May 17, 2018 at 5:07 AM, thierry bordaz
<tbordaz(a)redhat.com <mailto:tbordaz@redhat.com>
> <mailto:tbordaz@redhat.com <mailto:tbordaz@redhat.com>>>
wrote:
>
>
>
> On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
>> I've been just using the packages from Fedora. I
can build it
>> potentially but I don't have a cross build
environment set up at
>> the moment. From experience I'd want to do that
first because
>> building anything on the Pi usually takes ages.
>>
>> I'd been "redacting" the hostnames but I'll
stop
bothering since
>> it looks like we're getting far enough into the
weeds now that the
>> difference in string lengths after "redacting"
might actually be a
>> red herring.
>>
>> (gdb) p *agmt
>> $1 = {hostname = 0x1ef9be0
"ipa-12.creatuity.internal", port =
>> 389, transport_flags = 0, binddn = 0x1e8f650 "",
creds =
>> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
>> frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
>> frac_attr_total_defined = 1, schedule = 0x1a7f0c0,
auto_initialize
>> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
>> long_name = 0x1a7f100
"agmt=\"cn=meToipa-12.creatuity.internal\"
>> (ipa-12:5)", protocol = 0x19c2930, changecounters =
0x186d180,
>> num_changecounters = 0,
>> max_changecounters = 256, last_update_start_time
= 1526500697,
>> last_update_end_time = 1526500697,
>> last_update_status = "Error (0) Replica acquired
successfully:
>> Incremental update succeeded", '\000' <repeats
954
times>,
>> update_in_progress = 0, is_enabled = 1,
>> last_init_start_time = 0, last_init_end_time = 0,
>> last_init_status = '\000' <repeats 1023 times>,
lock = 0x1ee3740,
>> consumerRUV = 0x1f14e50,
>> consumerSchemaCSN = 0x317c520, consumerRID = 4,
tmpConsumerRID =
>> 0, timeout = 120, stop_in_progress = 0,
busywaittime = 0,
>> pausetime = 0, priv = 0x0,
>> attrs_to_strip = 0x1ef9ba0, agreement_type = 0,
protocol_timeout
>> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000,
>> flowControlPause = 2000, ignoreMissingChange = 0,
>> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
>> (gdb) p *agmt->replarea
>> $2 = {flag = 15 '\017', udn = 0x1efce80
>> "dc=creatuity,dc=internal", dn = 0x1ef9460
>> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
>> "dc=creatuity,dc=internal", ndn_len = 24}
>> (gdb) p *agmt->rdn
>> $3 = {flag = 0 '\000', rdn = 0x19c2840
>> "cn=meToipa-12.creatuity.internal", rdns = 0x0,
butcheredupto =
>> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
>>
>> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
>> /usr/include/inttypes.h:# define PRId64
__PRI64_PREFIX "d"
>> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
>> /usr/include/inttypes.h:# define PRIu16 "u"
>>
>>
>>
>> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
>> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>
<mailto:mreynolds@redhat.com
<mailto:mreynolds@redhat.com>>> wrote:
>>
>>
>>
>> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>>> The installed version of 389*
is 1.3.7.10-1.fc27 for armv7hl,
>>> which appears to be the latest available version.
>>
>> Perhaps something is off with the inttypes on
Raspberry. Are
>> you building this yourself on Raspberry? Can
we make code
>> changes and compile/install them?
>>
>> Before we do that though, in gdb can you run
these commands in
>> the same gdb frame:
>>
>> (gdb) p *agmt->replarea
>> (gdb) p *agmt->rdn
>>
>>
>> Then do:
>>
>> # grep -r PRId64 /usr/include/*
>> # grep -r PRIu16 /usr/include/*
>>
>>
>> So if you can compile the source, then change
this line in
>>
ldap/servers/plugins/replication/repl5_agmt.c:3036, but don't
>> do this yet until you get me the info I just
requested.
>>
>> From:
>>
>> agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%"
>> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replarea),
>>
>>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>> agmt->hostname,
>>
agmt->port,
>> agmt->consumerRID, maxcsn);
>>
>> To:
>>
>> agmt->maxcsn =
>> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>> slapi_sdn_get_dn(agmt->replarea),
>>
>>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>> agmt->hostname,
>>
>> (long)agmt->port, (int)agmt->consumerRID, maxcsn);
>>
>>
>> Thanks,
>> Mark
>>
>>>
>>> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
>>> <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>
<mailto:jonathan@creatuity.com
<mailto:jonathan@creatuity.com>>> wrote:
>>
>
> The agreement structure looks valid to me. it should
not lead to a
> crash.
>
> What looks weird to me is the order of the arguments
of cvt_s.
> It is called: rv = cvt_s(ss, u.s, width, prec, flags);
> But the crashing thread shows the opposite order:
flags, prec,width,
> str, ss
>
> The others frame do not show this change of order.
> Also 'str=4' that would be a meaningful value for
'prec=4'.
>
> From debug perspective I only imagine disassemble
the two last
> frames to confirm parameters.
> What are the nspr rpms ?
>
>
>
>
>>> (gdb) up
>>> #1 0xb6926b40 in cvt_s (flags=0,
prec=<optimized out>,
>>> width=0, str=0x4 <error: Cannot access
memory at address
>>> 0x4>, ss=<optimized out>)
>>> at ../../.././nspr/pr/src/io/prprf.c:374
>>> 374 slen = strlen(str);
>>> (gdb) up
>>> #2 dosprintf (ss=ss@entry=0x9e06e4bc,
fmt=0xb34b0df2 "",
>>> fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002", ap=...)
>>> at ../../.././nspr/pr/src/io/prprf.c:1018
>>> 1018 rv = cvt_s(ss, u.s,
width, prec, flags);
>>> (gdb) up
>>> #3 0xb6926c8c in PR_vsmprintf
(fmt=fmt@entry=0xb34da770
>>> <agmt_set> "\360\317p\002", ap=...,
ap@entry=...) at
>>> ../../.././nspr/pr/src/io/prprf.c:1184
>>> 1184 rv = dosprintf(&ss, fmt, ap);
>>> (gdb) up
>>> #4 0xb6de9d18 in slapi_ch_smprintf
(fmt=0xb34b0de0
>>> "%s;%s;%s;%ld;%d;%s") at
ldap/servers/slapd/ch_malloc.c:331
>>> 331 p = PR_vsmprintf(fmt, ap);
>>> (gdb) up
>>> #5 0xb34625a4 in agmt_update_maxcsn
>>> (r=r@entry=0x2737810, sdn=0x38d9a40,
sdn@entry=0x10,
>>> op=<optimized out>, mods=0x10,
mods@entry=0x27d0100,
>>> csn=csn@entry=0x38de350)
>>> at
ldap/servers/plugins/replication/repl5_agmt.c:3036
>>> 3036 agmt->maxcsn =
>>> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>> slapi_sdn_get_dn(agmt->replarea),
>>> (gdb) p *agmt
>>> $1 = {hostname = 0x2757be0
"ipa-12.company.internal",
>>> port = 389, transport_flags = 0, binddn =
0x26ed650 "",
>>> creds = 0x26ed7a0, bindmethod = 3,
replarea = 0x2757480,
>>> frac_attrs = 0x27579c0, frac_attrs_total
= 0x2757a40,
>>> frac_attr_total_defined = 1, schedule =
0x22dd0c0,
>>> auto_initialize = 502, dn = 0x2756d00, rdn
= 0x2756c20,
>>> long_name = 0x22dd100
>>>
"agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)",
>>> protocol = 0x2220930, changecounters =
0x20cb180,
>>> num_changecounters = 0,
>>> max_changecounters = 256,
last_update_start_time =
>>> 1526499214, last_update_end_time = 1526499214,
>>> last_update_status = "Error (0) Replica
acquired
>>> successfully: Incremental update
succeeded", '\000'
>>> <repeats 954 times>, update_in_progress =
0, is_enabled = 1,
>>> last_init_start_time = 0,
last_init_end_time = 0,
>>> last_init_status = '\000' <repeats 1023
times>, lock =
>>> 0x2741740, consumerRUV = 0x2772e50,
>>> consumerSchemaCSN = 0x39da520,
consumerRID = 4,
>>> tmpConsumerRID = 0, timeout = 120,
stop_in_progress = 0,
>>> busywaittime = 0, pausetime = 0, priv = 0x0,
>>> attrs_to_strip = 0x2757ba0,
agreement_type = 0,
>>> protocol_timeout = 0x26ed5f0, maxcsn = 0x0,
>>> flowControlWindow = 1000, flowControlPause
= 2000,
>>> ignoreMissingChange = 0,
>>> attr_lock = 0x2757c20,
WaitForAsyncResults = 100}
>>>
>>>
>>> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
>>> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>
<mailto:mreynolds@redhat.com
<mailto:mreynolds@redhat.com>>> wrote:
>>>
>>> This looks really familiar and I
thought it was
>>> fixed. It should have been fixed in
1.3.7.10-1
>>>
(https://pagure.io/389-ds-base/issue/49618
<https://pagure.io/389-ds-base/issue/49618>
>>>
<https://pagure.io/389-ds-base/issue/49618
<https://pagure.io/389-ds-base/issue/49618>>). In
>>> your debug session go "up" into
agmt_maxcsn_update()
>>> and do:
>>>
>>> (gdb) p *agmt
>>>
>>> Then send us that output please.
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>> On 05/15/2018 05:29 PM, Jonathan
Vaughn via
>>> FreeIPA-users wrote:
>>>> Here is a backtrace from live gdb
after the
>>>> segfault. Looks like things went
wrong somewhere
>>>> during in the replication code ?
>>>>
>>>> Thread 36 "ns-slapd" received
signal
SIGSEGV,
>>>> Segmentation fault.
>>>> [Switching to Thread 0x9e0bc280 (LWP
4662)]
>>>> strlen () at
../sysdeps/arm/armv6t2/strlen.S:142
>>>> 142 ldrd data1a,
data1b, [src]
>>>> (gdb) bt
>>>> #0 0xb6728f2e in strlen () at
>>>> ../sysdeps/arm/armv6t2/strlen.S:142
>>>> #1 0xb6973b40 in cvt_s (flags=0,
prec=<optimized
>>>> out>, width=0, str=0x4 <error: Cannot
access memory
>>>> at address 0x4>, ss=<optimized
out>)
>>>> at
../../.././nspr/pr/src/io/prprf.c:374
>>>> #2 0xb6973b40 in dosprintf
(ss=ss@entry=0x9e0bb4bc,
>>>> fmt=0xb34fddf2 "",
fmt@entry=0xb3527770 <agmt_set>
>>>> "\360\357\305\002", ap=...)
>>>> at
../../.././nspr/pr/src/io/prprf.c:1018
>>>> #3 0xb6973c8c in PR_vsmprintf
>>>> (fmt=fmt@entry=0xb3527770 <agmt_set>
>>>> "\360\357\305\002", ap=...,
ap@entry=...) at
>>>> ../../.././nspr/pr/src/io/prprf.c:1184
>>>> #4 0xb6e36d18 in slapi_ch_smprintf
(fmt=0xb34fdde0
>>>> "%s;%s;%s;%ld;%d;%s") at
>>>> ldap/servers/slapd/ch_malloc.c:331
>>>> #5 0xb34af5a4 in agmt_update_maxcsn
>>>> (r=r@entry=0x2c89810, sdn=0x3e2bac0,
sdn@entry=0x10,
>>>> op=<optimized out>, mods=0x10,
mods@entry=0x3eec100,
>>>> csn=csn@entry=0x3e30220)
>>>> at
>>>>
ldap/servers/plugins/replication/repl5_agmt.c:3036
>>>> #6 0xb34bd424 in write_changelog_and_ruv
>>>> (pb=pb@entry=0x2cb54a0) at
>>>>
ldap/servers/plugins/replication/repl5_plugins.c:1124
>>>> #7 0xb34be7ec in
multimaster_be_betxnpostop_add
>>>> (pb=0x2cb54a0) at
>>>>
ldap/servers/plugins/replication/repl5_plugins.c:855
>>>> #8 0xb34be880 in multimaster_mmr_postop
>>>> (pb=<optimized out>, flags=560) at
>>>>
ldap/servers/plugins/replication/repl5_plugins.c:616
>>>> #9 0xb6e8fcf0 in
plugin_call_mmr_plugin_postop
>>>> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0,
>>>> flags=flags@entry=560) at
>>>> ldap/servers/slapd/plugin_mmr.c:65
>>>> #10 0xb35ba870 in ldbm_back_add
(pb=0x2cb54a0) at
>>>>
ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>>>> #11 0xb6e2ce4c in op_shared_add
>>>> (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:679
>>>> #12 0xb6e2d35c in add_internal_pb
>>>> (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:407
>>>> #13 0xb6e2e0f8 in slapi_add_internal_pb
>>>> (pb=pb@entry=0x2cb54a0) at
ldap/servers/slapd/add.c:332
>>>> #14 0xb368db50 in
ipa_topo_util_segment_write
>>>> (tconf=tconf@entry=0x2cb5860,
>>>> tsegm=tsegm@entry=0x3f2d9a0) at
topology_util.c:1251
>>>> #15 0xb368e01c in
ipa_topo_util_update_agmt_list
>>>> (conf=0x2cb5860,
repl_segments=<optimized out>) at
>>>> topology_util.c:696
>>>> #16 0xb368891c in
ipa_topo_apply_shared_config () at
>>>> topology_init.c:165
>>>> #17 0xb6e4e65c in eq_call_all () at
>>>> ldap/servers/slapd/eventq.c:278
>>>> #18 0xb6e4e65c in eq_loop
(arg=<optimized out>) at
>>>> ldap/servers/slapd/eventq.c:323
>>>> #19 0xb6982d68 in _pt_root
(arg=0x2c8da40) at
>>>>
../../.././nspr/pr/src/pthreads/ptthread.c:201
>>>> #20 0xb6906ee8 in start_thread
(arg=0x9e0bc280) at
>>>> pthread_create.c:465
>>>> #21 0xb6785da8 in None () at
>>>> ../sysdeps/unix/sysv/linux/arm/clone.S:73
>>>>
>>>>
>>>> On Tue, May 15, 2018 at 3:01 AM,
thierry bordaz
>>>> <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>
<mailto:tbordaz@redhat.com <mailto:tbordaz@redhat.com>>>
wrote:
>>>>
>>>> Hi Jonathan,
>>>>
>>>> This problem looks new to me and
has something
>>>> specific to your environment.
>>>> I think the best approach is to
continue to
>>>> debug on your system if you have
the possibility
>>>> to do so.
>>>>
>>>> From strace we can see that DS
started smoothly
>>>> (created its pid file then
notified systemd it
>>>> was running fine). According to
the pstack
>>>> nunc-stans was running and was
able to accept
>>>> network events even if it appears
it detected no
>>>> incoming connection.
>>>> So the server should be ready to
serve for some
>>>> seconds (more than a minute),
then it crashed
>>>> with one thread dereferencing
likely a wrong
>>>> pointer.
>>>>
>>>> Could you attach a debugger when
the server is
>>>> started and wait for the sigsegv
to occur. Then
>>>> confirm the crashing thread
backstack.
>>>> If it is confirmed, I am afraid
this is a stack
>>>> corruption and valgrind could help
>>>>
(http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...
>>>>
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...).
>>>>
>>>> best regards
>>>> thierry
>>>>
>>>>
>>>> On 05/14/2018 10:20 PM, Jonathan
Vaughn wrote:
>>>>> Here's a strace from before it
dies. Most of
>>>>> the elapsed time is it waiting
on some futex
>>>>> call it looks like near the end,
when it
>>>>> finally "returns" (from
lack of
strace output
>>>>> for duration of call I assume it
didn't
>>>>> actually return but SIGSEGV in
that call) and
>>>>> strace prints ' = ?' on the
futex it then
>>>>> immediately reports SIGSEGV
after. So maybe the
>>>>> problem is that futex call,
which may mean the
>>>>> problem is not directly in 389DS
/ FreeIPA itself?
>>>>>
>>>>>
>>>>>
>>>>> 15:13:31.626587 (+ 0.000630)
listen(8, 128)
>>>>> = 0 <0.000068>
>>>>> 15:13:31.626857 (+ 0.000235)
listen(9, 128)
>>>>> = 0 <0.000048>
>>>>> 15:13:31.627111 (+ 0.000251)
>>>>> clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932,
>>>>> tv_nsec=41120614}) = 0
<0.000085>
>>>>> 15:13:31.627457 (+ 0.000356)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811,
tv_nsec=627560772}) = 0
>>>>> <0.000043>
>>>>> 15:13:31.631233 (+ 0.003839)
>>>>> clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932,
>>>>> tv_nsec=45286796}) = 0
<0.000077>
>>>>> 15:13:31.631720 (+ 0.000427)
>>>>> clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932,
>>>>> tv_nsec=45661430}) = 0
<0.000042>
>>>>> 15:13:31.631955 (+ 0.000220)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811,
tv_nsec=632049036}) = 0
>>>>> <0.000047>
>>>>> 15:13:31.635669 (+ 0.003785)
>>>>> clock_gettime(CLOCK_MONOTONIC,
{tv_sec=1464932,
>>>>> tv_nsec=49725840}) = 0
<0.000146>
>>>>> 15:13:31.636484 (+ 0.000784)
write(16, "a",
>>>>> 1) = 1 <0.000118>
>>>>> 15:13:31.636855 (+ 0.000341)
sched_yield()
>>>>> = 0 <0.000252>
>>>>> 15:13:31.637322 (+ 0.000470)
>>>>> futex(0x1cb57a0,
FUTEX_WAKE_PRIVATE, 1) = 1
>>>>> <0.000088>
>>>>> 15:13:31.637897 (+ 0.000610)
write(16, "a",
>>>>> 1) = 1 <0.000221>
>>>>> 15:13:31.638394 (+ 0.000467)
sched_yield()
>>>>> = 0 <0.000047>
>>>>> 15:13:31.638619 (+ 0.000202)
>>>>> futex(0x1cb5710,
FUTEX_WAKE_PRIVATE, 1) = 1
>>>>> <0.000065>
>>>>> 15:13:31.638908 (+ 0.000298)
>>>>> openat(AT_FDCWD,
>>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666)
= 33 <0.000831>
>>>>> 15:13:31.640260 (+ 0.001387)
getpid() =
>>>>> 32353 <0.000077>
>>>>> 15:13:31.640558 (+ 0.000256)
fstat64(33,
>>>>> {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0
>>>>> <0.000119>
>>>>> 15:13:31.641106 (+ 0.000556)
write(33,
>>>>> "32353\n", 6) = 6
<0.000127>
>>>>> 15:13:31.641472 (+ 0.000362)
close(33) = 0
>>>>> <0.000519>
>>>>> 15:13:31.642216 (+ 0.000758)
>>>>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>> 0644) = 0 <0.000152>
>>>>> 15:13:31.642900 (+ 0.000679)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811,
tv_nsec=643020294}) = 0
>>>>> <0.000056>
>>>>> 15:13:31.643495 (+ 0.000590)
write(2,
>>>>>
"[14/May/2018:15:13:31.643020294
"..., 134) =
>>>>> 134 <0.002697>
>>>>> 15:13:31.646515 (+ 0.003052)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811,
tv_nsec=646694394}) = 0
>>>>> <0.000075>
>>>>> 15:13:31.646892 (+ 0.000337)
write(4,
>>>>>
"[14/May/2018:15:13:31.646694394
"..., 134) =
>>>>> 134 <0.000522>
>>>>> 15:13:31.647841 (+ 0.000973)
fsync(4) = 0
>>>>> <0.005967>
>>>>> 15:13:31.654425 (+ 0.006617)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811,
tv_nsec=654598946}) = 0
>>>>> <0.000253>
>>>>> 15:13:31.655137 (+ 0.000717)
write(2,
>>>>>
"[14/May/2018:15:13:31.654598946
"..., 136) =
>>>>> 136 <0.002427>
>>>>> 15:13:31.658312 (+ 0.003165)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1526328811,
tv_nsec=658486117}) = 0
>>>>> <0.000251>
>>>>> 15:13:31.659032 (+ 0.000682)
write(4,
>>>>>
"[14/May/2018:15:13:31.658486117
"..., 136) =
>>>>> 136 <0.000346>
>>>>> 15:13:31.659623 (+ 0.000595)
fsync(4) = 0
>>>>> <0.003311>
>>>>> 15:13:31.663230 (+ 0.003642)
getpid() =
>>>>> 32353 <0.000045>
>>>>> 15:13:31.663732 (+ 0.000454)
>>>>> socket(AF_UNIX,
SOCK_DGRAM|SOCK_CLOEXEC, 0) =
>>>>> 33 <0.000296>
>>>>> 15:13:31.664760 (+ 0.001048)
getsockopt(33,
>>>>> SOL_SOCKET, SO_SNDBUF, [163840],
[4]) = 0
>>>>> <0.000108>
>>>>> 15:13:31.665141 (+ 0.000386)
setsockopt(33,
>>>>> SOL_SOCKET, SO_SNDBUFFORCE,
[8388608], 4) = -1
>>>>> EPERM (Operation not permitted)
<0.000051>
>>>>> 15:13:31.665500 (+ 0.000334)
setsockopt(33,
>>>>> SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0 <0.000229>
>>>>> 15:13:31.665973 (+ 0.000468)
sendmsg(33,
>>>>> {msg_name={sa_family=AF_UNIX,
>>>>>
sun_path="/run/systemd/notify"},
>>>>> msg_namelen=21,
>>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>>> started: Re"..., iov_len=69}],
msg_iovlen=1,
>>>>> msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL) =
>>>>> 69 <0.000600>
>>>>> 15:13:31.667270 (+ 0.001334)
close(33) = 0
>>>>> <0.003140>
>>>>> 15:13:31.677320 (+ 0.010146)
>>>>> futex(0xb31122e8, FUTEX_WAIT,
32357, NULL) = ?
>>>>> 15:14:27.661809 (+ 55.984448)
+++ killed by
>>>>> SIGSEGV (core dumped) +++
>>>>>
>>>>> On Mon, May 14, 2018 at 10:27
AM, thierry
>>>>> bordaz <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>
>>>>> <mailto:tbordaz@redhat.com
<mailto:tbordaz@redhat.com>>> wrote:
>>>>>
>>>>> Hi Jonathan,
>>>>>
>>>>> This is weird as the
crashing thread stack
>>>>> looks truncated (did you
copy/paste all of
>>>>> it ?)
>>>>>
>>>>> Thread 1 (Thread 0x9e13c280
(LWP 17245)):
>>>>> #0 0xb67bbf2e in strlen ()
at /lib/libc.so.6
>>>>> #1 0xb6a06b40 in dosprintf
() at
>>>>> /lib/libnspr4.so
>>>>> #2 0x00000000 in None ()
>>>>>
>>>>> Did you install
389-ds-base-debuginfo ?
>>>>> How did you get that
backtrace ? from a
>>>>> core dumped, pstack ? Can
you attach a
>>>>> debugger before the crash
occurs ?
>>>>>
>>>>> It looks it crashed soon at
startup, could
>>>>> it be related to a broken
dse.ldif. It
>>>>> should exists a dse.ldif.OK,
is it possibly
>>>>> to try to start with it ?
>>>>>
>>>>> best regards
>>>>> thierry
>>>>>
>>>>> On 05/12/2018 01:22 AM,
Jonathan Vaughn via
>>>>> FreeIPA-users wrote:
>>>>>> Not sure if it makes a
difference... I was
>>>>>> looking into this again and
realized I had
>>>>>> a bunch of messages from
gdb telling me to
>>>>>> install more debuginfo.
I've done that
>>>>>> now, here it is again
freshly run through
>>>>>> gdb
>>>>>>
>>>>>> GNU gdb (GDB) Fedora
8.0.1-36.fc27
>>>>>> Copyright (C) 2017 Free
Software
>>>>>> Foundation, Inc.
>>>>>> License GPLv3+: GNU GPL
version 3 or later
>>>>>>
<http://gnu.org/licenses/gpl.html
<http://gnu.org/licenses/gpl.html>
>>>>>>
<http://gnu.org/licenses/gpl.html
<http://gnu.org/licenses/gpl.html>>>
>>>>>> This is free software: you
are free to
>>>>>> change and redistribute it.
>>>>>> There is NO WARRANTY, to
the extent
>>>>>> permitted by law. Type
"show copying"
>>>>>> and "show
warranty" for
details.
>>>>>> This GDB was configured as
>>>>>>
"armv7hl-redhat-linux-gnueabi".
>>>>>> Type "show
configuration" for
>>>>>> configuration details.
>>>>>> For bug reporting
instructions, please see:
>>>>>>
<http://www.gnu.org/software/gdb/bugs/
<http://www.gnu.org/software/gdb/bugs/>
>>>>>>
<http://www.gnu.org/software/gdb/bugs/
<http://www.gnu.org/software/gdb/bugs/>>>;.
>>>>>> Find the GDB manual and
other
>>>>>> documentation resources
online at:
>>>>>>
<http://www.gnu.org/software/gdb/documentation/
<http://www.gnu.org/software/gdb/documentation/>
>>>>>>
<http://www.gnu.org/software/gdb/documentation/
<http://www.gnu.org/software/gdb/documentation/>>>;.
>>>>>> For help, type
"help".
>>>>>> Type "apropos
word" to
search for commands
>>>>>> related to
"word"...
>>>>>> Reading symbols from
>>>>>>
/usr/sbin/ns-slapd...Reading symbols from
>>>>>>
/usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>>>>>> done.
>>>>>> ...
>>>>>>
>>>>>> Thread 1 (Thread 0x9e13c280
(LWP 17245)):
>>>>>> #0 0xb67bbf2e in strlen ()
at /lib/libc.so.6
>>>>>> #1 0xb6a06b40 in dosprintf
() at
>>>>>> /lib/libnspr4.so
>>>>>> #2 0x00000000 in None ()
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 8, 2018 at
7:52 AM, Rob
>>>>>> Crittenden
<rcritten(a)redhat.com <mailto:rcritten@redhat.com>
>>>>>>
<mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>> wrote:
>>>>>>
>>>>>> Jonathan Vaughn via
FreeIPA-users
>>>>>> wrote:
>>>>>>
>>>>>> Still trying to
figure this
>>>>>> out. It looks
like slapd is
>>>>>> dying, I
thought it was still
>>>>>> running for
some reason.
>>>>>>
>>>>>> slapd is dying
to segfault.
>>>>>> strace of it
happening doesn't
>>>>>> seem to reveal
much:
>>>>>>
>>>>>>
>>>>>> A stack trace would
very much help
>>>>>> trying to track
down the cause.
>>>>>>
>>>>>>
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-cras...
>>>>>>
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-cras...
>>>>>>
>>>>>> rob
>>>>>>
>>>>>>
>>>>>> 18:32:41.543717
(+
>>>>>> 0.000801)
openat(AT_FDCWD,
>>>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>>>
O_WRONLY|O_CREAT|O_TRUNC,
>>>>>> 0666) = 32
>>>>>> 18:32:41.544907
(+
>>>>>> 0.001195)
getpid() = 16014
>>>>>> 18:32:41.545269
(+
>>>>>> 0.000329)
fstat64(32,
>>>>>>
{st_mode=S_IFREG|0644,
>>>>>> st_size=0,
...}) = 0
>>>>>> 18:32:41.545799
(+
>>>>>> 0.000536)
write(32,
>>>>>>
"16014\n", 6) = 6
>>>>>> 18:32:41.546603
(+
>>>>>> 0.000818)
close(32) = 0
>>>>>> 18:32:41.547061
(+
>>>>>> 0.000448)
>>>>>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>>> 0644) = 0
>>>>>> 18:32:41.547741
(+
>>>>>> 0.000676)
>>>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>>
{tv_sec=1525735961,
>>>>>>
tv_nsec=548030641}) = 0
>>>>>> 18:32:41.548324
(+
>>>>>> 0.000587)
write(2,
>>>>>>
"[07/May/2018:18:32:41.548030641
>>>>>> "..., 134)
= 134
>>>>>> 18:32:41.551096
(+
>>>>>> 0.002840)
>>>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>>
{tv_sec=1525735961,
>>>>>>
tv_nsec=551287555}) = 0
>>>>>> 18:32:41.551568
(+
>>>>>> 0.000406)
write(4,
>>>>>>
"[07/May/2018:18:32:41.551287555
>>>>>> "..., 134)
= 134
>>>>>> 18:32:41.552360
(+
>>>>>> 0.000811)
fsync(4) = 0
>>>>>> 18:32:41.558499
(+
>>>>>> 0.006170)
>>>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>>
{tv_sec=1525735961,
>>>>>>
tv_nsec=558678099}) = 0
>>>>>> 18:32:41.558901
(+
>>>>>> 0.000350)
write(2,
>>>>>>
"[07/May/2018:18:32:41.558678099
>>>>>> "..., 136)
= 136
>>>>>> 18:32:41.561537
(+
>>>>>> 0.002680)
>>>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>>
{tv_sec=1525735961,
>>>>>>
tv_nsec=561718659}) = 0
>>>>>> 18:32:41.562357
(+
>>>>>> 0.000793)
write(4,
>>>>>>
"[07/May/2018:18:32:41.561718659
>>>>>> "..., 136)
= 136
>>>>>> 18:32:41.563293
(+
>>>>>> 0.001148)
fsync(4) = 0
>>>>>> 18:32:41.566928
(+
>>>>>> 0.003452)
getpid() = 16014
>>>>>> 18:32:41.567712
(+
>>>>>> 0.000752)
socket(AF_UNIX,
>>>>>>
SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>>>>> 18:32:41.568628
(+
>>>>>> 0.000912)
getsockopt(32,
>>>>>> SOL_SOCKET,
SO_SNDBUF,
>>>>>> [163840], [4]) =
0
>>>>>> 18:32:41.568972
(+
>>>>>> 0.000319)
setsockopt(32,
>>>>>> SOL_SOCKET,
SO_SNDBUFFORCE,
>>>>>> [8388608], 4) =
-1 EPERM
>>>>>> (Operation not
permitted)
>>>>>> 18:32:41.569548
(+
>>>>>> 0.000589)
setsockopt(32,
>>>>>> SOL_SOCKET,
SO_SNDBUF,
>>>>>> [8388608], 4) =
0
>>>>>> 18:32:41.570064
(+
>>>>>> 0.000513)
sendmsg(32,
>>>>>>
{msg_name={sa_family=AF_UNIX,
>>>>>>
sun_path="/run/systemd/notify"},
>>>>>> msg_namelen=21,
>>>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>>>> started:
Re"..., iov_len=69}],
>>>>>> msg_iovlen=1,
>>>>>>
msg_controllen=0,
>>>>>> msg_flags=0},
MSG_NOSIGNAL) = 69
>>>>>> 18:32:41.570845
(+
>>>>>> 0.000789)
close(32) = 0
>>>>>> 18:32:41.576358
(+
>>>>>> 0.005575)
futex(0xb30ed2e8,
>>>>>> FUTEX_WAIT,
16016, NULL) = ?
>>>>>> 18:33:01.730774
(+
>>>>>> 20.154428) +++
killed by
>>>>>> SIGSEGV +++
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
_______________________________________________
>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
<mailto:freeipa-users@lists.fedorahosted.org>
>>>>
<mailto:freeipa-users@lists.fedorahosted.org
<mailto:freeipa-users@lists.fedorahosted.org>>
>>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
<mailto:freeipa-users-leave@lists.fedorahosted.org>
>>>>
<mailto:freeipa-users-leave@lists.fedorahosted.org
<mailto:freeipa-users-leave@lists.fedorahosted.org>>
>>>
>>>
>>>
>>
>>
>
>
>
>
> _______________________________________________
> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
<mailto:freeipa-users@lists.fedorahosted.org>
> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
<mailto:freeipa-users-leave@lists.fedorahosted.org>
> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
<https://getfedora.org/code-of-conduct.html>
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
<https://fedoraproject.org/wiki/Mailing_list_guidelines>
> List Archives:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorah...
<https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorah...
>
2:04 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Build completed, I tested it, got some error about libs... you had a rogue
lib64 in the build command given :) Rebuilt with that fixed to be regular
lib (since this is a 32-bit OS) and it appears to be running fine.
Well, it hasn't segfaulted. It complains about failing to get principles
from keytab and stuff... but that is probably because I could never
complete ipa-replica-install. It's still running. I'm going to uninstall
and reinstall replica and see what happens now.
On Thu, May 17, 2018 at 9:09 PM, Mark Reynolds <mreynolds(a)redhat.com> wrote:
On 05/17/2018 08:06 PM, Jonathan Vaughn wrote:
Sorry, missed your build instructions, only saw them just now. I've just
kicked off that build and I'll check it tomorrow once I'm back in.
One note, it appears it's ./autogen.sh not ./autogen.pl ...
Whoops! Good catch, and good luck...
On Thu, May 17, 2018 at 6:16 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
> Welp, I'm still getting the -fPIC error ... so I may need to figure out
> how to do it with COPR or something.
>
> On Thu, May 17, 2018 at 4:16 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
> wrote:
>
>> I've never used COPR. I've dabbled with RPMs in the past but that was...
>> CentOS 6 I think, and I wasn't making source code changes so much as just
>> copying and pasting SRPMs from another RPM platform to build for CentOS,
>> using the regular rpmbuild stuff.
>>
>> I did actually try just copying the configure line (manually expanding
>> the various vars) from the SRPM and just doing an old fashioned make, but
>> the first try bailed shy of an hour at linking with some error about not
>> having -fPIC specified, so I restarted it with -fPIC to see if it would at
>> least build. So if I should still try the code changes, if that build
>> finishes I can try it (I was planning to just copy the respective binaries
>> on top of the installed ones), but if it turned out the suggested code
>> changes were pointless then there's no point waiting for it to finish
>> building.
>>
>> On Thu, May 17, 2018 at 4:11 PM, Rob Crittenden <rcritten(a)redhat.com>
>> wrote:
>>
>>> Jonathan Vaughn via FreeIPA-users wrote:
>>> > Oops, hit reply instead of reply-all
>>> >
>>> > NSPR RPMs
>>> >
>>> > # yum list installed nspr*
>>> > Installed Packages
>>> > nspr.armv7hl
>>>
>>> > 4.19.0-1.fc27
>>>
>>> > @updates
>>> > nspr-debuginfo.armv7hl
>>>
>>> > 4.19.0-1.fc27
>>>
>>> > @updates-debuginfo
>>> > nspr-debugsource.armv7hl
>>>
>>> > 4.19.0-1.fc27
>>>
>>> > @updates-debuginfo
>>> > nspr-devel.armv7hl
>>>
>>> > 4.19.0-1.fc27
>>>
>>> > @updates
>>> >
>>> > As for building it with the changes you proposed previously - should I
>>> > not bother? I realized last night that the last and only time I built
>>> > anything *on* a Pi was a Pi Zero and that's why it was so terribly
>>> slow.
>>> > This Pi 3 ain't fast but it's fast enough to build stuff in far
less
>>> > than half a day (unlike the Zero)...
>>>
>>> If you are handy with rpms you could build it in copr.
>>>
>>> rob
>>>
>>> >
>>> >
>>> > On Thu, May 17, 2018 at 5:07 AM, thierry bordaz <tbordaz(a)redhat.com
>>> > <mailto:tbordaz@redhat.com>> wrote:
>>> >
>>> >
>>> >
>>> > On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
>>> >> I've been just using the packages from Fedora. I can build
it
>>> >> potentially but I don't have a cross build environment set
up at
>>> >> the moment. From experience I'd want to do that first
because
>>> >> building anything on the Pi usually takes ages.
>>> >>
>>> >> I'd been "redacting" the hostnames but I'll
stop bothering since
>>> >> it looks like we're getting far enough into the weeds now
that
>>> the
>>> >> difference in string lengths after "redacting" might
actually be
>>> a
>>> >> red herring.
>>> >>
>>> >> (gdb) p *agmt
>>> >> $1 = {hostname = 0x1ef9be0
"ipa-12.creatuity.internal", port =
>>> >> 389, transport_flags = 0, binddn = 0x1e8f650 "", creds
=
>>> >> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
>>> >> frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
>>> >> frac_attr_total_defined = 1, schedule = 0x1a7f0c0,
>>> auto_initialize
>>> >> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
>>> >> long_name = 0x1a7f100
"agmt=\"cn=meToipa-12.creatuit
>>> y.internal\"
>>> >> (ipa-12:5)", protocol = 0x19c2930, changecounters =
0x186d180,
>>> >> num_changecounters = 0,
>>> >> max_changecounters = 256, last_update_start_time =
1526500697,
>>> >> last_update_end_time = 1526500697,
>>> >> last_update_status = "Error (0) Replica acquired
successfully:
>>> >> Incremental update succeeded", '\000' <repeats
954 times>,
>>> >> update_in_progress = 0, is_enabled = 1,
>>> >> last_init_start_time = 0, last_init_end_time = 0,
>>> >> last_init_status = '\000' <repeats 1023 times>,
lock = 0x1ee3740,
>>> >> consumerRUV = 0x1f14e50,
>>> >> consumerSchemaCSN = 0x317c520, consumerRID = 4,
tmpConsumerRID
>>> =
>>> >> 0, timeout = 120, stop_in_progress = 0, busywaittime = 0,
>>> >> pausetime = 0, priv = 0x0,
>>> >> attrs_to_strip = 0x1ef9ba0, agreement_type = 0,
>>> protocol_timeout
>>> >> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000,
>>> >> flowControlPause = 2000, ignoreMissingChange = 0,
>>> >> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
>>> >> (gdb) p *agmt->replarea
>>> >> $2 = {flag = 15 '\017', udn = 0x1efce80
>>> >> "dc=creatuity,dc=internal", dn = 0x1ef9460
>>> >> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
>>> >> "dc=creatuity,dc=internal", ndn_len = 24}
>>> >> (gdb) p *agmt->rdn
>>> >> $3 = {flag = 0 '\000', rdn = 0x19c2840
>>> >> "cn=meToipa-12.creatuity.internal", rdns = 0x0,
butcheredupto =
>>> >> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
>>> >>
>>> >> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
>>> >> /usr/include/inttypes.h:# define PRId64 __PRI64_PREFIX
>>> "d"
>>> >> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
>>> >> /usr/include/inttypes.h:# define PRIu16 "u"
>>> >>
>>> >>
>>> >>
>>> >> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
>>> >> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>>
wrote:
>>> >>
>>> >>
>>> >>
>>> >> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>>> >>> The installed version of 389* is 1.3.7.10-1.fc27
>>> for armv7hl,
>>> >>> which appears to be the latest available version.
>>> >>
>>> >> Perhaps something is off with the inttypes on Raspberry.
Are
>>> >> you building this yourself on Raspberry? Can we make code
>>> >> changes and compile/install them?
>>> >>
>>> >> Before we do that though, in gdb can you run these commands
>>> in
>>> >> the same gdb frame:
>>> >>
>>> >> (gdb) p *agmt->replarea
>>> >> (gdb) p *agmt->rdn
>>> >>
>>> >>
>>> >> Then do:
>>> >>
>>> >> # grep -r PRId64 /usr/include/*
>>> >> # grep -r PRIu16 /usr/include/*
>>> >>
>>> >>
>>> >> So if you can compile the source, then change this line in
>>> >> ldap/servers/plugins/replication/repl5_agmt.c:3036, but
>>> don't
>>> >> do this yet until you get me the info I just requested.
>>> >>
>>> >> From:
>>> >>
>>> >> agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%"
>>> >> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replarea),
>>> >>
>>> >>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>>> >> agmt->hostname,
>>> >>
agmt->port,
>>> >> agmt->consumerRID, maxcsn);
>>> >>
>>> >> To:
>>> >>
>>> >> agmt->maxcsn =
>>> >> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>> >> slapi_sdn_get_dn(agmt->replarea),
>>> >>
>>> >>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>>> >> agmt->hostname,
>>> >>
>>> >> (long)agmt->port, (int)agmt->consumerRID, maxcsn);
>>> >>
>>> >>
>>> >> Thanks,
>>> >> Mark
>>> >>
>>> >>>
>>> >>> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
>>> >>> <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>>
>>> wrote:
>>> >>
>>> >
>>> > The agreement structure looks valid to me. it should not lead to a
>>> > crash.
>>> >
>>> > What looks weird to me is the order of the arguments of cvt_s.
>>> > It is called: rv = cvt_s(ss, u.s, width, prec, flags);
>>> > But the crashing thread shows the opposite order: flags,
>>> prec,width,
>>> > str, ss
>>> >
>>> > The others frame do not show this change of order.
>>> > Also 'str=4' that would be a meaningful value for
'prec=4'.
>>> >
>>> > From debug perspective I only imagine disassemble the two last
>>> > frames to confirm parameters.
>>> > What are the nspr rpms ?
>>> >
>>> >
>>> >
>>> >
>>> >>> (gdb) up
>>> >>> #1 0xb6926b40 in cvt_s (flags=0, prec=<optimized
out>,
>>> >>> width=0, str=0x4 <error: Cannot access memory at
address
>>> >>> 0x4>, ss=<optimized out>)
>>> >>> at ../../.././nspr/pr/src/io/prprf.c:374
>>> >>> 374 slen = strlen(str);
>>> >>> (gdb) up
>>> >>> #2 dosprintf (ss=ss@entry=0x9e06e4bc,
fmt=0xb34b0df2
>>> "",
>>> >>> fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002",
>>> ap=...)
>>> >>> at ../../.././nspr/pr/src/io/prprf.c:1018
>>> >>> 1018 rv = cvt_s(ss, u.s, width,
prec,
>>> flags);
>>> >>> (gdb) up
>>> >>> #3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry
>>> =0xb34da770
>>> >>> <agmt_set> "\360\317p\002", ap=...,
ap@entry=...) at
>>> >>> ../../.././nspr/pr/src/io/prprf.c:1184
>>> >>> 1184 rv = dosprintf(&ss, fmt, ap);
>>> >>> (gdb) up
>>> >>> #4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0
>>> >>> "%s;%s;%s;%ld;%d;%s") at
ldap/servers/slapd/ch_malloc.c
>>> :331
>>> >>> 331 p = PR_vsmprintf(fmt, ap);
>>> >>> (gdb) up
>>> >>> #5 0xb34625a4 in agmt_update_maxcsn
>>> >>> (r=r@entry=0x2737810, sdn=0x38d9a40,
sdn@entry=0x10,
>>> >>> op=<optimized out>, mods=0x10,
mods@entry=0x27d0100,
>>> >>> csn=csn@entry=0x38de350)
>>> >>> at ldap/servers/plugins/replicati
>>> on/repl5_agmt.c:3036
>>> >>> 3036 agmt->maxcsn =
>>> >>> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>> >>> slapi_sdn_get_dn(agmt->replarea),
>>> >>> (gdb) p *agmt
>>> >>> $1 = {hostname = 0x2757be0
"ipa-12.company.internal",
>>> >>> port = 389, transport_flags = 0, binddn = 0x26ed650
"",
>>> >>> creds = 0x26ed7a0, bindmethod = 3, replarea =
0x2757480,
>>> >>> frac_attrs = 0x27579c0, frac_attrs_total =
0x2757a40,
>>> >>> frac_attr_total_defined = 1, schedule = 0x22dd0c0,
>>> >>> auto_initialize = 502, dn = 0x2756d00, rdn =
0x2756c20,
>>> >>> long_name = 0x22dd100
>>> >>>
"agmt=\"cn=meToipa-12.company.internal\" (ipa-12:5)",
>>> >>> protocol = 0x2220930, changecounters = 0x20cb180,
>>> >>> num_changecounters = 0,
>>> >>> max_changecounters = 256, last_update_start_time
=
>>> >>> 1526499214, last_update_end_time = 1526499214,
>>> >>> last_update_status = "Error (0) Replica
acquired
>>> >>> successfully: Incremental update succeeded",
'\000'
>>> >>> <repeats 954 times>, update_in_progress = 0,
is_enabled
>>> = 1,
>>> >>> last_init_start_time = 0, last_init_end_time = 0,
>>> >>> last_init_status = '\000' <repeats 1023
times>, lock =
>>> >>> 0x2741740, consumerRUV = 0x2772e50,
>>> >>> consumerSchemaCSN = 0x39da520, consumerRID = 4,
>>> >>> tmpConsumerRID = 0, timeout = 120, stop_in_progress
= 0,
>>> >>> busywaittime = 0, pausetime = 0, priv = 0x0,
>>> >>> attrs_to_strip = 0x2757ba0, agreement_type = 0,
>>> >>> protocol_timeout = 0x26ed5f0, maxcsn = 0x0,
>>> >>> flowControlWindow = 1000, flowControlPause = 2000,
>>> >>> ignoreMissingChange = 0,
>>> >>> attr_lock = 0x2757c20, WaitForAsyncResults = 100}
>>> >>>
>>> >>>
>>> >>> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
>>> >>> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>>
>>> wrote:
>>> >>>
>>> >>> This looks really familiar and I thought it was
>>> >>> fixed. It should have been fixed in 1.3.7.10-1
>>> >>> (https://pagure.io/389-ds-base/issue/49618
>>> >>>
<https://pagure.io/389-ds-base/issue/49618>). In
>>> >>> your debug session go "up" into
agmt_maxcsn_update()
>>> >>> and do:
>>> >>>
>>> >>> (gdb) p *agmt
>>> >>>
>>> >>> Then send us that output please.
>>> >>>
>>> >>> Thanks,
>>> >>> Mark
>>> >>>
>>> >>>
>>> >>> On 05/15/2018 05:29 PM, Jonathan Vaughn via
>>> >>> FreeIPA-users wrote:
>>> >>>> Here is a backtrace from live gdb after the
>>> >>>> segfault. Looks like things went wrong
somewhere
>>> >>>> during in the replication code ?
>>> >>>>
>>> >>>> Thread 36 "ns-slapd" received
signal SIGSEGV,
>>> >>>> Segmentation fault.
>>> >>>> [Switching to Thread 0x9e0bc280 (LWP 4662)]
>>> >>>> strlen () at
../sysdeps/arm/armv6t2/strlen.S:142
>>> >>>> 142 ldrd data1a, data1b,
[src]
>>> >>>> (gdb) bt
>>> >>>> #0 0xb6728f2e in strlen () at
>>> >>>> ../sysdeps/arm/armv6t2/strlen.S:142
>>> >>>> #1 0xb6973b40 in cvt_s (flags=0,
prec=<optimized
>>> >>>> out>, width=0, str=0x4 <error: Cannot
access memory
>>> >>>> at address 0x4>, ss=<optimized
out>)
>>> >>>> at
../../.././nspr/pr/src/io/prprf.c:374
>>> >>>> #2 0xb6973b40 in dosprintf (ss=ss@entry
>>> =0x9e0bb4bc,
>>> >>>> fmt=0xb34fddf2 "",
fmt@entry=0xb3527770 <agmt_set>
>>> >>>> "\360\357\305\002", ap=...)
>>> >>>> at
../../.././nspr/pr/src/io/prprf.c:1018
>>> >>>> #3 0xb6973c8c in PR_vsmprintf
>>> >>>> (fmt=fmt@entry=0xb3527770 <agmt_set>
>>> >>>> "\360\357\305\002", ap=...,
ap@entry=...) at
>>> >>>> ../../.././nspr/pr/src/io/prprf.c:1184
>>> >>>> #4 0xb6e36d18 in slapi_ch_smprintf
(fmt=0xb34fdde0
>>> >>>> "%s;%s;%s;%ld;%d;%s") at
>>> >>>> ldap/servers/slapd/ch_malloc.c:331
>>> >>>> #5 0xb34af5a4 in agmt_update_maxcsn
>>> >>>> (r=r@entry=0x2c89810, sdn=0x3e2bac0,
sdn@entry
>>> =0x10,
>>> >>>> op=<optimized out>, mods=0x10,
mods@entry
>>> =0x3eec100,
>>> >>>> csn=csn@entry=0x3e30220)
>>> >>>> at
>>> >>>>
ldap/servers/plugins/replication/repl5_agmt.c:3036
>>> >>>> #6 0xb34bd424 in write_changelog_and_ruv
>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>> >>>> ldap/servers/plugins/replicat
>>> ion/repl5_plugins.c:1124
>>> >>>> #7 0xb34be7ec in
multimaster_be_betxnpostop_add
>>> >>>> (pb=0x2cb54a0) at
>>> >>>> ldap/servers/plugins/replicat
>>> ion/repl5_plugins.c:855
>>> >>>> #8 0xb34be880 in multimaster_mmr_postop
>>> >>>> (pb=<optimized out>, flags=560) at
>>> >>>> ldap/servers/plugins/replicat
>>> ion/repl5_plugins.c:616
>>> >>>> #9 0xb6e8fcf0 in
plugin_call_mmr_plugin_postop
>>> >>>> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0,
>>> >>>> flags=flags@entry=560) at
>>> >>>> ldap/servers/slapd/plugin_mmr.c:65
>>> >>>> #10 0xb35ba870 in ldbm_back_add
(pb=0x2cb54a0) at
>>> >>>>
ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>>> >>>> #11 0xb6e2ce4c in op_shared_add
>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>> ldap/servers/slapd/add.c:679
>>> >>>> #12 0xb6e2d35c in add_internal_pb
>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>> ldap/servers/slapd/add.c:407
>>> >>>> #13 0xb6e2e0f8 in slapi_add_internal_pb
>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>> ldap/servers/slapd/add.c:332
>>> >>>> #14 0xb368db50 in
ipa_topo_util_segment_write
>>> >>>> (tconf=tconf@entry=0x2cb5860,
>>> >>>> tsegm=tsegm@entry=0x3f2d9a0) at
>>> topology_util.c:1251
>>> >>>> #15 0xb368e01c in
ipa_topo_util_update_agmt_list
>>> >>>> (conf=0x2cb5860, repl_segments=<optimized
out>) at
>>> >>>> topology_util.c:696
>>> >>>> #16 0xb368891c in
ipa_topo_apply_shared_config ()
>>> at
>>> >>>> topology_init.c:165
>>> >>>> #17 0xb6e4e65c in eq_call_all () at
>>> >>>> ldap/servers/slapd/eventq.c:278
>>> >>>> #18 0xb6e4e65c in eq_loop (arg=<optimized
out>) at
>>> >>>> ldap/servers/slapd/eventq.c:323
>>> >>>> #19 0xb6982d68 in _pt_root (arg=0x2c8da40)
at
>>> >>>>
../../.././nspr/pr/src/pthreads/ptthread.c:201
>>> >>>> #20 0xb6906ee8 in start_thread
(arg=0x9e0bc280) at
>>> >>>> pthread_create.c:465
>>> >>>> #21 0xb6785da8 in None () at
>>> >>>> ../sysdeps/unix/sysv/linux/arm/clone.S:73
>>> >>>>
>>> >>>>
>>> >>>> On Tue, May 15, 2018 at 3:01 AM, thierry
bordaz
>>> >>>> <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>>
>>> wrote:
>>> >>>>
>>> >>>> Hi Jonathan,
>>> >>>>
>>> >>>> This problem looks new to me and has
something
>>> >>>> specific to your environment.
>>> >>>> I think the best approach is to continue
to
>>> >>>> debug on your system if you have the
>>> possibility
>>> >>>> to do so.
>>> >>>>
>>> >>>> From strace we can see that DS started
smoothly
>>> >>>> (created its pid file then notified
systemd it
>>> >>>> was running fine). According to the
pstack
>>> >>>> nunc-stans was running and was able to
accept
>>> >>>> network events even if it appears it
detected
>>> no
>>> >>>> incoming connection.
>>> >>>> So the server should be ready to serve
for some
>>> >>>> seconds (more than a minute), then it
crashed
>>> >>>> with one thread dereferencing likely a
wrong
>>> >>>> pointer.
>>> >>>>
>>> >>>> Could you attach a debugger when the
server is
>>> >>>> started and wait for the sigsegv to
occur. Then
>>> >>>> confirm the crashing thread backstack.
>>> >>>> If it is confirmed, I am afraid this is
a stack
>>> >>>> corruption and valgrind could help
>>> >>>> (http://www.port389.org/docs/
>>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>>> >>>> <http://www.port389.org/docs/
>>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>>> >).
>>> >>>>
>>> >>>> best regards
>>> >>>> thierry
>>> >>>>
>>> >>>>
>>> >>>> On 05/14/2018 10:20 PM, Jonathan Vaughn
wrote:
>>> >>>>> Here's a strace from before it
dies. Most of
>>> >>>>> the elapsed time is it waiting on
some futex
>>> >>>>> call it looks like near the end,
when it
>>> >>>>> finally "returns" (from
lack of strace output
>>> >>>>> for duration of call I assume it
didn't
>>> >>>>> actually return but SIGSEGV in that
call) and
>>> >>>>> strace prints ' = ?' on the
futex it then
>>> >>>>> immediately reports SIGSEGV after.
So maybe
>>> the
>>> >>>>> problem is that futex call, which
may mean the
>>> >>>>> problem is not directly in 389DS /
FreeIPA
>>> itself?
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> 15:13:31.626587 (+ 0.000630)
listen(8,
>>> 128)
>>> >>>>> = 0 <0.000068>
>>> >>>>> 15:13:31.626857 (+ 0.000235)
listen(9,
>>> 128)
>>> >>>>> = 0 <0.000048>
>>> >>>>> 15:13:31.627111 (+ 0.000251)
>>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>>> {tv_sec=1464932,
>>> >>>>> tv_nsec=41120614}) = 0
<0.000085>
>>> >>>>> 15:13:31.627457 (+ 0.000356)
>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>> >>>>> {tv_sec=1526328811,
tv_nsec=627560772}) = 0
>>> >>>>> <0.000043>
>>> >>>>> 15:13:31.631233 (+ 0.003839)
>>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>>> {tv_sec=1464932,
>>> >>>>> tv_nsec=45286796}) = 0
<0.000077>
>>> >>>>> 15:13:31.631720 (+ 0.000427)
>>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>>> {tv_sec=1464932,
>>> >>>>> tv_nsec=45661430}) = 0
<0.000042>
>>> >>>>> 15:13:31.631955 (+ 0.000220)
>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>> >>>>> {tv_sec=1526328811,
tv_nsec=632049036}) = 0
>>> >>>>> <0.000047>
>>> >>>>> 15:13:31.635669 (+ 0.003785)
>>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>>> {tv_sec=1464932,
>>> >>>>> tv_nsec=49725840}) = 0
<0.000146>
>>> >>>>> 15:13:31.636484 (+ 0.000784)
write(16,
>>> "a",
>>> >>>>> 1) = 1 <0.000118>
>>> >>>>> 15:13:31.636855 (+ 0.000341)
sched_yield()
>>> >>>>> = 0 <0.000252>
>>> >>>>> 15:13:31.637322 (+ 0.000470)
>>> >>>>> futex(0x1cb57a0, FUTEX_WAKE_PRIVATE,
1) = 1
>>> >>>>> <0.000088>
>>> >>>>> 15:13:31.637897 (+ 0.000610)
write(16,
>>> "a",
>>> >>>>> 1) = 1 <0.000221>
>>> >>>>> 15:13:31.638394 (+ 0.000467)
sched_yield()
>>> >>>>> = 0 <0.000047>
>>> >>>>> 15:13:31.638619 (+ 0.000202)
>>> >>>>> futex(0x1cb5710, FUTEX_WAKE_PRIVATE,
1) = 1
>>> >>>>> <0.000065>
>>> >>>>> 15:13:31.638908 (+ 0.000298)
>>> >>>>> openat(AT_FDCWD,
>>> >>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>> >>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) =
33
>>> <0.000831>
>>> >>>>> 15:13:31.640260 (+ 0.001387)
getpid() =
>>> >>>>> 32353 <0.000077>
>>> >>>>> 15:13:31.640558 (+ 0.000256)
fstat64(33,
>>> >>>>> {st_mode=S_IFREG|0644, st_size=0,
...}) = 0
>>> >>>>> <0.000119>
>>> >>>>> 15:13:31.641106 (+ 0.000556)
write(33,
>>> >>>>> "32353\n", 6) = 6
<0.000127>
>>> >>>>> 15:13:31.641472 (+ 0.000362)
close(33) = 0
>>> >>>>> <0.000519>
>>> >>>>> 15:13:31.642216 (+ 0.000758)
>>> >>>>> chmod("/var/run/dirsrv/slapd-
>>> COMPANY-INTERNAL.pid",
>>> >>>>> 0644) = 0 <0.000152>
>>> >>>>> 15:13:31.642900 (+ 0.000679)
>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>> >>>>> {tv_sec=1526328811,
tv_nsec=643020294}) = 0
>>> >>>>> <0.000056>
>>> >>>>> 15:13:31.643495 (+ 0.000590)
write(2,
>>> >>>>>
"[14/May/2018:15:13:31.643020294 "..., 134) =
>>> >>>>> 134 <0.002697>
>>> >>>>> 15:13:31.646515 (+ 0.003052)
>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>> >>>>> {tv_sec=1526328811,
tv_nsec=646694394}) = 0
>>> >>>>> <0.000075>
>>> >>>>> 15:13:31.646892 (+ 0.000337)
write(4,
>>> >>>>>
"[14/May/2018:15:13:31.646694394 "..., 134) =
>>> >>>>> 134 <0.000522>
>>> >>>>> 15:13:31.647841 (+ 0.000973)
fsync(4) = 0
>>> >>>>> <0.005967>
>>> >>>>> 15:13:31.654425 (+ 0.006617)
>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>> >>>>> {tv_sec=1526328811,
tv_nsec=654598946}) = 0
>>> >>>>> <0.000253>
>>> >>>>> 15:13:31.655137 (+ 0.000717)
write(2,
>>> >>>>>
"[14/May/2018:15:13:31.654598946 "..., 136) =
>>> >>>>> 136 <0.002427>
>>> >>>>> 15:13:31.658312 (+ 0.003165)
>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>> >>>>> {tv_sec=1526328811,
tv_nsec=658486117}) = 0
>>> >>>>> <0.000251>
>>> >>>>> 15:13:31.659032 (+ 0.000682)
write(4,
>>> >>>>>
"[14/May/2018:15:13:31.658486117 "..., 136) =
>>> >>>>> 136 <0.000346>
>>> >>>>> 15:13:31.659623 (+ 0.000595)
fsync(4) = 0
>>> >>>>> <0.003311>
>>> >>>>> 15:13:31.663230 (+ 0.003642)
getpid() =
>>> >>>>> 32353 <0.000045>
>>> >>>>> 15:13:31.663732 (+ 0.000454)
>>> >>>>> socket(AF_UNIX,
SOCK_DGRAM|SOCK_CLOEXEC, 0) =
>>> >>>>> 33 <0.000296>
>>> >>>>> 15:13:31.664760 (+ 0.001048)
>>> getsockopt(33,
>>> >>>>> SOL_SOCKET, SO_SNDBUF, [163840],
[4]) = 0
>>> >>>>> <0.000108>
>>> >>>>> 15:13:31.665141 (+ 0.000386)
>>> setsockopt(33,
>>> >>>>> SOL_SOCKET, SO_SNDBUFFORCE,
[8388608], 4) = -1
>>> >>>>> EPERM (Operation not permitted)
<0.000051>
>>> >>>>> 15:13:31.665500 (+ 0.000334)
>>> setsockopt(33,
>>> >>>>> SOL_SOCKET, SO_SNDBUF, [8388608], 4)
= 0
>>> <0.000229>
>>> >>>>> 15:13:31.665973 (+ 0.000468)
sendmsg(33,
>>> >>>>> {msg_name={sa_family=AF_UNIX,
>>> >>>>>
sun_path="/run/systemd/notify"},
>>> >>>>> msg_namelen=21,
>>> >>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>> >>>>> started: Re"..., iov_len=69}],
msg_iovlen=1,
>>> >>>>> msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL)
>>> =
>>> >>>>> 69 <0.000600>
>>> >>>>> 15:13:31.667270 (+ 0.001334)
close(33) = 0
>>> >>>>> <0.003140>
>>> >>>>> 15:13:31.677320 (+ 0.010146)
>>> >>>>> futex(0xb31122e8, FUTEX_WAIT, 32357,
NULL) = ?
>>> >>>>> 15:14:27.661809 (+ 55.984448) +++
killed by
>>> >>>>> SIGSEGV (core dumped) +++
>>> >>>>>
>>> >>>>> On Mon, May 14, 2018 at 10:27 AM,
thierry
>>> >>>>> bordaz <tbordaz(a)redhat.com
>>> >>>>>
<mailto:tbordaz@redhat.com>> wrote:
>>> >>>>>
>>> >>>>> Hi Jonathan,
>>> >>>>>
>>> >>>>> This is weird as the crashing
thread stack
>>> >>>>> looks truncated (did you
copy/paste all of
>>> >>>>> it ?)
>>> >>>>>
>>> >>>>> Thread 1 (Thread 0x9e13c280 (LWP
17245)):
>>> >>>>> #0 0xb67bbf2e in strlen () at
>>> /lib/libc.so.6
>>> >>>>> #1 0xb6a06b40 in dosprintf ()
at
>>> >>>>> /lib/libnspr4.so
>>> >>>>> #2 0x00000000 in None ()
>>> >>>>>
>>> >>>>> Did you install
389-ds-base-debuginfo ?
>>> >>>>> How did you get that backtrace ?
from a
>>> >>>>> core dumped, pstack ? Can you
attach a
>>> >>>>> debugger before the crash occurs
?
>>> >>>>>
>>> >>>>> It looks it crashed soon at
startup, could
>>> >>>>> it be related to a broken
dse.ldif. It
>>> >>>>> should exists a dse.ldif.OK, is
it
>>> possibly
>>> >>>>> to try to start with it ?
>>> >>>>>
>>> >>>>> best regards
>>> >>>>> thierry
>>> >>>>>
>>> >>>>> On 05/12/2018 01:22 AM, Jonathan
Vaughn
>>> via
>>> >>>>> FreeIPA-users wrote:
>>> >>>>>> Not sure if it makes a
difference... I
>>> was
>>> >>>>>> looking into this again and
realized I
>>> had
>>> >>>>>> a bunch of messages from gdb
telling me
>>> to
>>> >>>>>> install more debuginfo.
I've done that
>>> >>>>>> now, here it is again
freshly run through
>>> >>>>>> gdb
>>> >>>>>>
>>> >>>>>> GNU gdb (GDB) Fedora
8.0.1-36.fc27
>>> >>>>>> Copyright (C) 2017 Free
Software
>>> >>>>>> Foundation, Inc.
>>> >>>>>> License GPLv3+: GNU GPL
version 3 or
>>> later
>>> >>>>>>
<http://gnu.org/licenses/gpl.html
>>> >>>>>>
<http://gnu.org/licenses/gpl.html>>
>>> >>>>>> This is free software: you
are free to
>>> >>>>>> change and redistribute it.
>>> >>>>>> There is NO WARRANTY, to the
extent
>>> >>>>>> permitted by law. Type
"show copying"
>>> >>>>>> and "show
warranty" for details.
>>> >>>>>> This GDB was configured as
>>> >>>>>>
"armv7hl-redhat-linux-gnueabi".
>>> >>>>>> Type "show
configuration" for
>>> >>>>>> configuration details.
>>> >>>>>> For bug reporting
instructions, please
>>> see:
>>> >>>>>>
<http://www.gnu.org/software/gdb/bugs/
>>> >>>>>>
<http://www.gnu.org/software/gdb/bugs/
>>> >>.
>>> >>>>>> Find the GDB manual and
other
>>> >>>>>> documentation resources
online at:
>>> >>>>>>
<http://www.gnu.org/software/
>>> gdb/documentation/
>>> >>>>>>
<http://www.gnu.org/software/
>>> gdb/documentation/>>.
>>> >>>>>> For help, type
"help".
>>> >>>>>> Type "apropos
word" to search for
>>> commands
>>> >>>>>> related to
"word"...
>>> >>>>>> Reading symbols from
>>> >>>>>> /usr/sbin/ns-slapd...Reading
symbols from
>>> >>>>>>
/usr/lib/debug/usr/sbin/ns-sl
>>> apd-1.3.7.10-1.fc27.arm.debug...done.
>>> >>>>>> done.
>>> >>>>>> ...
>>> >>>>>>
>>> >>>>>> Thread 1 (Thread 0x9e13c280
(LWP 17245)):
>>> >>>>>> #0 0xb67bbf2e in strlen ()
at
>>> /lib/libc.so.6
>>> >>>>>> #1 0xb6a06b40 in dosprintf
() at
>>> >>>>>> /lib/libnspr4.so
>>> >>>>>> #2 0x00000000 in None ()
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Tue, May 8, 2018 at
7:52 AM, Rob
>>> >>>>>> Crittenden
<rcritten(a)redhat.com
>>> >>>>>>
<mailto:rcritten@redhat.com>> wrote:
>>> >>>>>>
>>> >>>>>> Jonathan Vaughn via
FreeIPA-users
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>> Still trying to
figure this
>>> >>>>>> out. It looks
like slapd is
>>> >>>>>> dying, I thought
it was still
>>> >>>>>> running for some
reason.
>>> >>>>>>
>>> >>>>>> slapd is dying
to segfault.
>>> >>>>>> strace of it
happening
>>> doesn't
>>> >>>>>> seem to reveal
much:
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> A stack trace would
very much
>>> help
>>> >
>>>
>> ...
[Message clipped]
2:10 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
On 05/18/2018 03:04 PM, Jonathan Vaughn wrote:
Build completed, I tested it, got some error about libs... you had a
rogue lib64 in the build command given :)
Ah, yeah sorry that configure command was
not fully tested I guess (I'll
update my notes)...
Rebuilt with that fixed to be regular lib (since this is a 32-bit
OS)
and it appears to be running fine.
Well, it hasn't segfaulted. It complains about failing to get
principles from keytab and stuff... but that is probably because I
could never complete ipa-replica-install. It's still running. I'm
going to uninstall and reinstall replica and see what happens now.
Keep us posted
Thanks,
Mark
On Thu, May 17, 2018 at 9:09 PM, Mark Reynolds <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>> wrote:
On 05/17/2018 08:06 PM, Jonathan Vaughn wrote:
> Sorry, missed your build instructions, only saw them just now.
> I've just kicked off that build and I'll check it tomorrow once
> I'm back in.
>
> One note, it appears it's ./autogen.sh not ./autogen.pl
> <http://autogen.pl> ...
Whoops! Good catch, and good luck...
>
> On Thu, May 17, 2018 at 6:16 PM, Jonathan Vaughn
> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
>
> Welp, I'm still getting the -fPIC error ... so I may need to
> figure out how to do it with COPR or something.
>
> On Thu, May 17, 2018 at 4:16 PM, Jonathan Vaughn
> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
>
> I've never used COPR. I've dabbled with RPMs in the past
> but that was... CentOS 6 I think, and I wasn't making
> source code changes so much as just copying and pasting
> SRPMs from another RPM platform to build for CentOS,
> using the regular rpmbuild stuff.
>
> I did actually try just copying the configure line
> (manually expanding the various vars) from the SRPM and
> just doing an old fashioned make, but the first try
> bailed shy of an hour at linking with some error about
> not having -fPIC specified, so I restarted it with -fPIC
> to see if it would at least build. So if I should still
> try the code changes, if that build finishes I can try it
> (I was planning to just copy the respective binaries on
> top of the installed ones), but if it turned out the
> suggested code changes were pointless then there's no
> point waiting for it to finish building.
>
> On Thu, May 17, 2018 at 4:11 PM, Rob Crittenden
> <rcritten(a)redhat.com <mailto:rcritten@redhat.com>> wrote:
>
> Jonathan Vaughn via FreeIPA-users wrote:
> > Oops, hit reply instead of reply-all
> >
> > NSPR RPMs
> >
> > # yum list installed nspr*
> > Installed Packages
> > nspr.armv7hl
>
> > 4.19.0-1.fc27
>
> > @updates
> > nspr-debuginfo.armv7hl
>
> > 4.19.0-1.fc27
>
> > @updates-debuginfo
> > nspr-debugsource.armv7hl
>
> > 4.19.0-1.fc27
>
> > @updates-debuginfo
> > nspr-devel.armv7hl
>
> > 4.19.0-1.fc27
>
> > @updates
> >
> > As for building it with the changes you proposed
> previously - should I
> > not bother? I realized last night that the last and
> only time I built
> > anything *on* a Pi was a Pi Zero and that's why it
> was so terribly slow.
> > This Pi 3 ain't fast but it's fast enough to build
> stuff in far less
> > than half a day (unlike the Zero)...
>
> If you are handy with rpms you could build it in copr.
>
> rob
>
> >
> >
> > On Thu, May 17, 2018 at 5:07 AM, thierry bordaz
> <tbordaz(a)redhat.com <mailto:tbordaz@redhat.com>
> > <mailto:tbordaz@redhat.com
> <mailto:tbordaz@redhat.com>>> wrote:
> >
> >
> >
> > On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
> >> I've been just using the packages from Fedora.
> I can build it
> >> potentially but I don't have a cross build
> environment set up at
> >> the moment. From experience I'd want to do
> that first because
> >> building anything on the Pi usually takes ages.
> >>
> >> I'd been "redacting" the hostnames but
I'll
> stop bothering since
> >> it looks like we're getting far enough into
> the weeds now that the
> >> difference in string lengths after
"redacting"
> might actually be a
> >> red herring.
> >>
> >> (gdb) p *agmt
> >> $1 = {hostname = 0x1ef9be0
> "ipa-12.creatuity.internal", port =
> >> 389, transport_flags = 0, binddn = 0x1e8f650
> "", creds =
> >> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
> >> frac_attrs = 0x1ef99c0, frac_attrs_total =
> 0x1ef9a40,
> >> frac_attr_total_defined = 1, schedule =
> 0x1a7f0c0, auto_initialize
> >> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
> >> long_name = 0x1a7f100
> "agmt=\"cn=meToipa-12.creatuity.internal\"
> >> (ipa-12:5)", protocol = 0x19c2930,
> changecounters = 0x186d180,
> >> num_changecounters = 0,
> >> max_changecounters = 256,
> last_update_start_time = 1526500697,
> >> last_update_end_time = 1526500697,
> >> last_update_status = "Error (0) Replica
> acquired successfully:
> >> Incremental update succeeded", '\000'
<repeats
> 954 times>,
> >> update_in_progress = 0, is_enabled = 1,
> >> last_init_start_time = 0, last_init_end_time
> = 0,
> >> last_init_status = '\000' <repeats 1023
> times>, lock = 0x1ee3740,
> >> consumerRUV = 0x1f14e50,
> >> consumerSchemaCSN = 0x317c520, consumerRID =
> 4, tmpConsumerRID =
> >> 0, timeout = 120, stop_in_progress = 0,
> busywaittime = 0,
> >> pausetime = 0, priv = 0x0,
> >> attrs_to_strip = 0x1ef9ba0, agreement_type =
> 0, protocol_timeout
> >> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow =
> 1000,
> >> flowControlPause = 2000, ignoreMissingChange = 0,
> >> attr_lock = 0x1ef9c20, WaitForAsyncResults =
> 100}
> >> (gdb) p *agmt->replarea
> >> $2 = {flag = 15 '\017', udn = 0x1efce80
> >> "dc=creatuity,dc=internal", dn = 0x1ef9460
> >> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
> >> "dc=creatuity,dc=internal", ndn_len = 24}
> >> (gdb) p *agmt->rdn
> >> $3 = {flag = 0 '\000', rdn = 0x19c2840
> >> "cn=meToipa-12.creatuity.internal", rdns =
> 0x0, butcheredupto =
> >> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
> >>
> >> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
> >> /usr/include/inttypes.h:# define PRId64
> __PRI64_PREFIX "d"
> >> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
> >> /usr/include/inttypes.h:# define PRIu16
> "u"
> >>
> >>
> >>
> >> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
> >> <mreynolds(a)redhat.com
> <mailto:mreynolds@redhat.com>
> <mailto:mreynolds@redhat.com
> <mailto:mreynolds@redhat.com>>> wrote:
> >>
> >>
> >>
> >> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
> >>> The installed version of 389*
> is 1.3.7.10-1.fc27 for armv7hl,
> >>> which appears to be the latest available
> version.
> >>
> >> Perhaps something is off with the inttypes
> on Raspberry. Are
> >> you building this yourself on Raspberry?
> Can we make code
> >> changes and compile/install them?
> >>
> >> Before we do that though, in gdb can you
> run these commands in
> >> the same gdb frame:
> >>
> >> (gdb) p *agmt->replarea
> >> (gdb) p *agmt->rdn
> >>
> >>
> >> Then do:
> >>
> >> # grep -r PRId64 /usr/include/*
> >> # grep -r PRIu16 /usr/include/*
> >>
> >>
> >> So if you can compile the source, then
> change this line in
> >>
> ldap/servers/plugins/replication/repl5_agmt.c:3036,
> but don't
> >> do this yet until you get me the info I
> just requested.
> >>
> >> From:
> >>
> >> agmt->maxcsn =
> slapi_ch_smprintf("%s;%s;%s;%"
> >> PRId64 ";%" PRIu16 ";%s",
> slapi_sdn_get_dn(agmt->replarea),
> >>
>
> >>
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
> >> agmt->hostname,
> >>
>
> agmt->port,
> >> agmt->consumerRID, maxcsn);
> >>
> >> To:
> >>
> >> agmt->maxcsn =
> >> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> >> slapi_sdn_get_dn(agmt->replarea),
> >>
>
> >>
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
> >> agmt->hostname,
> >>
>
> >> (long)agmt->port, (int)agmt->consumerRID,
> maxcsn);
> >>
> >>
> >> Thanks,
> >> Mark
> >>
> >>>
> >>> On Wed, May 16, 2018 at 2:38 PM, Jonathan
> Vaughn
> >>> <jonathan(a)creatuity.com
> <mailto:jonathan@creatuity.com>
> <mailto:jonathan@creatuity.com
> <mailto:jonathan@creatuity.com>>> wrote:
> >>
> >
> > The agreement structure looks valid to me. it
> should not lead to a
> > crash.
> >
> > What looks weird to me is the order of the
> arguments of cvt_s.
> > It is called: rv = cvt_s(ss, u.s, width, prec,
> flags);
> > But the crashing thread shows the opposite
> order: flags, prec,width,
> > str, ss
> >
> > The others frame do not show this change of order.
> > Also 'str=4' that would be a meaningful value
> for 'prec=4'.
> >
> > From debug perspective I only imagine
> disassemble the two last
> > frames to confirm parameters.
> > What are the nspr rpms ?
> >
> >
> >
> >
> >>> (gdb) up
> >>> #1 0xb6926b40 in cvt_s (flags=0,
> prec=<optimized out>,
> >>> width=0, str=0x4 <error: Cannot
> access memory at address
> >>> 0x4>, ss=<optimized out>)
> >>> at
> ../../.././nspr/pr/src/io/prprf.c:374
> >>> 374 slen = strlen(str);
> >>> (gdb) up
> >>> #2 dosprintf
> (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2 "",
> >>> fmt@entry=0xb34da770 <agmt_set>
> "\360\317p\002", ap=...)
> >>> at ../../.././nspr/pr/src/io/prprf.c:1018
> >>> 1018 rv = cvt_s(ss,
> u.s, width, prec, flags);
> >>> (gdb) up
> >>> #3 0xb6926c8c in PR_vsmprintf
> (fmt=fmt@entry=0xb34da770
> >>> <agmt_set> "\360\317p\002",
ap=...,
> ap@entry=...) at
> >>> ../../.././nspr/pr/src/io/prprf.c:1184
> >>> 1184 rv = dosprintf(&ss, fmt,
ap);
> >>> (gdb) up
> >>> #4 0xb6de9d18 in slapi_ch_smprintf
> (fmt=0xb34b0de0
> >>> "%s;%s;%s;%ld;%d;%s") at
> ldap/servers/slapd/ch_malloc.c:331
> >>> 331 p = PR_vsmprintf(fmt, ap);
> >>> (gdb) up
> >>> #5 0xb34625a4 in agmt_update_maxcsn
> >>> (r=r@entry=0x2737810, sdn=0x38d9a40,
> sdn@entry=0x10,
> >>> op=<optimized out>, mods=0x10,
> mods@entry=0x27d0100,
> >>> csn=csn@entry=0x38de350)
> >>> at
> ldap/servers/plugins/replication/repl5_agmt.c:3036
> >>> 3036 agmt->maxcsn =
> >>>
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> >>> slapi_sdn_get_dn(agmt->replarea),
> >>> (gdb) p *agmt
> >>> $1 = {hostname = 0x2757be0
> "ipa-12.company.internal",
> >>> port = 389, transport_flags = 0,
> binddn = 0x26ed650 "",
> >>> creds = 0x26ed7a0, bindmethod = 3,
> replarea = 0x2757480,
> >>> frac_attrs = 0x27579c0,
> frac_attrs_total = 0x2757a40,
> >>> frac_attr_total_defined = 1, schedule
> = 0x22dd0c0,
> >>> auto_initialize = 502, dn =
> 0x2756d00, rdn = 0x2756c20,
> >>> long_name = 0x22dd100
> >>>
> "agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)",
> >>> protocol = 0x2220930, changecounters
> = 0x20cb180,
> >>> num_changecounters = 0,
> >>> max_changecounters = 256,
> last_update_start_time =
> >>> 1526499214, last_update_end_time =
> 1526499214,
> >>> last_update_status = "Error (0)
> Replica acquired
> >>> successfully: Incremental update
> succeeded", '\000'
> >>> <repeats 954 times>,
> update_in_progress = 0, is_enabled = 1,
> >>> last_init_start_time = 0,
> last_init_end_time = 0,
> >>> last_init_status = '\000'
<repeats
> 1023 times>, lock =
> >>> 0x2741740, consumerRUV = 0x2772e50,
> >>> consumerSchemaCSN = 0x39da520,
> consumerRID = 4,
> >>> tmpConsumerRID = 0, timeout = 120,
> stop_in_progress = 0,
> >>> busywaittime = 0, pausetime = 0, priv
> = 0x0,
> >>> attrs_to_strip = 0x2757ba0,
> agreement_type = 0,
> >>> protocol_timeout = 0x26ed5f0, maxcsn
> = 0x0,
> >>> flowControlWindow = 1000,
> flowControlPause = 2000,
> >>> ignoreMissingChange = 0,
> >>> attr_lock = 0x2757c20,
> WaitForAsyncResults = 100}
> >>>
> >>>
> >>> On Tue, May 15, 2018 at 5:36 PM, Mark
> Reynolds
> >>> <mreynolds(a)redhat.com
> <mailto:mreynolds@redhat.com>
> <mailto:mreynolds@redhat.com
> <mailto:mreynolds@redhat.com>>> wrote:
> >>>
> >>> This looks really familiar and I
> thought it was
> >>> fixed. It should have been fixed
> in 1.3.7.10-1
> >>>
> (https://pagure.io/389-ds-base/issue/49618
> <https://pagure.io/389-ds-base/issue/49618>
> >>>
> <https://pagure.io/389-ds-base/issue/49618
> <https://pagure.io/389-ds-base/issue/49618>>). In
> >>> your debug session go "up"
into
> agmt_maxcsn_update()
> >>> and do:
> >>>
> >>> (gdb) p *agmt
> >>>
> >>> Then send us that output please.
> >>>
> >>> Thanks,
> >>> Mark
> >>>
> >>>
> >>> On 05/15/2018 05:29 PM, Jonathan
> Vaughn via
> >>> FreeIPA-users wrote:
> >>>> Here is a backtrace from live
> gdb after the
> >>>> segfault. Looks like things went
> wrong somewhere
> >>>> during in the replication code ?
> >>>>
> >>>> Thread 36 "ns-slapd"
received
> signal SIGSEGV,
> >>>> Segmentation fault.
> >>>> [Switching to Thread 0x9e0bc280
> (LWP 4662)]
> >>>> strlen () at
> ../sysdeps/arm/armv6t2/strlen.S:142
> >>>> 142 ldrd data1a,
> data1b, [src]
> >>>> (gdb) bt
> >>>> #0 0xb6728f2e in strlen () at
> >>>> ../sysdeps/arm/armv6t2/strlen.S:142
> >>>> #1 0xb6973b40 in cvt_s
> (flags=0, prec=<optimized
> >>>> out>, width=0, str=0x4
<error:
> Cannot access memory
> >>>> at address 0x4>, ss=<optimized
out>)
> >>>> at
> ../../.././nspr/pr/src/io/prprf.c:374
> >>>> #2 0xb6973b40 in dosprintf
> (ss=ss@entry=0x9e0bb4bc,
> >>>> fmt=0xb34fddf2 "",
> fmt@entry=0xb3527770 <agmt_set>
> >>>> "\360\357\305\002",
ap=...)
> >>>> at
> ../../.././nspr/pr/src/io/prprf.c:1018
> >>>> #3 0xb6973c8c in PR_vsmprintf
> >>>> (fmt=fmt@entry=0xb3527770
<agmt_set>
> >>>> "\360\357\305\002",
ap=...,
> ap@entry=...) at
> >>>>
> ../../.././nspr/pr/src/io/prprf.c:1184
> >>>> #4 0xb6e36d18 in
> slapi_ch_smprintf (fmt=0xb34fdde0
> >>>> "%s;%s;%s;%ld;%d;%s") at
> >>>> ldap/servers/slapd/ch_malloc.c:331
> >>>> #5 0xb34af5a4 in
agmt_update_maxcsn
> >>>> (r=r@entry=0x2c89810,
> sdn=0x3e2bac0, sdn@entry=0x10,
> >>>> op=<optimized out>,
mods=0x10,
> mods@entry=0x3eec100,
> >>>> csn=csn@entry=0x3e30220)
> >>>> at
> >>>>
> ldap/servers/plugins/replication/repl5_agmt.c:3036
> >>>> #6 0xb34bd424 in
> write_changelog_and_ruv
> >>>> (pb=pb@entry=0x2cb54a0) at
> >>>>
> ldap/servers/plugins/replication/repl5_plugins.c:1124
> >>>> #7 0xb34be7ec in
> multimaster_be_betxnpostop_add
> >>>> (pb=0x2cb54a0) at
> >>>>
> ldap/servers/plugins/replication/repl5_plugins.c:855
> >>>> #8 0xb34be880 in
> multimaster_mmr_postop
> >>>> (pb=<optimized out>,
flags=560) at
> >>>>
> ldap/servers/plugins/replication/repl5_plugins.c:616
> >>>> #9 0xb6e8fcf0 in
> plugin_call_mmr_plugin_postop
> >>>> (pb=pb@entry=0x2cb54a0,
> e=e@entry=0x0,
> >>>> flags=flags@entry=560) at
> >>>> ldap/servers/slapd/plugin_mmr.c:65
> >>>> #10 0xb35ba870 in ldbm_back_add
> (pb=0x2cb54a0) at
> >>>>
> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
> >>>> #11 0xb6e2ce4c in op_shared_add
> >>>> (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:679
> >>>> #12 0xb6e2d35c in add_internal_pb
> >>>> (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:407
> >>>> #13 0xb6e2e0f8 in
> slapi_add_internal_pb
> >>>> (pb=pb@entry=0x2cb54a0) at
> ldap/servers/slapd/add.c:332
> >>>> #14 0xb368db50 in
> ipa_topo_util_segment_write
> >>>> (tconf=tconf@entry=0x2cb5860,
> >>>> tsegm=tsegm@entry=0x3f2d9a0) at
> topology_util.c:1251
> >>>> #15 0xb368e01c in
> ipa_topo_util_update_agmt_list
> >>>> (conf=0x2cb5860,
> repl_segments=<optimized out>) at
> >>>> topology_util.c:696
> >>>> #16 0xb368891c in
> ipa_topo_apply_shared_config () at
> >>>> topology_init.c:165
> >>>> #17 0xb6e4e65c in eq_call_all () at
> >>>> ldap/servers/slapd/eventq.c:278
> >>>> #18 0xb6e4e65c in eq_loop
> (arg=<optimized out>) at
> >>>> ldap/servers/slapd/eventq.c:323
> >>>> #19 0xb6982d68 in _pt_root
> (arg=0x2c8da40) at
> >>>>
> ../../.././nspr/pr/src/pthreads/ptthread.c:201
> >>>> #20 0xb6906ee8 in start_thread
> (arg=0x9e0bc280) at
> >>>> pthread_create.c:465
> >>>> #21 0xb6785da8 in None () at
> >>>>
> ../sysdeps/unix/sysv/linux/arm/clone.S:73
> >>>>
> >>>>
> >>>> On Tue, May 15, 2018 at 3:01 AM,
> thierry bordaz
> >>>> <tbordaz(a)redhat.com
> <mailto:tbordaz@redhat.com>
> <mailto:tbordaz@redhat.com
> <mailto:tbordaz@redhat.com>>> wrote:
> >>>>
> >>>> Hi Jonathan,
> >>>>
> >>>> This problem looks new to me
> and has something
> >>>> specific to your environment.
> >>>> I think the best approach is
> to continue to
> >>>> debug on your system if you
> have the possibility
> >>>> to do so.
> >>>>
> >>>> From strace we can see that
> DS started smoothly
> >>>> (created its pid file then
> notified systemd it
> >>>> was running fine). According
> to the pstack
> >>>> nunc-stans was running and
> was able to accept
> >>>> network events even if it
> appears it detected no
> >>>> incoming connection.
> >>>> So the server should be
> ready to serve for some
> >>>> seconds (more than a
> minute), then it crashed
> >>>> with one thread
> dereferencing likely a wrong
> >>>> pointer.
> >>>>
> >>>> Could you attach a debugger
> when the server is
> >>>> started and wait for the
> sigsegv to occur. Then
> >>>> confirm the crashing thread
> backstack.
> >>>> If it is confirmed, I am
> afraid this is a stack
> >>>> corruption and valgrind
> could help
> >>>>
>
(http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...
> >>>>
>
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...).
> >>>>
> >>>> best regards
> >>>> thierry
> >>>>
> >>>>
> >>>> On 05/14/2018 10:20 PM,
> Jonathan Vaughn wrote:
> >>>>> Here's a strace from
before
> it dies. Most of
> >>>>> the elapsed time is it
> waiting on some futex
> >>>>> call it looks like near the
> end, when it
> >>>>> finally "returns"
(from
> lack of strace output
> >>>>> for duration of call I
> assume it didn't
> >>>>> actually return but SIGSEGV
> in that call) and
> >>>>> strace prints ' = ?'
on the
> futex it then
> >>>>> immediately reports SIGSEGV
> after. So maybe the
> >>>>> problem is that futex call,
> which may mean the
> >>>>> problem is not directly in
> 389DS / FreeIPA itself?
> >>>>>
> >>>>>
> >>>>>
> >>>>> 15:13:31.626587 (+
> 0.000630) listen(8, 128)
> >>>>> = 0 <0.000068>
> >>>>> 15:13:31.626857 (+
> 0.000235) listen(9, 128)
> >>>>> = 0 <0.000048>
> >>>>> 15:13:31.627111 (+
> 0.000251)
> >>>>>
> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
> >>>>> tv_nsec=41120614}) = 0
> <0.000085>
> >>>>> 15:13:31.627457 (+
> 0.000356)
> >>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811,
> tv_nsec=627560772}) = 0
> >>>>> <0.000043>
> >>>>> 15:13:31.631233 (+
> 0.003839)
> >>>>>
> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
> >>>>> tv_nsec=45286796}) = 0
> <0.000077>
> >>>>> 15:13:31.631720 (+
> 0.000427)
> >>>>>
> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
> >>>>> tv_nsec=45661430}) = 0
> <0.000042>
> >>>>> 15:13:31.631955 (+
> 0.000220)
> >>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811,
> tv_nsec=632049036}) = 0
> >>>>> <0.000047>
> >>>>> 15:13:31.635669 (+
> 0.003785)
> >>>>>
> clock_gettime(CLOCK_MONOTONIC, {tv_sec=1464932,
> >>>>> tv_nsec=49725840}) = 0
> <0.000146>
> >>>>> 15:13:31.636484 (+
> 0.000784) write(16, "a",
> >>>>> 1) = 1 <0.000118>
> >>>>> 15:13:31.636855 (+
> 0.000341) sched_yield()
> >>>>> = 0 <0.000252>
> >>>>> 15:13:31.637322 (+
> 0.000470)
> >>>>> futex(0x1cb57a0,
> FUTEX_WAKE_PRIVATE, 1) = 1
> >>>>> <0.000088>
> >>>>> 15:13:31.637897 (+
> 0.000610) write(16, "a",
> >>>>> 1) = 1 <0.000221>
> >>>>> 15:13:31.638394 (+
> 0.000467) sched_yield()
> >>>>> = 0 <0.000047>
> >>>>> 15:13:31.638619 (+
> 0.000202)
> >>>>> futex(0x1cb5710,
> FUTEX_WAKE_PRIVATE, 1) = 1
> >>>>> <0.000065>
> >>>>> 15:13:31.638908 (+
> 0.000298)
> >>>>> openat(AT_FDCWD,
> >>>>>
> "/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
> >>>>> O_WRONLY|O_CREAT|O_TRUNC,
> 0666) = 33 <0.000831>
> >>>>> 15:13:31.640260 (+
> 0.001387) getpid() =
> >>>>> 32353 <0.000077>
> >>>>> 15:13:31.640558 (+
> 0.000256) fstat64(33,
> >>>>> {st_mode=S_IFREG|0644,
> st_size=0, ...}) = 0
> >>>>> <0.000119>
> >>>>> 15:13:31.641106 (+
> 0.000556) write(33,
> >>>>> "32353\n", 6) = 6
<0.000127>
> >>>>> 15:13:31.641472 (+
> 0.000362) close(33) = 0
> >>>>> <0.000519>
> >>>>> 15:13:31.642216 (+
> 0.000758)
> >>>>>
> chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
> >>>>> 0644) = 0 <0.000152>
> >>>>> 15:13:31.642900 (+
> 0.000679)
> >>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811,
> tv_nsec=643020294}) = 0
> >>>>> <0.000056>
> >>>>> 15:13:31.643495 (+
> 0.000590) write(2,
> >>>>>
> "[14/May/2018:15:13:31.643020294 "..., 134) =
> >>>>> 134 <0.002697>
> >>>>> 15:13:31.646515 (+
> 0.003052)
> >>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811,
> tv_nsec=646694394}) = 0
> >>>>> <0.000075>
> >>>>> 15:13:31.646892 (+
> 0.000337) write(4,
> >>>>>
> "[14/May/2018:15:13:31.646694394 "..., 134) =
> >>>>> 134 <0.000522>
> >>>>> 15:13:31.647841 (+
> 0.000973) fsync(4) = 0
> >>>>> <0.005967>
> >>>>> 15:13:31.654425 (+
> 0.006617)
> >>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811,
> tv_nsec=654598946}) = 0
> >>>>> <0.000253>
> >>>>> 15:13:31.655137 (+
> 0.000717) write(2,
> >>>>>
> "[14/May/2018:15:13:31.654598946 "..., 136) =
> >>>>> 136 <0.002427>
> >>>>> 15:13:31.658312 (+
> 0.003165)
> >>>>>
clock_gettime(CLOCK_REALTIME,
> >>>>> {tv_sec=1526328811,
> tv_nsec=658486117}) = 0
> >>>>> <0.000251>
> >>>>> 15:13:31.659032 (+
> 0.000682) write(4,
> >>>>>
> "[14/May/2018:15:13:31.658486117 "..., 136) =
> >>>>> 136 <0.000346>
> >>>>> 15:13:31.659623 (+
> 0.000595) fsync(4) = 0
> >>>>> <0.003311>
> >>>>> 15:13:31.663230 (+
> 0.003642) getpid() =
> >>>>> 32353 <0.000045>
> >>>>> 15:13:31.663732 (+
> 0.000454)
> >>>>> socket(AF_UNIX,
> SOCK_DGRAM|SOCK_CLOEXEC, 0) =
> >>>>> 33 <0.000296>
> >>>>> 15:13:31.664760 (+
> 0.001048) getsockopt(33,
> >>>>> SOL_SOCKET, SO_SNDBUF,
> [163840], [4]) = 0
> >>>>> <0.000108>
> >>>>> 15:13:31.665141 (+
> 0.000386) setsockopt(33,
> >>>>> SOL_SOCKET, SO_SNDBUFFORCE,
> [8388608], 4) = -1
> >>>>> EPERM (Operation not
> permitted) <0.000051>
> >>>>> 15:13:31.665500 (+
> 0.000334) setsockopt(33,
> >>>>> SOL_SOCKET, SO_SNDBUF,
> [8388608], 4) = 0 <0.000229>
> >>>>> 15:13:31.665973 (+
> 0.000468) sendmsg(33,
> >>>>>
{msg_name={sa_family=AF_UNIX,
> >>>>>
> sun_path="/run/systemd/notify"},
> >>>>> msg_namelen=21,
> >>>>>
> msg_iov=[{iov_base="READY=1\nSTATUS=slapd
> >>>>> started: Re"...,
> iov_len=69}], msg_iovlen=1,
> >>>>> msg_controllen=0,
> msg_flags=0}, MSG_NOSIGNAL) =
> >>>>> 69 <0.000600>
> >>>>> 15:13:31.667270 (+
> 0.001334) close(33) = 0
> >>>>> <0.003140>
> >>>>> 15:13:31.677320 (+
> 0.010146)
> >>>>> futex(0xb31122e8,
> FUTEX_WAIT, 32357, NULL) = ?
> >>>>> 15:14:27.661809 (+
> 55.984448) +++ killed by
> >>>>> SIGSEGV (core dumped) +++
> >>>>>
> >>>>> On Mon, May 14, 2018 at
> 10:27 AM, thierry
> >>>>> bordaz
<tbordaz(a)redhat.com
> <mailto:tbordaz@redhat.com>
> >>>>>
<mailto:tbordaz@redhat.com
> <mailto:tbordaz@redhat.com>>> wrote:
> >>>>>
> >>>>> Hi Jonathan,
> >>>>>
> >>>>> This is weird as the
> crashing thread stack
> >>>>> looks truncated (did
> you copy/paste all of
> >>>>> it ?)
> >>>>>
> >>>>> Thread 1 (Thread
> 0x9e13c280 (LWP 17245)):
> >>>>> #0 0xb67bbf2e in
> strlen () at /lib/libc.so.6
> >>>>> #1 0xb6a06b40 in
> dosprintf () at
> >>>>> /lib/libnspr4.so
> >>>>> #2 0x00000000 in None
()
> >>>>>
> >>>>> Did you install
> 389-ds-base-debuginfo ?
> >>>>> How did you get that
> backtrace ? from a
> >>>>> core dumped, pstack ?
> Can you attach a
> >>>>> debugger before the
> crash occurs ?
> >>>>>
> >>>>> It looks it crashed
> soon at startup, could
> >>>>> it be related to a
> broken dse.ldif. It
> >>>>> should exists a
> dse.ldif.OK, is it possibly
> >>>>> to try to start with it
?
> >>>>>
> >>>>> best regards
> >>>>> thierry
> >>>>>
> >>>>> On 05/12/2018 01:22 AM,
> Jonathan Vaughn via
> >>>>> FreeIPA-users wrote:
> >>>>>> Not sure if it makes
a
> difference... I was
> >>>>>> looking into this
> again and realized I had
> >>>>>> a bunch of messages
> from gdb telling me to
> >>>>>> install more
> debuginfo. I've done that
> >>>>>> now, here it is
again
> freshly run through
> >>>>>> gdb
> >>>>>>
> >>>>>> GNU gdb (GDB)
Fedora
> 8.0.1-36.fc27
> >>>>>> Copyright (C) 2017
> Free Software
> >>>>>> Foundation, Inc.
> >>>>>> License GPLv3+: GNU
> GPL version 3 or later
> >>>>>>
> <http://gnu.org/licenses/gpl.html
> <http://gnu.org/licenses/gpl.html>
> >>>>>>
> <http://gnu.org/licenses/gpl.html
> <http://gnu.org/licenses/gpl.html>>>
> >>>>>> This is free
software:
> you are free to
> >>>>>> change and
> redistribute it.
> >>>>>> There is NO
WARRANTY,
> to the extent
> >>>>>> permitted by law.
> Type "show copying"
> >>>>>> and "show
warranty"
> for details.
> >>>>>> This GDB was
configured as
> >>>>>>
> "armv7hl-redhat-linux-gnueabi".
> >>>>>> Type "show
> configuration" for
> >>>>>> configuration
details.
> >>>>>> For bug reporting
> instructions, please see:
> >>>>>>
> <http://www.gnu.org/software/gdb/bugs/
> <http://www.gnu.org/software/gdb/bugs/>
> >>>>>>
> <http://www.gnu.org/software/gdb/bugs/
> <http://www.gnu.org/software/gdb/bugs/>>>;.
> >>>>>> Find the GDB manual
> and other
> >>>>>> documentation
> resources online at:
> >>>>>>
> <http://www.gnu.org/software/gdb/documentation/
> <http://www.gnu.org/software/gdb/documentation/>
> >>>>>>
> <http://www.gnu.org/software/gdb/documentation/
> <http://www.gnu.org/software/gdb/documentation/>>>;.
> >>>>>> For help, type
"help".
> >>>>>> Type "apropos
word" to
> search for commands
> >>>>>> related to
"word"...
> >>>>>> Reading symbols
from
> >>>>>>
> /usr/sbin/ns-slapd...Reading symbols from
> >>>>>>
> /usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
> >>>>>> done.
> >>>>>> ...
> >>>>>>
> >>>>>> Thread 1 (Thread
> 0x9e13c280 (LWP 17245)):
> >>>>>> #0 0xb67bbf2e in
> strlen () at /lib/libc.so.6
> >>>>>> #1 0xb6a06b40 in
> dosprintf () at
> >>>>>> /lib/libnspr4.so
> >>>>>> #2 0x00000000 in
None ()
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, May 8,
> 2018 at 7:52 AM, Rob
> >>>>>> Crittenden
> <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
> >>>>>>
> <mailto:rcritten@redhat.com
> <mailto:rcritten@redhat.com>>> wrote:
> >>>>>>
> >>>>>> Jonathan
> Vaughn via FreeIPA-users
> >>>>>> wrote:
> >>>>>>
> >>>>>> Still
> trying to figure this
> >>>>>> out. It
> looks like slapd is
> >>>>>> dying,
I
> thought it was still
> >>>>>> running
> for some reason.
> >>>>>>
> >>>>>> slapd
is
> dying to segfault.
> >>>>>> strace
of
> it happening doesn't
> >>>>>> seem to
> reveal much:
> >>>>>>
> >>>>>>
> >>>>>> A stack
trace
> would very much help
> >
>
...
[Message clipped]
3:20 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Looks like replica install finished just fine, and I can access the the web
UI on it. No obvious problems.
So it would appear that code change fixed 389DS for ARM.
I'm not running the Dogtag CA on the Pi, just on the non-Pi servers,
otherwise would need to probably adjust the startup timeout for it so that
the install script didn't give up waiting on it. When I was first trying to
play with this I tried installing the first replica/master on the Pi, and
ran into the Dogtag timeout problem. Even installing Oracle Java instead of
the OpenJDK version of Java didn't solve it, because somehow it always
picks the OpenJDK java instead of oracle even if I change the alternatives
setting to default to the Oracle one...
But as far as I can tell otherwise, everything is working as expected.
On Fri, May 18, 2018 at 2:10 PM, Mark Reynolds <mreynolds(a)redhat.com> wrote:
On 05/18/2018 03:04 PM, Jonathan Vaughn wrote:
Build completed, I tested it, got some error about libs... you had a rogue
lib64 in the build command given :)
Ah, yeah sorry that configure command was not fully tested I guess (I'll
update my notes)...
Rebuilt with that fixed to be regular lib (since this is a 32-bit OS) and
it appears to be running fine.
Well, it hasn't segfaulted. It complains about failing to get principles
from keytab and stuff... but that is probably because I could never
complete ipa-replica-install. It's still running. I'm going to uninstall
and reinstall replica and see what happens now.
Keep us posted
Thanks,
Mark
On Thu, May 17, 2018 at 9:09 PM, Mark Reynolds <mreynolds(a)redhat.com>
wrote:
>
>
> On 05/17/2018 08:06 PM, Jonathan Vaughn wrote:
>
> Sorry, missed your build instructions, only saw them just now. I've just
> kicked off that build and I'll check it tomorrow once I'm back in.
>
> One note, it appears it's ./autogen.sh not ./autogen.pl ...
>
> Whoops! Good catch, and good luck...
>
>
> On Thu, May 17, 2018 at 6:16 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
> wrote:
>
>> Welp, I'm still getting the -fPIC error ... so I may need to figure out
>> how to do it with COPR or something.
>>
>> On Thu, May 17, 2018 at 4:16 PM, Jonathan Vaughn <jonathan(a)creatuity.com
>> > wrote:
>>
>>> I've never used COPR. I've dabbled with RPMs in the past but that
>>> was... CentOS 6 I think, and I wasn't making source code changes so much
as
>>> just copying and pasting SRPMs from another RPM platform to build for
>>> CentOS, using the regular rpmbuild stuff.
>>>
>>> I did actually try just copying the configure line (manually expanding
>>> the various vars) from the SRPM and just doing an old fashioned make, but
>>> the first try bailed shy of an hour at linking with some error about not
>>> having -fPIC specified, so I restarted it with -fPIC to see if it would at
>>> least build. So if I should still try the code changes, if that build
>>> finishes I can try it (I was planning to just copy the respective binaries
>>> on top of the installed ones), but if it turned out the suggested code
>>> changes were pointless then there's no point waiting for it to finish
>>> building.
>>>
>>> On Thu, May 17, 2018 at 4:11 PM, Rob Crittenden <rcritten(a)redhat.com>
>>> wrote:
>>>
>>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>> > Oops, hit reply instead of reply-all
>>>> >
>>>> > NSPR RPMs
>>>> >
>>>> > # yum list installed nspr*
>>>> > Installed Packages
>>>> > nspr.armv7hl
>>>>
>>>> > 4.19.0-1.fc27
>>>>
>>>> > @updates
>>>> > nspr-debuginfo.armv7hl
>>>>
>>>> > 4.19.0-1.fc27
>>>>
>>>> > @updates-debuginfo
>>>> > nspr-debugsource.armv7hl
>>>>
>>>> > 4.19.0-1.fc27
>>>>
>>>> > @updates-debuginfo
>>>> > nspr-devel.armv7hl
>>>>
>>>> > 4.19.0-1.fc27
>>>>
>>>> > @updates
>>>> >
>>>> > As for building it with the changes you proposed previously -
should
>>>> I
>>>> > not bother? I realized last night that the last and only time I
built
>>>> > anything *on* a Pi was a Pi Zero and that's why it was so
terribly
>>>> slow.
>>>> > This Pi 3 ain't fast but it's fast enough to build stuff in
far less
>>>> > than half a day (unlike the Zero)...
>>>>
>>>> If you are handy with rpms you could build it in copr.
>>>>
>>>> rob
>>>>
>>>> >
>>>> >
>>>> > On Thu, May 17, 2018 at 5:07 AM, thierry bordaz
<tbordaz(a)redhat.com
>>>> > <mailto:tbordaz@redhat.com>> wrote:
>>>> >
>>>> >
>>>> >
>>>> > On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
>>>> >> I've been just using the packages from Fedora. I can
build it
>>>> >> potentially but I don't have a cross build environment
set up at
>>>> >> the moment. From experience I'd want to do that first
because
>>>> >> building anything on the Pi usually takes ages.
>>>> >>
>>>> >> I'd been "redacting" the hostnames but
I'll stop bothering since
>>>> >> it looks like we're getting far enough into the weeds
now that
>>>> the
>>>> >> difference in string lengths after "redacting"
might actually
>>>> be a
>>>> >> red herring.
>>>> >>
>>>> >> (gdb) p *agmt
>>>> >> $1 = {hostname = 0x1ef9be0
"ipa-12.creatuity.internal", port =
>>>> >> 389, transport_flags = 0, binddn = 0x1e8f650 "",
creds =
>>>> >> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
>>>> >> frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
>>>> >> frac_attr_total_defined = 1, schedule = 0x1a7f0c0,
>>>> auto_initialize
>>>> >> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
>>>> >> long_name = 0x1a7f100
"agmt=\"cn=meToipa-12.creatuit
>>>> y.internal\"
>>>> >> (ipa-12:5)", protocol = 0x19c2930, changecounters =
0x186d180,
>>>> >> num_changecounters = 0,
>>>> >> max_changecounters = 256, last_update_start_time =
1526500697,
>>>> >> last_update_end_time = 1526500697,
>>>> >> last_update_status = "Error (0) Replica acquired
successfully:
>>>> >> Incremental update succeeded", '\000'
<repeats 954 times>,
>>>> >> update_in_progress = 0, is_enabled = 1,
>>>> >> last_init_start_time = 0, last_init_end_time = 0,
>>>> >> last_init_status = '\000' <repeats 1023
times>, lock =
>>>> 0x1ee3740,
>>>> >> consumerRUV = 0x1f14e50,
>>>> >> consumerSchemaCSN = 0x317c520, consumerRID = 4,
>>>> tmpConsumerRID =
>>>> >> 0, timeout = 120, stop_in_progress = 0, busywaittime = 0,
>>>> >> pausetime = 0, priv = 0x0,
>>>> >> attrs_to_strip = 0x1ef9ba0, agreement_type = 0,
>>>> protocol_timeout
>>>> >> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000,
>>>> >> flowControlPause = 2000, ignoreMissingChange = 0,
>>>> >> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
>>>> >> (gdb) p *agmt->replarea
>>>> >> $2 = {flag = 15 '\017', udn = 0x1efce80
>>>> >> "dc=creatuity,dc=internal", dn = 0x1ef9460
>>>> >> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
>>>> >> "dc=creatuity,dc=internal", ndn_len = 24}
>>>> >> (gdb) p *agmt->rdn
>>>> >> $3 = {flag = 0 '\000', rdn = 0x19c2840
>>>> >> "cn=meToipa-12.creatuity.internal", rdns = 0x0,
butcheredupto =
>>>> >> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
>>>> >>
>>>> >> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
>>>> >> /usr/include/inttypes.h:# define PRId64
__PRI64_PREFIX
>>>> "d"
>>>> >> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
>>>> >> /usr/include/inttypes.h:# define PRIu16
"u"
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
>>>> >> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>> wrote:
>>>> >>
>>>> >>
>>>> >>
>>>> >> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>>>> >>> The installed version of 389* is 1.3.7.10-1.fc27
>>>> for armv7hl,
>>>> >>> which appears to be the latest available version.
>>>> >>
>>>> >> Perhaps something is off with the inttypes on
Raspberry.
>>>> Are
>>>> >> you building this yourself on Raspberry? Can we make
code
>>>> >> changes and compile/install them?
>>>> >>
>>>> >> Before we do that though, in gdb can you run these
commands
>>>> in
>>>> >> the same gdb frame:
>>>> >>
>>>> >> (gdb) p *agmt->replarea
>>>> >> (gdb) p *agmt->rdn
>>>> >>
>>>> >>
>>>> >> Then do:
>>>> >>
>>>> >> # grep -r PRId64 /usr/include/*
>>>> >> # grep -r PRIu16 /usr/include/*
>>>> >>
>>>> >>
>>>> >> So if you can compile the source, then change this line
in
>>>> >> ldap/servers/plugins/replication/repl5_agmt.c:3036, but
>>>> don't
>>>> >> do this yet until you get me the info I just requested.
>>>> >>
>>>> >> From:
>>>> >>
>>>> >> agmt->maxcsn =
>>>> slapi_ch_smprintf("%s;%s;%s;%"
>>>> >> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replarea),
>>>> >>
>>>> >>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>>>> >> agmt->hostname,
>>>> >>
>>>> agmt->port,
>>>> >> agmt->consumerRID, maxcsn);
>>>> >>
>>>> >> To:
>>>> >>
>>>> >> agmt->maxcsn =
>>>> >> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>>> >> slapi_sdn_get_dn(agmt->replarea),
>>>> >>
>>>> >>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>>>> >> agmt->hostname,
>>>> >>
>>>> >> (long)agmt->port, (int)agmt->consumerRID,
maxcsn);
>>>> >>
>>>> >>
>>>> >> Thanks,
>>>> >> Mark
>>>> >>
>>>> >>>
>>>> >>> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
>>>> >>> <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>>
>>>> wrote:
>>>> >>
>>>> >
>>>> > The agreement structure looks valid to me. it should not lead
to
>>>> a
>>>> > crash.
>>>> >
>>>> > What looks weird to me is the order of the arguments of cvt_s.
>>>> > It is called: rv = cvt_s(ss, u.s, width, prec, flags);
>>>> > But the crashing thread shows the opposite order: flags,
>>>> prec,width,
>>>> > str, ss
>>>> >
>>>> > The others frame do not show this change of order.
>>>> > Also 'str=4' that would be a meaningful value for
'prec=4'.
>>>> >
>>>> > From debug perspective I only imagine disassemble the two last
>>>> > frames to confirm parameters.
>>>> > What are the nspr rpms ?
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >>> (gdb) up
>>>> >>> #1 0xb6926b40 in cvt_s (flags=0,
prec=<optimized out>,
>>>> >>> width=0, str=0x4 <error: Cannot access memory
at
>>>> address
>>>> >>> 0x4>, ss=<optimized out>)
>>>> >>> at ../../.././nspr/pr/src/io/prprf.c:374
>>>> >>> 374 slen = strlen(str);
>>>> >>> (gdb) up
>>>> >>> #2 dosprintf (ss=ss@entry=0x9e06e4bc,
fmt=0xb34b0df2
>>>> "",
>>>> >>> fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002",
>>>> ap=...)
>>>> >>> at ../../.././nspr/pr/src/io/prprf.c:1018
>>>> >>> 1018 rv = cvt_s(ss, u.s, width,
prec,
>>>> flags);
>>>> >>> (gdb) up
>>>> >>> #3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry
>>>> =0xb34da770
>>>> >>> <agmt_set> "\360\317p\002",
ap=..., ap@entry=...) at
>>>> >>> ../../.././nspr/pr/src/io/prprf.c:1184
>>>> >>> 1184 rv = dosprintf(&ss, fmt, ap);
>>>> >>> (gdb) up
>>>> >>> #4 0xb6de9d18 in slapi_ch_smprintf
(fmt=0xb34b0de0
>>>> >>> "%s;%s;%s;%ld;%d;%s") at
ldap/servers/slapd/ch_malloc.c
>>>> :331
>>>> >>> 331 p = PR_vsmprintf(fmt, ap);
>>>> >>> (gdb) up
>>>> >>> #5 0xb34625a4 in agmt_update_maxcsn
>>>> >>> (r=r@entry=0x2737810, sdn=0x38d9a40,
sdn@entry=0x10,
>>>> >>> op=<optimized out>, mods=0x10,
mods@entry=0x27d0100,
>>>> >>> csn=csn@entry=0x38de350)
>>>> >>> at ldap/servers/plugins/replicati
>>>> on/repl5_agmt.c:3036
>>>> >>> 3036 agmt->maxcsn =
>>>> >>>
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>>> >>> slapi_sdn_get_dn(agmt->replarea),
>>>> >>> (gdb) p *agmt
>>>> >>> $1 = {hostname = 0x2757be0
"ipa-12.company.internal",
>>>> >>> port = 389, transport_flags = 0, binddn =
0x26ed650 "",
>>>> >>> creds = 0x26ed7a0, bindmethod = 3, replarea =
>>>> 0x2757480,
>>>> >>> frac_attrs = 0x27579c0, frac_attrs_total =
0x2757a40,
>>>> >>> frac_attr_total_defined = 1, schedule =
0x22dd0c0,
>>>> >>> auto_initialize = 502, dn = 0x2756d00, rdn =
0x2756c20,
>>>> >>> long_name = 0x22dd100
>>>> >>>
"agmt=\"cn=meToipa-12.company.internal\" (ipa-12:5)",
>>>> >>> protocol = 0x2220930, changecounters =
0x20cb180,
>>>> >>> num_changecounters = 0,
>>>> >>> max_changecounters = 256,
last_update_start_time =
>>>> >>> 1526499214, last_update_end_time = 1526499214,
>>>> >>> last_update_status = "Error (0) Replica
acquired
>>>> >>> successfully: Incremental update
succeeded", '\000'
>>>> >>> <repeats 954 times>, update_in_progress =
0,
>>>> is_enabled = 1,
>>>> >>> last_init_start_time = 0, last_init_end_time =
0,
>>>> >>> last_init_status = '\000' <repeats
1023 times>, lock =
>>>> >>> 0x2741740, consumerRUV = 0x2772e50,
>>>> >>> consumerSchemaCSN = 0x39da520, consumerRID =
4,
>>>> >>> tmpConsumerRID = 0, timeout = 120,
stop_in_progress =
>>>> 0,
>>>> >>> busywaittime = 0, pausetime = 0, priv = 0x0,
>>>> >>> attrs_to_strip = 0x2757ba0, agreement_type =
0,
>>>> >>> protocol_timeout = 0x26ed5f0, maxcsn = 0x0,
>>>> >>> flowControlWindow = 1000, flowControlPause =
2000,
>>>> >>> ignoreMissingChange = 0,
>>>> >>> attr_lock = 0x2757c20, WaitForAsyncResults =
100}
>>>> >>>
>>>> >>>
>>>> >>> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
>>>> >>> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>>
>>>> wrote:
>>>> >>>
>>>> >>> This looks really familiar and I thought it
was
>>>> >>> fixed. It should have been fixed in
1.3.7.10-1
>>>> >>> (https://pagure.io/389-ds-base/issue/49618
>>>> >>>
<https://pagure.io/389-ds-base/issue/49618>). In
>>>> >>> your debug session go "up" into
>>>> agmt_maxcsn_update()
>>>> >>> and do:
>>>> >>>
>>>> >>> (gdb) p *agmt
>>>> >>>
>>>> >>> Then send us that output please.
>>>> >>>
>>>> >>> Thanks,
>>>> >>> Mark
>>>> >>>
>>>> >>>
>>>> >>> On 05/15/2018 05:29 PM, Jonathan Vaughn via
>>>> >>> FreeIPA-users wrote:
>>>> >>>> Here is a backtrace from live gdb after
the
>>>> >>>> segfault. Looks like things went wrong
somewhere
>>>> >>>> during in the replication code ?
>>>> >>>>
>>>> >>>> Thread 36 "ns-slapd" received
signal SIGSEGV,
>>>> >>>> Segmentation fault.
>>>> >>>> [Switching to Thread 0x9e0bc280 (LWP
4662)]
>>>> >>>> strlen () at
../sysdeps/arm/armv6t2/strlen.S:142
>>>> >>>> 142 ldrd data1a, data1b,
[src]
>>>> >>>> (gdb) bt
>>>> >>>> #0 0xb6728f2e in strlen () at
>>>> >>>> ../sysdeps/arm/armv6t2/strlen.S:142
>>>> >>>> #1 0xb6973b40 in cvt_s (flags=0,
prec=<optimized
>>>> >>>> out>, width=0, str=0x4 <error:
Cannot access
>>>> memory
>>>> >>>> at address 0x4>, ss=<optimized
out>)
>>>> >>>> at
../../.././nspr/pr/src/io/prprf.c:374
>>>> >>>> #2 0xb6973b40 in dosprintf
(ss=ss@entry
>>>> =0x9e0bb4bc,
>>>> >>>> fmt=0xb34fddf2 "",
fmt@entry=0xb3527770
>>>> <agmt_set>
>>>> >>>> "\360\357\305\002", ap=...)
>>>> >>>> at
../../.././nspr/pr/src/io/prprf.c:1018
>>>> >>>> #3 0xb6973c8c in PR_vsmprintf
>>>> >>>> (fmt=fmt@entry=0xb3527770
<agmt_set>
>>>> >>>> "\360\357\305\002", ap=...,
ap@entry=...) at
>>>> >>>> ../../.././nspr/pr/src/io/prprf.c:1184
>>>> >>>> #4 0xb6e36d18 in slapi_ch_smprintf
>>>> (fmt=0xb34fdde0
>>>> >>>> "%s;%s;%s;%ld;%d;%s") at
>>>> >>>> ldap/servers/slapd/ch_malloc.c:331
>>>> >>>> #5 0xb34af5a4 in agmt_update_maxcsn
>>>> >>>> (r=r@entry=0x2c89810, sdn=0x3e2bac0,
sdn@entry
>>>> =0x10,
>>>> >>>> op=<optimized out>, mods=0x10,
mods@entry
>>>> =0x3eec100,
>>>> >>>> csn=csn@entry=0x3e30220)
>>>> >>>> at
>>>> >>>> ldap/servers/plugins/replicat
>>>> ion/repl5_agmt.c:3036
>>>> >>>> #6 0xb34bd424 in
write_changelog_and_ruv
>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>> >>>> ldap/servers/plugins/replicat
>>>> ion/repl5_plugins.c:1124
>>>> >>>> #7 0xb34be7ec in
multimaster_be_betxnpostop_add
>>>> >>>> (pb=0x2cb54a0) at
>>>> >>>> ldap/servers/plugins/replicat
>>>> ion/repl5_plugins.c:855
>>>> >>>> #8 0xb34be880 in
multimaster_mmr_postop
>>>> >>>> (pb=<optimized out>, flags=560)
at
>>>> >>>> ldap/servers/plugins/replicat
>>>> ion/repl5_plugins.c:616
>>>> >>>> #9 0xb6e8fcf0 in
plugin_call_mmr_plugin_postop
>>>> >>>> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0,
>>>> >>>> flags=flags@entry=560) at
>>>> >>>> ldap/servers/slapd/plugin_mmr.c:65
>>>> >>>> #10 0xb35ba870 in ldbm_back_add
(pb=0x2cb54a0) at
>>>> >>>>
ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>>>> >>>> #11 0xb6e2ce4c in op_shared_add
>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>> ldap/servers/slapd/add.c:679
>>>> >>>> #12 0xb6e2d35c in add_internal_pb
>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>> ldap/servers/slapd/add.c:407
>>>> >>>> #13 0xb6e2e0f8 in slapi_add_internal_pb
>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>> ldap/servers/slapd/add.c:332
>>>> >>>> #14 0xb368db50 in
ipa_topo_util_segment_write
>>>> >>>> (tconf=tconf@entry=0x2cb5860,
>>>> >>>> tsegm=tsegm@entry=0x3f2d9a0) at
>>>> topology_util.c:1251
>>>> >>>> #15 0xb368e01c in
ipa_topo_util_update_agmt_list
>>>> >>>> (conf=0x2cb5860,
repl_segments=<optimized out>) at
>>>> >>>> topology_util.c:696
>>>> >>>> #16 0xb368891c in
ipa_topo_apply_shared_config ()
>>>> at
>>>> >>>> topology_init.c:165
>>>> >>>> #17 0xb6e4e65c in eq_call_all () at
>>>> >>>> ldap/servers/slapd/eventq.c:278
>>>> >>>> #18 0xb6e4e65c in eq_loop
(arg=<optimized out>) at
>>>> >>>> ldap/servers/slapd/eventq.c:323
>>>> >>>> #19 0xb6982d68 in _pt_root
(arg=0x2c8da40) at
>>>> >>>>
../../.././nspr/pr/src/pthreads/ptthread.c:201
>>>> >>>> #20 0xb6906ee8 in start_thread
(arg=0x9e0bc280) at
>>>> >>>> pthread_create.c:465
>>>> >>>> #21 0xb6785da8 in None () at
>>>> >>>>
../sysdeps/unix/sysv/linux/arm/clone.S:73
>>>> >>>>
>>>> >>>>
>>>> >>>> On Tue, May 15, 2018 at 3:01 AM, thierry
bordaz
>>>> >>>> <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>>
>>>> wrote:
>>>> >>>>
>>>> >>>> Hi Jonathan,
>>>> >>>>
>>>> >>>> This problem looks new to me and has
something
>>>> >>>> specific to your environment.
>>>> >>>> I think the best approach is to
continue to
>>>> >>>> debug on your system if you have
the
>>>> possibility
>>>> >>>> to do so.
>>>> >>>>
>>>> >>>> From strace we can see that DS
started
>>>> smoothly
>>>> >>>> (created its pid file then notified
systemd it
>>>> >>>> was running fine). According to the
pstack
>>>> >>>> nunc-stans was running and was able
to accept
>>>> >>>> network events even if it appears it
detected
>>>> no
>>>> >>>> incoming connection.
>>>> >>>> So the server should be ready to
serve for
>>>> some
>>>> >>>> seconds (more than a minute), then
it crashed
>>>> >>>> with one thread dereferencing likely
a wrong
>>>> >>>> pointer.
>>>> >>>>
>>>> >>>> Could you attach a debugger when the
server is
>>>> >>>> started and wait for the sigsegv to
occur.
>>>> Then
>>>> >>>> confirm the crashing thread
backstack.
>>>> >>>> If it is confirmed, I am afraid this
is a
>>>> stack
>>>> >>>> corruption and valgrind could help
>>>> >>>> (http://www.port389.org/docs/
>>>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>>>> >>>> <http://www.port389.org/docs/
>>>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-with-valgrind
>>>> >).
>>>> >>>>
>>>> >>>> best regards
>>>> >>>> thierry
>>>> >>>>
>>>> >>>>
>>>> >>>> On 05/14/2018 10:20 PM, Jonathan
Vaughn wrote:
>>>> >>>>> Here's a strace from before
it dies. Most of
>>>> >>>>> the elapsed time is it waiting
on some futex
>>>> >>>>> call it looks like near the end,
when it
>>>> >>>>> finally "returns"
(from lack of strace output
>>>> >>>>> for duration of call I assume it
didn't
>>>> >>>>> actually return but SIGSEGV in
that call) and
>>>> >>>>> strace prints ' = ?' on
the futex it then
>>>> >>>>> immediately reports SIGSEGV
after. So maybe
>>>> the
>>>> >>>>> problem is that futex call,
which may mean
>>>> the
>>>> >>>>> problem is not directly in 389DS
/ FreeIPA
>>>> itself?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> 15:13:31.626587 (+ 0.000630)
listen(8,
>>>> 128)
>>>> >>>>> = 0 <0.000068>
>>>> >>>>> 15:13:31.626857 (+ 0.000235)
listen(9,
>>>> 128)
>>>> >>>>> = 0 <0.000048>
>>>> >>>>> 15:13:31.627111 (+
0.000251)
>>>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932,
>>>> >>>>> tv_nsec=41120614}) = 0
<0.000085>
>>>> >>>>> 15:13:31.627457 (+
0.000356)
>>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>>> >>>>> {tv_sec=1526328811,
tv_nsec=627560772}) = 0
>>>> >>>>> <0.000043>
>>>> >>>>> 15:13:31.631233 (+
0.003839)
>>>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932,
>>>> >>>>> tv_nsec=45286796}) = 0
<0.000077>
>>>> >>>>> 15:13:31.631720 (+
0.000427)
>>>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932,
>>>> >>>>> tv_nsec=45661430}) = 0
<0.000042>
>>>> >>>>> 15:13:31.631955 (+
0.000220)
>>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>>> >>>>> {tv_sec=1526328811,
tv_nsec=632049036}) = 0
>>>> >>>>> <0.000047>
>>>> >>>>> 15:13:31.635669 (+
0.003785)
>>>> >>>>> clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932,
>>>> >>>>> tv_nsec=49725840}) = 0
<0.000146>
>>>> >>>>> 15:13:31.636484 (+ 0.000784)
write(16,
>>>> "a",
>>>> >>>>> 1) = 1 <0.000118>
>>>> >>>>> 15:13:31.636855 (+
0.000341)
>>>> sched_yield()
>>>> >>>>> = 0 <0.000252>
>>>> >>>>> 15:13:31.637322 (+
0.000470)
>>>> >>>>> futex(0x1cb57a0,
FUTEX_WAKE_PRIVATE, 1) = 1
>>>> >>>>> <0.000088>
>>>> >>>>> 15:13:31.637897 (+ 0.000610)
write(16,
>>>> "a",
>>>> >>>>> 1) = 1 <0.000221>
>>>> >>>>> 15:13:31.638394 (+
0.000467)
>>>> sched_yield()
>>>> >>>>> = 0 <0.000047>
>>>> >>>>> 15:13:31.638619 (+
0.000202)
>>>> >>>>> futex(0x1cb5710,
FUTEX_WAKE_PRIVATE, 1) = 1
>>>> >>>>> <0.000065>
>>>> >>>>> 15:13:31.638908 (+
0.000298)
>>>> >>>>> openat(AT_FDCWD,
>>>> >>>>>
"/var/run/dirsrv/slapd-COMPAN
>>>> Y-INTERNAL.pid",
>>>> >>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666)
= 33
>>>> <0.000831>
>>>> >>>>> 15:13:31.640260 (+ 0.001387)
getpid() =
>>>> >>>>> 32353 <0.000077>
>>>> >>>>> 15:13:31.640558 (+ 0.000256)
fstat64(33,
>>>> >>>>> {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0
>>>> >>>>> <0.000119>
>>>> >>>>> 15:13:31.641106 (+ 0.000556)
write(33,
>>>> >>>>> "32353\n", 6) = 6
<0.000127>
>>>> >>>>> 15:13:31.641472 (+ 0.000362)
close(33) =
>>>> 0
>>>> >>>>> <0.000519>
>>>> >>>>> 15:13:31.642216 (+
0.000758)
>>>> >>>>>
chmod("/var/run/dirsrv/slapd-
>>>> COMPANY-INTERNAL.pid",
>>>> >>>>> 0644) = 0 <0.000152>
>>>> >>>>> 15:13:31.642900 (+
0.000679)
>>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>>> >>>>> {tv_sec=1526328811,
tv_nsec=643020294}) = 0
>>>> >>>>> <0.000056>
>>>> >>>>> 15:13:31.643495 (+ 0.000590)
write(2,
>>>> >>>>>
"[14/May/2018:15:13:31.643020294 "..., 134)
>>>> =
>>>> >>>>> 134 <0.002697>
>>>> >>>>> 15:13:31.646515 (+
0.003052)
>>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>>> >>>>> {tv_sec=1526328811,
tv_nsec=646694394}) = 0
>>>> >>>>> <0.000075>
>>>> >>>>> 15:13:31.646892 (+ 0.000337)
write(4,
>>>> >>>>>
"[14/May/2018:15:13:31.646694394 "..., 134)
>>>> =
>>>> >>>>> 134 <0.000522>
>>>> >>>>> 15:13:31.647841 (+ 0.000973)
fsync(4) = 0
>>>> >>>>> <0.005967>
>>>> >>>>> 15:13:31.654425 (+
0.006617)
>>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>>> >>>>> {tv_sec=1526328811,
tv_nsec=654598946}) = 0
>>>> >>>>> <0.000253>
>>>> >>>>> 15:13:31.655137 (+ 0.000717)
write(2,
>>>> >>>>>
"[14/May/2018:15:13:31.654598946 "..., 136)
>>>> =
>>>> >>>>> 136 <0.002427>
>>>> >>>>> 15:13:31.658312 (+
0.003165)
>>>> >>>>> clock_gettime(CLOCK_REALTIME,
>>>> >>>>> {tv_sec=1526328811,
tv_nsec=658486117}) = 0
>>>> >>>>> <0.000251>
>>>> >>>>> 15:13:31.659032 (+ 0.000682)
write(4,
>>>> >>>>>
"[14/May/2018:15:13:31.658486117 "..., 136)
>>>> =
>>>> >>>>> 136 <0.000346>
>>>> >>>>> 15:13:31.659623 (+ 0.000595)
fsync(4) = 0
>>>> >>>>> <0.003311>
>>>> >>>>> 15:13:31.663230 (+ 0.003642)
getpid() =
>>>> >>>>> 32353 <0.000045>
>>>> >>>>> 15:13:31.663732 (+
0.000454)
>>>> >>>>> socket(AF_UNIX,
SOCK_DGRAM|SOCK_CLOEXEC, 0) =
>>>> >>>>> 33 <0.000296>
>>>> >>>>> 15:13:31.664760 (+
0.001048)
>>>> getsockopt(33,
>>>> >>>>> SOL_SOCKET, SO_SNDBUF, [163840],
[4]) = 0
>>>> >>>>> <0.000108>
>>>> >>>>> 15:13:31.665141 (+
0.000386)
>>>> setsockopt(33,
>>>> >>>>> SOL_SOCKET, SO_SNDBUFFORCE,
[8388608], 4) =
>>>> -1
>>>> >>>>> EPERM (Operation not permitted)
<0.000051>
>>>> >>>>> 15:13:31.665500 (+
0.000334)
>>>> setsockopt(33,
>>>> >>>>> SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0
>>>> <0.000229>
>>>> >>>>> 15:13:31.665973 (+ 0.000468)
sendmsg(33,
>>>> >>>>> {msg_name={sa_family=AF_UNIX,
>>>> >>>>>
sun_path="/run/systemd/notify"},
>>>> >>>>> msg_namelen=21,
>>>> >>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>> >>>>> started: Re"...,
iov_len=69}], msg_iovlen=1,
>>>> >>>>> msg_controllen=0, msg_flags=0},
>>>> MSG_NOSIGNAL) =
>>>> >>>>> 69 <0.000600>
>>>> >>>>> 15:13:31.667270 (+ 0.001334)
close(33) =
>>>> 0
>>>> >>>>> <0.003140>
>>>> >>>>> 15:13:31.677320 (+
0.010146)
>>>> >>>>> futex(0xb31122e8, FUTEX_WAIT,
32357, NULL) =
>>>> ?
>>>> >>>>> 15:14:27.661809 (+ 55.984448)
+++ killed
>>>> by
>>>> >>>>> SIGSEGV (core dumped) +++
>>>> >>>>>
>>>> >>>>> On Mon, May 14, 2018 at 10:27
AM, thierry
>>>> >>>>> bordaz <tbordaz(a)redhat.com
>>>> >>>>>
<mailto:tbordaz@redhat.com>> wrote:
>>>> >>>>>
>>>> >>>>> Hi Jonathan,
>>>> >>>>>
>>>> >>>>> This is weird as the
crashing thread
>>>> stack
>>>> >>>>> looks truncated (did you
copy/paste all
>>>> of
>>>> >>>>> it ?)
>>>> >>>>>
>>>> >>>>> Thread 1 (Thread 0x9e13c280
(LWP 17245)):
>>>> >>>>> #0 0xb67bbf2e in strlen ()
at
>>>> /lib/libc.so.6
>>>> >>>>> #1 0xb6a06b40 in dosprintf
() at
>>>> >>>>> /lib/libnspr4.so
>>>> >>>>> #2 0x00000000 in None ()
>>>> >>>>>
>>>> >>>>> Did you install
389-ds-base-debuginfo ?
>>>> >>>>> How did you get that
backtrace ? from a
>>>> >>>>> core dumped, pstack ? Can
you attach a
>>>> >>>>> debugger before the crash
occurs ?
>>>> >>>>>
>>>> >>>>> It looks it crashed soon at
startup,
>>>> could
>>>> >>>>> it be related to a broken
dse.ldif. It
>>>> >>>>> should exists a dse.ldif.OK,
is it
>>>> possibly
>>>> >>>>> to try to start with it ?
>>>> >>>>>
>>>> >>>>> best regards
>>>> >>>>> thierry
>>>> >>>>>
>>>> >>>>> On 05/12/2018 01:22 AM,
Jonathan Vaughn
>>>> via
>>>> >>>>> FreeIPA-users wrote:
>>>> >>>>>> Not sure if it makes a
difference... I
>>>> was
>>>> >>>>>> looking into this again
and realized I
>>>> had
>>>> >>>>>> a bunch of messages from
gdb telling me
>>>> to
>>>> >>>>>> install more debuginfo.
I've done that
>>>> >>>>>> now, here it is again
freshly run
>>>> through
>>>> >>>>>> gdb
>>>> >>>>>>
>>>> >>>>>> GNU gdb (GDB) Fedora
8.0.1-36.fc27
>>>> >>>>>> Copyright (C) 2017 Free
Software
>>>> >>>>>> Foundation, Inc.
>>>> >>>>>> License GPLv3+: GNU GPL
version 3 or
>>>> later
>>>> >>>>>>
<http://gnu.org/licenses/gpl.html
>>>> >>>>>>
<http://gnu.org/licenses/gpl.html>>
>>>> >>>>>> This is free software:
you are free to
>>>> >>>>>> change and redistribute
it.
>>>> >>>>>> There is NO WARRANTY, to
the extent
>>>> >>>>>> permitted by law. Type
"show copying"
>>>> >>>>>> and "show
warranty" for details.
>>>> >>>>>> This GDB was configured
as
>>>> >>>>>>
"armv7hl-redhat-linux-gnueabi".
>>>> >>>>>> Type "show
configuration" for
>>>> >>>>>> configuration details.
>>>> >>>>>> For bug reporting
instructions, please
>>>> see:
>>>> >>>>>>
<http://www.gnu.org/software/gdb/bugs/
>>>> >>>>>>
<http://www.gnu.org/software/gdb/bugs/
>>>> >>.
>>>> >>>>>> Find the GDB manual and
other
>>>> >>>>>> documentation resources
online at:
>>>> >>>>>>
<http://www.gnu.org/software/
>>>> gdb/documentation/
>>>> >>>>>>
<http://www.gnu.org/software/
>>>> gdb/documentation/>>.
>>>> >>>>>> For help, type
"help".
>>>> >>>>>> Type "apropos
word" to search for
>>>> commands
>>>> >>>>>> related to
"word"...
>>>> >>>>>> Reading symbols from
>>>> >>>>>>
/usr/sbin/ns-slapd...Reading symbols
>>>> from
>>>> >>>>>>
/usr/lib/debug/usr/sbin/ns-sl
>>>> apd-1.3.7.10-1.fc27.arm.debug...done.
>>>> >>>>>> done.
>>>> >>>>>> ...
>>>> >>>>>>
>>>> >>>>>> Thread 1 (Thread
0x9e13c280 (LWP
>>>> 17245)):
>>>> >>>>>> #0 0xb67bbf2e in strlen
() at
>>>> /lib/libc.so.6
>>>> >>>>>> #1 0xb6a06b40 in
dosprintf () at
>>>> >>>>>> /lib/libnspr4.so
>>>> >>>>>> #2 0x00000000 in None
()
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Tue, May 8, 2018
at 7:52 AM, Rob
>>>> >>>>>> Crittenden
<rcritten(a)redhat.com
>>>> >>>>>>
<mailto:rcritten@redhat.com>>
>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> Jonathan Vaughn
via
>>>> FreeIPA-users
>>>> >>>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> Still trying
to figure this
>>>> >>>>>> out. It
looks like slapd is
>>>> >>>>>> dying, I
thought it was
>>>> still
>>>> >>>>>> running for
some reason.
>>>> >>>>>>
>>>> >>>>>> slapd is
dying to segfault.
>>>> >>>>>> strace of it
happening
>>>> doesn't
>>>> >>>>>> seem to
reveal much:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> A stack trace
would very much
>>>> help
>>>> >
>>>>
>>> ...
>
> [Message clipped]
3:23 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Actually - may have spoke too soon. I thought I'd signed in to the UI but I
was actually in another server's UI. When I try to log in it says that my
login has expired... so progress but something else is busted.
On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
Looks like replica install finished just fine, and I can access the
the
web UI on it. No obvious problems.
So it would appear that code change fixed 389DS for ARM.
I'm not running the Dogtag CA on the Pi, just on the non-Pi servers,
otherwise would need to probably adjust the startup timeout for it so that
the install script didn't give up waiting on it. When I was first trying to
play with this I tried installing the first replica/master on the Pi, and
ran into the Dogtag timeout problem. Even installing Oracle Java instead of
the OpenJDK version of Java didn't solve it, because somehow it always
picks the OpenJDK java instead of oracle even if I change the alternatives
setting to default to the Oracle one...
But as far as I can tell otherwise, everything is working as expected.
On Fri, May 18, 2018 at 2:10 PM, Mark Reynolds <mreynolds(a)redhat.com>
wrote:
>
>
> On 05/18/2018 03:04 PM, Jonathan Vaughn wrote:
>
> Build completed, I tested it, got some error about libs... you had a
> rogue lib64 in the build command given :)
>
> Ah, yeah sorry that configure command was not fully tested I guess (I'll
> update my notes)...
>
> Rebuilt with that fixed to be regular lib (since this is a 32-bit OS) and
> it appears to be running fine.
>
> Well, it hasn't segfaulted. It complains about failing to get principles
> from keytab and stuff... but that is probably because I could never
> complete ipa-replica-install. It's still running. I'm going to uninstall
> and reinstall replica and see what happens now.
>
> Keep us posted
>
> Thanks,
> Mark
>
>
> On Thu, May 17, 2018 at 9:09 PM, Mark Reynolds <mreynolds(a)redhat.com>
> wrote:
>
>>
>>
>> On 05/17/2018 08:06 PM, Jonathan Vaughn wrote:
>>
>> Sorry, missed your build instructions, only saw them just now. I've just
>> kicked off that build and I'll check it tomorrow once I'm back in.
>>
>> One note, it appears it's ./autogen.sh not ./autogen.pl ...
>>
>> Whoops! Good catch, and good luck...
>>
>>
>> On Thu, May 17, 2018 at 6:16 PM, Jonathan Vaughn <jonathan(a)creatuity.com
>> > wrote:
>>
>>> Welp, I'm still getting the -fPIC error ... so I may need to figure out
>>> how to do it with COPR or something.
>>>
>>> On Thu, May 17, 2018 at 4:16 PM, Jonathan Vaughn <
>>> jonathan(a)creatuity.com> wrote:
>>>
>>>> I've never used COPR. I've dabbled with RPMs in the past but
that
>>>> was... CentOS 6 I think, and I wasn't making source code changes so
much as
>>>> just copying and pasting SRPMs from another RPM platform to build for
>>>> CentOS, using the regular rpmbuild stuff.
>>>>
>>>> I did actually try just copying the configure line (manually expanding
>>>> the various vars) from the SRPM and just doing an old fashioned make,
but
>>>> the first try bailed shy of an hour at linking with some error about not
>>>> having -fPIC specified, so I restarted it with -fPIC to see if it would
at
>>>> least build. So if I should still try the code changes, if that build
>>>> finishes I can try it (I was planning to just copy the respective
binaries
>>>> on top of the installed ones), but if it turned out the suggested code
>>>> changes were pointless then there's no point waiting for it to
finish
>>>> building.
>>>>
>>>> On Thu, May 17, 2018 at 4:11 PM, Rob Crittenden
<rcritten(a)redhat.com>
>>>> wrote:
>>>>
>>>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>>> > Oops, hit reply instead of reply-all
>>>>> >
>>>>> > NSPR RPMs
>>>>> >
>>>>> > # yum list installed nspr*
>>>>> > Installed Packages
>>>>> > nspr.armv7hl
>>>>>
>>>>> > 4.19.0-1.fc27
>>>>>
>>>>> > @updates
>>>>> > nspr-debuginfo.armv7hl
>>>>>
>>>>> > 4.19.0-1.fc27
>>>>>
>>>>> > @updates-debuginfo
>>>>> > nspr-debugsource.armv7hl
>>>>>
>>>>> > 4.19.0-1.fc27
>>>>>
>>>>> > @updates-debuginfo
>>>>> > nspr-devel.armv7hl
>>>>>
>>>>> > 4.19.0-1.fc27
>>>>>
>>>>> > @updates
>>>>> >
>>>>> > As for building it with the changes you proposed previously -
>>>>> should I
>>>>> > not bother? I realized last night that the last and only time I
>>>>> built
>>>>> > anything *on* a Pi was a Pi Zero and that's why it was so
terribly
>>>>> slow.
>>>>> > This Pi 3 ain't fast but it's fast enough to build stuff
in far less
>>>>> > than half a day (unlike the Zero)...
>>>>>
>>>>> If you are handy with rpms you could build it in copr.
>>>>>
>>>>> rob
>>>>>
>>>>> >
>>>>> >
>>>>> > On Thu, May 17, 2018 at 5:07 AM, thierry bordaz
<tbordaz(a)redhat.com
>>>>> > <mailto:tbordaz@redhat.com>> wrote:
>>>>> >
>>>>> >
>>>>> >
>>>>> > On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
>>>>> >> I've been just using the packages from Fedora. I can
build it
>>>>> >> potentially but I don't have a cross build
environment set up
>>>>> at
>>>>> >> the moment. From experience I'd want to do that
first because
>>>>> >> building anything on the Pi usually takes ages.
>>>>> >>
>>>>> >> I'd been "redacting" the hostnames but
I'll stop bothering
>>>>> since
>>>>> >> it looks like we're getting far enough into the
weeds now that
>>>>> the
>>>>> >> difference in string lengths after "redacting"
might actually
>>>>> be a
>>>>> >> red herring.
>>>>> >>
>>>>> >> (gdb) p *agmt
>>>>> >> $1 = {hostname = 0x1ef9be0
"ipa-12.creatuity.internal", port =
>>>>> >> 389, transport_flags = 0, binddn = 0x1e8f650
"", creds =
>>>>> >> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
>>>>> >> frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
>>>>> >> frac_attr_total_defined = 1, schedule = 0x1a7f0c0,
>>>>> auto_initialize
>>>>> >> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
>>>>> >> long_name = 0x1a7f100
"agmt=\"cn=meToipa-12.creatuit
>>>>> y.internal\"
>>>>> >> (ipa-12:5)", protocol = 0x19c2930, changecounters =
0x186d180,
>>>>> >> num_changecounters = 0,
>>>>> >> max_changecounters = 256, last_update_start_time =
>>>>> 1526500697,
>>>>> >> last_update_end_time = 1526500697,
>>>>> >> last_update_status = "Error (0) Replica acquired
>>>>> successfully:
>>>>> >> Incremental update succeeded", '\000'
<repeats 954 times>,
>>>>> >> update_in_progress = 0, is_enabled = 1,
>>>>> >> last_init_start_time = 0, last_init_end_time = 0,
>>>>> >> last_init_status = '\000' <repeats 1023
times>, lock =
>>>>> 0x1ee3740,
>>>>> >> consumerRUV = 0x1f14e50,
>>>>> >> consumerSchemaCSN = 0x317c520, consumerRID = 4,
>>>>> tmpConsumerRID =
>>>>> >> 0, timeout = 120, stop_in_progress = 0, busywaittime =
0,
>>>>> >> pausetime = 0, priv = 0x0,
>>>>> >> attrs_to_strip = 0x1ef9ba0, agreement_type = 0,
>>>>> protocol_timeout
>>>>> >> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow = 1000,
>>>>> >> flowControlPause = 2000, ignoreMissingChange = 0,
>>>>> >> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
>>>>> >> (gdb) p *agmt->replarea
>>>>> >> $2 = {flag = 15 '\017', udn = 0x1efce80
>>>>> >> "dc=creatuity,dc=internal", dn = 0x1ef9460
>>>>> >> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
>>>>> >> "dc=creatuity,dc=internal", ndn_len = 24}
>>>>> >> (gdb) p *agmt->rdn
>>>>> >> $3 = {flag = 0 '\000', rdn = 0x19c2840
>>>>> >> "cn=meToipa-12.creatuity.internal", rdns =
0x0, butcheredupto
>>>>> =
>>>>> >> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
>>>>> >>
>>>>> >> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
>>>>> >> /usr/include/inttypes.h:# define PRId64
__PRI64_PREFIX
>>>>> "d"
>>>>> >> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
>>>>> >> /usr/include/inttypes.h:# define PRIu16
"u"
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
>>>>> >> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>> wrote:
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>>>>> >>> The installed version of 389* is
1.3.7.10-1.fc27
>>>>> for armv7hl,
>>>>> >>> which appears to be the latest available
version.
>>>>> >>
>>>>> >> Perhaps something is off with the inttypes on
Raspberry.
>>>>> Are
>>>>> >> you building this yourself on Raspberry? Can we
make code
>>>>> >> changes and compile/install them?
>>>>> >>
>>>>> >> Before we do that though, in gdb can you run these
>>>>> commands in
>>>>> >> the same gdb frame:
>>>>> >>
>>>>> >> (gdb) p *agmt->replarea
>>>>> >> (gdb) p *agmt->rdn
>>>>> >>
>>>>> >>
>>>>> >> Then do:
>>>>> >>
>>>>> >> # grep -r PRId64 /usr/include/*
>>>>> >> # grep -r PRIu16 /usr/include/*
>>>>> >>
>>>>> >>
>>>>> >> So if you can compile the source, then change this
line in
>>>>> >> ldap/servers/plugins/replication/repl5_agmt.c:3036,
but
>>>>> don't
>>>>> >> do this yet until you get me the info I just
requested.
>>>>> >>
>>>>> >> From:
>>>>> >>
>>>>> >> agmt->maxcsn =
>>>>> slapi_ch_smprintf("%s;%s;%s;%"
>>>>> >> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replare
>>>>> a),
>>>>> >>
>>>>> >>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>>>>> >> agmt->hostname,
>>>>> >>
>>>>> agmt->port,
>>>>> >> agmt->consumerRID, maxcsn);
>>>>> >>
>>>>> >> To:
>>>>> >>
>>>>> >> agmt->maxcsn =
>>>>> >> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>>>> >> slapi_sdn_get_dn(agmt->replarea),
>>>>> >>
>>>>> >>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>>>>> >> agmt->hostname,
>>>>> >>
>>>>> >> (long)agmt->port, (int)agmt->consumerRID,
maxcsn);
>>>>> >>
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Mark
>>>>> >>
>>>>> >>>
>>>>> >>> On Wed, May 16, 2018 at 2:38 PM, Jonathan
Vaughn
>>>>> >>> <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>>
>>>>> wrote:
>>>>> >>
>>>>> >
>>>>> > The agreement structure looks valid to me. it should not
lead
>>>>> to a
>>>>> > crash.
>>>>> >
>>>>> > What looks weird to me is the order of the arguments of
cvt_s.
>>>>> > It is called: rv = cvt_s(ss, u.s, width, prec, flags);
>>>>> > But the crashing thread shows the opposite order: flags,
>>>>> prec,width,
>>>>> > str, ss
>>>>> >
>>>>> > The others frame do not show this change of order.
>>>>> > Also 'str=4' that would be a meaningful value for
'prec=4'.
>>>>> >
>>>>> > From debug perspective I only imagine disassemble the two
last
>>>>> > frames to confirm parameters.
>>>>> > What are the nspr rpms ?
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >>> (gdb) up
>>>>> >>> #1 0xb6926b40 in cvt_s (flags=0,
prec=<optimized
>>>>> out>,
>>>>> >>> width=0, str=0x4 <error: Cannot access
memory at
>>>>> address
>>>>> >>> 0x4>, ss=<optimized out>)
>>>>> >>> at
../../.././nspr/pr/src/io/prprf.c:374
>>>>> >>> 374 slen = strlen(str);
>>>>> >>> (gdb) up
>>>>> >>> #2 dosprintf (ss=ss@entry=0x9e06e4bc,
>>>>> fmt=0xb34b0df2 "",
>>>>> >>> fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002",
>>>>> ap=...)
>>>>> >>> at ../../.././nspr/pr/src/io/prprf.c:1018
>>>>> >>> 1018 rv = cvt_s(ss, u.s,
width, prec,
>>>>> flags);
>>>>> >>> (gdb) up
>>>>> >>> #3 0xb6926c8c in PR_vsmprintf
(fmt=fmt@entry
>>>>> =0xb34da770
>>>>> >>> <agmt_set> "\360\317p\002",
ap=..., ap@entry=...) at
>>>>> >>> ../../.././nspr/pr/src/io/prprf.c:1184
>>>>> >>> 1184 rv = dosprintf(&ss, fmt,
ap);
>>>>> >>> (gdb) up
>>>>> >>> #4 0xb6de9d18 in slapi_ch_smprintf
(fmt=0xb34b0de0
>>>>> >>> "%s;%s;%s;%ld;%d;%s") at
>>>>> ldap/servers/slapd/ch_malloc.c:331
>>>>> >>> 331 p = PR_vsmprintf(fmt, ap);
>>>>> >>> (gdb) up
>>>>> >>> #5 0xb34625a4 in agmt_update_maxcsn
>>>>> >>> (r=r@entry=0x2737810, sdn=0x38d9a40,
sdn@entry=0x10,
>>>>> >>> op=<optimized out>, mods=0x10,
mods@entry=0x27d0100,
>>>>> >>> csn=csn@entry=0x38de350)
>>>>> >>> at ldap/servers/plugins/replicati
>>>>> on/repl5_agmt.c:3036
>>>>> >>> 3036 agmt->maxcsn =
>>>>> >>>
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>>>> >>> slapi_sdn_get_dn(agmt->replarea),
>>>>> >>> (gdb) p *agmt
>>>>> >>> $1 = {hostname = 0x2757be0
"ipa-12.company.internal",
>>>>> >>> port = 389, transport_flags = 0, binddn =
0x26ed650
>>>>> "",
>>>>> >>> creds = 0x26ed7a0, bindmethod = 3, replarea
=
>>>>> 0x2757480,
>>>>> >>> frac_attrs = 0x27579c0, frac_attrs_total
=
>>>>> 0x2757a40,
>>>>> >>> frac_attr_total_defined = 1, schedule =
0x22dd0c0,
>>>>> >>> auto_initialize = 502, dn = 0x2756d00, rdn
=
>>>>> 0x2756c20,
>>>>> >>> long_name = 0x22dd100
>>>>> >>>
"agmt=\"cn=meToipa-12.company.internal\" (ipa-12:5)",
>>>>> >>> protocol = 0x2220930, changecounters =
0x20cb180,
>>>>> >>> num_changecounters = 0,
>>>>> >>> max_changecounters = 256,
last_update_start_time =
>>>>> >>> 1526499214, last_update_end_time =
1526499214,
>>>>> >>> last_update_status = "Error (0)
Replica acquired
>>>>> >>> successfully: Incremental update
succeeded", '\000'
>>>>> >>> <repeats 954 times>,
update_in_progress = 0,
>>>>> is_enabled = 1,
>>>>> >>> last_init_start_time = 0,
last_init_end_time = 0,
>>>>> >>> last_init_status = '\000'
<repeats 1023 times>, lock =
>>>>> >>> 0x2741740, consumerRUV = 0x2772e50,
>>>>> >>> consumerSchemaCSN = 0x39da520, consumerRID
= 4,
>>>>> >>> tmpConsumerRID = 0, timeout = 120,
stop_in_progress =
>>>>> 0,
>>>>> >>> busywaittime = 0, pausetime = 0, priv =
0x0,
>>>>> >>> attrs_to_strip = 0x2757ba0, agreement_type
= 0,
>>>>> >>> protocol_timeout = 0x26ed5f0, maxcsn = 0x0,
>>>>> >>> flowControlWindow = 1000, flowControlPause =
2000,
>>>>> >>> ignoreMissingChange = 0,
>>>>> >>> attr_lock = 0x2757c20, WaitForAsyncResults
= 100}
>>>>> >>>
>>>>> >>>
>>>>> >>> On Tue, May 15, 2018 at 5:36 PM, Mark
Reynolds
>>>>> >>> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>>
>>>>> wrote:
>>>>> >>>
>>>>> >>> This looks really familiar and I thought
it was
>>>>> >>> fixed. It should have been fixed in
1.3.7.10-1
>>>>> >>>
(https://pagure.io/389-ds-base/issue/49618
>>>>> >>>
<https://pagure.io/389-ds-base/issue/49618>).
>>>>> In
>>>>> >>> your debug session go "up"
into
>>>>> agmt_maxcsn_update()
>>>>> >>> and do:
>>>>> >>>
>>>>> >>> (gdb) p *agmt
>>>>> >>>
>>>>> >>> Then send us that output please.
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> Mark
>>>>> >>>
>>>>> >>>
>>>>> >>> On 05/15/2018 05:29 PM, Jonathan Vaughn
via
>>>>> >>> FreeIPA-users wrote:
>>>>> >>>> Here is a backtrace from live gdb
after the
>>>>> >>>> segfault. Looks like things went
wrong somewhere
>>>>> >>>> during in the replication code ?
>>>>> >>>>
>>>>> >>>> Thread 36 "ns-slapd"
received signal SIGSEGV,
>>>>> >>>> Segmentation fault.
>>>>> >>>> [Switching to Thread 0x9e0bc280 (LWP
4662)]
>>>>> >>>> strlen () at
../sysdeps/arm/armv6t2/strlen.S:142
>>>>> >>>> 142 ldrd data1a,
data1b, [src]
>>>>> >>>> (gdb) bt
>>>>> >>>> #0 0xb6728f2e in strlen () at
>>>>> >>>> ../sysdeps/arm/armv6t2/strlen.S:142
>>>>> >>>> #1 0xb6973b40 in cvt_s (flags=0,
prec=<optimized
>>>>> >>>> out>, width=0, str=0x4 <error:
Cannot access
>>>>> memory
>>>>> >>>> at address 0x4>, ss=<optimized
out>)
>>>>> >>>> at
../../.././nspr/pr/src/io/prprf.c:374
>>>>> >>>> #2 0xb6973b40 in dosprintf
(ss=ss@entry
>>>>> =0x9e0bb4bc,
>>>>> >>>> fmt=0xb34fddf2 "",
fmt@entry=0xb3527770
>>>>> <agmt_set>
>>>>> >>>> "\360\357\305\002",
ap=...)
>>>>> >>>> at
../../.././nspr/pr/src/io/prprf.c:1018
>>>>> >>>> #3 0xb6973c8c in PR_vsmprintf
>>>>> >>>> (fmt=fmt@entry=0xb3527770
<agmt_set>
>>>>> >>>> "\360\357\305\002",
ap=..., ap@entry=...) at
>>>>> >>>>
../../.././nspr/pr/src/io/prprf.c:1184
>>>>> >>>> #4 0xb6e36d18 in slapi_ch_smprintf
>>>>> (fmt=0xb34fdde0
>>>>> >>>> "%s;%s;%s;%ld;%d;%s") at
>>>>> >>>> ldap/servers/slapd/ch_malloc.c:331
>>>>> >>>> #5 0xb34af5a4 in
agmt_update_maxcsn
>>>>> >>>> (r=r@entry=0x2c89810, sdn=0x3e2bac0,
sdn@entry
>>>>> =0x10,
>>>>> >>>> op=<optimized out>, mods=0x10,
mods@entry
>>>>> =0x3eec100,
>>>>> >>>> csn=csn@entry=0x3e30220)
>>>>> >>>> at
>>>>> >>>> ldap/servers/plugins/replicat
>>>>> ion/repl5_agmt.c:3036
>>>>> >>>> #6 0xb34bd424 in
write_changelog_and_ruv
>>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>>> >>>> ldap/servers/plugins/replicat
>>>>> ion/repl5_plugins.c:1124
>>>>> >>>> #7 0xb34be7ec in
multimaster_be_betxnpostop_add
>>>>> >>>> (pb=0x2cb54a0) at
>>>>> >>>> ldap/servers/plugins/replicat
>>>>> ion/repl5_plugins.c:855
>>>>> >>>> #8 0xb34be880 in
multimaster_mmr_postop
>>>>> >>>> (pb=<optimized out>,
flags=560) at
>>>>> >>>> ldap/servers/plugins/replicat
>>>>> ion/repl5_plugins.c:616
>>>>> >>>> #9 0xb6e8fcf0 in
plugin_call_mmr_plugin_postop
>>>>> >>>> (pb=pb@entry=0x2cb54a0,
e=e@entry=0x0,
>>>>> >>>> flags=flags@entry=560) at
>>>>> >>>> ldap/servers/slapd/plugin_mmr.c:65
>>>>> >>>> #10 0xb35ba870 in ldbm_back_add
(pb=0x2cb54a0) at
>>>>> >>>>
ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>>>>> >>>> #11 0xb6e2ce4c in op_shared_add
>>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>>> ldap/servers/slapd/add.c:679
>>>>> >>>> #12 0xb6e2d35c in add_internal_pb
>>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>>> ldap/servers/slapd/add.c:407
>>>>> >>>> #13 0xb6e2e0f8 in
slapi_add_internal_pb
>>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>>> ldap/servers/slapd/add.c:332
>>>>> >>>> #14 0xb368db50 in
ipa_topo_util_segment_write
>>>>> >>>> (tconf=tconf@entry=0x2cb5860,
>>>>> >>>> tsegm=tsegm@entry=0x3f2d9a0) at
>>>>> topology_util.c:1251
>>>>> >>>> #15 0xb368e01c in
ipa_topo_util_update_agmt_list
>>>>> >>>> (conf=0x2cb5860,
repl_segments=<optimized out>)
>>>>> at
>>>>> >>>> topology_util.c:696
>>>>> >>>> #16 0xb368891c in
ipa_topo_apply_shared_config
>>>>> () at
>>>>> >>>> topology_init.c:165
>>>>> >>>> #17 0xb6e4e65c in eq_call_all () at
>>>>> >>>> ldap/servers/slapd/eventq.c:278
>>>>> >>>> #18 0xb6e4e65c in eq_loop
(arg=<optimized out>)
>>>>> at
>>>>> >>>> ldap/servers/slapd/eventq.c:323
>>>>> >>>> #19 0xb6982d68 in _pt_root
(arg=0x2c8da40) at
>>>>> >>>>
../../.././nspr/pr/src/pthreads/ptthread.c:201
>>>>> >>>> #20 0xb6906ee8 in start_thread
(arg=0x9e0bc280)
>>>>> at
>>>>> >>>> pthread_create.c:465
>>>>> >>>> #21 0xb6785da8 in None () at
>>>>> >>>>
../sysdeps/unix/sysv/linux/arm/clone.S:73
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Tue, May 15, 2018 at 3:01 AM,
thierry bordaz
>>>>> >>>> <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> Hi Jonathan,
>>>>> >>>>
>>>>> >>>> This problem looks new to me and
has
>>>>> something
>>>>> >>>> specific to your environment.
>>>>> >>>> I think the best approach is to
continue to
>>>>> >>>> debug on your system if you have
the
>>>>> possibility
>>>>> >>>> to do so.
>>>>> >>>>
>>>>> >>>> From strace we can see that DS
started
>>>>> smoothly
>>>>> >>>> (created its pid file then
notified systemd
>>>>> it
>>>>> >>>> was running fine). According to
the pstack
>>>>> >>>> nunc-stans was running and was
able to accept
>>>>> >>>> network events even if it
appears it
>>>>> detected no
>>>>> >>>> incoming connection.
>>>>> >>>> So the server should be ready to
serve for
>>>>> some
>>>>> >>>> seconds (more than a minute),
then it crashed
>>>>> >>>> with one thread dereferencing
likely a wrong
>>>>> >>>> pointer.
>>>>> >>>>
>>>>> >>>> Could you attach a debugger when
the server
>>>>> is
>>>>> >>>> started and wait for the sigsegv
to occur.
>>>>> Then
>>>>> >>>> confirm the crashing thread
backstack.
>>>>> >>>> If it is confirmed, I am afraid
this is a
>>>>> stack
>>>>> >>>> corruption and valgrind could
help
>>>>> >>>> (http://www.port389.org/docs/
>>>>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-wit
>>>>> h-valgrind
>>>>> >>>>
<http://www.port389.org/docs/
>>>>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-wit
>>>>> h-valgrind>).
>>>>> >>>>
>>>>> >>>> best regards
>>>>> >>>> thierry
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On 05/14/2018 10:20 PM, Jonathan
Vaughn
>>>>> wrote:
>>>>> >>>>> Here's a strace from
before it dies. Most of
>>>>> >>>>> the elapsed time is it
waiting on some futex
>>>>> >>>>> call it looks like near the
end, when it
>>>>> >>>>> finally "returns"
(from lack of strace
>>>>> output
>>>>> >>>>> for duration of call I
assume it didn't
>>>>> >>>>> actually return but SIGSEGV
in that call)
>>>>> and
>>>>> >>>>> strace prints ' = ?'
on the futex it then
>>>>> >>>>> immediately reports SIGSEGV
after. So maybe
>>>>> the
>>>>> >>>>> problem is that futex call,
which may mean
>>>>> the
>>>>> >>>>> problem is not directly in
389DS / FreeIPA
>>>>> itself?
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> 15:13:31.626587 (+
0.000630) listen(8,
>>>>> 128)
>>>>> >>>>> = 0 <0.000068>
>>>>> >>>>> 15:13:31.626857 (+
0.000235) listen(9,
>>>>> 128)
>>>>> >>>>> = 0 <0.000048>
>>>>> >>>>> 15:13:31.627111 (+
0.000251)
>>>>> >>>>>
clock_gettime(CLOCK_MONOTONIC,
>>>>> {tv_sec=1464932,
>>>>> >>>>> tv_nsec=41120614}) = 0
<0.000085>
>>>>> >>>>> 15:13:31.627457 (+
0.000356)
>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=627560772}) = 0
>>>>> >>>>> <0.000043>
>>>>> >>>>> 15:13:31.631233 (+
0.003839)
>>>>> >>>>>
clock_gettime(CLOCK_MONOTONIC,
>>>>> {tv_sec=1464932,
>>>>> >>>>> tv_nsec=45286796}) = 0
<0.000077>
>>>>> >>>>> 15:13:31.631720 (+
0.000427)
>>>>> >>>>>
clock_gettime(CLOCK_MONOTONIC,
>>>>> {tv_sec=1464932,
>>>>> >>>>> tv_nsec=45661430}) = 0
<0.000042>
>>>>> >>>>> 15:13:31.631955 (+
0.000220)
>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=632049036}) = 0
>>>>> >>>>> <0.000047>
>>>>> >>>>> 15:13:31.635669 (+
0.003785)
>>>>> >>>>>
clock_gettime(CLOCK_MONOTONIC,
>>>>> {tv_sec=1464932,
>>>>> >>>>> tv_nsec=49725840}) = 0
<0.000146>
>>>>> >>>>> 15:13:31.636484 (+
0.000784) write(16,
>>>>> "a",
>>>>> >>>>> 1) = 1 <0.000118>
>>>>> >>>>> 15:13:31.636855 (+
0.000341)
>>>>> sched_yield()
>>>>> >>>>> = 0 <0.000252>
>>>>> >>>>> 15:13:31.637322 (+
0.000470)
>>>>> >>>>> futex(0x1cb57a0,
FUTEX_WAKE_PRIVATE, 1) = 1
>>>>> >>>>> <0.000088>
>>>>> >>>>> 15:13:31.637897 (+
0.000610) write(16,
>>>>> "a",
>>>>> >>>>> 1) = 1 <0.000221>
>>>>> >>>>> 15:13:31.638394 (+
0.000467)
>>>>> sched_yield()
>>>>> >>>>> = 0 <0.000047>
>>>>> >>>>> 15:13:31.638619 (+
0.000202)
>>>>> >>>>> futex(0x1cb5710,
FUTEX_WAKE_PRIVATE, 1) = 1
>>>>> >>>>> <0.000065>
>>>>> >>>>> 15:13:31.638908 (+
0.000298)
>>>>> >>>>> openat(AT_FDCWD,
>>>>> >>>>>
"/var/run/dirsrv/slapd-COMPAN
>>>>> Y-INTERNAL.pid",
>>>>> >>>>> O_WRONLY|O_CREAT|O_TRUNC,
0666) = 33
>>>>> <0.000831>
>>>>> >>>>> 15:13:31.640260 (+
0.001387) getpid() =
>>>>> >>>>> 32353 <0.000077>
>>>>> >>>>> 15:13:31.640558 (+
0.000256) fstat64(33,
>>>>> >>>>> {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0
>>>>> >>>>> <0.000119>
>>>>> >>>>> 15:13:31.641106 (+
0.000556) write(33,
>>>>> >>>>> "32353\n", 6) = 6
<0.000127>
>>>>> >>>>> 15:13:31.641472 (+
0.000362) close(33)
>>>>> = 0
>>>>> >>>>> <0.000519>
>>>>> >>>>> 15:13:31.642216 (+
0.000758)
>>>>> >>>>>
chmod("/var/run/dirsrv/slapd-
>>>>> COMPANY-INTERNAL.pid",
>>>>> >>>>> 0644) = 0 <0.000152>
>>>>> >>>>> 15:13:31.642900 (+
0.000679)
>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=643020294}) = 0
>>>>> >>>>> <0.000056>
>>>>> >>>>> 15:13:31.643495 (+
0.000590) write(2,
>>>>> >>>>>
"[14/May/2018:15:13:31.643020294 "...,
>>>>> 134) =
>>>>> >>>>> 134 <0.002697>
>>>>> >>>>> 15:13:31.646515 (+
0.003052)
>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=646694394}) = 0
>>>>> >>>>> <0.000075>
>>>>> >>>>> 15:13:31.646892 (+
0.000337) write(4,
>>>>> >>>>>
"[14/May/2018:15:13:31.646694394 "...,
>>>>> 134) =
>>>>> >>>>> 134 <0.000522>
>>>>> >>>>> 15:13:31.647841 (+
0.000973) fsync(4) =
>>>>> 0
>>>>> >>>>> <0.005967>
>>>>> >>>>> 15:13:31.654425 (+
0.006617)
>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=654598946}) = 0
>>>>> >>>>> <0.000253>
>>>>> >>>>> 15:13:31.655137 (+
0.000717) write(2,
>>>>> >>>>>
"[14/May/2018:15:13:31.654598946 "...,
>>>>> 136) =
>>>>> >>>>> 136 <0.002427>
>>>>> >>>>> 15:13:31.658312 (+
0.003165)
>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=658486117}) = 0
>>>>> >>>>> <0.000251>
>>>>> >>>>> 15:13:31.659032 (+
0.000682) write(4,
>>>>> >>>>>
"[14/May/2018:15:13:31.658486117 "...,
>>>>> 136) =
>>>>> >>>>> 136 <0.000346>
>>>>> >>>>> 15:13:31.659623 (+
0.000595) fsync(4) =
>>>>> 0
>>>>> >>>>> <0.003311>
>>>>> >>>>> 15:13:31.663230 (+
0.003642) getpid() =
>>>>> >>>>> 32353 <0.000045>
>>>>> >>>>> 15:13:31.663732 (+
0.000454)
>>>>> >>>>> socket(AF_UNIX,
SOCK_DGRAM|SOCK_CLOEXEC, 0)
>>>>> =
>>>>> >>>>> 33 <0.000296>
>>>>> >>>>> 15:13:31.664760 (+
0.001048)
>>>>> getsockopt(33,
>>>>> >>>>> SOL_SOCKET, SO_SNDBUF,
[163840], [4]) = 0
>>>>> >>>>> <0.000108>
>>>>> >>>>> 15:13:31.665141 (+
0.000386)
>>>>> setsockopt(33,
>>>>> >>>>> SOL_SOCKET, SO_SNDBUFFORCE,
[8388608], 4) =
>>>>> -1
>>>>> >>>>> EPERM (Operation not
permitted) <0.000051>
>>>>> >>>>> 15:13:31.665500 (+
0.000334)
>>>>> setsockopt(33,
>>>>> >>>>> SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0
>>>>> <0.000229>
>>>>> >>>>> 15:13:31.665973 (+
0.000468) sendmsg(33,
>>>>> >>>>>
{msg_name={sa_family=AF_UNIX,
>>>>> >>>>>
sun_path="/run/systemd/notify"},
>>>>> >>>>> msg_namelen=21,
>>>>> >>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>>> >>>>> started: Re"...,
iov_len=69}], msg_iovlen=1,
>>>>> >>>>> msg_controllen=0,
msg_flags=0},
>>>>> MSG_NOSIGNAL) =
>>>>> >>>>> 69 <0.000600>
>>>>> >>>>> 15:13:31.667270 (+
0.001334) close(33)
>>>>> = 0
>>>>> >>>>> <0.003140>
>>>>> >>>>> 15:13:31.677320 (+
0.010146)
>>>>> >>>>> futex(0xb31122e8,
FUTEX_WAIT, 32357, NULL)
>>>>> = ?
>>>>> >>>>> 15:14:27.661809 (+
55.984448) +++ killed
>>>>> by
>>>>> >>>>> SIGSEGV (core dumped) +++
>>>>> >>>>>
>>>>> >>>>> On Mon, May 14, 2018 at
10:27 AM, thierry
>>>>> >>>>> bordaz
<tbordaz(a)redhat.com
>>>>> >>>>>
<mailto:tbordaz@redhat.com>> wrote:
>>>>> >>>>>
>>>>> >>>>> Hi Jonathan,
>>>>> >>>>>
>>>>> >>>>> This is weird as the
crashing thread
>>>>> stack
>>>>> >>>>> looks truncated (did you
copy/paste all
>>>>> of
>>>>> >>>>> it ?)
>>>>> >>>>>
>>>>> >>>>> Thread 1 (Thread
0x9e13c280 (LWP
>>>>> 17245)):
>>>>> >>>>> #0 0xb67bbf2e in strlen
() at
>>>>> /lib/libc.so.6
>>>>> >>>>> #1 0xb6a06b40 in
dosprintf () at
>>>>> >>>>> /lib/libnspr4.so
>>>>> >>>>> #2 0x00000000 in None
()
>>>>> >>>>>
>>>>> >>>>> Did you install
389-ds-base-debuginfo ?
>>>>> >>>>> How did you get that
backtrace ? from a
>>>>> >>>>> core dumped, pstack ?
Can you attach a
>>>>> >>>>> debugger before the
crash occurs ?
>>>>> >>>>>
>>>>> >>>>> It looks it crashed soon
at startup,
>>>>> could
>>>>> >>>>> it be related to a
broken dse.ldif. It
>>>>> >>>>> should exists a
dse.ldif.OK, is it
>>>>> possibly
>>>>> >>>>> to try to start with it
?
>>>>> >>>>>
>>>>> >>>>> best regards
>>>>> >>>>> thierry
>>>>> >>>>>
>>>>> >>>>> On 05/12/2018 01:22 AM,
Jonathan Vaughn
>>>>> via
>>>>> >>>>> FreeIPA-users wrote:
>>>>> >>>>>> Not sure if it makes
a difference... I
>>>>> was
>>>>> >>>>>> looking into this
again and realized I
>>>>> had
>>>>> >>>>>> a bunch of messages
from gdb telling
>>>>> me to
>>>>> >>>>>> install more
debuginfo. I've done that
>>>>> >>>>>> now, here it is
again freshly run
>>>>> through
>>>>> >>>>>> gdb
>>>>> >>>>>>
>>>>> >>>>>> GNU gdb (GDB) Fedora
8.0.1-36.fc27
>>>>> >>>>>> Copyright (C) 2017
Free Software
>>>>> >>>>>> Foundation, Inc.
>>>>> >>>>>> License GPLv3+: GNU
GPL version 3 or
>>>>> later
>>>>> >>>>>>
<http://gnu.org/licenses/gpl.html
>>>>> >>>>>>
<http://gnu.org/licenses/gpl.html>>
>>>>> >>>>>> This is free
software: you are free to
>>>>> >>>>>> change and
redistribute it.
>>>>> >>>>>> There is NO
WARRANTY, to the extent
>>>>> >>>>>> permitted by law.
Type "show copying"
>>>>> >>>>>> and "show
warranty" for details.
>>>>> >>>>>> This GDB was
configured as
>>>>> >>>>>>
"armv7hl-redhat-linux-gnueabi".
>>>>> >>>>>> Type "show
configuration" for
>>>>> >>>>>> configuration
details.
>>>>> >>>>>> For bug reporting
instructions, please
>>>>> see:
>>>>> >>>>>>
<http://www.gnu.org/software/gdb/bugs/
>>>>> >>>>>>
<http://www.gnu.org/software/gdb/bugs/
>>>>> >>.
>>>>> >>>>>> Find the GDB manual
and other
>>>>> >>>>>> documentation
resources online at:
>>>>> >>>>>>
<http://www.gnu.org/software/
>>>>> gdb/documentation/
>>>>> >>>>>>
<http://www.gnu.org/software/
>>>>> gdb/documentation/>>.
>>>>> >>>>>> For help, type
"help".
>>>>> >>>>>> Type "apropos
word" to search for
>>>>> commands
>>>>> >>>>>> related to
"word"...
>>>>> >>>>>> Reading symbols
from
>>>>> >>>>>>
/usr/sbin/ns-slapd...Reading symbols
>>>>> from
>>>>> >>>>>>
/usr/lib/debug/usr/sbin/ns-sl
>>>>> apd-1.3.7.10-1.fc27.arm.debug...done.
>>>>> >>>>>> done.
>>>>> >>>>>> ...
>>>>> >>>>>>
>>>>> >>>>>> Thread 1 (Thread
0x9e13c280 (LWP
>>>>> 17245)):
>>>>> >>>>>> #0 0xb67bbf2e in
strlen () at
>>>>> /lib/libc.so.6
>>>>> >>>>>> #1 0xb6a06b40 in
dosprintf () at
>>>>> >>>>>> /lib/libnspr4.so
>>>>> >>>>>> #2 0x00000000 in
None ()
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> On Tue, May 8,
2018 at 7:52 AM, Rob
>>>>> >>>>>> Crittenden
<rcritten(a)redhat.com
>>>>> >>>>>>
<mailto:rcritten@redhat.com>>
>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Jonathan
Vaughn via
>>>>> FreeIPA-users
>>>>> >>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Still
trying to figure this
>>>>> >>>>>> out. It
looks like slapd is
>>>>> >>>>>> dying, I
thought it was
>>>>> still
>>>>> >>>>>> running
for some reason.
>>>>> >>>>>>
>>>>> >>>>>> slapd is
dying to segfault.
>>>>> >>>>>> strace
of it happening
>>>>> doesn't
>>>>> >>>>>> seem to
reveal much:
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> A stack
trace would very much
>>>>> help
>>>>> >
>>>>>
>>>> ...
>>
>> [Message clipped]
>
>
>
>
3:27 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
System time appears to be fine.
Here's the httpd access log (nothing in error log):
10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET /ipa/session/cookie
HTTP/1.1" 301 260 "-" "python-requests/2.18.4"
10.10.255.101 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "GET
/ipa/session/cookie HTTP/1.1" 200 -
10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
/ipa/session/login_password HTTP/1.1" 200 20
10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "POST
/ipa/session/json HTTP/1.1" 401 -
10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "GET
/ipa/session/login_kerberos?_=1526674892374 HTTP/1.1" 401 -
On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
Actually - may have spoke too soon. I thought I'd signed in to
the UI but
I was actually in another server's UI. When I try to log in it says that my
login has expired... so progress but something else is busted.
On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
> Looks like replica install finished just fine, and I can access the the
> web UI on it. No obvious problems.
>
> So it would appear that code change fixed 389DS for ARM.
>
> I'm not running the Dogtag CA on the Pi, just on the non-Pi servers,
> otherwise would need to probably adjust the startup timeout for it so that
> the install script didn't give up waiting on it. When I was first trying to
> play with this I tried installing the first replica/master on the Pi, and
> ran into the Dogtag timeout problem. Even installing Oracle Java instead of
> the OpenJDK version of Java didn't solve it, because somehow it always
> picks the OpenJDK java instead of oracle even if I change the alternatives
> setting to default to the Oracle one...
>
> But as far as I can tell otherwise, everything is working as expected.
>
> On Fri, May 18, 2018 at 2:10 PM, Mark Reynolds <mreynolds(a)redhat.com>
> wrote:
>
>>
>>
>> On 05/18/2018 03:04 PM, Jonathan Vaughn wrote:
>>
>> Build completed, I tested it, got some error about libs... you had a
>> rogue lib64 in the build command given :)
>>
>> Ah, yeah sorry that configure command was not fully tested I guess (I'll
>> update my notes)...
>>
>> Rebuilt with that fixed to be regular lib (since this is a 32-bit OS)
>> and it appears to be running fine.
>>
>> Well, it hasn't segfaulted. It complains about failing to get principles
>> from keytab and stuff... but that is probably because I could never
>> complete ipa-replica-install. It's still running. I'm going to uninstall
>> and reinstall replica and see what happens now.
>>
>> Keep us posted
>>
>> Thanks,
>> Mark
>>
>>
>> On Thu, May 17, 2018 at 9:09 PM, Mark Reynolds <mreynolds(a)redhat.com>
>> wrote:
>>
>>>
>>>
>>> On 05/17/2018 08:06 PM, Jonathan Vaughn wrote:
>>>
>>> Sorry, missed your build instructions, only saw them just now. I've
>>> just kicked off that build and I'll check it tomorrow once I'm back
in.
>>>
>>> One note, it appears it's ./autogen.sh not ./autogen.pl ...
>>>
>>> Whoops! Good catch, and good luck...
>>>
>>>
>>> On Thu, May 17, 2018 at 6:16 PM, Jonathan Vaughn <
>>> jonathan(a)creatuity.com> wrote:
>>>
>>>> Welp, I'm still getting the -fPIC error ... so I may need to figure
>>>> out how to do it with COPR or something.
>>>>
>>>> On Thu, May 17, 2018 at 4:16 PM, Jonathan Vaughn <
>>>> jonathan(a)creatuity.com> wrote:
>>>>
>>>>> I've never used COPR. I've dabbled with RPMs in the past but
that
>>>>> was... CentOS 6 I think, and I wasn't making source code changes
so much as
>>>>> just copying and pasting SRPMs from another RPM platform to build
for
>>>>> CentOS, using the regular rpmbuild stuff.
>>>>>
>>>>> I did actually try just copying the configure line (manually
>>>>> expanding the various vars) from the SRPM and just doing an old
fashioned
>>>>> make, but the first try bailed shy of an hour at linking with some
error
>>>>> about not having -fPIC specified, so I restarted it with -fPIC to see
if it
>>>>> would at least build. So if I should still try the code changes, if
that
>>>>> build finishes I can try it (I was planning to just copy the
respective
>>>>> binaries on top of the installed ones), but if it turned out the
suggested
>>>>> code changes were pointless then there's no point waiting for it
to finish
>>>>> building.
>>>>>
>>>>> On Thu, May 17, 2018 at 4:11 PM, Rob Crittenden
<rcritten(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Jonathan Vaughn via FreeIPA-users wrote:
>>>>>> > Oops, hit reply instead of reply-all
>>>>>> >
>>>>>> > NSPR RPMs
>>>>>> >
>>>>>> > # yum list installed nspr*
>>>>>> > Installed Packages
>>>>>> > nspr.armv7hl
>>>>>>
>>>>>> > 4.19.0-1.fc27
>>>>>>
>>>>>> > @updates
>>>>>> > nspr-debuginfo.armv7hl
>>>>>>
>>>>>> > 4.19.0-1.fc27
>>>>>>
>>>>>> > @updates-debuginfo
>>>>>> > nspr-debugsource.armv7hl
>>>>>>
>>>>>> > 4.19.0-1.fc27
>>>>>>
>>>>>> > @updates-debuginfo
>>>>>> > nspr-devel.armv7hl
>>>>>>
>>>>>> > 4.19.0-1.fc27
>>>>>>
>>>>>> > @updates
>>>>>> >
>>>>>> > As for building it with the changes you proposed previously
-
>>>>>> should I
>>>>>> > not bother? I realized last night that the last and only
time I
>>>>>> built
>>>>>> > anything *on* a Pi was a Pi Zero and that's why it was
so terribly
>>>>>> slow.
>>>>>> > This Pi 3 ain't fast but it's fast enough to build
stuff in far
>>>>>> less
>>>>>> > than half a day (unlike the Zero)...
>>>>>>
>>>>>> If you are handy with rpms you could build it in copr.
>>>>>>
>>>>>> rob
>>>>>>
>>>>>> >
>>>>>> >
>>>>>> > On Thu, May 17, 2018 at 5:07 AM, thierry bordaz <
>>>>>> tbordaz(a)redhat.com
>>>>>> > <mailto:tbordaz@redhat.com>> wrote:
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
>>>>>> >> I've been just using the packages from Fedora. I
can build it
>>>>>> >> potentially but I don't have a cross build
environment set up
>>>>>> at
>>>>>> >> the moment. From experience I'd want to do that
first because
>>>>>> >> building anything on the Pi usually takes ages.
>>>>>> >>
>>>>>> >> I'd been "redacting" the hostnames but
I'll stop bothering
>>>>>> since
>>>>>> >> it looks like we're getting far enough into the
weeds now
>>>>>> that the
>>>>>> >> difference in string lengths after
"redacting" might actually
>>>>>> be a
>>>>>> >> red herring.
>>>>>> >>
>>>>>> >> (gdb) p *agmt
>>>>>> >> $1 = {hostname = 0x1ef9be0
"ipa-12.creatuity.internal", port =
>>>>>> >> 389, transport_flags = 0, binddn = 0x1e8f650
"", creds =
>>>>>> >> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
>>>>>> >> frac_attrs = 0x1ef99c0, frac_attrs_total =
0x1ef9a40,
>>>>>> >> frac_attr_total_defined = 1, schedule = 0x1a7f0c0,
>>>>>> auto_initialize
>>>>>> >> = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
>>>>>> >> long_name = 0x1a7f100
"agmt=\"cn=meToipa-12.creatuit
>>>>>> y.internal\"
>>>>>> >> (ipa-12:5)", protocol = 0x19c2930,
changecounters = 0x186d180,
>>>>>> >> num_changecounters = 0,
>>>>>> >> max_changecounters = 256, last_update_start_time
=
>>>>>> 1526500697,
>>>>>> >> last_update_end_time = 1526500697,
>>>>>> >> last_update_status = "Error (0) Replica
acquired
>>>>>> successfully:
>>>>>> >> Incremental update succeeded", '\000'
<repeats 954 times>,
>>>>>> >> update_in_progress = 0, is_enabled = 1,
>>>>>> >> last_init_start_time = 0, last_init_end_time = 0,
>>>>>> >> last_init_status = '\000' <repeats 1023
times>, lock =
>>>>>> 0x1ee3740,
>>>>>> >> consumerRUV = 0x1f14e50,
>>>>>> >> consumerSchemaCSN = 0x317c520, consumerRID = 4,
>>>>>> tmpConsumerRID =
>>>>>> >> 0, timeout = 120, stop_in_progress = 0, busywaittime
= 0,
>>>>>> >> pausetime = 0, priv = 0x0,
>>>>>> >> attrs_to_strip = 0x1ef9ba0, agreement_type = 0,
>>>>>> protocol_timeout
>>>>>> >> = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow =
1000,
>>>>>> >> flowControlPause = 2000, ignoreMissingChange = 0,
>>>>>> >> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
>>>>>> >> (gdb) p *agmt->replarea
>>>>>> >> $2 = {flag = 15 '\017', udn = 0x1efce80
>>>>>> >> "dc=creatuity,dc=internal", dn =
0x1ef9460
>>>>>> >> "dc=creatuity,dc=internal", ndn =
0x1ef8ec0
>>>>>> >> "dc=creatuity,dc=internal", ndn_len = 24}
>>>>>> >> (gdb) p *agmt->rdn
>>>>>> >> $3 = {flag = 0 '\000', rdn = 0x19c2840
>>>>>> >> "cn=meToipa-12.creatuity.internal", rdns =
0x0,
>>>>>> butcheredupto =
>>>>>> >> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
>>>>>> >>
>>>>>> >> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
>>>>>> >> /usr/include/inttypes.h:# define PRId64
>>>>>> __PRI64_PREFIX "d"
>>>>>> >> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
>>>>>> >> /usr/include/inttypes.h:# define PRIu16
"u"
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
>>>>>> >> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>> wrote:
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>>>>>> >>> The installed version of 389* is
1.3.7.10-1.fc27
>>>>>> for armv7hl,
>>>>>> >>> which appears to be the latest available
version.
>>>>>> >>
>>>>>> >> Perhaps something is off with the inttypes on
Raspberry.
>>>>>> Are
>>>>>> >> you building this yourself on Raspberry? Can we
make code
>>>>>> >> changes and compile/install them?
>>>>>> >>
>>>>>> >> Before we do that though, in gdb can you run
these
>>>>>> commands in
>>>>>> >> the same gdb frame:
>>>>>> >>
>>>>>> >> (gdb) p *agmt->replarea
>>>>>> >> (gdb) p *agmt->rdn
>>>>>> >>
>>>>>> >>
>>>>>> >> Then do:
>>>>>> >>
>>>>>> >> # grep -r PRId64 /usr/include/*
>>>>>> >> # grep -r PRIu16 /usr/include/*
>>>>>> >>
>>>>>> >>
>>>>>> >> So if you can compile the source, then change
this line in
>>>>>> >>
ldap/servers/plugins/replication/repl5_agmt.c:3036, but
>>>>>> don't
>>>>>> >> do this yet until you get me the info I just
requested.
>>>>>> >>
>>>>>> >> From:
>>>>>> >>
>>>>>> >> agmt->maxcsn =
>>>>>> slapi_ch_smprintf("%s;%s;%s;%"
>>>>>> >> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replare
>>>>>> a),
>>>>>> >>
>>>>>> >>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>>>>>> >> agmt->hostname,
>>>>>> >>
>>>>>> agmt->port,
>>>>>> >> agmt->consumerRID, maxcsn);
>>>>>> >>
>>>>>> >> To:
>>>>>> >>
>>>>>> >> agmt->maxcsn =
>>>>>> >>
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>>>>> >> slapi_sdn_get_dn(agmt->replarea),
>>>>>> >>
>>>>>> >>
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
>>>>>> >> agmt->hostname,
>>>>>> >>
>>>>>> >> (long)agmt->port, (int)agmt->consumerRID,
maxcsn);
>>>>>> >>
>>>>>> >>
>>>>>> >> Thanks,
>>>>>> >> Mark
>>>>>> >>
>>>>>> >>>
>>>>>> >>> On Wed, May 16, 2018 at 2:38 PM, Jonathan
Vaughn
>>>>>> >>> <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>>
>>>>>> wrote:
>>>>>> >>
>>>>>> >
>>>>>> > The agreement structure looks valid to me. it should not
lead
>>>>>> to a
>>>>>> > crash.
>>>>>> >
>>>>>> > What looks weird to me is the order of the arguments of
cvt_s.
>>>>>> > It is called: rv = cvt_s(ss, u.s, width, prec, flags);
>>>>>> > But the crashing thread shows the opposite order:
flags,
>>>>>> prec,width,
>>>>>> > str, ss
>>>>>> >
>>>>>> > The others frame do not show this change of order.
>>>>>> > Also 'str=4' that would be a meaningful value
for 'prec=4'.
>>>>>> >
>>>>>> > From debug perspective I only imagine disassemble the
two last
>>>>>> > frames to confirm parameters.
>>>>>> > What are the nspr rpms ?
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >>> (gdb) up
>>>>>> >>> #1 0xb6926b40 in cvt_s (flags=0,
prec=<optimized
>>>>>> out>,
>>>>>> >>> width=0, str=0x4 <error: Cannot
access memory at
>>>>>> address
>>>>>> >>> 0x4>, ss=<optimized out>)
>>>>>> >>> at
../../.././nspr/pr/src/io/prprf.c:374
>>>>>> >>> 374 slen = strlen(str);
>>>>>> >>> (gdb) up
>>>>>> >>> #2 dosprintf (ss=ss@entry=0x9e06e4bc,
>>>>>> fmt=0xb34b0df2 "",
>>>>>> >>> fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002",
>>>>>> ap=...)
>>>>>> >>> at
../../.././nspr/pr/src/io/prprf.c:1018
>>>>>> >>> 1018 rv = cvt_s(ss, u.s,
width, prec,
>>>>>> flags);
>>>>>> >>> (gdb) up
>>>>>> >>> #3 0xb6926c8c in PR_vsmprintf
(fmt=fmt@entry
>>>>>> =0xb34da770
>>>>>> >>> <agmt_set>
"\360\317p\002", ap=..., ap@entry=...) at
>>>>>> >>> ../../.././nspr/pr/src/io/prprf.c:1184
>>>>>> >>> 1184 rv = dosprintf(&ss, fmt,
ap);
>>>>>> >>> (gdb) up
>>>>>> >>> #4 0xb6de9d18 in slapi_ch_smprintf
(fmt=0xb34b0de0
>>>>>> >>> "%s;%s;%s;%ld;%d;%s") at
>>>>>> ldap/servers/slapd/ch_malloc.c:331
>>>>>> >>> 331 p = PR_vsmprintf(fmt, ap);
>>>>>> >>> (gdb) up
>>>>>> >>> #5 0xb34625a4 in agmt_update_maxcsn
>>>>>> >>> (r=r@entry=0x2737810, sdn=0x38d9a40,
sdn@entry=0x10,
>>>>>> >>> op=<optimized out>, mods=0x10,
mods@entry=0x27d0100,
>>>>>> >>> csn=csn@entry=0x38de350)
>>>>>> >>> at ldap/servers/plugins/replicati
>>>>>> on/repl5_agmt.c:3036
>>>>>> >>> 3036 agmt->maxcsn
=
>>>>>> >>>
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>>>>>> >>> slapi_sdn_get_dn(agmt->replarea),
>>>>>> >>> (gdb) p *agmt
>>>>>> >>> $1 = {hostname = 0x2757be0
"ipa-12.company.internal",
>>>>>> >>> port = 389, transport_flags = 0, binddn
= 0x26ed650
>>>>>> "",
>>>>>> >>> creds = 0x26ed7a0, bindmethod = 3,
replarea =
>>>>>> 0x2757480,
>>>>>> >>> frac_attrs = 0x27579c0,
frac_attrs_total =
>>>>>> 0x2757a40,
>>>>>> >>> frac_attr_total_defined = 1, schedule =
0x22dd0c0,
>>>>>> >>> auto_initialize = 502, dn = 0x2756d00,
rdn =
>>>>>> 0x2756c20,
>>>>>> >>> long_name = 0x22dd100
>>>>>> >>>
"agmt=\"cn=meToipa-12.company.internal\"
>>>>>> (ipa-12:5)",
>>>>>> >>> protocol = 0x2220930, changecounters =
0x20cb180,
>>>>>> >>> num_changecounters = 0,
>>>>>> >>> max_changecounters = 256,
last_update_start_time =
>>>>>> >>> 1526499214, last_update_end_time =
1526499214,
>>>>>> >>> last_update_status = "Error (0)
Replica acquired
>>>>>> >>> successfully: Incremental update
succeeded", '\000'
>>>>>> >>> <repeats 954 times>,
update_in_progress = 0,
>>>>>> is_enabled = 1,
>>>>>> >>> last_init_start_time = 0,
last_init_end_time = 0,
>>>>>> >>> last_init_status = '\000'
<repeats 1023 times>, lock
>>>>>> =
>>>>>> >>> 0x2741740, consumerRUV = 0x2772e50,
>>>>>> >>> consumerSchemaCSN = 0x39da520,
consumerRID = 4,
>>>>>> >>> tmpConsumerRID = 0, timeout = 120,
stop_in_progress
>>>>>> = 0,
>>>>>> >>> busywaittime = 0, pausetime = 0, priv =
0x0,
>>>>>> >>> attrs_to_strip = 0x2757ba0,
agreement_type = 0,
>>>>>> >>> protocol_timeout = 0x26ed5f0, maxcsn =
0x0,
>>>>>> >>> flowControlWindow = 1000,
flowControlPause = 2000,
>>>>>> >>> ignoreMissingChange = 0,
>>>>>> >>> attr_lock = 0x2757c20,
WaitForAsyncResults = 100}
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Tue, May 15, 2018 at 5:36 PM, Mark
Reynolds
>>>>>> >>> <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> This looks really familiar and I
thought it was
>>>>>> >>> fixed. It should have been fixed in
1.3.7.10-1
>>>>>> >>>
(https://pagure.io/389-ds-base/issue/49618
>>>>>> >>>
<https://pagure.io/389-ds-base/issue/49618>).
>>>>>> In
>>>>>> >>> your debug session go "up"
into
>>>>>> agmt_maxcsn_update()
>>>>>> >>> and do:
>>>>>> >>>
>>>>>> >>> (gdb) p *agmt
>>>>>> >>>
>>>>>> >>> Then send us that output please.
>>>>>> >>>
>>>>>> >>> Thanks,
>>>>>> >>> Mark
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On 05/15/2018 05:29 PM, Jonathan
Vaughn via
>>>>>> >>> FreeIPA-users wrote:
>>>>>> >>>> Here is a backtrace from live
gdb after the
>>>>>> >>>> segfault. Looks like things went
wrong somewhere
>>>>>> >>>> during in the replication code
?
>>>>>> >>>>
>>>>>> >>>> Thread 36 "ns-slapd"
received signal SIGSEGV,
>>>>>> >>>> Segmentation fault.
>>>>>> >>>> [Switching to Thread 0x9e0bc280
(LWP 4662)]
>>>>>> >>>> strlen () at
../sysdeps/arm/armv6t2/strlen.
>>>>>> S:142
>>>>>> >>>> 142 ldrd data1a,
data1b, [src]
>>>>>> >>>> (gdb) bt
>>>>>> >>>> #0 0xb6728f2e in strlen () at
>>>>>> >>>>
../sysdeps/arm/armv6t2/strlen.S:142
>>>>>> >>>> #1 0xb6973b40 in cvt_s
(flags=0,
>>>>>> prec=<optimized
>>>>>> >>>> out>, width=0, str=0x4
<error: Cannot access
>>>>>> memory
>>>>>> >>>> at address 0x4>,
ss=<optimized out>)
>>>>>> >>>> at
../../.././nspr/pr/src/io/prprf.c:374
>>>>>> >>>> #2 0xb6973b40 in dosprintf
(ss=ss@entry
>>>>>> =0x9e0bb4bc,
>>>>>> >>>> fmt=0xb34fddf2 "",
fmt@entry=0xb3527770
>>>>>> <agmt_set>
>>>>>> >>>> "\360\357\305\002",
ap=...)
>>>>>> >>>> at
../../.././nspr/pr/src/io/prprf.c:1018
>>>>>> >>>> #3 0xb6973c8c in PR_vsmprintf
>>>>>> >>>> (fmt=fmt@entry=0xb3527770
<agmt_set>
>>>>>> >>>> "\360\357\305\002",
ap=..., ap@entry=...) at
>>>>>> >>>>
../../.././nspr/pr/src/io/prprf.c:1184
>>>>>> >>>> #4 0xb6e36d18 in
slapi_ch_smprintf
>>>>>> (fmt=0xb34fdde0
>>>>>> >>>> "%s;%s;%s;%ld;%d;%s")
at
>>>>>> >>>>
ldap/servers/slapd/ch_malloc.c:331
>>>>>> >>>> #5 0xb34af5a4 in
agmt_update_maxcsn
>>>>>> >>>> (r=r@entry=0x2c89810,
sdn=0x3e2bac0, sdn@entry
>>>>>> =0x10,
>>>>>> >>>> op=<optimized out>,
mods=0x10, mods@entry
>>>>>> =0x3eec100,
>>>>>> >>>> csn=csn@entry=0x3e30220)
>>>>>> >>>> at
>>>>>> >>>> ldap/servers/plugins/replicat
>>>>>> ion/repl5_agmt.c:3036
>>>>>> >>>> #6 0xb34bd424 in
write_changelog_and_ruv
>>>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>>>> >>>> ldap/servers/plugins/replicat
>>>>>> ion/repl5_plugins.c:1124
>>>>>> >>>> #7 0xb34be7ec in
multimaster_be_betxnpostop_add
>>>>>> >>>> (pb=0x2cb54a0) at
>>>>>> >>>> ldap/servers/plugins/replicat
>>>>>> ion/repl5_plugins.c:855
>>>>>> >>>> #8 0xb34be880 in
multimaster_mmr_postop
>>>>>> >>>> (pb=<optimized out>,
flags=560) at
>>>>>> >>>> ldap/servers/plugins/replicat
>>>>>> ion/repl5_plugins.c:616
>>>>>> >>>> #9 0xb6e8fcf0 in
plugin_call_mmr_plugin_postop
>>>>>> >>>> (pb=pb@entry=0x2cb54a0,
e=e@entry=0x0,
>>>>>> >>>> flags=flags@entry=560) at
>>>>>> >>>>
ldap/servers/slapd/plugin_mmr.c:65
>>>>>> >>>> #10 0xb35ba870 in ldbm_back_add
(pb=0x2cb54a0)
>>>>>> at
>>>>>> >>>>
ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>>>>>> >>>> #11 0xb6e2ce4c in op_shared_add
>>>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>>>> ldap/servers/slapd/add.c:679
>>>>>> >>>> #12 0xb6e2d35c in
add_internal_pb
>>>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>>>> ldap/servers/slapd/add.c:407
>>>>>> >>>> #13 0xb6e2e0f8 in
slapi_add_internal_pb
>>>>>> >>>> (pb=pb@entry=0x2cb54a0) at
>>>>>> ldap/servers/slapd/add.c:332
>>>>>> >>>> #14 0xb368db50 in
ipa_topo_util_segment_write
>>>>>> >>>> (tconf=tconf@entry=0x2cb5860,
>>>>>> >>>> tsegm=tsegm@entry=0x3f2d9a0) at
>>>>>> topology_util.c:1251
>>>>>> >>>> #15 0xb368e01c in
ipa_topo_util_update_agmt_list
>>>>>> >>>> (conf=0x2cb5860,
repl_segments=<optimized out>)
>>>>>> at
>>>>>> >>>> topology_util.c:696
>>>>>> >>>> #16 0xb368891c in
ipa_topo_apply_shared_config
>>>>>> () at
>>>>>> >>>> topology_init.c:165
>>>>>> >>>> #17 0xb6e4e65c in eq_call_all ()
at
>>>>>> >>>> ldap/servers/slapd/eventq.c:278
>>>>>> >>>> #18 0xb6e4e65c in eq_loop
(arg=<optimized out>)
>>>>>> at
>>>>>> >>>> ldap/servers/slapd/eventq.c:323
>>>>>> >>>> #19 0xb6982d68 in _pt_root
(arg=0x2c8da40) at
>>>>>> >>>>
../../.././nspr/pr/src/pthreads/ptthread.c:201
>>>>>> >>>> #20 0xb6906ee8 in start_thread
(arg=0x9e0bc280)
>>>>>> at
>>>>>> >>>> pthread_create.c:465
>>>>>> >>>> #21 0xb6785da8 in None () at
>>>>>> >>>>
../sysdeps/unix/sysv/linux/arm/clone.S:73
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> On Tue, May 15, 2018 at 3:01 AM,
thierry bordaz
>>>>>> >>>> <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> Hi Jonathan,
>>>>>> >>>>
>>>>>> >>>> This problem looks new to me
and has
>>>>>> something
>>>>>> >>>> specific to your
environment.
>>>>>> >>>> I think the best approach is
to continue to
>>>>>> >>>> debug on your system if you
have the
>>>>>> possibility
>>>>>> >>>> to do so.
>>>>>> >>>>
>>>>>> >>>> From strace we can see that
DS started
>>>>>> smoothly
>>>>>> >>>> (created its pid file then
notified systemd
>>>>>> it
>>>>>> >>>> was running fine). According
to the pstack
>>>>>> >>>> nunc-stans was running and
was able to
>>>>>> accept
>>>>>> >>>> network events even if it
appears it
>>>>>> detected no
>>>>>> >>>> incoming connection.
>>>>>> >>>> So the server should be
ready to serve for
>>>>>> some
>>>>>> >>>> seconds (more than a
minute), then it
>>>>>> crashed
>>>>>> >>>> with one thread
dereferencing likely a wrong
>>>>>> >>>> pointer.
>>>>>> >>>>
>>>>>> >>>> Could you attach a debugger
when the server
>>>>>> is
>>>>>> >>>> started and wait for the
sigsegv to occur.
>>>>>> Then
>>>>>> >>>> confirm the crashing thread
backstack.
>>>>>> >>>> If it is confirmed, I am
afraid this is a
>>>>>> stack
>>>>>> >>>> corruption and valgrind
could help
>>>>>> >>>>
(http://www.port389.org/docs/
>>>>>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-wit
>>>>>> h-valgrind
>>>>>> >>>>
<http://www.port389.org/docs/
>>>>>> 389ds/FAQ/faq.html#debugging-memory-growthinvalid-access-wit
>>>>>> h-valgrind>).
>>>>>> >>>>
>>>>>> >>>> best regards
>>>>>> >>>> thierry
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> On 05/14/2018 10:20 PM,
Jonathan Vaughn
>>>>>> wrote:
>>>>>> >>>>> Here's a strace from
before it dies. Most
>>>>>> of
>>>>>> >>>>> the elapsed time is it
waiting on some
>>>>>> futex
>>>>>> >>>>> call it looks like near
the end, when it
>>>>>> >>>>> finally
"returns" (from lack of strace
>>>>>> output
>>>>>> >>>>> for duration of call I
assume it didn't
>>>>>> >>>>> actually return but
SIGSEGV in that call)
>>>>>> and
>>>>>> >>>>> strace prints ' =
?' on the futex it then
>>>>>> >>>>> immediately reports
SIGSEGV after. So
>>>>>> maybe the
>>>>>> >>>>> problem is that futex
call, which may mean
>>>>>> the
>>>>>> >>>>> problem is not directly
in 389DS / FreeIPA
>>>>>> itself?
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> 15:13:31.626587 (+
0.000630) listen(8,
>>>>>> 128)
>>>>>> >>>>> = 0 <0.000068>
>>>>>> >>>>> 15:13:31.626857 (+
0.000235) listen(9,
>>>>>> 128)
>>>>>> >>>>> = 0 <0.000048>
>>>>>> >>>>> 15:13:31.627111 (+
0.000251)
>>>>>> >>>>>
clock_gettime(CLOCK_MONOTONIC,
>>>>>> {tv_sec=1464932,
>>>>>> >>>>> tv_nsec=41120614}) = 0
<0.000085>
>>>>>> >>>>> 15:13:31.627457 (+
0.000356)
>>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=627560772}) = 0
>>>>>> >>>>> <0.000043>
>>>>>> >>>>> 15:13:31.631233 (+
0.003839)
>>>>>> >>>>>
clock_gettime(CLOCK_MONOTONIC,
>>>>>> {tv_sec=1464932,
>>>>>> >>>>> tv_nsec=45286796}) = 0
<0.000077>
>>>>>> >>>>> 15:13:31.631720 (+
0.000427)
>>>>>> >>>>>
clock_gettime(CLOCK_MONOTONIC,
>>>>>> {tv_sec=1464932,
>>>>>> >>>>> tv_nsec=45661430}) = 0
<0.000042>
>>>>>> >>>>> 15:13:31.631955 (+
0.000220)
>>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=632049036}) = 0
>>>>>> >>>>> <0.000047>
>>>>>> >>>>> 15:13:31.635669 (+
0.003785)
>>>>>> >>>>>
clock_gettime(CLOCK_MONOTONIC,
>>>>>> {tv_sec=1464932,
>>>>>> >>>>> tv_nsec=49725840}) = 0
<0.000146>
>>>>>> >>>>> 15:13:31.636484 (+
0.000784) write(16,
>>>>>> "a",
>>>>>> >>>>> 1) = 1 <0.000118>
>>>>>> >>>>> 15:13:31.636855 (+
0.000341)
>>>>>> sched_yield()
>>>>>> >>>>> = 0 <0.000252>
>>>>>> >>>>> 15:13:31.637322 (+
0.000470)
>>>>>> >>>>> futex(0x1cb57a0,
FUTEX_WAKE_PRIVATE, 1) = 1
>>>>>> >>>>> <0.000088>
>>>>>> >>>>> 15:13:31.637897 (+
0.000610) write(16,
>>>>>> "a",
>>>>>> >>>>> 1) = 1 <0.000221>
>>>>>> >>>>> 15:13:31.638394 (+
0.000467)
>>>>>> sched_yield()
>>>>>> >>>>> = 0 <0.000047>
>>>>>> >>>>> 15:13:31.638619 (+
0.000202)
>>>>>> >>>>> futex(0x1cb5710,
FUTEX_WAKE_PRIVATE, 1) = 1
>>>>>> >>>>> <0.000065>
>>>>>> >>>>> 15:13:31.638908 (+
0.000298)
>>>>>> >>>>> openat(AT_FDCWD,
>>>>>> >>>>>
"/var/run/dirsrv/slapd-COMPAN
>>>>>> Y-INTERNAL.pid",
>>>>>> >>>>>
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
>>>>>> <0.000831>
>>>>>> >>>>> 15:13:31.640260 (+
0.001387) getpid() =
>>>>>> >>>>> 32353 <0.000077>
>>>>>> >>>>> 15:13:31.640558 (+
0.000256)
>>>>>> fstat64(33,
>>>>>> >>>>> {st_mode=S_IFREG|0644,
st_size=0, ...}) = 0
>>>>>> >>>>> <0.000119>
>>>>>> >>>>> 15:13:31.641106 (+
0.000556) write(33,
>>>>>> >>>>> "32353\n", 6)
= 6 <0.000127>
>>>>>> >>>>> 15:13:31.641472 (+
0.000362) close(33)
>>>>>> = 0
>>>>>> >>>>> <0.000519>
>>>>>> >>>>> 15:13:31.642216 (+
0.000758)
>>>>>> >>>>>
chmod("/var/run/dirsrv/slapd-
>>>>>> COMPANY-INTERNAL.pid",
>>>>>> >>>>> 0644) = 0
<0.000152>
>>>>>> >>>>> 15:13:31.642900 (+
0.000679)
>>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=643020294}) = 0
>>>>>> >>>>> <0.000056>
>>>>>> >>>>> 15:13:31.643495 (+
0.000590) write(2,
>>>>>> >>>>>
"[14/May/2018:15:13:31.643020294 "...,
>>>>>> 134) =
>>>>>> >>>>> 134 <0.002697>
>>>>>> >>>>> 15:13:31.646515 (+
0.003052)
>>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=646694394}) = 0
>>>>>> >>>>> <0.000075>
>>>>>> >>>>> 15:13:31.646892 (+
0.000337) write(4,
>>>>>> >>>>>
"[14/May/2018:15:13:31.646694394 "...,
>>>>>> 134) =
>>>>>> >>>>> 134 <0.000522>
>>>>>> >>>>> 15:13:31.647841 (+
0.000973) fsync(4)
>>>>>> = 0
>>>>>> >>>>> <0.005967>
>>>>>> >>>>> 15:13:31.654425 (+
0.006617)
>>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=654598946}) = 0
>>>>>> >>>>> <0.000253>
>>>>>> >>>>> 15:13:31.655137 (+
0.000717) write(2,
>>>>>> >>>>>
"[14/May/2018:15:13:31.654598946 "...,
>>>>>> 136) =
>>>>>> >>>>> 136 <0.002427>
>>>>>> >>>>> 15:13:31.658312 (+
0.003165)
>>>>>> >>>>>
clock_gettime(CLOCK_REALTIME,
>>>>>> >>>>> {tv_sec=1526328811,
tv_nsec=658486117}) = 0
>>>>>> >>>>> <0.000251>
>>>>>> >>>>> 15:13:31.659032 (+
0.000682) write(4,
>>>>>> >>>>>
"[14/May/2018:15:13:31.658486117 "...,
>>>>>> 136) =
>>>>>> >>>>> 136 <0.000346>
>>>>>> >>>>> 15:13:31.659623 (+
0.000595) fsync(4)
>>>>>> = 0
>>>>>> >>>>> <0.003311>
>>>>>> >>>>> 15:13:31.663230 (+
0.003642) getpid() =
>>>>>> >>>>> 32353 <0.000045>
>>>>>> >>>>> 15:13:31.663732 (+
0.000454)
>>>>>> >>>>> socket(AF_UNIX,
SOCK_DGRAM|SOCK_CLOEXEC,
>>>>>> 0) =
>>>>>> >>>>> 33 <0.000296>
>>>>>> >>>>> 15:13:31.664760 (+
0.001048)
>>>>>> getsockopt(33,
>>>>>> >>>>> SOL_SOCKET, SO_SNDBUF,
[163840], [4]) = 0
>>>>>> >>>>> <0.000108>
>>>>>> >>>>> 15:13:31.665141 (+
0.000386)
>>>>>> setsockopt(33,
>>>>>> >>>>> SOL_SOCKET,
SO_SNDBUFFORCE, [8388608], 4)
>>>>>> = -1
>>>>>> >>>>> EPERM (Operation not
permitted) <0.000051>
>>>>>> >>>>> 15:13:31.665500 (+
0.000334)
>>>>>> setsockopt(33,
>>>>>> >>>>> SOL_SOCKET, SO_SNDBUF,
[8388608], 4) = 0
>>>>>> <0.000229>
>>>>>> >>>>> 15:13:31.665973 (+
0.000468)
>>>>>> sendmsg(33,
>>>>>> >>>>>
{msg_name={sa_family=AF_UNIX,
>>>>>> >>>>>
sun_path="/run/systemd/notify"},
>>>>>> >>>>> msg_namelen=21,
>>>>>> >>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>>>> >>>>> started: Re"...,
iov_len=69}],
>>>>>> msg_iovlen=1,
>>>>>> >>>>> msg_controllen=0,
msg_flags=0},
>>>>>> MSG_NOSIGNAL) =
>>>>>> >>>>> 69 <0.000600>
>>>>>> >>>>> 15:13:31.667270 (+
0.001334) close(33)
>>>>>> = 0
>>>>>> >>>>> <0.003140>
>>>>>> >>>>> 15:13:31.677320 (+
0.010146)
>>>>>> >>>>> futex(0xb31122e8,
FUTEX_WAIT, 32357, NULL)
>>>>>> = ?
>>>>>> >>>>> 15:14:27.661809 (+
55.984448) +++
>>>>>> killed by
>>>>>> >>>>> SIGSEGV (core dumped)
+++
>>>>>> >>>>>
>>>>>> >>>>> On Mon, May 14, 2018 at
10:27 AM, thierry
>>>>>> >>>>> bordaz
<tbordaz(a)redhat.com
>>>>>> >>>>>
<mailto:tbordaz@redhat.com>> wrote:
>>>>>> >>>>>
>>>>>> >>>>> Hi Jonathan,
>>>>>> >>>>>
>>>>>> >>>>> This is weird as the
crashing thread
>>>>>> stack
>>>>>> >>>>> looks truncated (did
you copy/paste
>>>>>> all of
>>>>>> >>>>> it ?)
>>>>>> >>>>>
>>>>>> >>>>> Thread 1 (Thread
0x9e13c280 (LWP
>>>>>> 17245)):
>>>>>> >>>>> #0 0xb67bbf2e in
strlen () at
>>>>>> /lib/libc.so.6
>>>>>> >>>>> #1 0xb6a06b40 in
dosprintf () at
>>>>>> >>>>> /lib/libnspr4.so
>>>>>> >>>>> #2 0x00000000 in
None ()
>>>>>> >>>>>
>>>>>> >>>>> Did you install
389-ds-base-debuginfo ?
>>>>>> >>>>> How did you get that
backtrace ? from a
>>>>>> >>>>> core dumped, pstack
? Can you attach a
>>>>>> >>>>> debugger before the
crash occurs ?
>>>>>> >>>>>
>>>>>> >>>>> It looks it crashed
soon at startup,
>>>>>> could
>>>>>> >>>>> it be related to a
broken dse.ldif. It
>>>>>> >>>>> should exists a
dse.ldif.OK, is it
>>>>>> possibly
>>>>>> >>>>> to try to start with
it ?
>>>>>> >>>>>
>>>>>> >>>>> best regards
>>>>>> >>>>> thierry
>>>>>> >>>>>
>>>>>> >>>>> On 05/12/2018 01:22
AM, Jonathan
>>>>>> Vaughn via
>>>>>> >>>>> FreeIPA-users
wrote:
>>>>>> >>>>>> Not sure if it
makes a difference...
>>>>>> I was
>>>>>> >>>>>> looking into
this again and realized
>>>>>> I had
>>>>>> >>>>>> a bunch of
messages from gdb telling
>>>>>> me to
>>>>>> >>>>>> install more
debuginfo. I've done that
>>>>>> >>>>>> now, here it is
again freshly run
>>>>>> through
>>>>>> >>>>>> gdb
>>>>>> >>>>>>
>>>>>> >>>>>> GNU gdb (GDB)
Fedora 8.0.1-36.fc27
>>>>>> >>>>>> Copyright (C)
2017 Free Software
>>>>>> >>>>>> Foundation,
Inc.
>>>>>> >>>>>> License GPLv3+:
GNU GPL version 3 or
>>>>>> later
>>>>>> >>>>>>
<http://gnu.org/licenses/gpl.html
>>>>>> >>>>>>
<http://gnu.org/licenses/gpl.html>>
>>>>>> >>>>>> This is free
software: you are free to
>>>>>> >>>>>> change and
redistribute it.
>>>>>> >>>>>> There is NO
WARRANTY, to the extent
>>>>>> >>>>>> permitted by
law. Type "show copying"
>>>>>> >>>>>> and "show
warranty" for details.
>>>>>> >>>>>> This GDB was
configured as
>>>>>> >>>>>>
"armv7hl-redhat-linux-gnueabi".
>>>>>> >>>>>> Type "show
configuration" for
>>>>>> >>>>>> configuration
details.
>>>>>> >>>>>> For bug
reporting instructions,
>>>>>> please see:
>>>>>> >>>>>>
<http://www.gnu.org/software/
>>>>>> gdb/bugs/
>>>>>> >>>>>>
<http://www.gnu.org/software/
>>>>>> gdb/bugs/>>.
>>>>>> >>>>>> Find the GDB
manual and other
>>>>>> >>>>>> documentation
resources online at:
>>>>>> >>>>>>
<http://www.gnu.org/software/
>>>>>> gdb/documentation/
>>>>>> >>>>>>
<http://www.gnu.org/software/
>>>>>> gdb/documentation/>>.
>>>>>> >>>>>> For help, type
"help".
>>>>>> >>>>>> Type
"apropos word" to search for
>>>>>> commands
>>>>>> >>>>>> related to
"word"...
>>>>>> >>>>>> Reading symbols
from
>>>>>> >>>>>>
/usr/sbin/ns-slapd...Reading symbols
>>>>>> from
>>>>>> >>>>>>
/usr/lib/debug/usr/sbin/ns-sl
>>>>>> apd-1.3.7.10-1.fc27.arm.debug...done.
>>>>>> >>>>>> done.
>>>>>> >>>>>> ...
>>>>>> >>>>>>
>>>>>> >>>>>> Thread 1 (Thread
0x9e13c280 (LWP
>>>>>> 17245)):
>>>>>> >>>>>> #0 0xb67bbf2e
in strlen () at
>>>>>> /lib/libc.so.6
>>>>>> >>>>>> #1 0xb6a06b40
in dosprintf () at
>>>>>> >>>>>>
/lib/libnspr4.so
>>>>>> >>>>>> #2 0x00000000
in None ()
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> On Tue, May
8, 2018 at 7:52 AM,
>>>>>> Rob
>>>>>> >>>>>> Crittenden
<rcritten(a)redhat.com
>>>>>> >>>>>>
<mailto:rcritten@redhat.com>>
>>>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> Jonathan
Vaughn via
>>>>>> FreeIPA-users
>>>>>> >>>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>>
Still trying to figure
>>>>>> this
>>>>>> >>>>>> out.
It looks like slapd
>>>>>> is
>>>>>> >>>>>>
dying, I thought it was
>>>>>> still
>>>>>> >>>>>>
running for some reason.
>>>>>> >>>>>>
>>>>>> >>>>>>
slapd is dying to
>>>>>> segfault.
>>>>>> >>>>>>
strace of it happening
>>>>>> doesn't
>>>>>> >>>>>> seem
to reveal much:
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> A stack
trace would very much
>>>>>> help
>>>>>> >
>>>>>>
>>>>> ...
>>>
>>> [Message clipped]
>>
>>
>>
>>
>
3:53 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Jonathan Vaughn wrote:
System time appears to be fine.
Here's the httpd access log (nothing in error log):
10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET /ipa/session/cookie
HTTP/1.1" 301 260 "-" "python-requests/2.18.4"
10.10.255.101 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
"GET /ipa/session/cookie HTTP/1.1" 200 -
10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
/ipa/session/login_password HTTP/1.1" 200 20
10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "POST
/ipa/session/json HTTP/1.1" 401 -
10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "GET
/ipa/session/login_kerberos?_=1526674892374 HTTP/1.1" 401 -
You'll want to look at the error log for more details. For even more
output create /etc/ipa/server.conf with the contents:
[global]
debug = True
and restart httpd
rob
>
> On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn <jonathan(a)creatuity.com
> <mailto:jonathan@creatuity.com>> wrote:
>
> Actually - may have spoke too soon. I thought I'd signed in to the
> UI but I was actually in another server's UI. When I try to log in
> it says that my login has expired... so progress but something else
> is busted.
>
> On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn
> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
>
> Looks like replica install finished just fine, and I can access
> the the web UI on it. No obvious problems.
>
> So it would appear that code change fixed 389DS for ARM.
>
> I'm not running the Dogtag CA on the Pi, just on the non-Pi
> servers, otherwise would need to probably adjust the startup
> timeout for it so that the install script didn't give up waiting
> on it. When I was first trying to play with this I tried
> installing the first replica/master on the Pi, and ran into the
> Dogtag timeout problem. Even installing Oracle Java instead of
> the OpenJDK version of Java didn't solve it, because somehow it
> always picks the OpenJDK java instead of oracle even if I change
> the alternatives setting to default to the Oracle one...
>
> But as far as I can tell otherwise, everything is working as
> expected.
4:01 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
httpd error_log
[Fri May 18 15:58:29.546255 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Fri May 18 15:58:29.547217 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI login_password.__call__:
[Fri May 18 15:58:29.549799 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Obtaining armor in ccache
/var/run/ipa/ccaches/armor_14004
[Fri May 18 15:58:29.550812 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Initializing anonymous ccache
[Fri May 18 15:58:29.552268 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
[Fri May 18 15:58:29.553377 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit -n -c
/var/run/ipa/ccaches/armor_14004 -X
X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
[Fri May 18 15:58:30.486537 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
[Fri May 18 15:58:30.488137 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stdout=
[Fri May 18 15:58:30.489128 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.490955 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Initializing principal admin using
password
[Fri May 18 15:58:30.492060 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Using armor ccache
/var/run/ipa/ccaches/armor_14004 for FAST webauth
[Fri May 18 15:58:30.493262 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Using enterprise principal
[Fri May 18 15:58:30.494728 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
[Fri May 18 15:58:30.495799 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit admin -c
/var/run/ipa/ccaches/kinit_14004 -T /var/run/ipa/ccaches/armor_14004 -E
[Fri May 18 15:58:30.721637 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
[Fri May 18 15:58:30.723230 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stdout=Password for
admin(a)CREATUITY.INTERNAL:
[Fri May 18 15:58:30.723432 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284]
[Fri May 18 15:58:30.724345 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.725716 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Cleanup the armor ccache
[Fri May 18 15:58:30.727071 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
[Fri May 18 15:58:30.728152 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kdestroy -A -c
/var/run/ipa/ccaches/armor_14004
[Fri May 18 15:58:30.798642 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
[Fri May 18 15:58:30.800249 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stdout=
[Fri May 18 15:58:30.801224 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.857737 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTP connection (1):
ipa-11.creatuity.internal
[Fri May 18 15:58:30.867978 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: http://ipa-11.creatuity.internal:80
"GET /ipa/session/cookie HTTP/1.1" 301 260
[Fri May 18 15:58:30.889415 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTPS connection (1):
ipa-11.creatuity.internal
[Fri May 18 15:58:31.105092 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: https://ipa-11.creatuity.internal:443
"GET /ipa/session/cookie HTTP/1.1" 200 0
[Fri May 18 15:58:31.157295 2018] [wsgi:error] [pid 14003:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Fri May 18 15:58:31.158347 2018] [wsgi:error] [pid 14003:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI jsonserver_session.__call__:
[Fri May 18 15:58:31.159289 2018] [wsgi:error] [pid 14003:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
[Fri May 18 15:58:31.160349 2018] [wsgi:error] [pid 14003:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need login
[Fri May 18 15:58:31.175232 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Fri May 18 15:58:31.176208 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI KerberosLogin.__call__:
[Fri May 18 15:58:31.177060 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
[Fri May 18 15:58:31.178047 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need login
On Fri, May 18, 2018 at 3:53 PM, Rob Crittenden <rcritten(a)redhat.com> wrote:
Jonathan Vaughn wrote:
> System time appears to be fine.
>
> Here's the httpd access log (nothing in error log):
> 10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET /ipa/session/cookie
> HTTP/1.1" 301 260 "-" "python-requests/2.18.4"
> 10.10.255.101 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
> "GET /ipa/session/cookie HTTP/1.1" 200 -
> 10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
> /ipa/session/login_password HTTP/1.1" 200 20
> 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "POST
> /ipa/session/json HTTP/1.1" 401 -
> 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "GET
> /ipa/session/login_kerberos?_=1526674892374 HTTP/1.1" 401 -
>
You'll want to look at the error log for more details. For even more
output create /etc/ipa/server.conf with the contents:
[global]
debug = True
and restart httpd
rob
>
> On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn <jonathan(a)creatuity.com
> <mailto:jonathan@creatuity.com>> wrote:
>
> Actually - may have spoke too soon. I thought I'd signed in to the
> UI but I was actually in another server's UI. When I try to log in
> it says that my login has expired... so progress but something else
> is busted.
>
> On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn
> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
>
> Looks like replica install finished just fine, and I can access
> the the web UI on it. No obvious problems.
>
> So it would appear that code change fixed 389DS for ARM.
>
> I'm not running the Dogtag CA on the Pi, just on the non-Pi
> servers, otherwise would need to probably adjust the startup
> timeout for it so that the install script didn't give up waiting
> on it. When I was first trying to play with this I tried
> installing the first replica/master on the Pi, and ran into the
> Dogtag timeout problem. Even installing Oracle Java instead of
> the OpenJDK version of Java didn't solve it, because somehow it
> always picks the OpenJDK java instead of oracle even if I change
> the alternatives setting to default to the Oracle one...
>
> But as far as I can tell otherwise, everything is working as
> expected.
4:04 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Anyone have any further suggestions for troubleshooting this issue with the
web UI?
On Fri, May 18, 2018 at 4:01 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
httpd error_log
[Fri May 18 15:58:29.546255 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Fri May 18 15:58:29.547217 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI login_password.__call__:
[Fri May 18 15:58:29.549799 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Obtaining armor in ccache
/var/run/ipa/ccaches/armor_14004
[Fri May 18 15:58:29.550812 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Initializing anonymous ccache
[Fri May 18 15:58:29.552268 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
[Fri May 18 15:58:29.553377 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit -n -c
/var/run/ipa/ccaches/armor_14004 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt
-X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
[Fri May 18 15:58:30.486537 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
[Fri May 18 15:58:30.488137 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stdout=
[Fri May 18 15:58:30.489128 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.490955 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Initializing principal admin using
password
[Fri May 18 15:58:30.492060 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Using armor ccache
/var/run/ipa/ccaches/armor_14004 for FAST webauth
[Fri May 18 15:58:30.493262 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Using enterprise principal
[Fri May 18 15:58:30.494728 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
[Fri May 18 15:58:30.495799 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit admin -c
/var/run/ipa/ccaches/kinit_14004 -T /var/run/ipa/ccaches/armor_14004 -E
[Fri May 18 15:58:30.721637 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
[Fri May 18 15:58:30.723230 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stdout=Password for
admin(a)CREATUITY.INTERNAL:
[Fri May 18 15:58:30.723432 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284]
[Fri May 18 15:58:30.724345 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.725716 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Cleanup the armor ccache
[Fri May 18 15:58:30.727071 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
[Fri May 18 15:58:30.728152 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kdestroy -A -c
/var/run/ipa/ccaches/armor_14004
[Fri May 18 15:58:30.798642 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
[Fri May 18 15:58:30.800249 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stdout=
[Fri May 18 15:58:30.801224 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.857737 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTP connection (1):
ipa-11.creatuity.internal
[Fri May 18 15:58:30.867978 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: http://ipa-11.creatuity.internal:80
"GET /ipa/session/cookie HTTP/1.1" 301 260
[Fri May 18 15:58:30.889415 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTPS connection (1):
ipa-11.creatuity.internal
[Fri May 18 15:58:31.105092 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: https://ipa-11.creatuity.
internal:443 "GET /ipa/session/cookie HTTP/1.1" 200 0
[Fri May 18 15:58:31.157295 2018] [wsgi:error] [pid 14003:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Fri May 18 15:58:31.158347 2018] [wsgi:error] [pid 14003:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI jsonserver_session.__call__:
[Fri May 18 15:58:31.159289 2018] [wsgi:error] [pid 14003:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
[Fri May 18 15:58:31.160349 2018] [wsgi:error] [pid 14003:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need login
[Fri May 18 15:58:31.175232 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Fri May 18 15:58:31.176208 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: WSGI KerberosLogin.__call__:
[Fri May 18 15:58:31.177060 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
[Fri May 18 15:58:31.178047 2018] [wsgi:error] [pid 14004:tid 2903716672]
[remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need login
On Fri, May 18, 2018 at 3:53 PM, Rob Crittenden <rcritten(a)redhat.com>
wrote:
> Jonathan Vaughn wrote:
> > System time appears to be fine.
> >
> > Here's the httpd access log (nothing in error log):
> > 10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET /ipa/session/cookie
> > HTTP/1.1" 301 260 "-" "python-requests/2.18.4"
> > 10.10.255.101 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
> > "GET /ipa/session/cookie HTTP/1.1" 200 -
> > 10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
> > /ipa/session/login_password HTTP/1.1" 200 20
> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
> "POST
> > /ipa/session/json HTTP/1.1" 401 -
> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "GET
> > /ipa/session/login_kerberos?_=1526674892374 HTTP/1.1" 401 -
> >
>
> You'll want to look at the error log for more details. For even more
> output create /etc/ipa/server.conf with the contents:
>
> [global]
> debug = True
>
> and restart httpd
>
> rob
>
> >
> > On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn <
> jonathan(a)creatuity.com
> > <mailto:jonathan@creatuity.com>> wrote:
> >
> > Actually - may have spoke too soon. I thought I'd signed in to the
> > UI but I was actually in another server's UI. When I try to log in
> > it says that my login has expired... so progress but something else
> > is busted.
> >
> > On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn
> > <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
> >
> > Looks like replica install finished just fine, and I can access
> > the the web UI on it. No obvious problems.
> >
> > So it would appear that code change fixed 389DS for ARM.
> >
> > I'm not running the Dogtag CA on the Pi, just on the non-Pi
> > servers, otherwise would need to probably adjust the startup
> > timeout for it so that the install script didn't give up waiting
> > on it. When I was first trying to play with this I tried
> > installing the first replica/master on the Pi, and ran into the
> > Dogtag timeout problem. Even installing Oracle Java instead of
> > the OpenJDK version of Java didn't solve it, because somehow it
> > always picks the OpenJDK java instead of oracle even if I change
> > the alternatives setting to default to the Oracle one...
> >
> > But as far as I can tell otherwise, everything is working as
> > expected.
>
3:32 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Alright, I think I've got everything* working. (* Not running the CA server
on the Arm device, not tested, but from what I've read before I would need
to adjust the startup timeout since OpenJDK is so slow).
1) I removed the Arm replica from the cluster and reimaged the device
entirely and started over.
2) Rebuilt 389-ds-base using the patch by Mark Reynolds, and replica
install succeeded (just as it did with the patch before). I did use the
next version (1.3.8) of 389-ds-base because the package installed version
bumped up to 1.3.8.1 - patch worked same as on 1.3.7
3) I also had to apply the fix to /etc/httpd/conf.d/ipa.conf from
https://pagure.io/freeipa/issue/7337 so that Web UI login could work (I did
this previously on the Arm device but still couldn't login - likely just
too many things were tried and something else was busted elsewhere).
As far as I can tell everything seems to be working now.
I'm not sure if the patch provided would work as-is or if it would break on
non-Arm architectures.
The patch :
diff --git a/ldap/servers/plugins/replication/repl5_agmt.c
b/ldap/servers/plugins/replication/repl5_agmt.c
index d71d3f38b..e0f1f41bd 100644
--- a/ldap/servers/plugins/replication/repl5_agmt.c
+++ b/ldap/servers/plugins/replication/repl5_agmt.c
@@ -3035,7 +3035,7 @@ agmt_update_maxcsn(Replica *r, Slapi_DN *sdn, int op,
LDAPMod **mods, CSN *csn)
slapi_ch_free_string(&agmt->maxcsn);
agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
slapi_sdn_get_dn(agmt->replarea),
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
- agmt->port,
agmt->consumerRID, maxcsn);
+ (long)agmt->port,
(int)agmt->consumerRID, maxcsn);
}
PR_Unlock(agmt->lock);
}
To get this fixed properly (to avoid building from source), should I open
an issue in Pagure.io ? Or will someone else handle that?
On Wed, May 23, 2018 at 4:04 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
Anyone have any further suggestions for troubleshooting this issue
with
the web UI?
On Fri, May 18, 2018 at 4:01 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
> httpd error_log
>
> [Fri May 18 15:58:29.546255 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
> [Fri May 18 15:58:29.547217 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: WSGI login_password.__call__:
> [Fri May 18 15:58:29.549799 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Obtaining armor in ccache
> /var/run/ipa/ccaches/armor_14004
> [Fri May 18 15:58:29.550812 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Initializing anonymous ccache
> [Fri May 18 15:58:29.552268 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
> [Fri May 18 15:58:29.553377 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit -n -c
> /var/run/ipa/ccaches/armor_14004 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt
> -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
> [Fri May 18 15:58:30.486537 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
> [Fri May 18 15:58:30.488137 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: stdout=
> [Fri May 18 15:58:30.489128 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
> [Fri May 18 15:58:30.490955 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Initializing principal admin using
> password
> [Fri May 18 15:58:30.492060 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Using armor ccache
> /var/run/ipa/ccaches/armor_14004 for FAST webauth
> [Fri May 18 15:58:30.493262 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Using enterprise principal
> [Fri May 18 15:58:30.494728 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
> [Fri May 18 15:58:30.495799 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit admin -c
> /var/run/ipa/ccaches/kinit_14004 -T /var/run/ipa/ccaches/armor_14004 -E
> [Fri May 18 15:58:30.721637 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
> [Fri May 18 15:58:30.723230 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: stdout=Password for
> admin(a)CREATUITY.INTERNAL:
> [Fri May 18 15:58:30.723432 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284]
> [Fri May 18 15:58:30.724345 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
> [Fri May 18 15:58:30.725716 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Cleanup the armor ccache
> [Fri May 18 15:58:30.727071 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
> [Fri May 18 15:58:30.728152 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kdestroy -A -c
> /var/run/ipa/ccaches/armor_14004
> [Fri May 18 15:58:30.798642 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
> [Fri May 18 15:58:30.800249 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: stdout=
> [Fri May 18 15:58:30.801224 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
> [Fri May 18 15:58:30.857737 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTP connection (1):
> ipa-11.creatuity.internal
> [Fri May 18 15:58:30.867978 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: http://ipa-11.creatuity.internal:80
> "GET /ipa/session/cookie HTTP/1.1" 301 260
> [Fri May 18 15:58:30.889415 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTPS connection (1):
> ipa-11.creatuity.internal
> [Fri May 18 15:58:31.105092 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: https://ipa-11.creatuity.inter
> nal:443 "GET /ipa/session/cookie HTTP/1.1" 200 0
> [Fri May 18 15:58:31.157295 2018] [wsgi:error] [pid 14003:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
> [Fri May 18 15:58:31.158347 2018] [wsgi:error] [pid 14003:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: WSGI jsonserver_session.__call__:
> [Fri May 18 15:58:31.159289 2018] [wsgi:error] [pid 14003:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
> [Fri May 18 15:58:31.160349 2018] [wsgi:error] [pid 14003:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need login
> [Fri May 18 15:58:31.175232 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
> [Fri May 18 15:58:31.176208 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: WSGI KerberosLogin.__call__:
> [Fri May 18 15:58:31.177060 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
> [Fri May 18 15:58:31.178047 2018] [wsgi:error] [pid 14004:tid 2903716672]
> [remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need login
>
>
> On Fri, May 18, 2018 at 3:53 PM, Rob Crittenden <rcritten(a)redhat.com>
> wrote:
>
>> Jonathan Vaughn wrote:
>> > System time appears to be fine.
>> >
>> > Here's the httpd access log (nothing in error log):
>> > 10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET
/ipa/session/cookie
>> > HTTP/1.1" 301 260 "-" "python-requests/2.18.4"
>> > 10.10.255.101 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
>> > "GET /ipa/session/cookie HTTP/1.1" 200 -
>> > 10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
>> > /ipa/session/login_password HTTP/1.1" 200 20
>> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
>> "POST
>> > /ipa/session/json HTTP/1.1" 401 -
>> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
>> "GET
>> > /ipa/session/login_kerberos?_=1526674892374 HTTP/1.1" 401 -
>> >
>>
>> You'll want to look at the error log for more details. For even more
>> output create /etc/ipa/server.conf with the contents:
>>
>> [global]
>> debug = True
>>
>> and restart httpd
>>
>> rob
>>
>> >
>> > On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn <
>> jonathan(a)creatuity.com
>> > <mailto:jonathan@creatuity.com>> wrote:
>> >
>> > Actually - may have spoke too soon. I thought I'd signed in to the
>> > UI but I was actually in another server's UI. When I try to log in
>> > it says that my login has expired... so progress but something else
>> > is busted.
>> >
>> > On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn
>> > <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>>
wrote:
>> >
>> > Looks like replica install finished just fine, and I can access
>> > the the web UI on it. No obvious problems.
>> >
>> > So it would appear that code change fixed 389DS for ARM.
>> >
>> > I'm not running the Dogtag CA on the Pi, just on the non-Pi
>> > servers, otherwise would need to probably adjust the startup
>> > timeout for it so that the install script didn't give up
>> waiting
>> > on it. When I was first trying to play with this I tried
>> > installing the first replica/master on the Pi, and ran into the
>> > Dogtag timeout problem. Even installing Oracle Java instead of
>> > the OpenJDK version of Java didn't solve it, because somehow it
>> > always picks the OpenJDK java instead of oracle even if I
>> change
>> > the alternatives setting to default to the Oracle one...
>> >
>> > But as far as I can tell otherwise, everything is working as
>> > expected.
>>
>
>
3:54 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
On 06/01/2018 04:32 PM, Jonathan Vaughn wrote:
Alright, I think I've got everything* working. (* Not running the
CA
server on the Arm device, not tested, but from what I've read before I
would need to adjust the startup timeout since OpenJDK is so slow).
1) I removed the Arm replica from the cluster and reimaged the device
entirely and started over.
2) Rebuilt 389-ds-base using the patch by Mark Reynolds, and replica
install succeeded (just as it did with the patch before). I did use
the next version (1.3.8) of 389-ds-base because the package installed
version bumped up to 1.3.8.1 - patch worked same as on 1.3.7
3) I also had to apply the fix to /etc/httpd/conf.d/ipa.conf from
https://pagure.io/freeipa/issue/7337 so that Web UI login could work
(I did this previously on the Arm device but still couldn't login -
likely just too many things were tried and something else was busted
elsewhere).
As far as I can tell everything seems to be working now.
I'm not sure if the patch provided would work as-is or if it would
break on non-Arm architectures.
The patch :
diff --git a/ldap/servers/plugins/replication/repl5_agmt.c
b/ldap/servers/plugins/replication/repl5_agmt.c
index d71d3f38b..e0f1f41bd 100644
--- a/ldap/servers/plugins/replication/repl5_agmt.c
+++ b/ldap/servers/plugins/replication/repl5_agmt.c
@@ -3035,7 +3035,7 @@ agmt_update_maxcsn(Replica *r, Slapi_DN *sdn,
int op, LDAPMod **mods, CSN *csn)
slapi_ch_free_string(&agmt->maxcsn);
agmt->maxcsn =
slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s", slapi_sdn_get_dn(agmt->replarea),
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
- agmt->port, agmt->consumerRID, maxcsn);
+ (long)agmt->port, (int)agmt->consumerRID, maxcsn);
}
PR_Unlock(agmt->lock);
}
To get this fixed properly (to avoid building from source), should I
open an issue in Pagure.io ? Or will someone else handle that?
Please file ticket:
https://pagure.io/389-ds-base/new_issue
Can you also do me a favor please? Please build the server again, but
without my patch, and send the build output to me. I want to see if
there are any compiler errors/warnings. The issue is that I fixed all
the issues on ARM, including this exact issue you ran into. Its working
for others without issue. I was wondering if it was perhaps Raspberry's
version of ARM that was causing issues. Anyway the build output
(without my fix) would be very interesting to look at.
Thanks,
Mark
On Wed, May 23, 2018 at 4:04 PM, Jonathan Vaughn
<jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
Anyone have any further suggestions for troubleshooting this issue
with the web UI?
On Fri, May 18, 2018 at 4:01 PM, Jonathan Vaughn
<jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
httpd error_log
[Fri May 18 15:58:29.546255 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: WSGI
wsgi_dispatch.__call__:
[Fri May 18 15:58:29.547217 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: WSGI
login_password.__call__:
[Fri May 18 15:58:29.549799 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Obtaining armor in
ccache /var/run/ipa/ccaches/armor_14004
[Fri May 18 15:58:29.550812 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Initializing anonymous
ccache
[Fri May 18 15:58:29.552268 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Starting external process
[Fri May 18 15:58:29.553377 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: args=/usr/bin/kinit -n
-c /var/run/ipa/ccaches/armor_14004 -X
X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
[Fri May 18 15:58:30.486537 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Process finished,
return code=0
[Fri May 18 15:58:30.488137 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: stdout=
[Fri May 18 15:58:30.489128 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.490955 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Initializing principal
admin using password
[Fri May 18 15:58:30.492060 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Using armor ccache
/var/run/ipa/ccaches/armor_14004 for FAST webauth
[Fri May 18 15:58:30.493262 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Using enterprise principal
[Fri May 18 15:58:30.494728 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Starting external process
[Fri May 18 15:58:30.495799 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: args=/usr/bin/kinit
admin -c /var/run/ipa/ccaches/kinit_14004 -T
/var/run/ipa/ccaches/armor_14004 -E
[Fri May 18 15:58:30.721637 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Process finished,
return code=0
[Fri May 18 15:58:30.723230 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: stdout=Password for
admin(a)CREATUITY.INTERNAL:
[Fri May 18 15:58:30.723432 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284 <http://10.10.11.3:59284>]
[Fri May 18 15:58:30.724345 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.725716 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Cleanup the armor ccache
[Fri May 18 15:58:30.727071 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Starting external process
[Fri May 18 15:58:30.728152 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: args=/usr/bin/kdestroy
-A -c /var/run/ipa/ccaches/armor_14004
[Fri May 18 15:58:30.798642 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Process finished,
return code=0
[Fri May 18 15:58:30.800249 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: stdout=
[Fri May 18 15:58:30.801224 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.857737 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Starting new HTTP
connection (1): ipa-11.creatuity.internal
[Fri May 18 15:58:30.867978 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG:
http://ipa-11.creatuity.internal:80
<http://ipa-11.creatuity.internal:80> "GET /ipa/session/cookie
HTTP/1.1" 301 260
[Fri May 18 15:58:30.889415 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: Starting new HTTPS
connection (1): ipa-11.creatuity.internal
[Fri May 18 15:58:31.105092 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG:
https://ipa-11.creatuity.internal:443
<https://ipa-11.creatuity.internal:443> "GET
/ipa/session/cookie HTTP/1.1" 200 0
[Fri May 18 15:58:31.157295 2018] [wsgi:error] [pid 14003:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: WSGI
wsgi_dispatch.__call__:
[Fri May 18 15:58:31.158347 2018] [wsgi:error] [pid 14003:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: WSGI
jsonserver_session.__call__:
[Fri May 18 15:58:31.159289 2018] [wsgi:error] [pid 14003:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: no ccache, need login
[Fri May 18 15:58:31.160349 2018] [wsgi:error] [pid 14003:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: 401 Unauthorized need login
[Fri May 18 15:58:31.175232 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: WSGI
wsgi_dispatch.__call__:
[Fri May 18 15:58:31.176208 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: WSGI
KerberosLogin.__call__:
[Fri May 18 15:58:31.177060 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: no ccache, need login
[Fri May 18 15:58:31.178047 2018] [wsgi:error] [pid 14004:tid
2903716672] [remote 10.10.11.3:59284
<http://10.10.11.3:59284>] ipa: DEBUG: 401 Unauthorized need login
On Fri, May 18, 2018 at 3:53 PM, Rob Crittenden
<rcritten(a)redhat.com <mailto:rcritten@redhat.com>> wrote:
Jonathan Vaughn wrote:
> System time appears to be fine.
>
> Here's the httpd access log (nothing in error log):
> 10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET
/ipa/session/cookie
> HTTP/1.1" 301 260 "-" "python-requests/2.18.4"
> 10.10.255.101 - admin(a)CREATUITY.INTERNAL
[18/May/2018:15:26:20 -0500]
> "GET /ipa/session/cookie HTTP/1.1" 200 -
> 10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
> /ipa/session/login_password HTTP/1.1" 200 20
> 10.10.11.3 - admin(a)CREATUITY.INTERNAL
[18/May/2018:15:26:20 -0500] "POST
> /ipa/session/json HTTP/1.1" 401 -
> 10.10.11.3 - admin(a)CREATUITY.INTERNAL
[18/May/2018:15:26:20 -0500] "GET
> /ipa/session/login_kerberos?_=1526674892374 HTTP/1.1" 401 -
>
You'll want to look at the error log for more details. For
even more
output create /etc/ipa/server.conf with the contents:
[global]
debug = True
and restart httpd
rob
>
> On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn
<jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>
> <mailto:jonathan@creatuity.com
<mailto:jonathan@creatuity.com>>>
wrote:
>
> Actually - may have spoke too soon. I thought I'd
signed in to the
> UI but I was actually in another server's UI. When I
try to log in
> it says that my login has expired... so progress but
something else
> is busted.
>
> On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn
> <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>
<mailto:jonathan@creatuity.com
<mailto:jonathan@creatuity.com>>> wrote:
>
> Looks like replica install finished just fine,
and I can access
> the the web UI on it. No obvious problems.
>
> So it would appear that code change fixed 389DS
for ARM.
>
> I'm not running the Dogtag CA on the Pi, just on
the non-Pi
> servers, otherwise would need to probably adjust
the startup
> timeout for it so that the install script didn't
give up waiting
> on it. When I was first trying to play with this
I tried
> installing the first replica/master on the Pi,
and ran into the
> Dogtag timeout problem. Even installing Oracle
Java instead of
> the OpenJDK version of Java didn't solve it,
because somehow it
> always picks the OpenJDK java instead of oracle
even if I change
> the alternatives setting to default to the
Oracle one...
>
> But as far as I can tell otherwise, everything
is working as
> expected.
4:38 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Created https://pagure.io/389-ds-base/issue/49746
I'll get you the build log once it's done building without the patch.
On Fri, Jun 1, 2018 at 3:54 PM, Mark Reynolds <mreynolds(a)redhat.com> wrote:
On 06/01/2018 04:32 PM, Jonathan Vaughn wrote:
Alright, I think I've got everything* working. (* Not running the CA
server on the Arm device, not tested, but from what I've read before I
would need to adjust the startup timeout since OpenJDK is so slow).
1) I removed the Arm replica from the cluster and reimaged the device
entirely and started over.
2) Rebuilt 389-ds-base using the patch by Mark Reynolds, and replica
install succeeded (just as it did with the patch before). I did use the
next version (1.3.8) of 389-ds-base because the package installed version
bumped up to 1.3.8.1 - patch worked same as on 1.3.7
3) I also had to apply the fix to /etc/httpd/conf.d/ipa.conf from
https://pagure.io/freeipa/issue/7337 so that Web UI login could work (I
did this previously on the Arm device but still couldn't login - likely
just too many things were tried and something else was busted elsewhere).
As far as I can tell everything seems to be working now.
I'm not sure if the patch provided would work as-is or if it would break
on non-Arm architectures.
The patch :
diff --git a/ldap/servers/plugins/replication/repl5_agmt.c
b/ldap/servers/plugins/replication/repl5_agmt.c
index d71d3f38b..e0f1f41bd 100644
--- a/ldap/servers/plugins/replication/repl5_agmt.c
+++ b/ldap/servers/plugins/replication/repl5_agmt.c
@@ -3035,7 +3035,7 @@ agmt_update_maxcsn(Replica *r, Slapi_DN *sdn, int
op, LDAPMod **mods, CSN *csn)
slapi_ch_free_string(&agmt->maxcsn);
agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
slapi_sdn_get_dn(agmt->replarea),
slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
- agmt->port,
agmt->consumerRID, maxcsn);
+ (long)agmt->port,
(int)agmt->consumerRID, maxcsn);
}
PR_Unlock(agmt->lock);
}
To get this fixed properly (to avoid building from source), should I open
an issue in Pagure.io ? Or will someone else handle that?
Please file ticket:
https://pagure.io/389-ds-base/new_issue
Can you also do me a favor please? Please build the server again, but
without my patch, and send the build output to me. I want to see if there
are any compiler errors/warnings. The issue is that I fixed all the issues
on ARM, including this exact issue you ran into. Its working for others
without issue. I was wondering if it was perhaps Raspberry's version of
ARM that was causing issues. Anyway the build output (without my fix)
would be very interesting to look at.
Thanks,
Mark
On Wed, May 23, 2018 at 4:04 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
wrote:
> Anyone have any further suggestions for troubleshooting this issue with
> the web UI?
>
> On Fri, May 18, 2018 at 4:01 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
> wrote:
>
>> httpd error_log
>>
>> [Fri May 18 15:58:29.546255 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>> wsgi_dispatch.__call__:
>> [Fri May 18 15:58:29.547217 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>> login_password.__call__:
>> [Fri May 18 15:58:29.549799 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Obtaining armor in
>> ccache /var/run/ipa/ccaches/armor_14004
>> [Fri May 18 15:58:29.550812 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Initializing
>> anonymous ccache
>> [Fri May 18 15:58:29.552268 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting external
>> process
>> [Fri May 18 15:58:29.553377 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit
>> -n -c /var/run/ipa/ccaches/armor_14004 -X
X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt
>> -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
>> [Fri May 18 15:58:30.486537 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Process finished,
>> return code=0
>> [Fri May 18 15:58:30.488137 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stdout=
>> [Fri May 18 15:58:30.489128 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
>> [Fri May 18 15:58:30.490955 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Initializing
>> principal admin using password
>> [Fri May 18 15:58:30.492060 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Using armor ccache
>> /var/run/ipa/ccaches/armor_14004 for FAST webauth
>> [Fri May 18 15:58:30.493262 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Using enterprise
>> principal
>> [Fri May 18 15:58:30.494728 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting external
>> process
>> [Fri May 18 15:58:30.495799 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit
>> admin -c /var/run/ipa/ccaches/kinit_14004 -T
>> /var/run/ipa/ccaches/armor_14004 -E
>> [Fri May 18 15:58:30.721637 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Process finished,
>> return code=0
>> [Fri May 18 15:58:30.723230 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stdout=Password for
>> admin(a)CREATUITY.INTERNAL:
>> [Fri May 18 15:58:30.723432 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284]
>> [Fri May 18 15:58:30.724345 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
>> [Fri May 18 15:58:30.725716 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Cleanup the armor
>> ccache
>> [Fri May 18 15:58:30.727071 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting external
>> process
>> [Fri May 18 15:58:30.728152 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG:
>> args=/usr/bin/kdestroy -A -c /var/run/ipa/ccaches/armor_14004
>> [Fri May 18 15:58:30.798642 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Process finished,
>> return code=0
>> [Fri May 18 15:58:30.800249 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stdout=
>> [Fri May 18 15:58:30.801224 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
>> [Fri May 18 15:58:30.857737 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTP
>> connection (1): ipa-11.creatuity.internal
>> [Fri May 18 15:58:30.867978 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG:
>> http://ipa-11.creatuity.internal:80 "GET /ipa/session/cookie HTTP/1.1"
>> 301 260
>> [Fri May 18 15:58:30.889415 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTPS
>> connection (1): ipa-11.creatuity.internal
>> [Fri May 18 15:58:31.105092 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG:
>> https://ipa-11.creatuity.internal:443 "GET /ipa/session/cookie
>> HTTP/1.1" 200 0
>> [Fri May 18 15:58:31.157295 2018] [wsgi:error] [pid 14003:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>> wsgi_dispatch.__call__:
>> [Fri May 18 15:58:31.158347 2018] [wsgi:error] [pid 14003:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>> jsonserver_session.__call__:
>> [Fri May 18 15:58:31.159289 2018] [wsgi:error] [pid 14003:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
>> [Fri May 18 15:58:31.160349 2018] [wsgi:error] [pid 14003:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need
>> login
>> [Fri May 18 15:58:31.175232 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>> wsgi_dispatch.__call__:
>> [Fri May 18 15:58:31.176208 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>> KerberosLogin.__call__:
>> [Fri May 18 15:58:31.177060 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
>> [Fri May 18 15:58:31.178047 2018] [wsgi:error] [pid 14004:tid
>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need
>> login
>>
>>
>> On Fri, May 18, 2018 at 3:53 PM, Rob Crittenden <rcritten(a)redhat.com>
>> wrote:
>>
>>> Jonathan Vaughn wrote:
>>> > System time appears to be fine.
>>> >
>>> > Here's the httpd access log (nothing in error log):
>>> > 10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET
>>> /ipa/session/cookie
>>> > HTTP/1.1" 301 260 "-" "python-requests/2.18.4"
>>> > 10.10.255.101 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
>>> > "GET /ipa/session/cookie HTTP/1.1" 200 -
>>> > 10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
>>> > /ipa/session/login_password HTTP/1.1" 200 20
>>> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
>>> "POST
>>> > /ipa/session/json HTTP/1.1" 401 -
>>> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
>>> "GET
>>> > /ipa/session/login_kerberos?_=1526674892374 HTTP/1.1" 401 -
>>> >
>>>
>>> You'll want to look at the error log for more details. For even more
>>> output create /etc/ipa/server.conf with the contents:
>>>
>>> [global]
>>> debug = True
>>>
>>> and restart httpd
>>>
>>> rob
>>>
>>> >
>>> > On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn <
>>> jonathan(a)creatuity.com
>>> > <mailto:jonathan@creatuity.com>> wrote:
>>> >
>>> > Actually - may have spoke too soon. I thought I'd signed in to
the
>>> > UI but I was actually in another server's UI. When I try to log
in
>>> > it says that my login has expired... so progress but something
>>> else
>>> > is busted.
>>> >
>>> > On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn
>>> > <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>>
wrote:
>>> >
>>> > Looks like replica install finished just fine, and I can
>>> access
>>> > the the web UI on it. No obvious problems.
>>> >
>>> > So it would appear that code change fixed 389DS for ARM.
>>> >
>>> > I'm not running the Dogtag CA on the Pi, just on the non-Pi
>>> > servers, otherwise would need to probably adjust the startup
>>> > timeout for it so that the install script didn't give up
>>> waiting
>>> > on it. When I was first trying to play with this I tried
>>> > installing the first replica/master on the Pi, and ran into
>>> the
>>> > Dogtag timeout problem. Even installing Oracle Java instead of
>>> > the OpenJDK version of Java didn't solve it, because
somehow
>>> it
>>> > always picks the OpenJDK java instead of oracle even if I
>>> change
>>> > the alternatives setting to default to the Oracle one...
>>> >
>>> > But as far as I can tell otherwise, everything is working as
>>> > expected.
>>>
>>
>>
>
8:07 a.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
I was right I did fix all the ARM issues, but not in 1.3.8, only in
1.4.0. It was a large change though that required a few patches. I'll
see what I can do about backporting the changes...
On 06/01/2018 05:38 PM, Jonathan Vaughn wrote:
Created https://pagure.io/389-ds-base/issue/49746
I'll get you the build log once it's done building without the patch.
On Fri, Jun 1, 2018 at 3:54 PM, Mark Reynolds <mreynolds(a)redhat.com
<mailto:mreynolds@redhat.com>> wrote:
On 06/01/2018 04:32 PM, Jonathan Vaughn wrote:
> Alright, I think I've got everything* working. (* Not running the
> CA server on the Arm device, not tested, but from what I've read
> before I would need to adjust the startup timeout since OpenJDK
> is so slow).
>
> 1) I removed the Arm replica from the cluster and reimaged the
> device entirely and started over.
>
> 2) Rebuilt 389-ds-base using the patch by Mark Reynolds, and
> replica install succeeded (just as it did with the patch before).
> I did use the next version (1.3.8) of 389-ds-base because the
> package installed version bumped up to 1.3.8.1 - patch worked
> same as on 1.3.7
>
> 3) I also had to apply the fix to /etc/httpd/conf.d/ipa.conf from
> https://pagure.io/freeipa/issue/7337
> <https://pagure.io/freeipa/issue/7337> so that Web UI login could
> work (I did this previously on the Arm device but still couldn't
> login - likely just too many things were tried and something else
> was busted elsewhere).
>
> As far as I can tell everything seems to be working now.
>
> I'm not sure if the patch provided would work as-is or if it
> would break on non-Arm architectures.
>
> The patch :
>
> diff --git a/ldap/servers/plugins/replication/repl5_agmt.c
> b/ldap/servers/plugins/replication/repl5_agmt.c
> index d71d3f38b..e0f1f41bd 100644
> --- a/ldap/servers/plugins/replication/repl5_agmt.c
> +++ b/ldap/servers/plugins/replication/repl5_agmt.c
> @@ -3035,7 +3035,7 @@ agmt_update_maxcsn(Replica *r, Slapi_DN
> *sdn, int op, LDAPMod **mods, CSN *csn)
> slapi_ch_free_string(&agmt->maxcsn);
> agmt->maxcsn =
> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> slapi_sdn_get_dn(agmt->replarea),
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
> agmt->hostname,
> - agmt->port, agmt->consumerRID, maxcsn);
> + (long)agmt->port, (int)agmt->consumerRID, maxcsn);
> }
> PR_Unlock(agmt->lock);
> }
>
> To get this fixed properly (to avoid building from source),
> should I open an issue in Pagure.io ? Or will someone else handle
> that?
Please file ticket:
https://pagure.io/389-ds-base/new_issue
<https://pagure.io/389-ds-base/new_issue>
Can you also do me a favor please? Please build the server again,
but without my patch, and send the build output to me. I want to
see if there are any compiler errors/warnings. The issue is that
I fixed all the issues on ARM, including this exact issue you ran
into. Its working for others without issue. I was wondering if
it was perhaps Raspberry's version of ARM that was causing
issues. Anyway the build output (without my fix) would be very
interesting to look at.
Thanks,
Mark
>
> On Wed, May 23, 2018 at 4:04 PM, Jonathan Vaughn
> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
>
> Anyone have any further suggestions for troubleshooting this
> issue with the web UI?
>
> On Fri, May 18, 2018 at 4:01 PM, Jonathan Vaughn
> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>> wrote:
>
> httpd error_log
>
> [Fri May 18 15:58:29.546255 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: WSGI
> wsgi_dispatch.__call__:
> [Fri May 18 15:58:29.547217 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: WSGI
> login_password.__call__:
> [Fri May 18 15:58:29.549799 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Obtaining armor in
> ccache /var/run/ipa/ccaches/armor_14004
> [Fri May 18 15:58:29.550812 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Initializing
> anonymous ccache
> [Fri May 18 15:58:29.552268 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Starting external
> process
> [Fri May 18 15:58:29.553377 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG:
> args=/usr/bin/kinit -n -c
> /var/run/ipa/ccaches/armor_14004 -X
> X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
> X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
> [Fri May 18 15:58:30.486537 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Process finished,
> return code=0
> [Fri May 18 15:58:30.488137 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: stdout=
> [Fri May 18 15:58:30.489128 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: stderr=
> [Fri May 18 15:58:30.490955 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Initializing
> principal admin using password
> [Fri May 18 15:58:30.492060 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Using armor ccache
> /var/run/ipa/ccaches/armor_14004 for FAST webauth
> [Fri May 18 15:58:30.493262 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Using enterprise
> principal
> [Fri May 18 15:58:30.494728 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Starting external
> process
> [Fri May 18 15:58:30.495799 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG:
> args=/usr/bin/kinit admin -c
> /var/run/ipa/ccaches/kinit_14004 -T
> /var/run/ipa/ccaches/armor_14004 -E
> [Fri May 18 15:58:30.721637 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Process finished,
> return code=0
> [Fri May 18 15:58:30.723230 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: stdout=Password
> for admin(a)CREATUITY.INTERNAL
> <mailto:admin@CREATUITY.INTERNAL>:
> [Fri May 18 15:58:30.723432 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>]
> [Fri May 18 15:58:30.724345 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: stderr=
> [Fri May 18 15:58:30.725716 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Cleanup the armor
> ccache
> [Fri May 18 15:58:30.727071 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Starting external
> process
> [Fri May 18 15:58:30.728152 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG:
> args=/usr/bin/kdestroy -A -c /var/run/ipa/ccaches/armor_14004
> [Fri May 18 15:58:30.798642 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Process finished,
> return code=0
> [Fri May 18 15:58:30.800249 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: stdout=
> [Fri May 18 15:58:30.801224 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: stderr=
> [Fri May 18 15:58:30.857737 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Starting new HTTP
> connection (1): ipa-11.creatuity.internal
> [Fri May 18 15:58:30.867978 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG:
> http://ipa-11.creatuity.internal:80
> <http://ipa-11.creatuity.internal:80> "GET
> /ipa/session/cookie HTTP/1.1" 301 260
> [Fri May 18 15:58:30.889415 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: Starting new HTTPS
> connection (1): ipa-11.creatuity.internal
> [Fri May 18 15:58:31.105092 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG:
> https://ipa-11.creatuity.internal:443
> <https://ipa-11.creatuity.internal:443> "GET
> /ipa/session/cookie HTTP/1.1" 200 0
> [Fri May 18 15:58:31.157295 2018] [wsgi:error] [pid
> 14003:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: WSGI
> wsgi_dispatch.__call__:
> [Fri May 18 15:58:31.158347 2018] [wsgi:error] [pid
> 14003:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: WSGI
> jsonserver_session.__call__:
> [Fri May 18 15:58:31.159289 2018] [wsgi:error] [pid
> 14003:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: no ccache, need login
> [Fri May 18 15:58:31.160349 2018] [wsgi:error] [pid
> 14003:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: 401 Unauthorized
> need login
> [Fri May 18 15:58:31.175232 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: WSGI
> wsgi_dispatch.__call__:
> [Fri May 18 15:58:31.176208 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: WSGI
> KerberosLogin.__call__:
> [Fri May 18 15:58:31.177060 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: no ccache, need login
> [Fri May 18 15:58:31.178047 2018] [wsgi:error] [pid
> 14004:tid 2903716672] [remote 10.10.11.3:59284
> <http://10.10.11.3:59284>] ipa: DEBUG: 401 Unauthorized
> need login
>
>
> On Fri, May 18, 2018 at 3:53 PM, Rob Crittenden
> <rcritten(a)redhat.com <mailto:rcritten@redhat.com>> wrote:
>
> Jonathan Vaughn wrote:
> > System time appears to be fine.
> >
> > Here's the httpd access log (nothing in error log):
> > 10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET
> /ipa/session/cookie
> > HTTP/1.1" 301 260 "-"
"python-requests/2.18.4"
> > 10.10.255.101 - admin(a)CREATUITY.INTERNAL
> <mailto:admin@CREATUITY.INTERNAL>
> [18/May/2018:15:26:20 -0500]
> > "GET /ipa/session/cookie HTTP/1.1" 200 -
> > 10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
> > /ipa/session/login_password HTTP/1.1" 200 20
> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL
> <mailto:admin@CREATUITY.INTERNAL>
> [18/May/2018:15:26:20 -0500] "POST
> > /ipa/session/json HTTP/1.1" 401 -
> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL
> <mailto:admin@CREATUITY.INTERNAL>
> [18/May/2018:15:26:20 -0500] "GET
> > /ipa/session/login_kerberos?_=1526674892374
> HTTP/1.1" 401 -
> >
>
> You'll want to look at the error log for more
> details. For even more
> output create /etc/ipa/server.conf with the contents:
>
> [global]
> debug = True
>
> and restart httpd
>
> rob
>
> >
> > On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn
> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>
> > <mailto:jonathan@creatuity.com
> <mailto:jonathan@creatuity.com>>> wrote:
> >
> > Actually - may have spoke too soon. I thought
> I'd signed in to the
> > UI but I was actually in another server's UI.
> When I try to log in
> > it says that my login has expired... so
> progress but something else
> > is busted.
> >
> > On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn
> > <jonathan(a)creatuity.com
> <mailto:jonathan@creatuity.com>
> <mailto:jonathan@creatuity.com
> <mailto:jonathan@creatuity.com>>> wrote:
> >
> > Looks like replica install finished just
> fine, and I can access
> > the the web UI on it. No obvious problems.
> >
> > So it would appear that code change fixed
> 389DS for ARM.
> >
> > I'm not running the Dogtag CA on the Pi,
> just on the non-Pi
> > servers, otherwise would need to probably
> adjust the startup
> > timeout for it so that the install script
> didn't give up waiting
> > on it. When I was first trying to play with
> this I tried
> > installing the first replica/master on the Pi, and
> ran into the
> > Dogtag timeout problem. Even installing
> Oracle Java instead of
> > the OpenJDK version of Java didn't solve
> it, because somehow it
> > always picks the OpenJDK java instead of
> oracle even if I change
> > the alternatives setting to default to the
> Oracle one...
> >
> > But as far as I can tell otherwise,
> everything is working as
> > expected.
>
>
>
>
4:37 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
Thanks! I'm looking forward to seeing these fixes hit the Fedora repo :)
On Sat, Jun 2, 2018 at 8:07 AM, Mark Reynolds <mreynolds(a)redhat.com> wrote:
I was right I did fix all the ARM issues, but not in 1.3.8, only in
1.4.0. It was a large change though that required a few patches. I'll see
what I can do about backporting the changes...
On 06/01/2018 05:38 PM, Jonathan Vaughn wrote:
Created https://pagure.io/389-ds-base/issue/49746
I'll get you the build log once it's done building without the patch.
On Fri, Jun 1, 2018 at 3:54 PM, Mark Reynolds <mreynolds(a)redhat.com>
wrote:
>
>
> On 06/01/2018 04:32 PM, Jonathan Vaughn wrote:
>
> Alright, I think I've got everything* working. (* Not running the CA
> server on the Arm device, not tested, but from what I've read before I
> would need to adjust the startup timeout since OpenJDK is so slow).
>
> 1) I removed the Arm replica from the cluster and reimaged the device
> entirely and started over.
>
> 2) Rebuilt 389-ds-base using the patch by Mark Reynolds, and replica
> install succeeded (just as it did with the patch before). I did use the
> next version (1.3.8) of 389-ds-base because the package installed version
> bumped up to 1.3.8.1 - patch worked same as on 1.3.7
>
> 3) I also had to apply the fix to /etc/httpd/conf.d/ipa.conf from
> https://pagure.io/freeipa/issue/7337 so that Web UI login could work (I
> did this previously on the Arm device but still couldn't login - likely
> just too many things were tried and something else was busted elsewhere).
>
> As far as I can tell everything seems to be working now.
>
> I'm not sure if the patch provided would work as-is or if it would break
> on non-Arm architectures.
>
> The patch :
>
> diff --git a/ldap/servers/plugins/replication/repl5_agmt.c
> b/ldap/servers/plugins/replication/repl5_agmt.c
> index d71d3f38b..e0f1f41bd 100644
> --- a/ldap/servers/plugins/replication/repl5_agmt.c
> +++ b/ldap/servers/plugins/replication/repl5_agmt.c
> @@ -3035,7 +3035,7 @@ agmt_update_maxcsn(Replica *r, Slapi_DN *sdn, int
> op, LDAPMod **mods, CSN *csn)
> slapi_ch_free_string(&agmt->maxcsn);
> agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> slapi_sdn_get_dn(agmt->replarea),
>
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
> - agmt->port,
> agmt->consumerRID, maxcsn);
> + (long)agmt->port,
> (int)agmt->consumerRID, maxcsn);
> }
> PR_Unlock(agmt->lock);
> }
>
> To get this fixed properly (to avoid building from source), should I open
> an issue in Pagure.io ? Or will someone else handle that?
>
>
> Please file ticket:
>
> https://pagure.io/389-ds-base/new_issue
>
> Can you also do me a favor please? Please build the server again, but
> without my patch, and send the build output to me. I want to see if there
> are any compiler errors/warnings. The issue is that I fixed all the issues
> on ARM, including this exact issue you ran into. Its working for others
> without issue. I was wondering if it was perhaps Raspberry's version of
> ARM that was causing issues. Anyway the build output (without my fix)
> would be very interesting to look at.
>
> Thanks,
> Mark
>
>
> On Wed, May 23, 2018 at 4:04 PM, Jonathan Vaughn <jonathan(a)creatuity.com>
> wrote:
>
>> Anyone have any further suggestions for troubleshooting this issue with
>> the web UI?
>>
>> On Fri, May 18, 2018 at 4:01 PM, Jonathan Vaughn <jonathan(a)creatuity.com
>> > wrote:
>>
>>> httpd error_log
>>>
>>> [Fri May 18 15:58:29.546255 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>>> wsgi_dispatch.__call__:
>>> [Fri May 18 15:58:29.547217 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>>> login_password.__call__:
>>> [Fri May 18 15:58:29.549799 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Obtaining armor in
>>> ccache /var/run/ipa/ccaches/armor_14004
>>> [Fri May 18 15:58:29.550812 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Initializing
>>> anonymous ccache
>>> [Fri May 18 15:58:29.552268 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting external
>>> process
>>> [Fri May 18 15:58:29.553377 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit
>>> -n -c /var/run/ipa/ccaches/armor_14004 -X X509_anchors=
>>> FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa
>>> -client/pki/kdc-ca-bundle.pem
>>> [Fri May 18 15:58:30.486537 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Process finished,
>>> return code=0
>>> [Fri May 18 15:58:30.488137 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stdout=
>>> [Fri May 18 15:58:30.489128 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
>>> [Fri May 18 15:58:30.490955 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Initializing
>>> principal admin using password
>>> [Fri May 18 15:58:30.492060 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Using armor ccache
>>> /var/run/ipa/ccaches/armor_14004 for FAST webauth
>>> [Fri May 18 15:58:30.493262 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Using enterprise
>>> principal
>>> [Fri May 18 15:58:30.494728 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting external
>>> process
>>> [Fri May 18 15:58:30.495799 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit
>>> admin -c /var/run/ipa/ccaches/kinit_14004 -T
>>> /var/run/ipa/ccaches/armor_14004 -E
>>> [Fri May 18 15:58:30.721637 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Process finished,
>>> return code=0
>>> [Fri May 18 15:58:30.723230 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stdout=Password for
>>> admin(a)CREATUITY.INTERNAL:
>>> [Fri May 18 15:58:30.723432 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284]
>>> [Fri May 18 15:58:30.724345 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
>>> [Fri May 18 15:58:30.725716 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Cleanup the armor
>>> ccache
>>> [Fri May 18 15:58:30.727071 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting external
>>> process
>>> [Fri May 18 15:58:30.728152 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG:
>>> args=/usr/bin/kdestroy -A -c /var/run/ipa/ccaches/armor_14004
>>> [Fri May 18 15:58:30.798642 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Process finished,
>>> return code=0
>>> [Fri May 18 15:58:30.800249 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stdout=
>>> [Fri May 18 15:58:30.801224 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
>>> [Fri May 18 15:58:30.857737 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTP
>>> connection (1): ipa-11.creatuity.internal
>>> [Fri May 18 15:58:30.867978 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG:
>>> http://ipa-11.creatuity.internal:80 "GET /ipa/session/cookie
HTTP/1.1"
>>> 301 260
>>> [Fri May 18 15:58:30.889415 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTPS
>>> connection (1): ipa-11.creatuity.internal
>>> [Fri May 18 15:58:31.105092 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG:
>>> https://ipa-11.creatuity.internal:443 "GET /ipa/session/cookie
>>> HTTP/1.1" 200 0
>>> [Fri May 18 15:58:31.157295 2018] [wsgi:error] [pid 14003:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>>> wsgi_dispatch.__call__:
>>> [Fri May 18 15:58:31.158347 2018] [wsgi:error] [pid 14003:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>>> jsonserver_session.__call__:
>>> [Fri May 18 15:58:31.159289 2018] [wsgi:error] [pid 14003:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
>>> [Fri May 18 15:58:31.160349 2018] [wsgi:error] [pid 14003:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized
>>> need login
>>> [Fri May 18 15:58:31.175232 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>>> wsgi_dispatch.__call__:
>>> [Fri May 18 15:58:31.176208 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI
>>> KerberosLogin.__call__:
>>> [Fri May 18 15:58:31.177060 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
>>> [Fri May 18 15:58:31.178047 2018] [wsgi:error] [pid 14004:tid
>>> 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized
>>> need login
>>>
>>>
>>> On Fri, May 18, 2018 at 3:53 PM, Rob Crittenden <rcritten(a)redhat.com>
>>> wrote:
>>>
>>>> Jonathan Vaughn wrote:
>>>> > System time appears to be fine.
>>>> >
>>>> > Here's the httpd access log (nothing in error log):
>>>> > 10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET
>>>> /ipa/session/cookie
>>>> > HTTP/1.1" 301 260 "-"
"python-requests/2.18.4"
>>>> > 10.10.255.101 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20
>>>> -0500]
>>>> > "GET /ipa/session/cookie HTTP/1.1" 200 -
>>>> > 10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
>>>> > /ipa/session/login_password HTTP/1.1" 200 20
>>>> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
>>>> "POST
>>>> > /ipa/session/json HTTP/1.1" 401 -
>>>> > 10.10.11.3 - admin(a)CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
>>>> "GET
>>>> > /ipa/session/login_kerberos?_=1526674892374 HTTP/1.1" 401 -
>>>> >
>>>>
>>>> You'll want to look at the error log for more details. For even more
>>>> output create /etc/ipa/server.conf with the contents:
>>>>
>>>> [global]
>>>> debug = True
>>>>
>>>> and restart httpd
>>>>
>>>> rob
>>>>
>>>> >
>>>> > On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn <
>>>> jonathan(a)creatuity.com
>>>> > <mailto:jonathan@creatuity.com>> wrote:
>>>> >
>>>> > Actually - may have spoke too soon. I thought I'd signed in
to
>>>> the
>>>> > UI but I was actually in another server's UI. When I try to
log
>>>> in
>>>> > it says that my login has expired... so progress but something
>>>> else
>>>> > is busted.
>>>> >
>>>> > On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn
>>>> > <jonathan(a)creatuity.com
<mailto:jonathan@creatuity.com>> wrote:
>>>> >
>>>> > Looks like replica install finished just fine, and I can
>>>> access
>>>> > the the web UI on it. No obvious problems.
>>>> >
>>>> > So it would appear that code change fixed 389DS for ARM.
>>>> >
>>>> > I'm not running the Dogtag CA on the Pi, just on the
non-Pi
>>>> > servers, otherwise would need to probably adjust the
startup
>>>> > timeout for it so that the install script didn't give
up
>>>> waiting
>>>> > on it. When I was first trying to play with this I tried
>>>> > installing the first replica/master on the Pi, and ran into
>>>> the
>>>> > Dogtag timeout problem. Even installing Oracle Java instead
>>>> of
>>>> > the OpenJDK version of Java didn't solve it, because
somehow
>>>> it
>>>> > always picks the OpenJDK java instead of oracle even if I
>>>> change
>>>> > the alternatives setting to default to the Oracle one...
>>>> >
>>>> > But as far as I can tell otherwise, everything is working
as
>>>> > expected.
>>>>
>>>
>>>
>>
>
>
5:34 p.m.
New subject: Problems setting up replica on Raspberry Pi 3B (ARM)
On 05/17/2018 05:07 PM, Jonathan Vaughn wrote:
Oops, hit reply instead of reply-all
NSPR RPMs
# yum list installed nspr*
Installed Packages
nspr.armv7hl
4.19.0-1.fc27
@updates
nspr-debuginfo.armv7hl
4.19.0-1.fc27
@updates-debuginfo
nspr-debugsource.armv7hl
4.19.0-1.fc27
@updates-debuginfo
nspr-devel.armv7hl
4.19.0-1.fc27
@updates
As for building it with the changes you proposed previously - should I
not bother? I realized last night that the last and only time I built
anything *on* a Pi was a Pi Zero and that's why it was so terribly
slow. This Pi 3 ain't fast but it's fast enough to build stuff in far
less than half a day (unlike the Zero)...
I know Thierry said the agmt looked fine but I fixed this exact crash on
arm before, and it was a casting issue with sprintf. So something is
off on raspberry, and this is just to work around it.
Anyway this is how I build DS:
# mkdir source
# cd source
# mkdir BUILD
# git clone https://git@pagure.io/389-ds-base.git
# cd 389-ds-base
# git checkout 389-ds-base-1.3.7
---> make suggested code changes in repl5_agmt.c
# ./autogen.pl
# cd ../BUILD
# CFLAGS='-O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -march=armv7-a
-mfloat-abi=hard' CXXFLAGS='-O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -march=armv7-a
-mfloat-abi=hard' ../389-ds-base/configure --enable-autobind
--with-selinux --with-openldap --with-tmpfiles-d=/etc/tmpfiles.d
--with-systemdsystemunitdir=/usr/lib/systemd/system
--with-systemdsystemconfdir=/etc/systemd/system
--with-systemdgroupname=dirsrv.target --with-fhs --libdir=/usr/lib64
--enable-nunc-stans --with-systemd
# make install
DONE!
On Thu, May 17, 2018 at 5:07 AM, thierry bordaz <tbordaz(a)redhat.com
<mailto:tbordaz@redhat.com>> wrote:
On 05/16/2018 10:03 PM, Jonathan Vaughn wrote:
> I've been just using the packages from Fedora. I can build it
> potentially but I don't have a cross build environment set up at
> the moment. From experience I'd want to do that first because
> building anything on the Pi usually takes ages.
>
> I'd been "redacting" the hostnames but I'll stop bothering
since
> it looks like we're getting far enough into the weeds now that
> the difference in string lengths after "redacting" might actually
> be a red herring.
>
> (gdb) p *agmt
> $1 = {hostname = 0x1ef9be0 "ipa-12.creatuity.internal", port =
> 389, transport_flags = 0, binddn = 0x1e8f650 "", creds =
> 0x1e8f7a0, bindmethod = 3, replarea = 0x1ef9480,
> frac_attrs = 0x1ef99c0, frac_attrs_total = 0x1ef9a40,
> frac_attr_total_defined = 1, schedule = 0x1a7f0c0,
> auto_initialize = 502, dn = 0x1ef8d00, rdn = 0x1ef8c20,
> long_name = 0x1a7f100
> "agmt=\"cn=meToipa-12.creatuity.internal\" (ipa-12:5)",
protocol
> = 0x19c2930, changecounters = 0x186d180, num_changecounters = 0,
> max_changecounters = 256, last_update_start_time = 1526500697,
> last_update_end_time = 1526500697,
> last_update_status = "Error (0) Replica acquired successfully:
> Incremental update succeeded", '\000' <repeats 954 times>,
> update_in_progress = 0, is_enabled = 1,
> last_init_start_time = 0, last_init_end_time = 0,
> last_init_status = '\000' <repeats 1023 times>, lock = 0x1ee3740,
> consumerRUV = 0x1f14e50,
> consumerSchemaCSN = 0x317c520, consumerRID = 4, tmpConsumerRID
> = 0, timeout = 120, stop_in_progress = 0, busywaittime = 0,
> pausetime = 0, priv = 0x0,
> attrs_to_strip = 0x1ef9ba0, agreement_type = 0,
> protocol_timeout = 0x1e8f5f0, maxcsn = 0x0, flowControlWindow =
> 1000, flowControlPause = 2000, ignoreMissingChange = 0,
> attr_lock = 0x1ef9c20, WaitForAsyncResults = 100}
> (gdb) p *agmt->replarea
> $2 = {flag = 15 '\017', udn = 0x1efce80
> "dc=creatuity,dc=internal", dn = 0x1ef9460
> "dc=creatuity,dc=internal", ndn = 0x1ef8ec0
> "dc=creatuity,dc=internal", ndn_len = 24}
> (gdb) p *agmt->rdn
> $3 = {flag = 0 '\000', rdn = 0x19c2840
> "cn=meToipa-12.creatuity.internal", rdns = 0x0, butcheredupto =
> -1, nrdn = 0x0, all_rdns = 0x0, all_nrdns = 0x0}
>
> [root@ipa-11 ~]# grep -r PRId64 /usr/include/*
> /usr/include/inttypes.h:# define PRId64 __PRI64_PREFIX "d"
> [root@ipa-11 ~]# grep -r PRIu16 /usr/include/*
> /usr/include/inttypes.h:# define PRIu16 "u"
>
>
>
> On Wed, May 16, 2018 at 2:55 PM, Mark Reynolds
> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>> wrote:
>
>
>
> On 05/16/2018 03:43 PM, Jonathan Vaughn wrote:
>> The installed version of 389* is 1.3.7.10-1.fc27
>> for armv7hl, which appears to be the latest available version.
>
> Perhaps something is off with the inttypes on Raspberry. Are
> you building this yourself on Raspberry? Can we make code
> changes and compile/install them?
>
> Before we do that though, in gdb can you run these commands
> in the same gdb frame:
>
> (gdb) p *agmt->replarea
> (gdb) p *agmt->rdn
>
>
> Then do:
>
> # grep -r PRId64 /usr/include/*
> # grep -r PRIu16 /usr/include/*
>
>
> So if you can compile the source, then change this line in
> ldap/servers/plugins/replication/repl5_agmt.c:3036, but don't
> do this yet until you get me the info I just requested.
>
> From:
>
> agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%"
> PRId64 ";%" PRIu16 ";%s",
slapi_sdn_get_dn(agmt->replarea),
>
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
> agmt->hostname,
> agmt->port,
> agmt->consumerRID, maxcsn);
>
> To:
>
> agmt->maxcsn =
> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
> slapi_sdn_get_dn(agmt->replarea),
>
> slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)),
> agmt->hostname,
>
> (long)agmt->port, (int)agmt->consumerRID, maxcsn);
>
>
> Thanks,
> Mark
>
>>
>> On Wed, May 16, 2018 at 2:38 PM, Jonathan Vaughn
>> <jonathan(a)creatuity.com <mailto:jonathan@creatuity.com>>
wrote:
>
The agreement structure looks valid to me. it should not lead to a
crash.
What looks weird to me is the order of the arguments of cvt_s.
It is called: rv = cvt_s(ss, u.s, width, prec, flags);
But the crashing thread shows the opposite order: flags,
prec,width, str, ss
The others frame do not show this change of order.
Also 'str=4' that would be a meaningful value for 'prec=4'.
From debug perspective I only imagine disassemble the two last
frames to confirm parameters.
What are the nspr rpms ?
>> (gdb) up
>> #1 0xb6926b40 in cvt_s (flags=0, prec=<optimized out>,
>> width=0, str=0x4 <error: Cannot access memory at address
>> 0x4>, ss=<optimized out>)
>> at ../../.././nspr/pr/src/io/prprf.c:374
>> 374 slen = strlen(str);
>> (gdb) up
>> #2 dosprintf (ss=ss@entry=0x9e06e4bc, fmt=0xb34b0df2
>> "", fmt@entry=0xb34da770 <agmt_set>
"\360\317p\002",
>> ap=...) at ../../.././nspr/pr/src/io/prprf.c:1018
>> 1018 rv = cvt_s(ss, u.s, width, prec, flags);
>> (gdb) up
>> #3 0xb6926c8c in PR_vsmprintf (fmt=fmt@entry=0xb34da770
>> <agmt_set> "\360\317p\002", ap=..., ap@entry=...) at
>> ../../.././nspr/pr/src/io/prprf.c:1184
>> 1184 rv = dosprintf(&ss, fmt, ap);
>> (gdb) up
>> #4 0xb6de9d18 in slapi_ch_smprintf (fmt=0xb34b0de0
>> "%s;%s;%s;%ld;%d;%s") at
ldap/servers/slapd/ch_malloc.c:331
>> 331 p = PR_vsmprintf(fmt, ap);
>> (gdb) up
>> #5 0xb34625a4 in agmt_update_maxcsn
>> (r=r@entry=0x2737810, sdn=0x38d9a40, sdn@entry=0x10,
>> op=<optimized out>, mods=0x10, mods@entry=0x27d0100,
>> csn=csn@entry=0x38de350)
>> at ldap/servers/plugins/replication/repl5_agmt.c:3036
>> 3036 agmt->maxcsn =
>> slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s",
>> slapi_sdn_get_dn(agmt->replarea),
>> (gdb) p *agmt
>> $1 = {hostname = 0x2757be0 "ipa-12.company.internal",
>> port = 389, transport_flags = 0, binddn = 0x26ed650 "",
>> creds = 0x26ed7a0, bindmethod = 3, replarea = 0x2757480,
>> frac_attrs = 0x27579c0, frac_attrs_total = 0x2757a40,
>> frac_attr_total_defined = 1, schedule = 0x22dd0c0,
>> auto_initialize = 502, dn = 0x2756d00, rdn = 0x2756c20,
>> long_name = 0x22dd100
>> "agmt=\"cn=meToipa-12.company.internal\"
(ipa-12:5)",
>> protocol = 0x2220930, changecounters = 0x20cb180,
>> num_changecounters = 0,
>> max_changecounters = 256, last_update_start_time =
>> 1526499214, last_update_end_time = 1526499214,
>> last_update_status = "Error (0) Replica acquired
>> successfully: Incremental update succeeded", '\000'
>> <repeats 954 times>, update_in_progress = 0, is_enabled = 1,
>> last_init_start_time = 0, last_init_end_time = 0,
>> last_init_status = '\000' <repeats 1023 times>, lock =
>> 0x2741740, consumerRUV = 0x2772e50,
>> consumerSchemaCSN = 0x39da520, consumerRID = 4,
>> tmpConsumerRID = 0, timeout = 120, stop_in_progress = 0,
>> busywaittime = 0, pausetime = 0, priv = 0x0,
>> attrs_to_strip = 0x2757ba0, agreement_type = 0,
>> protocol_timeout = 0x26ed5f0, maxcsn = 0x0,
>> flowControlWindow = 1000, flowControlPause = 2000,
>> ignoreMissingChange = 0,
>> attr_lock = 0x2757c20, WaitForAsyncResults = 100}
>>
>>
>> On Tue, May 15, 2018 at 5:36 PM, Mark Reynolds
>> <mreynolds(a)redhat.com <mailto:mreynolds@redhat.com>>
wrote:
>>
>> This looks really familiar and I thought it was
>> fixed. It should have been fixed in 1.3.7.10-1
>> (https://pagure.io/389-ds-base/issue/49618
>> <https://pagure.io/389-ds-base/issue/49618>). In
>> your debug session go "up" into agmt_maxcsn_update()
>> and do:
>>
>> (gdb) p *agmt
>>
>> Then send us that output please.
>>
>> Thanks,
>> Mark
>>
>>
>> On 05/15/2018 05:29 PM, Jonathan Vaughn via
>> FreeIPA-users wrote:
>>> Here is a backtrace from live gdb after the
>>> segfault. Looks like things went wrong somewhere
>>> during in the replication code ?
>>>
>>> Thread 36 "ns-slapd" received signal SIGSEGV,
>>> Segmentation fault.
>>> [Switching to Thread 0x9e0bc280 (LWP 4662)]
>>> strlen () at ../sysdeps/arm/armv6t2/strlen.S:142
>>> 142 ldrd data1a, data1b, [src]
>>> (gdb) bt
>>> #0 0xb6728f2e in strlen () at
>>> ../sysdeps/arm/armv6t2/strlen.S:142
>>> #1 0xb6973b40 in cvt_s (flags=0, prec=<optimized
>>> out>, width=0, str=0x4 <error: Cannot access memory
>>> at address 0x4>, ss=<optimized out>)
>>> at ../../.././nspr/pr/src/io/prprf.c:374
>>> #2 0xb6973b40 in dosprintf
>>> (ss=ss@entry=0x9e0bb4bc, fmt=0xb34fddf2 "",
>>> fmt@entry=0xb3527770 <agmt_set>
"\360\357\305\002",
>>> ap=...)
>>> at ../../.././nspr/pr/src/io/prprf.c:1018
>>> #3 0xb6973c8c in PR_vsmprintf
>>> (fmt=fmt@entry=0xb3527770 <agmt_set>
>>> "\360\357\305\002", ap=..., ap@entry=...) at
>>> ../../.././nspr/pr/src/io/prprf.c:1184
>>> #4 0xb6e36d18 in slapi_ch_smprintf (fmt=0xb34fdde0
>>> "%s;%s;%s;%ld;%d;%s") at
>>> ldap/servers/slapd/ch_malloc.c:331
>>> #5 0xb34af5a4 in agmt_update_maxcsn
>>> (r=r@entry=0x2c89810, sdn=0x3e2bac0,
>>> sdn@entry=0x10, op=<optimized out>, mods=0x10,
>>> mods@entry=0x3eec100, csn=csn@entry=0x3e30220)
>>> at
>>> ldap/servers/plugins/replication/repl5_agmt.c:3036
>>> #6 0xb34bd424 in write_changelog_and_ruv
>>> (pb=pb@entry=0x2cb54a0) at
>>> ldap/servers/plugins/replication/repl5_plugins.c:1124
>>> #7 0xb34be7ec in multimaster_be_betxnpostop_add
>>> (pb=0x2cb54a0) at
>>> ldap/servers/plugins/replication/repl5_plugins.c:855
>>> #8 0xb34be880 in multimaster_mmr_postop
>>> (pb=<optimized out>, flags=560) at
>>> ldap/servers/plugins/replication/repl5_plugins.c:616
>>> #9 0xb6e8fcf0 in plugin_call_mmr_plugin_postop
>>> (pb=pb@entry=0x2cb54a0, e=e@entry=0x0,
>>> flags=flags@entry=560) at
>>> ldap/servers/slapd/plugin_mmr.c:65
>>> #10 0xb35ba870 in ldbm_back_add (pb=0x2cb54a0) at
>>> ldap/servers/slapd/back-ldbm/ldbm_add.c:1218
>>> #11 0xb6e2ce4c in op_shared_add
>>> (pb=pb@entry=0x2cb54a0) at ldap/servers/slapd/add.c:679
>>> #12 0xb6e2d35c in add_internal_pb
>>> (pb=pb@entry=0x2cb54a0) at ldap/servers/slapd/add.c:407
>>> #13 0xb6e2e0f8 in slapi_add_internal_pb
>>> (pb=pb@entry=0x2cb54a0) at ldap/servers/slapd/add.c:332
>>> #14 0xb368db50 in ipa_topo_util_segment_write
>>> (tconf=tconf@entry=0x2cb5860,
>>> tsegm=tsegm@entry=0x3f2d9a0) at topology_util.c:1251
>>> #15 0xb368e01c in ipa_topo_util_update_agmt_list
>>> (conf=0x2cb5860, repl_segments=<optimized out>) at
>>> topology_util.c:696
>>> #16 0xb368891c in ipa_topo_apply_shared_config ()
>>> at topology_init.c:165
>>> #17 0xb6e4e65c in eq_call_all () at
>>> ldap/servers/slapd/eventq.c:278
>>> #18 0xb6e4e65c in eq_loop (arg=<optimized out>) at
>>> ldap/servers/slapd/eventq.c:323
>>> #19 0xb6982d68 in _pt_root (arg=0x2c8da40) at
>>> ../../.././nspr/pr/src/pthreads/ptthread.c:201
>>> #20 0xb6906ee8 in start_thread (arg=0x9e0bc280) at
>>> pthread_create.c:465
>>> #21 0xb6785da8 in None () at
>>> ../sysdeps/unix/sysv/linux/arm/clone.S:73
>>>
>>>
>>> On Tue, May 15, 2018 at 3:01 AM, thierry bordaz
>>> <tbordaz(a)redhat.com <mailto:tbordaz@redhat.com>>
wrote:
>>>
>>> Hi Jonathan,
>>>
>>> This problem looks new to me and has something
>>> specific to your environment.
>>> I think the best approach is to continue to
>>> debug on your system if you have the
>>> possibility to do so.
>>>
>>> From strace we can see that DS started smoothly
>>> (created its pid file then notified systemd it
>>> was running fine). According to the pstack
>>> nunc-stans was running and was able to accept
>>> network events even if it appears it detected
>>> no incoming connection.
>>> So the server should be ready to serve for some
>>> seconds (more than a minute), then it crashed
>>> with one thread dereferencing likely a wrong
>>> pointer.
>>>
>>> Could you attach a debugger when the server is
>>> started and wait for the sigsegv to occur. Then
>>> confirm the crashing thread backstack.
>>> If it is confirmed, I am afraid this is a stack
>>> corruption and valgrind could help
>>>
(http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...
>>>
<http://www.port389.org/docs/389ds/FAQ/faq.html#debugging-memory-growthinv...>).
>>>
>>> best regards
>>> thierry
>>>
>>>
>>> On 05/14/2018 10:20 PM, Jonathan Vaughn wrote:
>>>> Here's a strace from before it dies. Most of
>>>> the elapsed time is it waiting on some futex
>>>> call it looks like near the end, when it
>>>> finally "returns" (from lack of strace
output
>>>> for duration of call I assume it didn't
>>>> actually return but SIGSEGV in that call) and
>>>> strace prints ' = ?' on the futex it then
>>>> immediately reports SIGSEGV after. So maybe
>>>> the problem is that futex call, which may mean
>>>> the problem is not directly in 389DS / FreeIPA
>>>> itself?
>>>>
>>>>
>>>>
>>>> 15:13:31.626587 (+ 0.000630) listen(8,
>>>> 128) = 0 <0.000068>
>>>> 15:13:31.626857 (+ 0.000235) listen(9,
>>>> 128) = 0 <0.000048>
>>>> 15:13:31.627111 (+ 0.000251)
>>>> clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932, tv_nsec=41120614}) = 0
<0.000085>
>>>> 15:13:31.627457 (+ 0.000356)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=627560772}) = 0
>>>> <0.000043>
>>>> 15:13:31.631233 (+ 0.003839)
>>>> clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932, tv_nsec=45286796}) = 0
<0.000077>
>>>> 15:13:31.631720 (+ 0.000427)
>>>> clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932, tv_nsec=45661430}) = 0
<0.000042>
>>>> 15:13:31.631955 (+ 0.000220)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=632049036}) = 0
>>>> <0.000047>
>>>> 15:13:31.635669 (+ 0.003785)
>>>> clock_gettime(CLOCK_MONOTONIC,
>>>> {tv_sec=1464932, tv_nsec=49725840}) = 0
<0.000146>
>>>> 15:13:31.636484 (+ 0.000784) write(16,
>>>> "a", 1) = 1 <0.000118>
>>>> 15:13:31.636855 (+ 0.000341) sched_yield()
>>>> = 0 <0.000252>
>>>> 15:13:31.637322 (+ 0.000470)
>>>> futex(0x1cb57a0, FUTEX_WAKE_PRIVATE, 1) = 1
>>>> <0.000088>
>>>> 15:13:31.637897 (+ 0.000610) write(16,
>>>> "a", 1) = 1 <0.000221>
>>>> 15:13:31.638394 (+ 0.000467) sched_yield()
>>>> = 0 <0.000047>
>>>> 15:13:31.638619 (+ 0.000202)
>>>> futex(0x1cb5710, FUTEX_WAKE_PRIVATE, 1) = 1
>>>> <0.000065>
>>>> 15:13:31.638908 (+ 0.000298)
>>>> openat(AT_FDCWD,
>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> O_WRONLY|O_CREAT|O_TRUNC, 0666) = 33
<0.000831>
>>>> 15:13:31.640260 (+ 0.001387) getpid() =
>>>> 32353 <0.000077>
>>>> 15:13:31.640558 (+ 0.000256) fstat64(33,
>>>> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
>>>> <0.000119>
>>>> 15:13:31.641106 (+ 0.000556) write(33,
>>>> "32353\n", 6) = 6 <0.000127>
>>>> 15:13:31.641472 (+ 0.000362) close(33) = 0
>>>> <0.000519>
>>>> 15:13:31.642216 (+ 0.000758)
>>>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>> 0644) = 0 <0.000152>
>>>> 15:13:31.642900 (+ 0.000679)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=643020294}) = 0
>>>> <0.000056>
>>>> 15:13:31.643495 (+ 0.000590) write(2,
>>>> "[14/May/2018:15:13:31.643020294 "..., 134)
=
>>>> 134 <0.002697>
>>>> 15:13:31.646515 (+ 0.003052)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=646694394}) = 0
>>>> <0.000075>
>>>> 15:13:31.646892 (+ 0.000337) write(4,
>>>> "[14/May/2018:15:13:31.646694394 "..., 134)
=
>>>> 134 <0.000522>
>>>> 15:13:31.647841 (+ 0.000973) fsync(4) = 0
>>>> <0.005967>
>>>> 15:13:31.654425 (+ 0.006617)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=654598946}) = 0
>>>> <0.000253>
>>>> 15:13:31.655137 (+ 0.000717) write(2,
>>>> "[14/May/2018:15:13:31.654598946 "..., 136)
=
>>>> 136 <0.002427>
>>>> 15:13:31.658312 (+ 0.003165)
>>>> clock_gettime(CLOCK_REALTIME,
>>>> {tv_sec=1526328811, tv_nsec=658486117}) = 0
>>>> <0.000251>
>>>> 15:13:31.659032 (+ 0.000682) write(4,
>>>> "[14/May/2018:15:13:31.658486117 "..., 136)
=
>>>> 136 <0.000346>
>>>> 15:13:31.659623 (+ 0.000595) fsync(4) = 0
>>>> <0.003311>
>>>> 15:13:31.663230 (+ 0.003642) getpid() =
>>>> 32353 <0.000045>
>>>> 15:13:31.663732 (+ 0.000454)
>>>> socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) =
>>>> 33 <0.000296>
>>>> 15:13:31.664760 (+ 0.001048)
>>>> getsockopt(33, SOL_SOCKET, SO_SNDBUF,
>>>> [163840], [4]) = 0 <0.000108>
>>>> 15:13:31.665141 (+ 0.000386)
>>>> setsockopt(33, SOL_SOCKET, SO_SNDBUFFORCE,
>>>> [8388608], 4) = -1 EPERM (Operation not
>>>> permitted) <0.000051>
>>>> 15:13:31.665500 (+ 0.000334)
>>>> setsockopt(33, SOL_SOCKET, SO_SNDBUF,
>>>> [8388608], 4) = 0 <0.000229>
>>>> 15:13:31.665973 (+ 0.000468) sendmsg(33,
>>>> {msg_name={sa_family=AF_UNIX,
>>>> sun_path="/run/systemd/notify"},
>>>> msg_namelen=21,
>>>> msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>> started: Re"..., iov_len=69}], msg_iovlen=1,
>>>> msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL)
>>>> = 69 <0.000600>
>>>> 15:13:31.667270 (+ 0.001334) close(33) = 0
>>>> <0.003140>
>>>> 15:13:31.677320 (+ 0.010146)
>>>> futex(0xb31122e8, FUTEX_WAIT, 32357, NULL) = ?
>>>> 15:14:27.661809 (+ 55.984448) +++ killed by
>>>> SIGSEGV (core dumped) +++
>>>>
>>>> On Mon, May 14, 2018 at 10:27 AM, thierry
>>>> bordaz <tbordaz(a)redhat.com
>>>> <mailto:tbordaz@redhat.com>> wrote:
>>>>
>>>> Hi Jonathan,
>>>>
>>>> This is weird as the crashing thread stack
>>>> looks truncated (did you copy/paste all of
>>>> it ?)
>>>>
>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>> #0 0xb67bbf2e in strlen () at /lib/libc.so.6
>>>> #1 0xb6a06b40 in dosprintf () at
>>>> /lib/libnspr4.so
>>>> #2 0x00000000 in None ()
>>>>
>>>> Did you install 389-ds-base-debuginfo ?
>>>> How did you get that backtrace ? from a
>>>> core dumped, pstack ? Can you attach a
>>>> debugger before the crash occurs ?
>>>>
>>>> It looks it crashed soon at startup, could
>>>> it be related to a broken dse.ldif. It
>>>> should exists a dse.ldif.OK, is it
>>>> possibly to try to start with it ?
>>>>
>>>> best regards
>>>> thierry
>>>>
>>>> On 05/12/2018 01:22 AM, Jonathan Vaughn
>>>> via FreeIPA-users wrote:
>>>>> Not sure if it makes a difference... I
>>>>> was looking into this again and realized
>>>>> I had a bunch of messages from gdb
>>>>> telling me to install more debuginfo.
>>>>> I've done that now, here it is again
>>>>> freshly run through gdb
>>>>>
>>>>> GNU gdb (GDB) Fedora 8.0.1-36.fc27
>>>>> Copyright (C) 2017 Free Software
>>>>> Foundation, Inc.
>>>>> License GPLv3+: GNU GPL version 3 or
>>>>> later <http://gnu.org/licenses/gpl.html
>>>>> <http://gnu.org/licenses/gpl.html>>
>>>>> This is free software: you are free to
>>>>> change and redistribute it.
>>>>> There is NO WARRANTY, to the extent
>>>>> permitted by law. Type "show
copying"
>>>>> and "show warranty" for details.
>>>>> This GDB was configured as
>>>>> "armv7hl-redhat-linux-gnueabi".
>>>>> Type "show configuration" for
>>>>> configuration details.
>>>>> For bug reporting instructions, please see:
>>>>> <http://www.gnu.org/software/gdb/bugs/
>>>>>
<http://www.gnu.org/software/gdb/bugs/>>;.
>>>>> Find the GDB manual and other
>>>>> documentation resources online at:
>>>>>
<http://www.gnu.org/software/gdb/documentation/
>>>>>
<http://www.gnu.org/software/gdb/documentation/>>;.
>>>>> For help, type "help".
>>>>> Type "apropos word" to search for
>>>>> commands related to "word"...
>>>>> Reading symbols from
>>>>> /usr/sbin/ns-slapd...Reading symbols from
>>>>>
/usr/lib/debug/usr/sbin/ns-slapd-1.3.7.10-1.fc27.arm.debug...done.
>>>>> done.
>>>>> ...
>>>>>
>>>>> Thread 1 (Thread 0x9e13c280 (LWP 17245)):
>>>>> #0 0xb67bbf2e in strlen () at
/lib/libc.so.6
>>>>> #1 0xb6a06b40 in dosprintf () at
>>>>> /lib/libnspr4.so
>>>>> #2 0x00000000 in None ()
>>>>>
>>>>>
>>>>>
>>>>> On Tue, May 8, 2018 at 7:52 AM, Rob
>>>>> Crittenden <rcritten(a)redhat.com
>>>>> <mailto:rcritten@redhat.com>>
wrote:
>>>>>
>>>>> Jonathan Vaughn via FreeIPA-users
>>>>> wrote:
>>>>>
>>>>> Still trying to figure this
>>>>> out. It looks like slapd is
>>>>> dying, I thought it was still
>>>>> running for some reason.
>>>>>
>>>>> slapd is dying to segfault.
>>>>> strace of it happening
>>>>> doesn't seem to reveal much:
>>>>>
>>>>>
>>>>> A stack trace would very much
>>>>> help trying to track down the cause.
>>>>>
>>>>>
http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-crashes
>>>>>
<http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-cras...
>>>>>
>>>>> rob
>>>>>
>>>>>
>>>>> 18:32:41.543717 (+
>>>>> 0.000801) openat(AT_FDCWD,
>>>>>
"/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>> O_WRONLY|O_CREAT|O_TRUNC,
>>>>> 0666) = 32
>>>>> 18:32:41.544907 (+
>>>>> 0.001195) getpid() = 16014
>>>>> 18:32:41.545269 (+
>>>>> 0.000329) fstat64(32,
>>>>> {st_mode=S_IFREG|0644,
>>>>> st_size=0, ...}) = 0
>>>>> 18:32:41.545799 (+
>>>>> 0.000536) write(32,
>>>>> "16014\n", 6) = 6
>>>>> 18:32:41.546603 (+
>>>>> 0.000818) close(32) = 0
>>>>> 18:32:41.547061 (+
>>>>> 0.000448)
>>>>>
chmod("/var/run/dirsrv/slapd-COMPANY-INTERNAL.pid",
>>>>> 0644) = 0
>>>>> 18:32:41.547741 (+
>>>>> 0.000676)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961,
>>>>> tv_nsec=548030641}) = 0
>>>>> 18:32:41.548324 (+
>>>>> 0.000587) write(2,
>>>>>
"[07/May/2018:18:32:41.548030641
>>>>> "..., 134) = 134
>>>>> 18:32:41.551096 (+
>>>>> 0.002840)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961,
>>>>> tv_nsec=551287555}) = 0
>>>>> 18:32:41.551568 (+
>>>>> 0.000406) write(4,
>>>>>
"[07/May/2018:18:32:41.551287555
>>>>> "..., 134) = 134
>>>>> 18:32:41.552360 (+
>>>>> 0.000811) fsync(4) = 0
>>>>> 18:32:41.558499 (+
>>>>> 0.006170)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961,
>>>>> tv_nsec=558678099}) = 0
>>>>> 18:32:41.558901 (+
>>>>> 0.000350) write(2,
>>>>>
"[07/May/2018:18:32:41.558678099
>>>>> "..., 136) = 136
>>>>> 18:32:41.561537 (+
>>>>> 0.002680)
>>>>> clock_gettime(CLOCK_REALTIME,
>>>>> {tv_sec=1525735961,
>>>>> tv_nsec=561718659}) = 0
>>>>> 18:32:41.562357 (+
>>>>> 0.000793) write(4,
>>>>>
"[07/May/2018:18:32:41.561718659
>>>>> "..., 136) = 136
>>>>> 18:32:41.563293 (+
>>>>> 0.001148) fsync(4) = 0
>>>>> 18:32:41.566928 (+
>>>>> 0.003452) getpid() = 16014
>>>>> 18:32:41.567712 (+
>>>>> 0.000752) socket(AF_UNIX,
>>>>> SOCK_DGRAM|SOCK_CLOEXEC, 0) = 32
>>>>> 18:32:41.568628 (+
>>>>> 0.000912) getsockopt(32,
>>>>> SOL_SOCKET, SO_SNDBUF,
>>>>> [163840], [4]) = 0
>>>>> 18:32:41.568972 (+
>>>>> 0.000319) setsockopt(32,
>>>>> SOL_SOCKET, SO_SNDBUFFORCE,
>>>>> [8388608], 4) = -1 EPERM
>>>>> (Operation not permitted)
>>>>> 18:32:41.569548 (+
>>>>> 0.000589) setsockopt(32,
>>>>> SOL_SOCKET, SO_SNDBUF,
>>>>> [8388608], 4) = 0
>>>>> 18:32:41.570064 (+
>>>>> 0.000513) sendmsg(32,
>>>>> {msg_name={sa_family=AF_UNIX,
>>>>>
sun_path="/run/systemd/notify"},
>>>>> msg_namelen=21,
>>>>>
msg_iov=[{iov_base="READY=1\nSTATUS=slapd
>>>>> started: Re"...,
>>>>> iov_len=69}], msg_iovlen=1,
>>>>> msg_controllen=0,
>>>>> msg_flags=0}, MSG_NOSIGNAL) = 69
>>>>> 18:32:41.570845 (+
>>>>> 0.000789) close(32) = 0
>>>>> 18:32:41.576358 (+
>>>>> 0.005575) futex(0xb30ed2e8,
>>>>> FUTEX_WAIT, 16016, NULL) = ?
>>>>> 18:33:01.730774 (+
>>>>> 20.154428) +++ killed by
>>>>> SIGSEGV +++
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>> <mailto:freeipa-users@lists.fedorahosted.org>
>>> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
>>> <mailto:freeipa-users-leave@lists.fedorahosted.org>
>>
>>
>>
>
>
2146
days inactive
2178
days old
freeipa-users@lists.fedorahosted.org
36 comments
4 participants
participants (4)
-
Jonathan Vaughn -
Mark Reynolds -
Rob Crittenden -
thierry bordaz