kernel: ns-slapd[5865]: segfault at 5603c0ee2000 ip 00007fe3ba3975ba sp 00007fe3bdbd28a8 error 4 in libc-2.17.so[7fe3ba242000
Hey All,
I've upgrade one side of my two node cluster. However, the secondary won't come even though the manual upgrade apparently went well.
[root@idmipa04 ~]# ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/9]: saving configuration [2/9]: disabling listeners [3/9]: enabling DS global lock [4/9]: disabling Schema Compat [5/9]: starting directory server [6/9]: updating schema [7/9]: upgrading server [8/9]: stopping directory server [9/9]: restoring configuration Done. Update complete Upgrading IPA services Upgrading the configuration of the IPA services IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. Unexpected error - see /var/log/ipaupgrade.log for details: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information [root@idmipa04 ~]#
Additional information:
[root@idmipa04 ~]# cat /var/log/ipaupgrade.log|tail -n 50 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG duration: 0 seconds 2020-05-19T04:18:10Z DEBUG Done. 2020-05-19T04:18:10Z INFO Update complete 2020-05-19T04:18:10Z INFO Upgrading the configuration of the IPA services 2020-05-19T04:18:10Z DEBUG IPA version 4.6.6-11.el7.centos 2020-05-19T04:18:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2020-05-19T04:18:10Z DEBUG Starting external process 2020-05-19T04:18:10Z DEBUG args=/bin/systemctl is-active dirsrv@MWS-MDS-XYZ.service 2020-05-19T04:18:10Z DEBUG Process finished, return code=3 2020-05-19T04:18:10Z DEBUG stdout=unknown
2020-05-19T04:18:10Z DEBUG stderr= 2020-05-19T04:18:10Z DEBUG Starting external process 2020-05-19T04:18:10Z DEBUG args=/bin/systemctl start dirsrv@MWS-MDS-XYZ.service 2020-05-19T04:19:55Z DEBUG Process finished, return code=1 2020-05-19T04:19:55Z DEBUG stdout= 2020-05-19T04:19:55Z DEBUG stderr=Job for dirsrv@MWS-MDS-XYZ.service failed because a fatal signal was delivered to the control process. See "systemctl status dirsrv@MWS-MDS-XYZ.service" and "journalctl -xe" for details.
2020-05-19T04:19:56Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2020-05-19T04:19:56Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2166, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1791, in upgrade_configuration ds.start(ds_serverid) File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 656, in start super(DsInstance, self).start(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 464, in start self.service.start(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 136, in start instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 303, in start skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 563, in run raise CalledProcessError(p.returncode, arg_string, str(output))
2020-05-19T04:19:56Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 2020-05-19T04:19:56Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 2020-05-19T04:19:56Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# /bin/systemctl start dirsrv@MWS-MDS-XYZ.service [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service ● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ. Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 10s ago Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS) Main PID: 4664 (ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service └─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:49.380321307 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:49.381268146 -0400] - ERR - NSMMReplicationPl...r) () May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:49.602418963 -0400] - ERR - schema-compat-plu...onds! May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plu...c=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plu...c=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plu...tion. May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - ...ealm) Hint: Some lines were ellipsized, use -l to show in full. [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service -l ● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ. Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 16s ago Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS) Main PID: 4664 (ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service └─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=mws,dc=mds,dc=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=mws,dc=mds,dc=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plugin - Finished plugin initialization. May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 1 May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 2 May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 3 [root@idmipa04 ~]# [root@idmipa04 ~]#
---------------------------------------------------------------------
Looking at the service, appears libc hit a SEGFAULT.
[root@idmipa04 ~]# journalctl -xe May 19 00:34:03 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]: /usr/bin/killall -0 haproxy exited with status 1 May 19 00:34:05 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]: /usr/bin/killall -0 haproxy exited with status 1 May 19 00:34:06 idmipa04.mws.mds.xyz ns-slapd[5745]: [19/May/2020:00:34:06.272901079 -0400] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRU May 19 00:34:06 idmipa04.mws.mds.xyz kernel: ns-slapd[5865]: segfault at 5603c0ee2000 ip 00007fe3ba3975ba sp 00007fe3bdbd28a8 error 4 in libc-2.17.so[7fe3ba242000 May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: dirsrv@MWS-MDS-XYZ.service: main process exited, code=killed, status=11/SEGV May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: Failed to start 389 Directory Server MWS-MDS-XYZ.. -- Subject: Unit dirsrv@MWS-MDS-XYZ.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit dirsrv@MWS-MDS-XYZ.service has failed. -- -- The result is failed.
Wondering what should my next steps be from here?
TomK via FreeIPA-users wrote:
Hey All,
I've upgrade one side of my two node cluster. However, the secondary won't come even though the manual upgrade apparently went well.
[root@idmipa04 ~]# ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/9]: saving configuration [2/9]: disabling listeners [3/9]: enabling DS global lock [4/9]: disabling Schema Compat [5/9]: starting directory server [6/9]: updating schema [7/9]: upgrading server [8/9]: stopping directory server [9/9]: restoring configuration Done. Update complete Upgrading IPA services Upgrading the configuration of the IPA services IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. Unexpected error - see /var/log/ipaupgrade.log for details: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information [root@idmipa04 ~]#
Additional information:
[root@idmipa04 ~]# cat /var/log/ipaupgrade.log|tail -n 50 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG duration: 0 seconds 2020-05-19T04:18:10Z DEBUG Done. 2020-05-19T04:18:10Z INFO Update complete 2020-05-19T04:18:10Z INFO Upgrading the configuration of the IPA services 2020-05-19T04:18:10Z DEBUG IPA version 4.6.6-11.el7.centos 2020-05-19T04:18:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2020-05-19T04:18:10Z DEBUG Starting external process 2020-05-19T04:18:10Z DEBUG args=/bin/systemctl is-active dirsrv@MWS-MDS-XYZ.service 2020-05-19T04:18:10Z DEBUG Process finished, return code=3 2020-05-19T04:18:10Z DEBUG stdout=unknown
2020-05-19T04:18:10Z DEBUG stderr= 2020-05-19T04:18:10Z DEBUG Starting external process 2020-05-19T04:18:10Z DEBUG args=/bin/systemctl start dirsrv@MWS-MDS-XYZ.service 2020-05-19T04:19:55Z DEBUG Process finished, return code=1 2020-05-19T04:19:55Z DEBUG stdout= 2020-05-19T04:19:55Z DEBUG stderr=Job for dirsrv@MWS-MDS-XYZ.service failed because a fatal signal was delivered to the control process. See "systemctl status dirsrv@MWS-MDS-XYZ.service" and "journalctl -xe" for details.
2020-05-19T04:19:56Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2020-05-19T04:19:56Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2166, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1791, in upgrade_configuration ds.start(ds_serverid) File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 656, in start super(DsInstance, self).start(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 464, in start self.service.start(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 136, in start instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 303, in start skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 563, in run raise CalledProcessError(p.returncode, arg_string, str(output))
2020-05-19T04:19:56Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 2020-05-19T04:19:56Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 2020-05-19T04:19:56Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# /bin/systemctl start dirsrv@MWS-MDS-XYZ.service [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service ● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ. Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 10s ago Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS) Main PID: 4664 (ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service └─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:49.380321307 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:49.381268146 -0400] - ERR - NSMMReplicationPl...r) () May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:49.602418963 -0400] - ERR - schema-compat-plu...onds! May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plu...c=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plu...c=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plu...tion. May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - ...ealm) Hint: Some lines were ellipsized, use -l to show in full. [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service -l ● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ. Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 16s ago Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS) Main PID: 4664 (ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service └─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=mws,dc=mds,dc=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=mws,dc=mds,dc=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plugin - Finished plugin initialization. May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 1 May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 2 May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 3 [root@idmipa04 ~]# [root@idmipa04 ~]#
Looking at the service, appears libc hit a SEGFAULT.
[root@idmipa04 ~]# journalctl -xe May 19 00:34:03 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]: /usr/bin/killall -0 haproxy exited with status 1 May 19 00:34:05 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]: /usr/bin/killall -0 haproxy exited with status 1 May 19 00:34:06 idmipa04.mws.mds.xyz ns-slapd[5745]: [19/May/2020:00:34:06.272901079 -0400] - NOTICE - NSMMReplicationPlugin
- changelog program - _cl5ConstructRU
May 19 00:34:06 idmipa04.mws.mds.xyz kernel: ns-slapd[5865]: segfault at 5603c0ee2000 ip 00007fe3ba3975ba sp 00007fe3bdbd28a8 error 4 in libc-2.17.so[7fe3ba242000 May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: dirsrv@MWS-MDS-XYZ.service: main process exited, code=killed, status=11/SEGV May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: Failed to start 389 Directory Server MWS-MDS-XYZ.. -- Subject: Unit dirsrv@MWS-MDS-XYZ.service has failed -- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Unit dirsrv@MWS-MDS-XYZ.service has failed.
-- The result is failed.
Wondering what should my next steps be from here?
Install the debuginfo packages and get a stack trace.
https://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes
rob
On 5/19/2020 8:21 AM, Rob Crittenden via FreeIPA-users wrote:
TomK via FreeIPA-users wrote:
Hey All,
I've upgrade one side of my two node cluster. However, the secondary won't come even though the manual upgrade apparently went well.
[root@idmipa04 ~]# ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/9]: saving configuration [2/9]: disabling listeners [3/9]: enabling DS global lock [4/9]: disabling Schema Compat [5/9]: starting directory server [6/9]: updating schema [7/9]: upgrading server [8/9]: stopping directory server [9/9]: restoring configuration Done. Update complete Upgrading IPA services Upgrading the configuration of the IPA services IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. Unexpected error - see /var/log/ipaupgrade.log for details: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information [root@idmipa04 ~]#
Additional information:
[root@idmipa04 ~]# cat /var/log/ipaupgrade.log|tail -n 50 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG duration: 0 seconds 2020-05-19T04:18:10Z DEBUG Done. 2020-05-19T04:18:10Z INFO Update complete 2020-05-19T04:18:10Z INFO Upgrading the configuration of the IPA services 2020-05-19T04:18:10Z DEBUG IPA version 4.6.6-11.el7.centos 2020-05-19T04:18:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2020-05-19T04:18:10Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2020-05-19T04:18:10Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2020-05-19T04:18:10Z DEBUG Starting external process 2020-05-19T04:18:10Z DEBUG args=/bin/systemctl is-active dirsrv@MWS-MDS-XYZ.service 2020-05-19T04:18:10Z DEBUG Process finished, return code=3 2020-05-19T04:18:10Z DEBUG stdout=unknown
2020-05-19T04:18:10Z DEBUG stderr= 2020-05-19T04:18:10Z DEBUG Starting external process 2020-05-19T04:18:10Z DEBUG args=/bin/systemctl start dirsrv@MWS-MDS-XYZ.service 2020-05-19T04:19:55Z DEBUG Process finished, return code=1 2020-05-19T04:19:55Z DEBUG stdout= 2020-05-19T04:19:55Z DEBUG stderr=Job for dirsrv@MWS-MDS-XYZ.service failed because a fatal signal was delivered to the control process. See "systemctl status dirsrv@MWS-MDS-XYZ.service" and "journalctl -xe" for details.
2020-05-19T04:19:56Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2020-05-19T04:19:56Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2166, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1791, in upgrade_configuration ds.start(ds_serverid) File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line 656, in start super(DsInstance, self).start(*args, **kwargs) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 464, in start self.service.start(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 136, in start instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 303, in start skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 563, in run raise CalledProcessError(p.returncode, arg_string, str(output))
2020-05-19T04:19:56Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 2020-05-19T04:19:56Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details: CalledProcessError: Command '/bin/systemctl start dirsrv@MWS-MDS-XYZ.service' returned non-zero exit status 1 2020-05-19T04:19:56Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# /bin/systemctl start dirsrv@MWS-MDS-XYZ.service [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service ● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ. Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 10s ago Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS) Main PID: 4664 (ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service └─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:49.380321307 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:49.381268146 -0400] - ERR - NSMMReplicationPl...r) () May 19 00:21:49 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:49.602418963 -0400] - ERR - schema-compat-plu...onds! May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plu...c=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plu...c=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plu...tion. May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - ...ealm) May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - ...ealm) Hint: Some lines were ellipsized, use -l to show in full. [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# [root@idmipa04 ~]# systemctl status dirsrv@MWS-MDS-XYZ.service -l ● dirsrv@MWS-MDS-XYZ.service - 389 Directory Server MWS-MDS-XYZ. Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-05-19 00:21:49 EDT; 16s ago Process: 4657 ExecStartPre=/usr/sbin/ds_systemd_ask_password_acl /etc/dirsrv/slapd-%i/dse.ldif (code=exited, status=0/SUCCESS) Main PID: 4664 (ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /system.slice/system-dirsrv.slice/dirsrv@MWS-MDS-XYZ.service └─4664 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-MWS-MDS-XYZ -i /var/run/dirsrv/slapd-MWS-MDS-XYZ.pid
May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.607265030 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:21:52 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:52.609340757 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.625706606 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=ng, cn=compat,dc=mws,dc=mds,dc=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.758965595 -0400] - ERR - schema-compat-plugin - warning: no entries set up under cn=computers, cn=compat,dc=mws,dc=mds,dc=xyz May 19 00:21:54 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:54.759530547 -0400] - ERR - schema-compat-plugin - Finished plugin initialization. May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.612054116 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:21:58 idmipa04.mws.mds.xyz ns-slapd[4664]: [19/May/2020:00:21:58.613830783 -0400] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/idmipa04.mws.mds.xyz@MWS.MDS.XYZ] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm) May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 1 May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 2 May 19 00:22:03 idmipa04.mws.mds.xyz ns-slapd[4664]: GSSAPI server step 3 [root@idmipa04 ~]# [root@idmipa04 ~]#
Looking at the service, appears libc hit a SEGFAULT.
[root@idmipa04 ~]# journalctl -xe May 19 00:34:03 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]: /usr/bin/killall -0 haproxy exited with status 1 May 19 00:34:05 idmipa04.mws.mds.xyz Keepalived_vrrp[5424]: /usr/bin/killall -0 haproxy exited with status 1 May 19 00:34:06 idmipa04.mws.mds.xyz ns-slapd[5745]: [19/May/2020:00:34:06.272901079 -0400] - NOTICE - NSMMReplicationPlugin
- changelog program - _cl5ConstructRU
May 19 00:34:06 idmipa04.mws.mds.xyz kernel: ns-slapd[5865]: segfault at 5603c0ee2000 ip 00007fe3ba3975ba sp 00007fe3bdbd28a8 error 4 in libc-2.17.so[7fe3ba242000 May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: dirsrv@MWS-MDS-XYZ.service: main process exited, code=killed, status=11/SEGV May 19 00:34:06 idmipa04.mws.mds.xyz systemd[1]: Failed to start 389 Directory Server MWS-MDS-XYZ.. -- Subject: Unit dirsrv@MWS-MDS-XYZ.service has failed -- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Unit dirsrv@MWS-MDS-XYZ.service has failed.
-- The result is failed.
Wondering what should my next steps be from here?
Install the debuginfo packages and get a stack trace.
https://www.port389.org/docs/389ds/FAQ/faq.html#debugging-crashes
Couldn't get the debug tools to install on this CentOS 7. Had a closer look at all the log files and what I saw was memory allocation issues:
[root@idmipa04 slapd-MWS-MDS-XYZ]# [root@idmipa04 slapd-MWS-MDS-XYZ]# cat errors|tail -n 100^C [root@idmipa04 slapd-MWS-MDS-XYZ]# pwd /var/log/dirsrv/slapd-MWS-MDS-XYZ [root@idmipa04 slapd-MWS-MDS-XYZ]# [root@idmipa04 slapd-MWS-MDS-XYZ]# cat errors|tail -n 30 [23/May/2020:16:33:18.519974074 -0400] - WARN - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist [23/May/2020:16:33:18.522332851 -0400] - WARN - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mws,dc=mds,dc=xyz does not exist [23/May/2020:16:33:18.759212393 -0400] - WARN - NSACLPlugin - acl_parse - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist [23/May/2020:16:33:18.773571691 -0400] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=mws,dc=mds,dc=xyz--no CoS Templates found, which should be added before the CoS Definition. [23/May/2020:16:33:18.820082920 -0400] - NOTICE - NSMMReplicationPlugin - changelog program - _cl5ConstructRUV - Rebuilding the replication changelog RUV, this may take several minutes... [23/May/2020:16:39:06.851785150 -0400] - ERR - memory allocator - malloc of 2152941454 bytes failed; OS error 12 (Cannot allocate memory) The server has probably allocated all available virtual memory. To solve this problem, make more virtual memory available to your server, or reduce one or more of the following server configuration settings: nsslapd-cachesize (Database Settings - Maximum entries in cache) nsslapd-cachememsize (Database Settings - Memory available for cache) nsslapd-dbcachesize (LDBM Plug-in Settings - Maximum cache size) nsslapd-import-cachesize (LDBM Plug-in Settings - Import cache size). Can't recover; calling exit(1).
[root@idmipa04 slapd-MWS-MDS-XYZ]# free -g total used free shared buff/cache available Mem: 1 0 1 0 0 1 Swap: 3 0 3 [root@idmipa04 slapd-MWS-MDS-XYZ]# free -m total used free shared buff/cache available Mem: 1838 120 1520 2 196 1574 Swap: 3967 175 3792 [root@idmipa04 slapd-MWS-MDS-XYZ]# cat /proc/meminfo |grep -Ei "inactive|active" Active: 11216 kB Inactive: 174824 kB Active(anon): 2372 kB Inactive(anon): 10396 kB Active(file): 8844 kB Inactive(file): 164428 kB [root@idmipa04 slapd-MWS-MDS-XYZ]#
Posting back in case someone else has the same issue.
rob _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
freeipa-users@lists.fedorahosted.org
-
Rob Crittenden -
TomK