Hello everone, we have an issue/problem that seems highly related to the current "timeout and offline mode behaviour". We have several SSSD 1.9.5 systems with "cache_credentials = true", which experience periodic blocking/pausing of the complete X session for several seconds (~5-10) when these machines are not connected to their home network. The user is not able to use the machine at all during these lockups, which is very annoying... These are some of the details: 1) The LDAP server has a local 192.168.x.x IP, which is resolvable globally. 2) The LDAP server is behind a firewall, dropping packets from outside networks.

3) The lockups seem to occur if the clients are not connected to this "home" network but to a different network with similar configuration (192.168.x.x, which is pretty common). I am still trying to track down the issue with "debug_level = 9" and get the logs of such a lockup. @Jakub: Could you let me know which openLDAP version you are using and suspecting to be the problem? Could you let me know how to trigger/test the issue reliably (the DNS setup you are using to trigger it)? I would like to confirm that we are having the same or possibly different problem... Best regards, Joschi Brauchle _______________________________________________ sssd-users mailing list sssd-users(a)lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-users

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlOMZLUACgkQeiVVYja6o6PoVACfXIRsk1+k4ldjTD/Tp/sJJeJR Fx8An2qk7/s7Y4kRfEXhc2lMDgI8Axsd =oNT6 -----END PGP SIGNATURE-----

On Mon, 2 Jun 2014, Stephen Gallagher wrote: > This is the real problem. If SSSD can route to the IP address, > then we have to proceed assuming that the LDAP server should be > available (thereby attempting to connect to it and perform > online authentication). There's really no way to determine ahead > of time whether the service is "supposed" to be available. > > You may want to play with the option 'ldap_opt_timeout' (see > sssd-ldap(5)). It controls how long the OpenLDAP client libraries > will wait for a response (in your case, how long it will wait > while the packets are dropped. It defaults to 6s). This should be a one off hit though, right? If I discover the LDAP server is offline, I should remember this, admittedly recheck periodically, but never cause another delay waiting for it to spring back into life. Given the way some of these laptops are used, I'd even quite like to configure it to default to this state. When I last tried this (which was a while ago) these delays would happen repeatedly, so the setup was unusable, and I had to ditch sssd on the laptop.

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlOMadkACgkQeiVVYja6o6MAcwCeNIGfCAQyhaKagU1vPt5hm1aj RroAnjebqWadxdN01yL+Sdspms1jDFVh =UAAQ -----END PGP SIGNATURE-----

On Mon, 2014-06-02 at 10:34 +0200, Joschi Brauchle wrote: > These are some of the details: > 1) The LDAP server has a local 192.168.x.x IP, which is resolvable globally. IPs in the 129.168 are not resolvable globally, they are private addresses and routing is not allowed outside private networks.

On 3 June 2014 06:25, Joschi Brauchle <joschi.brauchle(a)tum.de&gt; wrote: On 06/02/2014 05:39 PM, Simo Sorce wrote: On Mon, 2014-06-02 at 10:34 +0200, Joschi Brauchle wrote: These are some of the details: 1) The LDAP server has a local 192.168.x.x IP, which is resolvable globally. IPs in the 129.168 are not resolvable globally, they are private addresses and routing is not allowed outside private networks. What I meant is that the DNS hostname of the LDAP server 'gemini.lnt.ei.tum.de' is globally resolvable to its private '192.168.109.3' IP address. Unfortunately, this DNS setup cannot be changed.. And neither can corporate firewalls be easily changed to return failures instead of dropping packets. (our IPs are non-rfc1918) I know it's well meant advice from Simo, but reality disagrees with the implementation. Also if I read Joschi's further part of the mail correctly it might even happen if the IP stack thinks it's unroutable.

I agree it would be nice and preferable to just adjust the environment to fit this, but many users might not be at liberty to do so.

I have a script that adds blackholes to the routing table just to work around this brokenness if it detects a non-corporate network. I'd rather not roll that to the corporate landscape.

On Tue, Jun 03, 2014 at 08:40:15AM -0400, Simo Sorce wrote: > On Tue, 2014-06-03 at 06:36 +0000, Thomas Rücker wrote: >> On 3 June 2014 06:25, Joschi Brauchle <joschi.brauchle(a)tum.de&gt; wrote: >> On 06/02/2014 05:39 PM, Simo Sorce wrote: >> >> On Mon, 2014-06-02 at 10:34 +0200, Joschi Brauchle >> wrote: >> >> These are some of the details: >> 1) The LDAP server has a local 192.168.x.x IP, >> which is resolvable globally. >> >> IPs in the 129.168 are not resolvable globally, they >> are private >> addresses and routing is not allowed outside private >> networks. >> >> >> What I meant is that the DNS hostname of the LDAP server >> 'gemini.lnt.ei.tum.de' is globally resolvable to its private >> '192.168.109.3' IP address. Unfortunately, this DNS setup >> cannot be changed.. >> >> >> And neither can corporate firewalls be easily changed to return >> failures instead of dropping packets. (our IPs are non-rfc1918) I know >> it's well meant advice from Simo, but reality disagrees with the >> implementation. Also if I read Joschi's further part of the mail >> correctly it might even happen if the IP stack thinks it's unroutable. > Understandable, but there are some conditions SSSD simply cannot know > about w/o outside knowledge. > >> I agree it would be nice and preferable to just adjust the environment >> to fit this, but many users might not be at liberty to do so. > True, but then they need to be able to do something else at the system > level. > >> I have a script that adds blackholes to the routing table just to work >> around this brokenness if it detects a non-corporate network. I'd >> rather not roll that to the corporate landscape. > Well, one way could be to use a DNS name that simply does not resolve > outside your network, maybe that is also not feasible, and is subject to > caching issues too. > > Perhaps we could provide a way yo tell sssd to go permanently offline so > you can build a script to turn this on/off based on your external > knowledge of the network environment, would that work ? That's an interesting thought. Do you mean having a mode (using an option) where an offline failure would be permanent /except/ for receiving signals? Then the user could drop NM dispatcher scripts that would SIGUSR2 SSSD on joining the right network.

On 06/02/2014 05:39 PM, Simo Sorce wrote: > On Mon, 2014-06-02 at 10:34 +0200, Joschi Brauchle wrote: >> These are some of the details: >> 1) The LDAP server has a local 192.168.x.x IP, which is resolvable >> globally. > > IPs in the 129.168 are not resolvable globally, they are private > addresses and routing is not allowed outside private networks. What I meant is that the DNS hostname of the LDAP server 'gemini.lnt.ei.tum.de' is globally resolvable to its private '192.168.109.3' IP address. Unfortunately, this DNS setup cannot be changed.. So I finally have a "debug_level = 9" log output from a lockup: ---------------------------- (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sbus_message_handler] (0x4000): Received SBUS method [getAccountInfo] (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [be_get_account_info] (0x0100): Got request for [4097][1][idnumber=10011] (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [be_get_account_info] (0x0100): Request processed. Returned 1,11,Fast reply - offline (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sdap_id_op_connect_step] (0x4000): beginning to connect (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP' (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [get_server_status] (0x1000): Status of server 'gemini.lnt.ei.tum.de' is 'name resolved' (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [get_port_status] (0x1000): Port status of port 636 for server 'gemini.lnt.ei.tum.de' is 'not working' (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [get_port_status] (0x0100): Reseting the status of port 636 for server 'gemini.lnt.ei.tum.de' (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [fo_resolve_service_activate_timeout] (0x2000): Resolve timeout set to 10 seconds (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [get_server_status] (0x1000): Status of server 'gemini.lnt.ei.tum.de' is 'name resolved' (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [be_resolve_server_process] (0x1000): Saving the first resolved server (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [be_resolve_server_process] (0x0200): Found address for server gemini.lnt.ei.tum.de: [192.168.109.3] TTL 7200 (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sss_ldap_init_send] (0x4000): Using file descriptor [21] for LDAP connection. (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sss_ldap_init_send] (0x0400): Setting 6 seconds timeout for connecting (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sbus_dispatch] (0x4000): dbus conn: 18E8420 (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sbus_dispatch] (0x4000): Dispatching. (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sbus_message_handler] (0x4000): Received SBUS method [getAccountInfo] (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [be_get_account_info] (0x0100): Got request for [4097][1][name=brauchle] (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sdap_id_op_connect_step] (0x4000): waiting for connection to complete (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sbus_dispatch] (0x4000): dbus conn: 18E8420 (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sbus_dispatch] (0x4000): Dispatching. (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sbus_message_handler] (0x4000): Received SBUS method [getAccountInfo] (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [be_get_account_info] (0x0100): Got request for [4097][1][idnumber=10011] (Tue Jun 3 08:16:42 2014) [sssd[be[default]]] [sdap_id_op_connect_step] (0x4000): waiting for connection to complete (Tue Jun 3 08:16:47 2014) [sssd[be[default]]] [sbus_dispatch] (0x4000): dbus conn: 18BFD50 (Tue Jun 3 08:16:47 2014) [sssd[be[default]]] [sbus_dispatch] (0x4000): Dispatching. (Tue Jun 3 08:16:47 2014) [sssd[be[default]]] [sbus_message_handler] (0x4000): Received SBUS method [ping] (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [sdap_async_sys_connect_timeout] (0x0100): The LDAP connection timed out (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [sss_ldap_init_sys_connect_done] (0x0020): sdap_async_sys_connect request failed. (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [sdap_sys_connect_done] (0x0020): sdap_async_connect_call request failed. (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [sdap_handle_release] (0x2000): Trace: sh[0x191fdb0], connected[0], ops[(nil)], ldap[(nil)], destructor_lock[0], release_memory[0] (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [fo_set_port_status] (0x0100): Marking port 636 of server 'gemini.lnt.ei.tum.de' as 'not working' (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP' (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [get_server_status] (0x1000): Status of server 'gemini.lnt.ei.tum.de' is 'name resolved' (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [get_port_status] (0x1000): Port status of port 636 for server 'gemini.lnt.ei.tum.de' is 'not working' (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [fo_resolve_service_send] (0x0020): No available servers for service 'LDAP' (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [be_resolve_server_done] (0x1000): Server resolution failed: 5 (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [sdap_id_op_connect_done] (0x0020): Failed to connect, going offline (5 [Input/output error]) (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [be_mark_offline] (0x2000): Going offline! (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [be_run_offline_cb] (0x0080): Going offline. Running callbacks. (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [sdap_id_op_connect_done] (0x4000): notify offline to op #1 (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [sdap_id_op_connect_done] (0x4000): notify offline to op #2 (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [acctinfo_callback] (0x0100): Request processed. Returned 1,11,Offline (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [sdap_id_op_connect_done] (0x4000): notify offline to op #3 (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [acctinfo_callback] (0x0100): Request processed. Returned 1,11,Offline (Tue Jun 3 08:16:48 2014) [sssd[be[default]]] [sdap_id_release_conn_data] (0x4000): releasing unused connection ---------------------------- As the logs are showing, the system was completely unresponsive for about 5 seconds form 08:16:42 to 08:16:47 when the LDAP connection timed out. I am currently NOT connected to my home '192.168.109.0/24' network, but this is my current network config: ---------------------------- # ifconfig wlan0 Link encap:Ethernet HWaddr ... inet addr:192.168.1.11 Bcast:192.168.1.255 Mask:255.255.255.0 # route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default SL2141I.home 0.0.0.0 UG 0 0 0 wlan0 192.168.1.0 * 255.255.255.0 U 9 0 0 wlan0 ---------------------------- Pinging the LDAP server 'gemini.lnt.ei.tum.de' results in: ---------------------------- # ping gemini.lnt.ei.tum.de PING gemini.lnt.ei.tum.de (192.168.109.3) 56(84) bytes of data. From 62.53.2.254 icmp_seq=1 Destination Net Unreachable ---------------------------- So why does the LDAP connection not terminate immediately with the same 'destination net unreachable' status but time out instead?

So why not keep the "offline" flag set to "true" until the LDAP connection succeeds and only if so, go to online mode? As the first request (triggering the reconnection) is answered from the cache anyway, I dont see the point why one needs to keep the subsequent ones pending until the connection is established successfully...