[Fedora-directory-users] directory server setting fail to terminate idle connections

I ran into issues hitting the max filedescriptors setting and found that it was because the server never terminates idle connections. I have an idle timeout setting of 1200 seconds (20min). If I make an LDAP request from a client to the directory server, the tcp connection stays in ESTABLISHED state on the server side forever. I ran tcpdump on the client side and not a single packet of traffic was sent to the server during for hours. Any idea why this connection would not be terminated after 1200 sec?