Firewall Trouble: Dropped Sessions versus High Latency
Bill Alderson, Technology Consulting Officer, NetQoS, Inc.

We have seen an alarming number of firewalls that maintain state information for only 5 minutes of inactivity and then lose important connections to systems across firewalls. We have also seen firewalls that maintain state information for so long (up to 1 hour or more) that latency through the firewall becomes so high that performance suffers.

What is happening? Why do some firewalls keep state information for long periods and others for only a few minutes?

If your web server accesses your database server across a firewall and is inactive for 5 minutes, the firewall drops the connection. The web server must start a new connection or your application fails intermittently after 5 minutes of inactivity.

To combat this problem, security folks simply increase the timeout from a default of 5 minutes of holding state information to something higher. After they do this, the number of sessions in the state cache is so high that lookups delay packets through the firewall.

Here is the scoop. TCP sessions use a 2-hour keep-alive-by-RFC default. For every 2 hours of no activity, a TCP session sends a TCP-ACK and the other side replies with an ACK to keep the session open between them. Because firewalls have a short period of state information, sessions drop well before the 2 hour keep-alive occurs.

To fix problems of high latency through the firewall and dropped connections between inactive TCP sessions, do the following: Recommend that your desktops change their TCP keep-alive value from 2 hours to 3 minutes. Great solution, right? All you have to do is touch all your end stations. Wrong! The solution is to change the TCP keep-alive only on the server. This way You have to change only your servers and not your end stations, some of which you might not even manage if they are Internet clients.

By changing your server's TCP keep-alive, the sessions initiates an ACK...ACK exchange that keeps the state alive for however long the session is connected. You can then lower the time you maintain the state cache to something more reasonable to reduce firewall latency.