Gateway RTT / packet loss
First post, please let me know if I need to include more information.
pfSense firewall is 2.0.2-RELEASE, Intel Xeon E3-1230 3.30GHz, 16GB memory, Broadcom NetXtreme 5709. I have a limiter for 300Mbps each direction.
I have a environment that handles thousands of HTTP requests per second with no specific source. Traffic comes into a VIP on the firewall and is NAT to some internal IP.
Few days ago, I started experiencing network latency issues. First thing I noticed is the WANGW and LANGW RTT go from <2ms to upwards of >100ms with 2-4% pack loss sometimes. From all the graphs, I can see the problem started on the day when PPS went over 40k (we avg 30k and this day showed max of 66K). Firewall optimization option was set to normal and we typically range from 500k-1M states, and this day I saw a vertical jump to 2M states. I raised the state table size, but no help, so I have since changed to use aggressive option, however still seeing the same problem.
Any insight to help me troubleshoot this?