Super High Latency From Host to Host

[fannet]

We are seeing crazy high latency (25-45MS) pinging from our PFSENSE gateway to any other host (and vice versa) on the same layer-2 domain (same switch). The machine has 16 cores (AMD) 32 GB ram and a dual 10gb NIC. The total traffic going through the NIC is less than 1 gigabit/s. We have NAT enabled and have about 1500 users going through the box. Between any other two hosts on the same switch the ping is < 0.9ms

Any suggestions?

Some information from the box:

State Table                          Total             Rate
 current entries                   220113              
 searches                     26480915673       182833.8/s
 inserts                        454504314         3138.1/s
 removals                       454284201         3136.5/s
ounters
 match                          602771956         4160.0/s
 bad-offset                             0            0.0/s
 fragment                         1535799           10.6/s
 short                               4776            0.0/s
 normalize                          21462            0.1/s
 memory                          80657911          556.7/s
 bad-timestamp                          0            0.0/s
 congestion                             0            0.0/s
 ip-option                          27088            0.2/s
 proto-cksum                        68710            0.5/s
 state-mismatch                   1629077           11.2/s
 state-insert                         378            0.0/s
 state-limit                            0            0.0/s
 src-limit                              0            0.0/s
 synproxy                               0            0.0/s
 divert                                 0            0.0/s

load averages:  0.55,  0.64,  0.54  up 1+16:23:01    10:52:54
238 processes: 18 running, 147 sleeping, 73 waiting

Mem: 383M Active, 29M Inact, 807M Wired, 132K Cache, 66M Buf, 30G Free
Swap: 32G Total, 32G Free

[Lee Sharp]

Same behaviour on both LAN and WAN?  What if you ping the WAN IP from the LAN interface on the firewall?  And is it the same for a client on the LAN pinging IPs on the WAN subnet?

[dhatz]

@fannet:

We are seeing crazy high latency (25-45MS) pinging from our PFSENSE gateway to any other host (and vice versa) on the same layer-2 domain (same switch). The machine has 16 cores (AMD) 32 GB ram and a dual 10gb NIC. The total traffic going through the NIC is less than 1 gigabit/s. We have NAT enabled and have about 1500 users going through the box. Between any other two hosts on the same switch the ping is < 0.9ms

Any suggestions?

Did your particular hardware configuration ever work well in the past ?