Slow traffic in one direction
I have 4 locations with T1 connections to the internet, running pfSense 2.0.1 on Alix boxes. All four locations have IPsec VPNs between them. Everything has been running smoothly for over a year, until last week, when I started having problems between sites A and B. Whenever there would be any significant traffic from B to A, the bandwidth from A to B would plummet, and I'd start seeing connection errors on client workstations. I installed iperf on servers in each location to measure the extent of the problem. Between all pairs other than A and B, I could run bi-directional simultaneous tests with iperf and get 1.25 - 1.50 Mb/sec throughput every time. Between A and B, if I ran the tests sequentially, I could get similar numbers. However, if I ran the tests sequentially, I'd get the same 1.25 - 1.50 Mb/sec from B to A, but only 10 - 100 kb/sec from A to B. Restarting IPsec on the router at B seems to clear up the problem for a short period of time, but the problem always comes back. It might also be coincidence, because I think the problem has come and gone without restarting IPsec as well.
I forwarded port 5001 at each end, and tested iperf over a plain internet connection without going through the VPN, and the problem never manifested itself in that configuration. At first, I thought that indicated that it was a problem with the firewall itself; but now I'm starting to wonder if there's something wrong with the internet connection between the two locations that is causing the VPN to misbehave, but I have no idea what to look for. The two sites have different ISPs, so I need to have a good idea of what's wrong before I go down that path.
I also investigated whether I have my VPN misconfigured. I've deleted and reconfigured the IPsec configurations at both ends several times; compared the settings to the other locations that aren't having problems, and even restored the configuration from a backup that predated the start of this issue, to no avail.
Does anybody have an idea of where I should look next?
MTU problems maybe?
You can try the "MSS clamping on VPN traffic" option on site A.
Otherwise, do packet capture on the WAN while testing the port forwarding way and a pcap on the IPSec interface when testing iperf through the tunnel. You should see what's happening