Routing issue on LAN interface

ssheikh

I have 3 offices. One each in VA, TX, and CA. The three offices are hooked up via MPLS. The MPLS routers are managed by the service provider and they are:

CA: 10.100.4.4/23
VA: 10.100.6.4/23
TX: 10.100.8.4/23

Each office has its own internet connection and a pfSense firewall running 1.2.3-release. The respective pfSense firewall is the default gateway on all the systems in an office. pfSense LAN ips are:

CA: 10.100.4.2/23
VA: 10.100.6.2/23
TX: 10.100.8.2/23

Each pfSense device has a static route on it that routes data destined for the other offices through the MPLS router (i.e. the .4 address.) So for example, on the VA pfSense, my static routes look as follows:

# netstat -rn -f inet | grep UGS
default            A.B.C.D      UGS         0 63558580   fxp1
10.100.4.0/23      10.100.6.4         UGS         0  5493991   fxp0
10.100.8.0/23      10.100.6.4         UGS         0 87553042   fxp0

In System | Advanced "Bypass firewall rules for traffic on the same interface" is checked.

The problem is that connections to machines in the other offices is unreliable. TCP sessions reset routinely and UDP packets start dropping out of the blue.

For testing if I added specific routes on two test machines in different offices to talk to each other via the MPLS router eliminating the hop through the pfSense machine. My connections stay solid and I do not see any TCP disconnects.

Any ideas why pfSense is having problems forwarding traffic through the LAN interface to the MPLS router at the .4 address? Suggestions on how to troubleshoot?

Thanks,

Shahid