IPsec not allowing multiple simultaneous protocols

[ervetzin]

I’m a bit of an IPSec novice, but at least somewhat experienced with networking in general.  My previous attempts to use ipsec for site-to-site have “just worked.”  However, this time I am trying to connect to a non-pfsense firewall and am having some rather… unusual issues.  I’ve done my best to run this down online, but come up empty.  Hopefully, one of you can point me in the right direction.

Here’s the basics of the situation:

Site 1 (me) pfsense 2.1  - Site 2 (them) Fortigate firewall

IpSec appears to be the only VPN option they have available.

We have set up a tunnel which appears to connect OK.

Pings to the far host seem to work (apart from the first couple of packets frequently dropping)

Other protocols have seemed intermittent (they’ll work sometimes and then stop for a while).

I’ve tried numerous “tweaks” I have seen online (listed below) with no positive results.

Today I noticed (followed by throwing up my hands & typing this) that it appears that the VPN only allows one type of connection/protocol at a time.  That is, in my tests, I could get ping or nslookup to work but not both at the same time (The other one just times out).  When I stopped using one, the other starts working after a minute or so.

I have done packet dumps & I can see what appear to be responses coming back from the other system for everything (as ESP packets).  However, they do not appear to make it back out the internal interface (ie. they appear to die in the pfSense).  Is there any way to see what is happening with those packets within the pfSense?

If I restart raccoon, everything seems to work for a few seconds (~10).

Here is what I have tried, based on various online troubleshooting lists:

Turned Nat Translation on/off on both sides
  Enabled/disablded DPD
  Added/removed keepalive
  Switched from Main to Aggressive mode
  Removed all but one phase 2 instance
  Verified keys and timeouts match on both sides
  Turned on raccoon debugging to look for any additional errors (haven’t found anything)
  Enabled MMS clamping & turned it down to 1000
  Replaced the NIC cards

Other settings:

3Des encryption
  SHA1 Hash
  DHKey Group 2
  Default Policy & Proposal checking
  Pfs keygroup is off.

Thanks for your help.