Firewall apparently dropping reply fragments
linuxclient –> pfsense --> JuniperFW - - - (VPN) - - - JuniperFW --> DellDrac5
The problem appears to be that pfsense is dropping replies to largish packets that have been fragmented in transit. This has been tested using "ping -s 1419 -M dont"
Having done some packet captures on both interfaces of the pfsense machine, I can see the pings going out and a fragmented reply returning (which pfsense appears to reassemble). However at this point pfsense does 1 of 2 things:
1. if scrubbing is left enabled, the reply packet is (presumably) scrubbed, and passed, however doing this mucks up the checksum and the client doesn't recognise it as a reply.
2. if scrubbing is disabled the reply packet is dropped.
I have a pcap file from the WAN interface if that helps.
The scrubbing implies to me that pfsense thought that there was something wrong with the packet, but I don't know what, could someone more bsd leaning help me out?
This pfsense cluster replaced a rather old and crusty Linux box which never had this problem. I've been really impressed with pfsense so far and I would really like to solve this niggly problem.
Do you have checksum offloading enabled on your nic?
If you do, outbound packets will fail cksum when viewed with tcpdump. Th nic should put the correct checksum in the packet which happens after you see the packet with tcpdump. If you do a dump on the client machine, do you see a checksum problem on the received packet there?
You can disable checksums using ifconfig. Something like: ifconfig bge0 -rxcsum -txcsum to see if that helps.
System -> Advanced has an option for disabling offload Csum. Might be worth a shot to enable that option.