CARP and asymetric routing issues: ICMP redirect + dropped connection
-
Hi community! We are having a weird issue here (both version 2.1 and 2.1.3).
We have a couple of servers here that has a pfSense CARP VIP as default gateway, and it leads us sometimes to connection to be dropped by pfSense during bulk TCP transfer, when communicating with another VLAN. We have an asymetrical routing setup, with static routes in the pfSense that points to our layer 3 switch. Of course, the Bypass firewall rules for traffic on the same interface checkbox is active, and problem is occuring whether scrubbing is enabled or not.
First issue: the pfSense does sends ICMP redirect properly for static routes that lands within the same interface as the sending device. Unfortunately, the ICMP redirect is coming from the pfsense Interface's main IP address, instead of the CARP IP address. Consequence: the ICMP redirect packet is being rejected by most kernels. Is there a way to fix that?
Second issue: TCP connections works properly up to a certain point. The pfSense stops forwarding to the layer3 switch after a certain amount of payload data sent through the TCP connection (around 90 kB). We are able to reproduce a connection drop on demnad (puppet agent to puppet master communication).
If I insert a firewall rule on the LAN interface that matches puppet server TCP port, and select Sloppy state, the problem is gone. So I don't understand why state matters in that situation, since I checked the bypass firewall rules for traffic on the same interface. Any idea?
Thanks