ICMP Fragmentation Needed sent from real IP instead of VIP

corentin.deboisset

Hello,

I have two pfSense boxes used as router on my local network in high availability. A VIPs with CARP are set up on the LAN and WAN interfaces, and the local machines connected to the network use the LAN_VIP as gateway. The WAN network has a supported MTU of 1500, but the machines on the LAN network automatically have jumbo frames enabled with a MTU of 9000 (our cloud provider does this by default).

I noticed that when a VM sends a packet too large through the pfSense, an ICMP packet Fragmentation needed is sent but it originates from the real IP of the primary Pfsense, not from the VIP, so the VM doesn't act on it and sends smaller packets.

It is possible to configure the PfSense to send those ICMP from the VIP instead?

(Note: I managed to resolve the situation using MSS clamping, but the MTU issue remains for UDP for instance)

Here are some example logs for a tcpdump on the VM:

14:03:52.257176 ens3  Out IP 10.0.0.50.56666 > 141.95.161.68.https: Flags [P.], seq 1245:7219, ack 7454, win 442, length 5974
14:03:52.258122 ens3  In  IP 10.0.0.11 > 10.0.0.50: ICMP 141.95.161.68 unreachable - need to frag (mtu 1500), length 576
14:03:53.121154 ens3  Out IP 10.0.0.50.56666 > 141.95.161.68.https: Flags [P.], seq 1245:7219, ack 7454, win 442, length 5974
14:03:53.121940 ens3  In  IP 10.0.0.11 > 10.0.0.50: ICMP 141.95.161.68 unreachable - need to frag (mtu 1500), length 576

For context the IPs are following:

The primary PfSense has the real IP 10.0.0.11
The PfSense VIP is 10.0.0.1 and is the gateway for the VMs of the network
The VM where this record has been made is 10.0.0.50

We can see that the packets are too large (length 5974), the PfSense sends an ICMP from its real IP but it's discarded: the next packet has the same size.

corentin.deboisset

This post is deleted!