Large packet sizes fail to send to internet

oggsct

Recently I discovered my pfsense no longer fragments packets properly. As an example if I ping with a packet size greater than 1472, the request times out. I tried to reboot and ran into a different issue (see https://forum.netgate.com/topic/187550/exceeded-input-buffer-on-reboot) so I restored to 2.6.0 (from 2.7.0) using a USB thumb drive and older config from that thumb drive. To my surprise I could suddently ping beyond 1472 packets again. I did a side by side comparison of the configs and don't see an obvious setting that would be different. The only noticeable differences are the installation of wireguard, and 2.7.0 being based on FreeBSD 14. I looked through the release notes of pfsense and don't see anything obvious that matches my issue, I have skimmed FreeBSD 14 release notes but I don't see anything in there as well. The hardware is a Supermicro 1U Server X10SLH-N6 E3-1231v3

Other tests I have tried

From pfsense ssh ping -s 4000 1,1,1,1 - works
From pfsense ssh ping -s 4000 -S LAN_IP 1.1.1.1 - works
From pfsense ssh ping -s 4000 -S WAN_IP 1.1.1.1 - works
ping (4000 size) across vlans from pfsense IP to pfsense IP - works
ping (4000 size) from desktop on vlan A to device on vlan B - works
ping (4000 size) from desktop on vlan A to vlan A IP on pfsense - works
ping (4000 size) from desktop on vlan A to vlan B IP on pfsense - works

Yes I realize it makes it seem like it is a firewall rule going out to WAN but it happens on both WAN connections from any internal vlan. Again the problem does not exist in 2.6.0 with a restored config from last year.

stephenw10

You've labeled all those tests as 'works'. So that was in 2.6? What doesn't work?

oggsct

All of the other tests are in 2.7.2, tried to edit the post to reflect that but too much time had past.

@stephenw10 said in Large packet sizes fail to send to internet:

You've labeled all those tests as 'works'. So that was in 2.6? What doesn't work?

stephenw10

So the only thing that doesn't work is pinging with some large packet size from a host behind pfSense to some external target?

johnpoz

@oggsct I just fired up my 2.7.2 vm, not having any issues with large pings.

$ ping -l 4000 1.1.1.1

Pinging 1.1.1.1 with 4000 bytes of data:
Reply from 1.1.1.1: bytes=4000 time=22ms TTL=56
Reply from 1.1.1.1: bytes=4000 time=21ms TTL=56
Reply from 1.1.1.1: bytes=4000 time=21ms TTL=56

oggsct

@stephenw10 said in Large packet sizes fail to send to internet:

So the only thing that doesn't work is pinging with some large packet size from a host behind pfSense to some external target?

Correct and since your reply I have done the followings tests all on 2.7..2

Clean install pfsense, no config restore, simply setup into an igb interface for both WAN and LAN. - Success for 4000 size packets

Restored previous config via web interface and rebooted

Add new empty interface on igb0 for a LAN network (direct connect), basic static IP, basic dhcp, simple firewall rule allowing everything - Success for 4000 size packets
Test from host on LAN interface (through switch) of restored config - Failure for anything past 1472
Uninstall wireguard and repeat above test - Failure for anything past 1472
Add new empty interface on ix0 for a LAN network (direct connect), basic static IP, basic dhcp, simple firewall rule allowing everything - Success for 4000 size packets
Use same settings as previous but modify firewall rule to use WAN1 to WAN2 failover routing - Success for 4000 size packets
Use same settings as previous but modify firewall rule to use WAN2 to WAN1failover routing - Success for 4000 size packets
Connect to an existing wifi network that is guest - Success for 4000 size packets

The only commonality I am now seeing is the new networks and the existing wifi network are not included in any VPN configuration. I have tried stopping IPSec and Wireguard services and there is no change to the exiting primary LAN subnet's ability to ping beyond 1472, same with secondary LAN.

stephenw10

I'd try running pcaps and see what's actually being sent and where.

oggsct

@stephenw10 said in Large packet sizes fail to send to internet:

I'd try running pcaps and see what's actually being sent and where.

I have done a number of packet captures and I can see the packet on the LAN side of the interface and then if I capture on the WAN interface it doesn't show the WAN IP going to the ping destination, it shows the LAN IP going to the ping destination, despite being on the WAN interface.

In contrast the LAN segment (not in any VLAN config) that is successful shows the large icmp request and response on LAN interface of pfsense, and on WAN interface I can see the request/response with proper WAN IP translations.

Somewhat SOLVED
I dug into pfctl to see what it was writing for rules and one that stuck out to me was:

scrub from any to <vpn_networks> max-mss 1300 fragment no reassemble
scrub from <vpn_networks> to any max-mss 1300 fragment no reassemble

Based on that I found the Reassemble IP Fragments until they form a complete packet under System -> Advanced -> Firewall & NAT -> VPN Packet Processing. Once I checked that box I was able to ping larger than 1472 from all LAN segments out to the internet. (See https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html?highlight=fragments#ip-fragment-reassemble)

The questions that remain

Why would it fail to perform NAT translation?
Why is a VPN Packet Processing setting affecting LAN to WAN traffic?
How sever of a memory cost is it going to be if it does this on so many packets?
Will this affect my IPSec tunnels? I have a mix of VTI style route based and more traditional policy based

w0w

@oggsct
Not sure, really, but since 2.7 is based on freebsd 14 then maybe this
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276856

stephenw10

Hmm, that option should only apply to VPN traffic, things that are matched by those 'vpn_networks' rules. Do you have VPNs configured that could be over-matching?

Otherwise pfscrub would usually reassemble everything. You could also try disabling pfscrub and it would then pass fragments. But that also disables other things so I wouldn't do that permanently.

I wouldn't be concerned by memory usage when re-assembling packets. As I said that is the default.

oggsct

@w0w said in Large packet sizes fail to send to internet:

@oggsct
Not sure, really, but since 2.7 is based on freebsd 14 then maybe this
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276856

Yes it seems that may be related but the reassembly setting for VPN seems suspect too since I wouldn't expect local networks in the vpn_networks variable of pf

oggsct

@stephenw10 said in Large packet sizes fail to send to internet:

Hmm, that option should only apply to VPN traffic, things that are matched by those 'vpn_networks' rules. Do you have VPNs configured that could be over-matching?

Otherwise pfscrub would usually reassemble everything. You could also try disabling pfscrub and it would then pass fragments. But that also disables other things so I wouldn't do that permanently.

I wouldn't be concerned by memory usage when re-assembling packets. As I said that is the default.

I will go review the VPN configs but I don't believe they are overmatching. The pfsense in question utilizes a 10.X.Y.Z format such that X is the location (0 for this pfsense), Y is the vlan and Z is the device IP. With that said there is a 10.0.0.0/16 in that vpn_networks which certainly covers all of the internal networks at this location, excluding a guest VLAN that is 192.168.20.0/24, which is the one that has worked t hroughout. They are all mostly policy-based IPSec with 2 route-based IPSec. I think what may be more suspect is the wireguard settings. Is there code that I can look at to see how pfSense is building the vpn_networks for pf?

UPDATE: I went through all IPSec configs and found a tunnel that is disabled for both P1 and P2 and the P2 policy had the local and remote reversed, which is what put the 10.0.0.0/16 into the vpn_networks. Now what is still surprising is that a disabled policy is affecting the running configuration of pf.

stephenw10

Hmm, now that does sounds familiar....

stephenw10

Hmm, can't find a bug for that now. However I can say it doesn't happen in 24.03 so it's fixed now.

oggsct

@stephenw10 said in Large packet sizes fail to send to internet:

Hmm, can't find a bug for that now. However I can say it doesn't happen in 24.03 so it's fixed now.

I am not sure I can get to 24.03. I lost track of the policy for non-netgate hardware on pfSense plus. I also have the other open ticket I am chasing that may be tied to upgrades. I can possibly try to get to 23.09.1 but I would like to know I will have a future path on that codebase. I also don't see a way to download USB installs for that codebase. I already have an activation token for this pfsense but had issues last year trying to go to 23.x so I stayed where I was for the time being.

stephenw10

If you can see 23.0X available you would be able to reach 24.03-RC (or release by then hopefully).

oggsct

@stephenw10 said in Large packet sizes fail to send to internet:

If you can see 23.0X available you would be able to reach 24.03-RC (or release by then hopefully).

Yes I can see it as an option. The concern lies in I have had to do re-installs from USB due to another bug I am still troubleshooting. Going to 23.0X doesn't seem to have a easy download for a USB recovery install. If that is still available, even by support request, I would simply keep that on hand so I know that I can recover at any time instead of going all the way back to 2.7.2 or similar.

stephenw10

You can use the new Net Installer to install Plus directly if the NDI is eligible.

oggsct

@stephenw10 said in Large packet sizes fail to send to internet:

You can use the new Net Installer to install Plus directly if the NDI is eligible.

I had missed that post when it came out. That certainly resolves my concerns once it makes it out of beta. In the meantime it looks like things are stable again and we found the oddities that were causing issues. Thank you for your assistance.