Large packet sizes fail to send to internet
-
@stephenw10 said in Large packet sizes fail to send to internet:
I'd try running pcaps and see what's actually being sent and where.
I have done a number of packet captures and I can see the packet on the LAN side of the interface and then if I capture on the WAN interface it doesn't show the WAN IP going to the ping destination, it shows the LAN IP going to the ping destination, despite being on the WAN interface.
In contrast the LAN segment (not in any VLAN config) that is successful shows the large icmp request and response on LAN interface of pfsense, and on WAN interface I can see the request/response with proper WAN IP translations.
Somewhat SOLVED
I dug into pfctl to see what it was writing for rules and one that stuck out to me was:scrub from any to <vpn_networks> max-mss 1300 fragment no reassemble scrub from <vpn_networks> to any max-mss 1300 fragment no reassembleBased on that I found the
Reassemble IP Fragments until they form a complete packetunderSystem -> Advanced -> Firewall & NAT -> VPN Packet Processing. Once I checked that box I was able to ping larger than 1472 from all LAN segments out to the internet. (See https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html?highlight=fragments#ip-fragment-reassemble)The questions that remain
- Why would it fail to perform NAT translation?
- Why is a VPN Packet Processing setting affecting LAN to WAN traffic?
- How sever of a memory cost is it going to be if it does this on so many packets?
- Will this affect my IPSec tunnels? I have a mix of VTI style route based and more traditional policy based
-
@oggsct
Not sure, really, but since 2.7 is based on freebsd 14 then maybe this
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276856 -
Hmm, that option should only apply to VPN traffic, things that are matched by those 'vpn_networks' rules. Do you have VPNs configured that could be over-matching?
Otherwise pfscrub would usually reassemble everything. You could also try disabling pfscrub and it would then pass fragments. But that also disables other things so I wouldn't do that permanently.
I wouldn't be concerned by memory usage when re-assembling packets. As I said that is the default.
-
@w0w said in Large packet sizes fail to send to internet:
@oggsct
Not sure, really, but since 2.7 is based on freebsd 14 then maybe this
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276856Yes it seems that may be related but the reassembly setting for VPN seems suspect too since I wouldn't expect local networks in the vpn_networks variable of pf
-
@stephenw10 said in Large packet sizes fail to send to internet:
Hmm, that option should only apply to VPN traffic, things that are matched by those 'vpn_networks' rules. Do you have VPNs configured that could be over-matching?
Otherwise pfscrub would usually reassemble everything. You could also try disabling pfscrub and it would then pass fragments. But that also disables other things so I wouldn't do that permanently.
I wouldn't be concerned by memory usage when re-assembling packets. As I said that is the default.
I will go review the VPN configs but I don't believe they are overmatching. The pfsense in question utilizes a 10.X.Y.Z format such that X is the location (0 for this pfsense), Y is the vlan and Z is the device IP. With that said there is a 10.0.0.0/16 in that
vpn_networkswhich certainly covers all of the internal networks at this location, excluding a guest VLAN that is 192.168.20.0/24, which is the one that has worked t hroughout. They are all mostly policy-based IPSec with 2 route-based IPSec. I think what may be more suspect is the wireguard settings. Is there code that I can look at to see how pfSense is building thevpn_networksfor pf?UPDATE: I went through all IPSec configs and found a tunnel that is disabled for both P1 and P2 and the P2 policy had the local and remote reversed, which is what put the 10.0.0.0/16 into the
vpn_networks. Now what is still surprising is that a disabled policy is affecting the running configuration of pf. -
Hmm, now that does sounds familiar....
-
Hmm, can't find a bug for that now. However I can say it doesn't happen in 24.03 so it's fixed now.
-
@stephenw10 said in Large packet sizes fail to send to internet:
Hmm, can't find a bug for that now. However I can say it doesn't happen in 24.03 so it's fixed now.
I am not sure I can get to 24.03. I lost track of the policy for non-netgate hardware on pfSense plus. I also have the other open ticket I am chasing that may be tied to upgrades. I can possibly try to get to 23.09.1 but I would like to know I will have a future path on that codebase. I also don't see a way to download USB installs for that codebase. I already have an activation token for this pfsense but had issues last year trying to go to 23.x so I stayed where I was for the time being.
-
If you can see 23.0X available you would be able to reach 24.03-RC (or release by then hopefully).
-
@stephenw10 said in Large packet sizes fail to send to internet:
If you can see 23.0X available you would be able to reach 24.03-RC (or release by then hopefully).
Yes I can see it as an option. The concern lies in I have had to do re-installs from USB due to another bug I am still troubleshooting. Going to 23.0X doesn't seem to have a easy download for a USB recovery install. If that is still available, even by support request, I would simply keep that on hand so I know that I can recover at any time instead of going all the way back to 2.7.2 or similar.
-
You can use the new Net Installer to install Plus directly if the NDI is eligible.
-
@stephenw10 said in Large packet sizes fail to send to internet:
You can use the new Net Installer to install Plus directly if the NDI is eligible.
I had missed that post when it came out. That certainly resolves my concerns once it makes it out of beta. In the meantime it looks like things are stable again and we found the oddities that were causing issues. Thank you for your assistance.
