Strange issue with IPv4 packet fragmentation

ChrisJenk

I've been experimenting with MTU path discovery, MSS sees etc. (only from a measurement perspective, not changing any config values anywhere).

I came across these test sites published by CLoudflare:

http://icmpcheck.popcount.org

http://icmpcheckv6.popcount.org

If I go to the first site (IPv4) I get:

ICMP path MTU packet delivery
✓ All good! ICMP path MTU message was successfully delivered to you.

In this test your browser uploads a sizeable payload to our server, while your long TCP segments are being dropped. Our server will generate an ICMP message asking you to reduce the MTU to a laughingly small value.

ICMP is critical to a well functioning Internet connections, yet networks are often misconfigured. Broken ICMP (being an ICMP blackhole) can lead to serious issues:

Slow DNS lookups.
Glitches and lag when playing games using UDP.
TCP connections failing occasionally for no apparent reason.

IP fragmented packet delivery
✗ The request timed out. Looks like IP fragments failed to be delivered to you.

In this test we send a large payload from the server, to your browser. Each TCP segment sent by the server will be fragmented into multiple IP packets.

A failure of this test means:

Some UDP based protocols (like games, NTP) might occasionally fail.
Certain DNS queries may time out, slowing down your internet experience.
Your ISP, firewall or CPE is not supporting basic Internet Protocol features.

Whereas if I go to the second one (IPv6) I get:

ICMP path MTU packet delivery
✓ All good! ICMP path MTU message was successfully delivered to you.
...
IP fragmented packet delivery
✓ All good! IP fragments were successfully delivered to your host.
...

This seems to suggest I might have some issue in regard to fragmented IPv4 packets. However I am not aware of any issues and everything seems to be working as it should (as far as I can tell!).

Is the 'error' from the IPv4 test bogus? Or...? Are there even any pfSense settings that are relevant in this regard?

All thoughts appreciated...

stephenw10

If Path MTU discovery is working then you shouldn't see fragmented traffic for most things. Almost everything. Also if your actual path MTU is a standard size you might not see any issues even for things that don't try to discover it, like UDP streams. But you should be able to pass fragmented packets.
Try running very large pings to something external like:

[23.09.1-RELEASE][admin@fw1.stevew.lan]/root: ping -s 1600 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 1600 data bytes
1608 bytes from 1.1.1.1: icmp_seq=0 ttl=57 time=7.185 ms
1608 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=7.021 ms
1608 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=6.790 ms
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 6.790/6.999/7.185/0.162 ms

Since those are bigger than the link MTU (1500) they are fragmented. A packet capture shows that:

15:48:15.520278 IP 192.168.1.5 > 1.1.1.1: ICMP echo request, id 20946, seq 0, length 1480
15:48:15.520288 IP 192.168.1.5 > 1.1.1.1: ip-proto-1
15:48:15.527148 IP 1.1.1.1 > 192.168.1.5: ICMP echo reply, id 20946, seq 0, length 1480
15:48:15.527162 IP 1.1.1.1 > 192.168.1.5: ip-proto-1
15:48:16.521571 IP 192.168.1.5 > 1.1.1.1: ICMP echo request, id 20946, seq 1, length 1480
15:48:16.521576 IP 192.168.1.5 > 1.1.1.1: ip-proto-1
15:48:16.528868 IP 1.1.1.1 > 192.168.1.5: ICMP echo reply, id 20946, seq 1, length 1480
15:48:16.528881 IP 1.1.1.1 > 192.168.1.5: ip-proto-1
15:48:17.522566 IP 192.168.1.5 > 1.1.1.1: ICMP echo request, id 20946, seq 2, length 1480
15:48:17.522572 IP 192.168.1.5 > 1.1.1.1: ip-proto-1
15:48:17.529618 IP 1.1.1.1 > 192.168.1.5: ICMP echo reply, id 20946, seq 2, length 1480
15:48:17.529628 IP 1.1.1.1 > 192.168.1.5: ip-proto-1

Steve

JKnott

@ChrisJenk

Any chance you're blocking ICMP on IPv4?

Here are my ICMP rules on the WAN:

As you can see, I'm not blocking anything. Some people think blocking ICMP is a security feature, but not really and it will really kill things on IPv6. At most, you might want to block pings.

ChrisJenk

@JKnott I have the same rules and ICMP traffic (v4 and v6) flows freely. The more I think about it the more I think that the Cloudflare test may be somewhat flawed (which would be unusual for them).

Squuiid

I get the same for both IPv4 and 6…

ICMP path MTU packet delivery
✓ All good! ICMP path MTU message was successfully delivered to you.

And

IP fragmented packet delivery
✗ The request timed out. Looks like IP fragments failed to be delivered to you.

ChrisJenk

@Squuiid UPDATE: Since changing my ISP several weeks ago (no change to pfSense configuration) both tests now work for me. So whatever may original problem with IPv4 was, it seems it was probably something to do with my ISP's network...

adude42069

Got the same issue, except I get the same result on both IPv4 and IPv6.

I even added the same rule as JKnott

@JKnott said in Strange issue with IPv4 packet fragmentation:

but no difference. Is it safe to say it's ISP related? Using their mobile network, the issue could not be reproduced, as all tests succeed. This could also mean their mobile network is configured correctly and their fixed internet is not, idk.

Firewall was also temporably disabled on the host. The tests were also performed with an android device, exact same issue. First test passes (v4 + v6), second tests fails (v4 + v6).
It's very weird. Am I missing something, or am I overthinking it? Simply opening ICMP is all that's needed?

Thank you

johnpoz

@adude42069 You really shouldn't have to open up all of icmp.. I don't have all open, only echo.

And no rules should be required to be honest.. The required rules for IPv6 are hidden and allowed look in your /tmp/rules.debug to see them.

You have an isp issue.. Unless you have really jacked up something in the config. Because it should work out of the box for fragmented packets. Have you messed with Firewall Maximum Fragment Entries, in the advanced firewall & nat section?

I just ran that test and works just fine..

" All good! IP fragments were successfully delivered to your host."

adude42069

@johnpoz said in Strange issue with IPv4 packet fragmentation:

@adude42069 You really shouldn't have to open up all of icmp.. I don't have all open, only echo.

To be more precise, I added that rule temporarely for testing, but removed it afterwards.

@johnpoz said in Strange issue with IPv4 packet fragmentation:

And no rules should be required to be honest.. The required rules for IPv6 are hidden and allowed look in your /tmp/rules.debug to see them.

Did not know they were hidden rules. Will look into that, thanks!

@johnpoz said in Strange issue with IPv4 packet fragmentation:

Have you messed with Firewall Maximum Fragment Entries, in the advanced firewall & nat section?

Not that I know of. I planned to rebuild my config "soon", as it was just upgrade after upgrade from 2.3 or something, but I did not actively change those settings.
These are the current settings:
packet processing settings screenshot