Pfsense not responding to large packet pings

GemeenAapje · Sep 16, 2021, 2:18 PM

Thanks for the replies.

I think the problem is with the speed reassembling the packets. I don't know why though.

Forgot to mention I have a HP NIC:
https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=c04111479&doctype=quickspecs&doclang=EN_US&searchquery=&cc=th&lc=en

It has 2 10gbps SFP+ adapters (1 fibre and 1 ethernet).

I have the hardware offload settings disabled. I think that was default.

I'm not actually sure at what level the packets are reassembled if fragmented. Can anyone advise?

Gertjan · Sep 16, 2021, 2:44 PM

@gemeenaapje

Read : https://en.wikipedia.org/wiki/OSI_model : Layer 3 - Network Layer.

stephenw10 · Sep 16, 2021, 3:12 PM

Mmm, I'd guess it exhausted the frags table if you're sending a lot of pings. Though that is normally logged in the system log.

You can increase it in Sys > Adv > Firewall > 'Firewall Maximum Fragment Entries'.
5000 is normally more than sufficient though.

Why are you sending huge packets like that?

Steve

johnpoz · Sep 16, 2021, 3:41 PM

@stephenw10 said in Pfsense not responding to large packet pings:

Why are you sending huge packets like that?

Exactly

🔒 Log in to view

GemeenAapje · Sep 17, 2021, 6:12 AM

@stephenw10
Morning
I'm trying to do some testing for my day job. I support a bunch of products for hospitals, one of which is very old (like 25 years old foundation software). It's known to have problems when large packets don't make it through the switches.
I also found it weird, so I started to do some checks on my own network at home.
To my surprise I found that I also had problems with fragmented packets not being reassembled.

Now, if I can figure out how to fix it on my own network at home, I'll be better positioned to help our customer out. Not only that but I will have learned something new.

Thanks

Gertjan · Sep 17, 2021, 6:30 AM

@gemeenaapje said in Pfsense not responding to large packet pings:

It's known to have problems when large packets don't make it through the switches.

That is, if a switch support 'RFC' Jumbo frames, then you'll find it in their product description. I can imagine some just don't.

johnpoz · Sep 17, 2021, 10:28 AM

@gertjan 25 year old software wouldn't be doing jumbo ;)

@GemeenAapje I don't think the switch would be the problem, it wouldn't be fragmenting or reassembling - so its not the switches that would be an issue.. It would be the routers. Or the end device.

The device putting the data on the wire would break it up according to its mtu. If it put larger sized than the switch supports ie a jumbo then yeah that would be problematic. But you testing large pings to pfsense, wanting an answer - isn't the switch having an issue.

As you see on the sniff I did on pfsense - he got the fragments at 1514. A better test might be to send the packets to something through the router with such a large size..

JKnott · Sep 17, 2021, 10:46 AM

@gemeenaapje said in Pfsense not responding to large packet pings:

It's known to have problems when large packets don't make it through the switches.

Why is it sending large packets? Is it using jumbo frames? What is the MTU? Other than on token ring, back when I was at IBM, I've never seen MTUs set beyond 1500, except for my own testing. If it is using jumbo frames, then you have to make sure the switch can handle them. IIRC, when jumbo frames are used, the usual MTU is 9000. Even then, you have to make sure everything on the local network can handle that MTU.

Gertjan · Sep 17, 2021, 12:32 PM

@johnpoz said in Pfsense not responding to large packet pings:

25 year old software wouldn't be doing jumbo ;)

25 year old software isn't jumbo aware. Software from 2021 : same thing.
The ISO network stack is far older then that. I recall, somewhere in the eighties, @school, that they started to tell me about this new 7 layer model (some say 8 layers) thing.
So, the program would hand over 'the file' to be transmitted to the OS.
And deep down somewhere, the data stream is cut down in chunks of XX bytes, as that is the way how info is send over.
The program doesn't need to know about headers, sessions, MAC addresses or even IP addresses. It doesn't care that 'TCP' or IPv4 or IPv6 is used.

I guess @GemeenAapje is talking about Jumbo frames because he want to push to the limit his local traffic from/to a file server or a NAS.
With these jumbo frame settings, locally, on PC's using Macrium (backup disk clone tool) I can backup a disk to my Syno NAS with true 100 % 1 Gigabit / sec. With classic 1500 byte frames there is little bit more overhead. Jumbo frame cram out that extra zero dot x % speed gain.

johnpoz · Sep 17, 2021, 12:46 PM

@gertjan good point actually.. Really need to just sniff on the device sending the traffic and see what its putting on the wire.

Are they jumbo frame or just fragmented down to 1500 mtu.. If the sender is putting jumbo on the wire - then yeah your going to need jumbo support on the switches. And the router as well..

And you can have all kinds of issues when you have one network doing jumbo, and then another network your trying to route to for the receiver device that isn't using jumbo, etc.

@GemeenAapje we really need to more info - but causing a problem by sending large pings to pfsense is prob not related to the problem..

stephenw10 · Sep 17, 2021, 2:29 PM

@johnpoz said in Pfsense not responding to large packet pings:

but causing a problem by sending large pings to pfsense is prob not related to the problem..

...though I would expect it to work with arbitrarily sized packets. I have never tried 64K though. Until now.

[2.5.2-RELEASE][admin@t70.stevew.lan]/root: ping -s 65500 172.21.16.246
PING 172.21.16.246 (172.21.16.246): 65500 data bytes
65508 bytes from 172.21.16.246: icmp_seq=0 ttl=64 time=1.942 ms
65508 bytes from 172.21.16.246: icmp_seq=1 ttl=64 time=1.825 ms
65508 bytes from 172.21.16.246: icmp_seq=2 ttl=64 time=1.878 ms
65508 bytes from 172.21.16.246: icmp_seq=3 ttl=64 time=1.764 ms
^C
--- 172.21.16.246 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.764/1.852/1.942/0.066 ms

A similar test to another device in my network fails:

[2.5.2-RELEASE][admin@t70.stevew.lan]/root: ping -s 2000 172.21.16.185
PING 172.21.16.185 (172.21.16.185): 2000 data bytes
^C
--- 172.21.16.185 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

The difference there is that 172.21.16.185 is behind a PoS TP-Link switch which doesn't pass packet fragments. Because.... no clue!

But bad switches aside it should work.

Steve

johnpoz · Sep 17, 2021, 2:35 PM

@stephenw10 said in Pfsense not responding to large packet pings:

PoS TP-Link switch which doesn't pass packet fragments.

What kind of shit switch? Yeah that belongs in the trash!

So what I have been able to duplicate.. Is if I try it "through" pfsense it fails.. But if on the same L2 network then works

$ ping -l 65500 ntp

Pinging ntp.local.lan [192.168.3.32] with 65500 bytes of data:
Request timed out.
Request timed out.

root@pi-hole:/# ping -s 65500 ntp
PING ntp.local.lan (192.168.3.32) 65500(65528) bytes of data.
65508 bytes from ntp.local.lan (192.168.3.32): icmp_seq=1 ttl=64 time=12.1 ms
65508 bytes from ntp.local.lan (192.168.3.32): icmp_seq=2 ttl=64 time=12.1 ms
65508 bytes from ntp.local.lan (192.168.3.32): icmp_seq=3 ttl=64 time=12.1 ms

stephenw10 · Sep 17, 2021, 2:38 PM

Same one that loves VLAN 1.

Yeah, needs recycling!

Just for reference here's two 3100s on different subnets routed through a 2.5.2 box:

[21.05.1-RELEASE][admin@fw1.stevew.lan]/root: ping -s 65500 -c 3 3100-2.fire.box
PING 3100-2.fire.box (192.168.10.103): 65500 data bytes
65508 bytes from 192.168.10.103: icmp_seq=0 ttl=63 time=3.832 ms
65508 bytes from 192.168.10.103: icmp_seq=1 ttl=63 time=3.512 ms
65508 bytes from 192.168.10.103: icmp_seq=2 ttl=63 time=3.583 ms

--- 3100-2.fire.box ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.512/3.642/3.832/0.137 ms

That's also through some VLANs over a LAGG just for fun.

Steve

johnpoz · Sep 17, 2021, 2:44 PM

Odd so through pfsense seems largest I can get is 34276

$ ping -l 34276 ntp

Pinging ntp.local.lan [192.168.3.32] with 34276 bytes of data:
Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63
Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63

$ ping -l 34277 ntp

Pinging ntp.local.lan [192.168.3.32] with 34277 bytes of data:
Request timed out.
Request timed out.

problem is I think if you loose like 1 packet - you run into problem where they all have to be transmitted again..

$ ping -l 34276 ntp -t

Pinging ntp.local.lan [192.168.3.32] with 34276 bytes of data:
Request timed out.
Request timed out.
Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63
Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63
Reply from 192.168.3.32: bytes=34276 time=8ms TTL=63
Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63
Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63
Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63
Request timed out.
Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63
Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63

stephenw10 · Sep 17, 2021, 2:50 PM

That's between local subnets? The ping time is very high compared with what I'm seeing at double the packet size. So presumably twice the number of fragments.

Unable to replicate here.

johnpoz · Sep 17, 2021, 2:56 PM

@stephenw10 yeah that is between 2 local segments..

Its to a pi, so the pi might have some issues with reassembly that is slowing up the response?

I do see sub ms response normal sized pings

C:\tools>hrping ntp
This is hrPING v5.07.1148 by cFos Software GmbH -- http://www.cfos.de

Source address is 192.168.9.100; using ICMP echo-request, ID=fc2f
Pinging ntp [192.168.3.32]
with 32 bytes data (60 bytes IP):

From 192.168.3.32: bytes=60 seq=0001 TTL=63 ID=f41c time=0.931ms
From 192.168.3.32: bytes=60 seq=0002 TTL=63 ID=f42d time=0.808ms
From 192.168.3.32: bytes=60 seq=0003 TTL=63 ID=f45b time=0.829ms
From 192.168.3.32: bytes=60 seq=0004 TTL=63 ID=f481 time=0.820ms

Packets: sent=4, rcvd=4, error=0, lost=0 (0.0% loss) in 1.511180 sec
RTTs in ms: min/avg/max/dev: 0.808 / 0.847 / 0.931 / 0.049
Bandwidth in kbytes/sec: sent=0.158, rcvd=0.158

edit: You know what also could be contributing?? Pi is only connected at 100, while my pc is at gig.. hmmmm

bingo600 · Sep 17, 2021, 3:00 PM

I can do 25100 to the pfsense , 25200 fails

$ ping -s 25100 fw-01
PING fw-01 (10.17.11.1) 25100(25128) bytes of data.
25108 bytes from fw-01 (10.17.11.1): icmp_seq=1 ttl=64 time=3.40 ms

I do 25200 wo probs to my RasPi3 , on another subnet.

$ ping -s 65500 raspi3
PING raspi3. (192.168.17.34) 65500(65528) bytes of data.
65508 bytes from raspi3. (192.168.17.34): icmp_seq=1 ttl=63 time=24.0 ms

Tests done via wifi C2702i , and passing two HP 1820 switches (No Jumbo)

Gertjan · Sep 17, 2021, 2:58 PM

ping -l 65500 192.168.2.2

Where 192.168.2.2 is an ancient Linksys E1200 access point, cabled up with 100 Mb - not 1 Gb.
This device is 10,12 years old.

I ping from 192.168.1.15, a windows 10 on LAN, through pfSense, to the other network 192.168.2.0.

From 192.168.2.2 : bytes=65500 time=15 ms TTL=63

"15 ms" probably because the10x slower "192.168.2.x network.

Windows - the ping command, doesn't allow me to use values above "65500".

johnpoz · Sep 17, 2021, 3:09 PM

Well this sure seems like a crazy rabbit hole to have gone down ;) hehehe

So my 2 pis can ping each other at 65500, but neither can ping pfsense interface on that same network.. But my PC can ping pfsense on my lan can ping just fine at 65500

but a vm I have on a different vlan can ping pfsense IP on its vlan just fine with 65500..

Yeah this is a crazy rabbit hole ;)

stephenw10 · Sep 17, 2021, 4:23 PM

@johnpoz said in Pfsense not responding to large packet pings:

Well this sure seems like a crazy rabbit hole to have gone down ;)

Ha, indeed! 64K packets.... who knew.