Pfsense not responding to large packet pings
-
@gemeenaapje said in Pfsense not responding to large packet pings:
It's known to have problems when large packets don't make it through the switches.
Why is it sending large packets? Is it using jumbo frames? What is the MTU? Other than on token ring, back when I was at IBM, I've never seen MTUs set beyond 1500, except for my own testing. If it is using jumbo frames, then you have to make sure the switch can handle them. IIRC, when jumbo frames are used, the usual MTU is 9000. Even then, you have to make sure everything on the local network can handle that MTU.
-
@johnpoz said in Pfsense not responding to large packet pings:
25 year old software wouldn't be doing jumbo ;)
25 year old software isn't jumbo aware. Software from 2021 : same thing.
The ISO network stack is far older then that. I recall, somewhere in the eighties, @school, that they started to tell me about this new 7 layer model (some say 8 layers) thing.
So, the program would hand over 'the file' to be transmitted to the OS.
And deep down somewhere, the data stream is cut down in chunks of XX bytes, as that is the way how info is send over.
The program doesn't need to know about headers, sessions, MAC addresses or even IP addresses. It doesn't care that 'TCP' or IPv4 or IPv6 is used.I guess @GemeenAapje is talking about Jumbo frames because he want to push to the limit his local traffic from/to a file server or a NAS.
With these jumbo frame settings, locally, on PC's using Macrium (backup disk clone tool) I can backup a disk to my Syno NAS with true 100 % 1 Gigabit / sec. With classic 1500 byte frames there is little bit more overhead. Jumbo frame cram out that extra zero dot x % speed gain. -
@gertjan good point actually.. Really need to just sniff on the device sending the traffic and see what its putting on the wire.
Are they jumbo frame or just fragmented down to 1500 mtu.. If the sender is putting jumbo on the wire - then yeah your going to need jumbo support on the switches. And the router as well..
And you can have all kinds of issues when you have one network doing jumbo, and then another network your trying to route to for the receiver device that isn't using jumbo, etc.
@GemeenAapje we really need to more info - but causing a problem by sending large pings to pfsense is prob not related to the problem..
-
@johnpoz said in Pfsense not responding to large packet pings:
but causing a problem by sending large pings to pfsense is prob not related to the problem..
...though I would expect it to work with arbitrarily sized packets. I have never tried 64K though. Until now.
[2.5.2-RELEASE][admin@t70.stevew.lan]/root: ping -s 65500 172.21.16.246 PING 172.21.16.246 (172.21.16.246): 65500 data bytes 65508 bytes from 172.21.16.246: icmp_seq=0 ttl=64 time=1.942 ms 65508 bytes from 172.21.16.246: icmp_seq=1 ttl=64 time=1.825 ms 65508 bytes from 172.21.16.246: icmp_seq=2 ttl=64 time=1.878 ms 65508 bytes from 172.21.16.246: icmp_seq=3 ttl=64 time=1.764 ms ^C --- 172.21.16.246 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.764/1.852/1.942/0.066 msA similar test to another device in my network fails:
[2.5.2-RELEASE][admin@t70.stevew.lan]/root: ping -s 2000 172.21.16.185 PING 172.21.16.185 (172.21.16.185): 2000 data bytes ^C --- 172.21.16.185 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet lossThe difference there is that 172.21.16.185 is behind a PoS TP-Link switch which doesn't pass packet fragments. Because.... no clue!
But bad switches aside it should work.
Steve
-
@stephenw10 said in Pfsense not responding to large packet pings:
PoS TP-Link switch which doesn't pass packet fragments.
What kind of shit switch? Yeah that belongs in the trash!
So what I have been able to duplicate.. Is if I try it "through" pfsense it fails.. But if on the same L2 network then works
$ ping -l 65500 ntp Pinging ntp.local.lan [192.168.3.32] with 65500 bytes of data: Request timed out. Request timed out.root@pi-hole:/# ping -s 65500 ntp PING ntp.local.lan (192.168.3.32) 65500(65528) bytes of data. 65508 bytes from ntp.local.lan (192.168.3.32): icmp_seq=1 ttl=64 time=12.1 ms 65508 bytes from ntp.local.lan (192.168.3.32): icmp_seq=2 ttl=64 time=12.1 ms 65508 bytes from ntp.local.lan (192.168.3.32): icmp_seq=3 ttl=64 time=12.1 ms -
stephenw10 Netgate Administratorlast edited by stephenw10 Sep 17, 2021, 2:38 PM Sep 17, 2021, 2:37 PM
Same one that loves VLAN 1.
Yeah, needs recycling!
Just for reference here's two 3100s on different subnets routed through a 2.5.2 box:
[21.05.1-RELEASE][admin@fw1.stevew.lan]/root: ping -s 65500 -c 3 3100-2.fire.box PING 3100-2.fire.box (192.168.10.103): 65500 data bytes 65508 bytes from 192.168.10.103: icmp_seq=0 ttl=63 time=3.832 ms 65508 bytes from 192.168.10.103: icmp_seq=1 ttl=63 time=3.512 ms 65508 bytes from 192.168.10.103: icmp_seq=2 ttl=63 time=3.583 ms --- 3100-2.fire.box ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 3.512/3.642/3.832/0.137 msThat's also through some VLANs over a LAGG just for fun.
Steve
-
Odd so through pfsense seems largest I can get is 34276
$ ping -l 34276 ntp Pinging ntp.local.lan [192.168.3.32] with 34276 bytes of data: Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63 Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63$ ping -l 34277 ntp Pinging ntp.local.lan [192.168.3.32] with 34277 bytes of data: Request timed out. Request timed out.problem is I think if you loose like 1 packet - you run into problem where they all have to be transmitted again..
$ ping -l 34276 ntp -t Pinging ntp.local.lan [192.168.3.32] with 34276 bytes of data: Request timed out. Request timed out. Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63 Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63 Reply from 192.168.3.32: bytes=34276 time=8ms TTL=63 Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63 Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63 Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63 Request timed out. Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63 Reply from 192.168.3.32: bytes=34276 time=7ms TTL=63 -
That's between local subnets? The ping time is very high compared with what I'm seeing at double the packet size. So presumably twice the number of fragments.
Unable to replicate here.
-
@stephenw10 yeah that is between 2 local segments..
Its to a pi, so the pi might have some issues with reassembly that is slowing up the response?
I do see sub ms response normal sized pings
C:\tools>hrping ntp This is hrPING v5.07.1148 by cFos Software GmbH -- http://www.cfos.de Source address is 192.168.9.100; using ICMP echo-request, ID=fc2f Pinging ntp [192.168.3.32] with 32 bytes data (60 bytes IP): From 192.168.3.32: bytes=60 seq=0001 TTL=63 ID=f41c time=0.931ms From 192.168.3.32: bytes=60 seq=0002 TTL=63 ID=f42d time=0.808ms From 192.168.3.32: bytes=60 seq=0003 TTL=63 ID=f45b time=0.829ms From 192.168.3.32: bytes=60 seq=0004 TTL=63 ID=f481 time=0.820ms Packets: sent=4, rcvd=4, error=0, lost=0 (0.0% loss) in 1.511180 sec RTTs in ms: min/avg/max/dev: 0.808 / 0.847 / 0.931 / 0.049 Bandwidth in kbytes/sec: sent=0.158, rcvd=0.158edit: You know what also could be contributing?? Pi is only connected at 100, while my pc is at gig.. hmmmm
-
I can do 25100 to the pfsense , 25200 fails
$ ping -s 25100 fw-01 PING fw-01 (10.17.11.1) 25100(25128) bytes of data. 25108 bytes from fw-01 (10.17.11.1): icmp_seq=1 ttl=64 time=3.40 msI do 25200 wo probs to my RasPi3 , on another subnet.
$ ping -s 65500 raspi3 PING raspi3. (192.168.17.34) 65500(65528) bytes of data. 65508 bytes from raspi3. (192.168.17.34): icmp_seq=1 ttl=63 time=24.0 msTests done via wifi C2702i , and passing two HP 1820 switches (No Jumbo)
-
ping -l 65500 192.168.2.2
Where 192.168.2.2 is an ancient Linksys E1200 access point, cabled up with 100 Mb - not 1 Gb.
This device is 10,12 years old.I ping from 192.168.1.15, a windows 10 on LAN, through pfSense, to the other network 192.168.2.0.
From 192.168.2.2 : bytes=65500 time=15 ms TTL=63
"15 ms" probably because the10x slower "192.168.2.x network.
Windows - the ping command, doesn't allow me to use values above "65500".
-
Well this sure seems like a crazy rabbit hole to have gone down ;) hehehe
So my 2 pis can ping each other at 65500, but neither can ping pfsense interface on that same network.. But my PC can ping pfsense on my lan can ping just fine at 65500
but a vm I have on a different vlan can ping pfsense IP on its vlan just fine with 65500..
Yeah this is a crazy rabbit hole ;)
-
@johnpoz said in Pfsense not responding to large packet pings:
Well this sure seems like a crazy rabbit hole to have gone down ;)
Ha, indeed! 64K packets.... who knew.
-
@gertjan said in Pfsense not responding to large packet pings:
Jumbo frame cram out that extra zero dot x % speed gain.
Actually, it's due to the way switches handle the data. Switch specs talk about frames, not bits, as it takes them just as much effort to handle a small frame as a big one. The buffer receives the data, triggers the switch to move it and sends it to the appropriate port and it does this an entire frame at a time. So, the bigger the frame, the more data is handled each time.
-
@johnpoz said in Pfsense not responding to large packet pings:
What kind of shit switch?
Your favourite, TP-Link.
-
@johnpoz said in Pfsense not responding to large packet pings:
problem is I think if you loose like 1 packet - you run into problem where they all have to be transmitted again..
That depends on the software. TCP can recover from missing pieces, without starting from the beginning of a packet. However, it depends on the particular version. I don't know that UDP or ICMP can recover.
-
@gertjan said in Pfsense not responding to large packet pings:
Windows - the ping command, doesn't allow me to use values above "65500"
That's because 65K is the maximum packet size. So, that would be 65536 less overhead.
-
Well, that is the maximum allowed for IP.
-
Good morning guys.
I'm so sorry for sending you all down the rabbit hole :-DWell I tried some things in the weekend too...
-Upgraded to 2.5.2
-Tried unchecking the "disable hardware offload" options.
-Disabled suricata and pfBlockerNG
-Pinging various other devices in different directionsBasically, nothing helps. It's only pfSense I have a problem with. All other devices can ping 65500 in any direction with almost zero latency.
It's connected with 2 glass fibre cables (sorry I stated this wrong before).
1 glass fibre from the ISP and the other glass fibre going to a 10gb/s switch.Even just trying to send a large ping packet to the directly-connected switch, over fibre, is impossible. Works up to about 20,000 which give a (broken) response in about 1.8 seconds (not milliseconds!). Above 20k it just fails.
I can ping the pfsense from itself using it's own IP, which is successful (doesn't go out over the network I suppose).
I'm running all good hardware. The NIC is connected via direct passthrough from the VM Host to the pfSense box so the VM has complete control of it.
Something just isn't right though.
I'm not using Jumbo packets by the way, just 1500. I expect the switch the just fragment any large packets.
Anyone got anymore ideas?
Thanks
MattPS: anyone know how to reload the Menu after using the shell in pfsense?
-
Your pfSense runs from a VM ?
So, for you the 'how many bytes per packet' is hard coded in the NIC device driver. And the VM "virtual" NIC driver.
The smallest wins, of course.My pfSense, 2.5.2 CE, a bare bone install, using a quad Intel NIC card, works fine.
@gemeenaapje said in Pfsense not responding to large packet pings:
I can ping the pfsense from itself using it's own IP
Aha : the local - for pfSense - NIC driver is used, the transport stays purely 'in software' to physical or VM rules don't apply. Again : go bare bone and you have your 'close to "65535"'.
@gemeenaapje said in Pfsense not responding to large packet pings:
PS: anyone know how to reload the Menu after using the shell in pfsense?
Console ? SSH ?
This menu :
0) Logout (SSH only) 9) pfTop 1) Assign Interfaces 10) Filter Logs 2) Set interface(s) IP address 11) Restart webConfigurator 3) Reset webConfigurator password 12) PHP shell + pfSense tools 4) Reset to factory defaults 13) Update from console 5) Reboot system 14) Disable Secure Shell (sshd) 6) Halt system 15) Restore recent configuration 7) Ping host 16) Restart PHP-FPM 8) Shell?
Press Enter an it shows up.
When you entered the 8 or Shell option just type "exit" + enter.
