Ping spikes on new install



  • Hi all, this is my first pfSense build and have noticed some issues with ping spikes while gaming. The issue is not severe - it seems that I have random ping spikes (around three in my last test of 100 pings to 8.8.8.8 ). The latency will be around 24 or 25 and then will shoot up to the hundreds or even 200 ms then go straight back down again.

    My hardware is:
    Hynix 8GB DDR3 1600Mhz SODIMM
    J3455B-ITX
    Game Max GP400A 400W 80+ Bronze PSU
    Crucial MX200 250GB SATA 6Gb/s 2.5" SSD
    DELL INTEL 1000 PRO PCI-E QUAD GIGABIT NETWORK CARD - YT674 - D96950-006

    My setup is:
    TalkTalk router in bridged mode –-> <wan>pfSense <lan>-> Switch -> WAP / wired devices.

    I have narrowed down the problems with ping latency to the pfSense. I did a long ping test using just my ISP modem and it all came back fine. The problems I describe are after SSHing to my pfSense and running ping 8.8.8.8 from the shell.

    I am really lost as to how to investigate this and would really appreciate any help you the community may be able to offer. I have not really set much up with pfSense so far, just OpenVPN (not currently running).

    One thing coming to mind is maybe it's my second-hand Intel NIC, but I do not know how to test if this is the issue or not.

    Thank you

    Edit: Also noticing when I'm in SSH for pfSense that when I ping a device on my LAN I also have latency spikes (weirdly, pinging my Macbook Pro has constant latency over 10ms, between 10 and 100). Pinging another device gives me around 0-2ms and then spike to 100 or so.

    Edit2: I've tried using the Motherboard NIC as my WAN interface, and still when I ssh to pfSense and ping out I get the random ping spikes like 23 23 23 23 23 84 66 23 23 23 23 23 23 23 23 23 23 23 23 145 etc.</lan></wan>



  • Still facing this issue and really struggling how to diagnose :( any advice here greatly appreciated

    Thank you very much



  • disable any c-states on cpu, also if powerd is running either set it to performance mode or shut it down, this is so we can rule out power saving technologies misbehaving.
    disable hyperhtreading if enabled and supported
    consider disabling checksum offloading on nic, although as its intel I consider this unlikely to be the problem
    make sure large receive offload and send equivelent is disabled, and promiscious mode.
    google how to do it, but set the intel nic to a single queue mode only.  (loader.conf.custom).

    This is to test and resolve the lan side spikes, ignore whats going on wan side for now.



  • @chrcoluk:

    disable any c-states on cpu, also if powerd is running either set it to performance mode or shut it down, this is so we can rule out power saving technologies misbehaving.
    disable hyperhtreading if enabled and supported
    consider disabling checksum offloading on nic, although as its intel I consider this unlikely to be the problem
    make sure large receive offload and send equivelent is disabled, and promiscious mode.
    google how to do it, but set the intel nic to a single queue mode only.  (loader.conf.custom).

    This is to test and resolve the lan side spikes, ignore whats going on wan side for now.

    • powerd is disabled already

    • Diabling checksum offboarding - I haven't figured out how to do this but please note I have also tested using the motherboard NIC as my WAN interface and see the same issues.

    • Large receive offload is disabled. I can't find an option for promiscuous mode or send equivalent but I haven't changed these.

    • Disabling hyperthreading

    • Disabling c-states



  • offloading is in the pfsense gui under advanced system settings.

    you can disable hyperthreading in the loader.conf

    add this to /boot/loader.conf.local and reboot

    also Ill add lines to force your network over single threaded.  Note the igb line is only useful if your intel uses the igb driver, it may be using the em driver, which can be confirmed by running ifconfig.

    net.isr.maxthreads=1
    hw.igb.num_queues=1
    machdep.hyperthreading_allowed="0"

    I would say the LAN debugging takes priority over the WAN, the LAN spikes shouldnt be happening period, whilst WAN can be caused by all sorts of things outside your control.



  • The particular board in question unless your running has a pair of RTL8111GR based nics on it which "should" work. That being said the standard answer to rule it out is try switching over to using Intel nics…FreeBSD drivers for those nics are just that much better, there's also better quality control during fab.

    On the lan side when start seeing latency issues start removing hops until it's stable. What kind of switch & AP are you using? Are you running on the 2.4GHz or 5GHz band? Check for other devices broadcasting on the same channel. Period 100ms latency spikes given the original could be anything from a buggy switch firmware, buggy drivers, crap nics, failing AP, DC motor interference, microwave ovens, neighborhood AP's on the same channel, or flaky MBP wireless cards (PITA) depending on the year.



  • Also to add if my tip doesnt fix it then go back to defaults, what I offered is troubleshooting tips and could possibly be cause of problem, but dont set the changes if the defaults are ok.



  • Definately NIC offloading that causes this - seen it so many times.

    Try disabling all three options for hardware offloading on the NIC in advanced settings. This will fix your issue, and then selectively start enabling it and which one/combination is causing the issue.



  • Thank you all for your helpful input

    @Patrick_:

    The particular board in question unless your running has a pair of RTL8111GR based nics on it which "should" work. That being said the standard answer to rule it out is try switching over to using Intel nics…FreeBSD drivers for those nics are just that much better, there's also better quality control during fab.

    On the lan side when start seeing latency issues start removing hops until it's stable. What kind of switch & AP are you using? Are you running on the 2.4GHz or 5GHz band? Check for other devices broadcasting on the same channel. Period 100ms latency spikes given the original could be anything from a buggy switch firmware, buggy drivers, crap nics, failing AP, DC motor interference, microwave ovens, neighborhood AP's on the same channel, or flaky MBP wireless cards (PITA) depending on the year.

    My motherboard only has one NIC. I did however use this and the Intel NIC as well and same problems on each. Regarding the LAN side, the latency seems to only be TO my macbook. If I ping pfSense from my macbook I see 1-2 ms latency. So I imagine it's probably something locally on my Macbook. Likewise, pinging from pfSense to another device on DHCP also sees 1,2ms. I am using a Netgear R7000 DD-WRT for WAP and a Netgear 5port unmanaged switch. I have tested the same by using a cable to my macbook, and still same very high LAN latency.

    I sat with my Macbook plugged directly into LAN port on pfSense (no WAP or switch involved). latency not anywhere near 100 but still interesting to see it is spiking, similar to with the WAN..
    https://pastebin.com/A32yrq5v

    @chrcoluk:

    add this to /boot/loader.conf.local and reboot

    net.isr.maxthreads=1
    hw.igb.num_queues=1
    machdep.hyperthreading_allowed="0"

    I have done this but still same issue. Granted, for the test I have just run I didn't see any spikes over 100ms like I did yesterday but I wouldn't say that is a result of changing this file. I did 100 pings and put it on pastebin just to show, the pattern does seem random
    https://pastebin.com/CabxSFpy
    Again, tested on WiFi and the response time seems very random. Could easily be problem with my macbook though
    https://pastebin.com/ShyzBXfh

    @chrcoluk:

    whilst WAN can be caused by all sorts of things outside your control.

    Strange thing is when I tested from my talktalk router I saw no issues, which leads me to assume it's something with my pfSense.

    @keyser:

    Try disabling all three options for hardware offloading on the NIC in advanced settings. This will fix your issue, and then selectively start enabling it and which one/combination is causing the issue.

    Hardware Large Receive and Hardware TCP Segmentation Offloading were already selected 'Disable'. I've additionally disabled Hardware Checksum Offloading but the problems on LAN and WAN persist unfortunately.



  • Still the same issues :(

    Just did some more tests to show the issue. Really appreciate any other advice as to what the problem could be? I feel like it's not a hardware issue, I have tried WAN on the Intel card I have and also onboard motherboard NIC.

    Pinging out from laptop plugged in to Talktalk ISP router
    64 bytes from 8.8.8.8: icmp_seq=0 ttl=59 time=25.146 ms
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=25.048 ms
    64 bytes from 8.8.8.8: icmp_seq=2 ttl=59 time=24.956 ms
    64 bytes from 8.8.8.8: icmp_seq=3 ttl=59 time=24.973 ms
    64 bytes from 8.8.8.8: icmp_seq=4 ttl=59 time=25.198 ms
    64 bytes from 8.8.8.8: icmp_seq=5 ttl=59 time=24.880 ms
    64 bytes from 8.8.8.8: icmp_seq=6 ttl=59 time=25.087 ms
    64 bytes from 8.8.8.8: icmp_seq=7 ttl=59 time=25.050 ms
    64 bytes from 8.8.8.8: icmp_seq=8 ttl=59 time=24.885 ms
    64 bytes from 8.8.8.8: icmp_seq=9 ttl=59 time=25.227 ms
    64 bytes from 8.8.8.8: icmp_seq=10 ttl=59 time=24.880 ms
    64 bytes from 8.8.8.8: icmp_seq=11 ttl=59 time=24.729 ms
    64 bytes from 8.8.8.8: icmp_seq=12 ttl=59 time=24.975 ms
    64 bytes from 8.8.8.8: icmp_seq=13 ttl=59 time=24.889 ms
    64 bytes from 8.8.8.8: icmp_seq=14 ttl=59 time=25.303 ms
    64 bytes from 8.8.8.8: icmp_seq=15 ttl=59 time=24.928 ms
    64 bytes from 8.8.8.8: icmp_seq=16 ttl=59 time=24.848 ms
    64 bytes from 8.8.8.8: icmp_seq=17 ttl=59 time=25.005 ms

    Pinging out from laptop on pfSense LAN
    PING 8.8.8.8 (8.8.8.8): 56 data bytes
    64 bytes from 8.8.8.8: icmp_seq=0 ttl=58 time=37.218 ms
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=35.790 ms
    64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=26.519 ms
    64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=31.367 ms
    64 bytes from 8.8.8.8: icmp_seq=4 ttl=58 time=320.857 ms
    64 bytes from 8.8.8.8: icmp_seq=5 ttl=58 time=64.461 ms
    64 bytes from 8.8.8.8: icmp_seq=6 ttl=58 time=79.875 ms
    64 bytes from 8.8.8.8: icmp_seq=7 ttl=58 time=27.106 ms
    64 bytes from 8.8.8.8: icmp_seq=8 ttl=58 time=308.835 ms
    64 bytes from 8.8.8.8: icmp_seq=9 ttl=58 time=28.096 ms
    64 bytes from 8.8.8.8: icmp_seq=10 ttl=58 time=27.234 ms
    64 bytes from 8.8.8.8: icmp_seq=11 ttl=58 time=28.600 ms
    64 bytes from 8.8.8.8: icmp_seq=12 ttl=58 time=118.843 ms
    64 bytes from 8.8.8.8: icmp_seq=13 ttl=58 time=30.498 ms

    Pinging out from pfSense itself, through the WAN to my talktalk router
    PING 8.8.8.8 (8.8.8.8): 56 data bytes
    64 bytes from 8.8.8.8: icmp_seq=0 ttl=59 time=25.000 ms
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=25.163 ms
    64 bytes from 8.8.8.8: icmp_seq=2 ttl=59 time=99.020 ms
    64 bytes from 8.8.8.8: icmp_seq=3 ttl=59 time=25.020 ms
    64 bytes from 8.8.8.8: icmp_seq=4 ttl=59 time=25.013 ms
    64 bytes from 8.8.8.8: icmp_seq=5 ttl=59 time=25.466 ms
    64 bytes from 8.8.8.8: icmp_seq=6 ttl=59 time=32.813 ms
    64 bytes from 8.8.8.8: icmp_seq=7 ttl=59 time=53.171 ms
    64 bytes from 8.8.8.8: icmp_seq=8 ttl=59 time=61.021 ms

    Edit: Going to try a factory reset tomorrow and then test the WAN from there.



  • I performed a factory reset, and configured ONLY the WAN as a DHCP interface.

    I am still seeing latency issues when pinging the internet (23ms, 23ms, 23ms, 106ms, 23ms, 23ms, 23ms, 23ms, 44ms, 24ms, 23ms, 23ms, 33ms, 23ms, 101ms) etc.

    I see the same results when using my Intel PCI card NIC as WAN and also the onboard motherboard NIC.

    I also see the same results after I make sure all Hardware Offloading is dissabled in the Advanced menu. My changes to /boot/loader.conf.local persisted through the factory reset. Still same latency issues.

    I do not see these issues when am pinging directly from the TalkTalk modem. The cable between the two is around 1M long. Really can't think what the issue is.. Any other ideas here would be greatly appreciated. Could it be an issue with the Apollo Lake? I read some people have issues with this with newest pfSense. Didn't read anything about latency spikes though.

    Edit: Managed to get into the BIOS and changed C-States from C6 to Disabled. STILL getting the latency spikes. Anything else I may be able to change in the BIOS? Everything will be default apart from now c-states.
    Thanks



  • Are you using any of the altq traffic shapers?



  • @NaterGator:

    Are you using any of the altq traffic shapers?

    Everything is unchecked in firewall-> traffic shaper. I am using a completely out of the box install here with WAN and LAN configured.
    Thank you



    • Fresh install 2.4.2. Only config is LAN/WAN ports and:

    • Everything is unchecked in firewall-> traffic shaper->By interface

    • All three Hardware Offloading settings are Disabled

    • Added the following to /boot/loader.conf.local:
      net.isr.maxthreads=1
      hw.igb.num_queues=1
      machdep.hyperthreading_allowed="0"

    • powerd is not running

    • BIOS: CPU C States Support = Disabled



  • So having read all this, the only thing I can think of is the ethernet port on your ISP router maybe?

    i.e. when you test from your ISP router, are you testing from a device plugged into it, or from it directly?

    What I mean is that running a ping on the router itself might be telling you "the router is fine" when actually if there's a problem with the router that causes latency/issues only on it's "LAN" port (the port you plug your pfSense into) you won't see it.

    What if you plug a laptop into the TeamTalk device and test using it?  Does that show you the latency spikes?

    Also, what if you remove "the internet" from the equation and configure two laptops.  One pretends to be the Internet and one is a LAN client, then you ping between the laptops via the pfsense (you'll need two NICs on your pfSense box)

    Does that show you latency?



  • @muppet:

    So having read all this, the only thing I can think of is the ethernet port on your ISP router maybe?

    i.e. when you test from your ISP router, are you testing from a device plugged into it, or from it directly?

    What I mean is that running a ping on the router itself might be telling you "the router is fine" when actually if there's a problem with the router that causes latency/issues only on it's "LAN" port (the port you plug your pfSense into) you won't see it.

    What if you plug a laptop into the TeamTalk device and test using it?  Does that show you the latency spikes?

    Also, what if you remove "the internet" from the equation and configure two laptops.  One pretends to be the Internet and one is a LAN client, then you ping between the laptops via the pfsense (you'll need two NICs on your pfSense box)

    Does that show you latency?

    Hi muppet, thanks for your time. I did come to the same thought but unfortunately disproved it by:
    *** Plugging Windows laptop into TalkTalk Router directly, and pings were fine

    • I also have latency on the LAN

    • Running FreeBSD on the server seems to work fine regarding latency**



  • So even ping between two laptops, via pfSense, shows the problem?

    Can you post the output of dmesg? I'm not a FreeBSD guru, but maybe we can spot something in there that might be the cause of the problems.



  • @muppet:

    So even ping between two laptops, via pfSense, shows the problem?

    Can you post the output of dmesg? I'm not a FreeBSD guru, but maybe we can spot something in there that might be the cause of the problems.

    Unfortunately so :( and strange that FreeBSD live USB didn't have the problems! Must be some sort of setting in PfSense

    Me neither re FreeBSD guru but I thank you for having a skim over incase!
    Here's the output https://pastebin.com/3wCFpKqW



  • I'm completely lost for ideas now, may have to scrap pfSense and go back to DD-WRT :'( Or at least buy a new motherboard/processor combination…

    Any suggestions for ITX motherboard/CPU that has AES-NI and a proven record of being good for pfSense? Or alternatively any other troubleshooting options to test the LAN/WAN latency issues

    I've also tried setting the Cryptographic Hardware from BSD Crypto Device to None, to no avail :(
    Thank you all



  • Maybe try disabling MSX-I and/or Flow Control?

    https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

    Also try running using a RAM disk (System->Advanced->Misc), maybe your pfSense writing logs to disk causes the system to pause/freeze up for a moment?

    These are all just stab-in-dark guesses now :(



  • @nazuro:

    Hi all, this is my first pfSense build and have noticed some issues with ping spikes while gaming. The issue is not severe - it seems that I have random ping spikes (around three in my last test of 100 pings to 8.8.8.8 ). The latency will be around 24 or 25 and then will shoot up to the hundreds or even 200 ms then go straight back down again.

    Hi,

    Delayed answer, but in case someone else is struckling with same problem…

    I had exactly same issue for months, and tried to find cause from every component in network. Yet it was my pfsense hardware afterall.

    Solution:
    disable 'Monitor M-Wait' in bios
    Could be under Features tab > CPU Configuration

    At least work with my Zotac Ci-327