• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

[SOLVED] Upgraded to 2.5.0 Now Seeing Ping Spikes

Scheduled Pinned Locked Moved Routing and Multi WAN
13 Posts 3 Posters 2.5k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • T
    tman222
    last edited by Feb 20, 2021, 3:47 PM

    After spending a couple more days pouring over network settings and hardware tunables, unfortunately I have still not been able to resolve this ping spike issue. However, it does look like it is limited to the Chelsio card in my system so not sure if this is a driver issue or something related to FreeBSD 12.2 that changed from FreeBSD 11.3. Thankfully the intermittent spikes appears to be limited to ICMP traffic - i.e. I haven't noticed any slow down or excessive jitter on regular UDP/TCP traffic. One thing I might try is a fresh install of 2.5.0 to see if that resolves it unless someone out there has any suggestions on what to try next to troubleshoot this further. Thanks again for your help.

    T 1 Reply Last reply Feb 20, 2021, 7:59 PM Reply Quote 0
    • T
      tman222 @tman222
      last edited by Feb 20, 2021, 7:59 PM

      Ran a few ping flood tests between two hosts on different subnets (each subnet connected to a different port on the Chelsio card) and over the course of 25k packets or so there was 0% packet loss (good news) but a decent amount variability (jitter) from sub 0.1 ms to 100ms+ (bad news). Watching the ping tests I could see that ping requests were being sent and queued up a few packets deep before being answered and queue being emptied. This repeated over and over during the test. The hardware I'm using is plenty fast - could this be pointing towards some sort of buffering or CPU interrupt problem perhaps?

      1 Reply Last reply Reply Quote 0
      • T
        tman222
        last edited by Feb 21, 2021, 9:37 PM

        Found a useful tool called qperf (similar to iperf) that allows one to measure latency and transfer speeds between two hosts on a network:

        https://www.opsdash.com/blog/network-performance-linux.html

        Ran a number of UDP and TCP qperf tests between two 10Gbit hosts that are located in different network segments (which then uplink via a separate port on the Chelsio card to the pfSense firewall). Transfer speeds and latency looked perfectly normal across a variety of different data transfer sizes and test lengths. At this point I do think the ping spike/jitter I'm observing may just be limited to ICMP packets which, while a bit annoying, is something I can deal with. Is it possible for those packets to be getting de-prioritized somehow? Thanks again for your help.

        1 Reply Last reply Reply Quote 0
        • T
          tman222
          last edited by Feb 27, 2021, 5:28 PM

          Well, perhaps a step closer to finding a resolution. Found out today that the Chelsio firmware / driver was updated in FreeBSD 12.2 (on which pfSense 2.5.0 is based):

          https://www.freebsd.org/releases/12.2R/relnotes/

          So, all this could be a driver related issue - will try to reach out to Chelsio support directly.

          1 Reply Last reply Reply Quote 0
          • ?
            A Former User
            last edited by Mar 1, 2021, 3:25 AM

            We use neither Snort nor pfBlocker-NG and we’re seeing the same behaviour on a beefy system with Intel NIC. The spikes become more frequent the more users are on during the day. I’ll report back if we find the underlying cause of it.

            1 Reply Last reply Reply Quote 0
            • T
              tman222
              last edited by tman222 Mar 1, 2021, 4:00 PM Mar 1, 2021, 3:49 PM

              Update: I heard back from Chelsio support this morning and was told that there was a bug in the driver's transmit path that they have fixed recently (looks like a couple weeks ago), which seems directly applicable given the behavior I've observed / described:

              cxgbe(4): Fixes to tx coalescing.
                  
                  - The behavior implemented in r362905 resulted in delayed transmission
                    of packets in some cases, causing performance issues.  Use a different
                    heuristic to predict tx requests.
                  
                  - Add a tunable/sysctl (hw.cxgbe.tx_coalesce) to disable tx coalescing
                    entirely.  It can be changed at any time.  There is no change in
                    default behavior.
              

              FreeBSD 14-current and FreeBSD 13:
              https://lists.freebsd.org/pipermail/dev-commits-src-all/2021-February/002084.html
              https://lists.freebsd.org/pipermail/dev-commits-src-all/2021-February/002263.html

              FreeBSD 12.2 (relevant to pfSense 2.5.0):
              https://lists.freebsd.org/pipermail/dev-commits-src-all/2021-February/002794.html
              https://lists.freebsd.org/pipermail/dev-commits-src-all/2021-February/002795.html

              How do I apply this driver fix to my current install of pfSense 2.5.0? Is there an easy way for me to do that using the System Patches package? Or do I need to raise a request on Redmine to include in the next release?

              Thanks again for all your help.

              1 Reply Last reply Reply Quote 0
              • S
                stephenw10 Netgate Administrator
                last edited by Mar 1, 2021, 5:50 PM

                That is already in master: https://github.com/pfsense/FreeBSD-src/commits/devel-12/sys/dev/cxgbe

                So should be in 2.6 snapshots. Can you test one? Other things may be broken there....

                Steve

                T 1 Reply Last reply Mar 1, 2021, 5:54 PM Reply Quote 0
                • T
                  tman222 @stephenw10
                  last edited by Mar 1, 2021, 5:54 PM

                  @stephenw10 said in Upgraded to 2.5.0 Now Seeing Ping Spikes:

                  That is already in master: https://github.com/pfsense/FreeBSD-src/commits/devel-12/sys/dev/cxgbe

                  So should be in 2.6 snapshots. Can you test one? Other things may be broken there....

                  Steve

                  Unfortunately I only have one production system with that card that I would reluctant to upgrade to the unstable development branch. Do I have any other options bring this fix into my current 2.5.0 environment? Thanks again.

                  1 Reply Last reply Reply Quote 0
                  • T
                    tman222
                    last edited by Mar 5, 2021, 4:38 AM

                    So I tried grabbing the Chelsio drivers modules from the latest FreeBSD 12.2 stable release:

                    t4fw_cfg.ko
                    t5fw_cfg.ko
                    t6fw_cfg.ko
                    if_cxgbe.ko
                    

                    I copied them over to /boot/modules made sure permissions were correct and then added the necessary load lines to loader.conf.local as described here:

                    https://www.freebsd.org/cgi/man.cgi?query=if_cxgbe&apropos=0&sektion=4&manpath=FreeBSD+12.2-stable&arch=default&format=html

                    However, upon reboot the updated driver modules aren't actually being loaded, which I can see from kldstat. It seems that the system still prefers the older Chelsio driver that was compiled into the kernel vs. these updated driver modules. If I try to load the updated modules manually using kdload I get error messages that the interfaces already exist in the kernel.

                    Am I out of luck here, or is there any way for the system to use the newer driver modules instead of what was compiled into the kernel?

                    Thanks in advance for your help.

                    1 Reply Last reply Reply Quote 0
                    • T
                      tman222
                      last edited by Mar 5, 2021, 2:56 PM

                      Well, unless I'm misunderstanding this thread, I think I may just be out of luck unless I recompile the kernel with the newer modules (which wouldn't really be an option):

                      https://forums.freebsd.org/threads/custom-kernel-driver-modules.14998/

                      Does anyone else have any thoughts / suggestions? Thanks in advance.

                      1 Reply Last reply Reply Quote 0
                      • T
                        tman222
                        last edited by Mar 19, 2021, 10:43 PM

                        Happy to say that this issue has been fixed in the latest 2.5.1 snapshots:

                        https://redmine.pfsense.org/issues/11602

                        Before - 2.5.0:

                        --- X.X.X.X ping statistics ---
                        500 packets transmitted, 500 received, 0% packet loss, time 701ms
                        rtt min/avg/max/mdev = 0.175/25.373/109.791/27.343 ms, pipe 8
                        

                        After - 2.5.1-RC:

                        --- X.X.X.X ping statistics ---
                        500 packets transmitted, 500 received, 0% packet loss, time 96ms
                        rtt min/avg/max/mdev = 0.181/0.267/0.365/0.040 ms
                        

                        A big thank you to entire Netgate / pfSense team for addressing this so quickly.

                        1 Reply Last reply Reply Quote 1
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                          This community forum collects and processes your personal information.
                          consent.not_received