Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfSense underperforming, high jitter + random packet loss

    Scheduled Pinned Locked Moved General pfSense Questions
    27 Posts 3 Posters 6.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      Chrismallia
      last edited by

      UT proof

      utCapture.PNG
      utCapture.PNG_thumb

      1 Reply Last reply Reply Quote 0
      • N
        NaterGator
        last edited by

        @Chrismallia:

        Dude tested UT on same HW it first errored the test for a few times then I got these, pfsense did better with HFSC and Codel

        http://www.dslreports.com/speedtest/26821430

        Bizarre. What NIC are you running?

        1 Reply Last reply Reply Quote 0
        • C
          Chrismallia
          last edited by

          Nic is Intel dual port server grade

          1 Reply Last reply Reply Quote 0
          • N
            NaterGator
            last edited by

            Tested using a brand new Intel I350T2V2, exactly the same results.

            1 Reply Last reply Reply Quote 0
            • C
              Chrismallia
              last edited by

              My ping spikes up to 300ms sure but it goes down and I get A with no interruption to the services  same on UT,  can you post your results with Intel nic ? was the internet slow while performing the test ? try traffic shaper with HFSC and enable codel on every q and post your results,

              1 Reply Last reply Reply Quote 0
              • N
                NaterGator
                last edited by

                The issue seems to be entirely with ALTQ shaping.

                I decided to spend the day booted natively into pfSense (home alone, so nobody to be bothered with intermittent internet and no access to the TV) to troubleshoot this.

                Ultimately after different iterations of ALTQ shapers with and without codel I couldn't find a single one that offered even remotely acceptable performance and that didn't introduce gigantic latency / bufferings spikes.

                I decide to try this: https://forum.pfsense.org/index.php?topic=126637.0

                Lo-and-behold, it worked like a charm. Using dummynet and real fq_codel on limiters gives me results I would expect without the altq insanity.
                https://www.dslreports.com/speedtest/26865693

                I don't know if I'm the only one experiencing thing, but it honestly seems like currently altq is introducing side effects worse than the problems it is supposed to fix.

                1 Reply Last reply Reply Quote 0
                • w0wW
                  w0w
                  last edited by

                  pfSense 2.4.3 alphabeta built on Sat Dec 16 11:23:26 CST 2017,
                  Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz e3c226d2i (2xi210 LAN)
                  tunables
                  kern.ipc.maxsockbuf 256000000
                  hw.igb.rxd="4096"
                  hw.igb.txd="4096"
                  net.inet.tcp.syncache.hashsize=1024
                  net.inet.tcp.syncache.bucketlimit=100
                  net.isr.defaultqlimit=4096
                  net.link.ifqmaxlen=10240
                  hw.igb.rx_process_limit="-1"
                  hw.igb.num_queues=2
                  dev.igb.0.fc=0
                  dev.igb.1.fc=0
                  kern.ipc.nmbjumbo9="20000"
                  kern.ipc.nmbclusters="1000000"
                  WAN is PPPoE 300/300Mbit over gigabit LAN to ISP router (some CISCO with 10G fiber optic connection)

                  FQ_CODEL enabled,  Hi-Res bufferbloat and other settings as posted by NaterGator:

                  http://www.dslreports.com/speedtest/26877901

                  FQ_CODEL enabled,  Hi-Res bufferbloat and 30/30 streams:

                  http://www.dslreports.com/speedtest/26877933

                  FQ_CODEL disabled, Hi-Res bufferbloat and other settings as posted by NaterGator:

                  http://www.dslreports.com/speedtest/26877771

                  FQ_CODEL disabled, Hi-Res bufferbloat and 30/30 streams:

                  http://www.dslreports.com/speedtest/26877806

                  FQ_CODEL disabled, no tunables, Hi-Res bufferbloat and other settings as posted by NaterGator:

                  http://www.dslreports.com/speedtest/26877572

                  FQ_CODEL disabled, no tunables, Hi-Res bufferbloat and 30/30 streams:

                  http://www.dslreports.com/speedtest/26877682

                  I do not see any huge difference, just some fluctuations that are mostly on ISP side I think.

                  If you want me to test ALTQ shaper, please provide some sample configuration. But really, I have had some not very good experience with ALTQ at least it have twice as much overhead bandwidth comparing to IPFW shaper.

                  1 Reply Last reply Reply Quote 0
                  • N
                    NaterGator
                    last edited by

                    Interesting results… I wonder if asymmetric link bandwidth is having a greater influence?

                    This was my "typical" basic altq test with no limiter/fq_codel: https://i.imgur.com/d1vQLFc.png (only the one shaper on the WAN interface)

                    I also tried the configuration outlined here: http://www.speedtest.net/insights/blog/maximized-speed-non-gigabit-internet-connection/

                    Also...go bolts?

                    1 Reply Last reply Reply Quote 0
                    • w0wW
                      w0w
                      last edited by

                      ALTQ CODELQ, NaterGator settings — http://www.dslreports.com/speedtest/27005845  As you can see dslreports automatucally dropped to 18 : 6 streams.
                      And for the 30/30 streams we have a problem! Triple test start ended with stuck on idle latency testing with spikes (failed due to overall timeout. error:2) and at the end I've got this with 24/24 http://www.dslreports.com/speedtest/27006168
                      And repeat test with FQ_CODEL and 30/30 — http://www.dslreports.com/speedtest/27006586
                      There is something broken in ALTQ CODELQ…

                      1 Reply Last reply Reply Quote 0
                      • N
                        NaterGator
                        last edited by

                        Thanks for the extra effort and offering some level of confirmation that I'm not totally crazy. I'm not sure if this is an issue I should submit to the pfSense tracker or if this belongs upstream on FreeBSD's end.

                        FWIW: To reduce variables I use the preferences on the dslreports test to set fixed servers that I know are close by and a fixed number of streams.

                        1 Reply Last reply Reply Quote 0
                        • C
                          Chrismallia
                          last edited by

                          @w0w

                          You get great results :) using fq_codel. The minimum  ping spike I could get was 150 something just on download, upload is fine , but I think ISP matters and also that you have a symmetrical  speed  makes a difference

                          1 Reply Last reply Reply Quote 0
                          • w0wW
                            w0w
                            last edited by

                            Chrismallia, yes it's ISP, just very good ISP network at least in my location.
                            NaterGator, it's FreeBSD, but I don't think anybody cares ALTQ CODELQ, you have alternative with HFSC and codel enabled queue. I think next 3-5 years we will see some progress for IPFW or ALTQ — it does not matter they both need code to be rewritten from scratch, because of used 32-bit integers they both do not support modern traffic bandwidth (over 4 Gigs/sec).

                            1 Reply Last reply Reply Quote 0
                            • N
                              NaterGator
                              last edited by

                              @w0w:

                              NaterGator, it's FreeBSD, but I don't think anybody cares ALTQ CODELQ, you have alternative with HFSC and codel enabled queue. I think next 3-5 years we will see some progress for IPFW or ALTQ — it does not matter they both need code to be rewritten from scratch, because of used 32-bit integers they both do not support modern traffic bandwidth (over 4 Gigs/sec).

                              Hmm, I do see this issue in HFSC with and without codel enabled. What I'm saying is any altq enabled shaping at all triggers the issue.

                              1 Reply Last reply Reply Quote 0
                              • C
                                Chrismallia
                                last edited by

                                w0w

                                What is strange for me is that with no traffic shaping I get low ping spikes  on download and high ping spikes on upload, when enabling any shaping including fq  I get low ping spikes on upload but then get high ping spikes on download,  see my results in post 3 if you may, I cant understand it

                                1 Reply Last reply Reply Quote 0
                                • w0wW
                                  w0w
                                  last edited by

                                  NaterGator, I did not tested HFSC for a long time, but I can test it also, later this week. Can you provide more settings regarding queues you have used? Did you change any other settings like RED or ECN, priority, queue limit in packets?

                                  1 Reply Last reply Reply Quote 0
                                  • w0wW
                                    w0w
                                    last edited by

                                    @Chrismallia:

                                    w0w

                                    What is strange for me is that with no traffic shaping I get low ping spikes  on download and high ping spikes on upload, when enabling any shaping including fq  I get low ping spikes on upload but then get high ping spikes on download,  see my results in post 3 if you may, I cant understand it

                                    I have had some similar results, but I am not sure if it's some DSLreports bug or anything else, their server, or other problems, ex. ISP.
                                    You should repeat your tests 10 times at least with traffic shaping and without, to make some conclusions.

                                    1 Reply Last reply Reply Quote 0
                                    • w0wW
                                      w0w
                                      last edited by

                                      http://www.dslreports.com/speedtest/27223186 ALTQ HFSC codel enabled queue + ECN, dslreports 24/6 HD enabled
                                      https://www.dslreports.com/speedtest/27223235 ALTQ HFSC codel enabled queue + ECN, dslreports 30/30 HD enabled — twice got failed due to overall timeout. error:2
                                      http://www.dslreports.com/speedtest/27223412 FQ_CODEL again

                                      NaterGator did you try to set bandwidth limit for the ALQ shapers more tighten then for FQ_CODEL? I think it can be possible reason for the spikes you've got. The overhead for the ALTQ is slightly bigger so it's possible that it fails on a bandwidth limit. To make sure put the bandwidth to 50% of overall.

                                      I can not explain why dslreports often fails to start with ALTQ CODEL and FQ_CODEL mostly never fails. May be it's by design of DSLreports or ALQ CODEL.  :D

                                      1 Reply Last reply Reply Quote 0
                                      • N
                                        NaterGator
                                        last edited by

                                        @w0w:

                                        NaterGator did you try to set bandwidth limit for the ALQ shapers more tighten then for FQ_CODEL? I think it can be possible reason for the spikes you've got. The overhead for the ALTQ is slightly bigger so it's possible that it fails on a bandwidth limit. To make sure put the bandwidth to 50% of overall.

                                        Oh yes, I tried some absurdly low limits just to be absolutely positive I had plenty of headroom, for example:
                                        https://www.dslreports.com/speedtest/25228491

                                        The insane spikey buffering effect from ALTQ is definitely still there.

                                        1 Reply Last reply Reply Quote 0
                                        • w0wW
                                          w0w
                                          last edited by

                                          Did you try ECN on ALTQ? It is enabled by default on IPFW FQ_CODEL.

                                          1 Reply Last reply Reply Quote 0
                                          • N
                                            NaterGator
                                            last edited by

                                            Yes, I tried with and without ECN.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.