Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    600MB down to 10MB Performance Issue

    General pfSense Questions
    5
    14
    1756
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      adfischer last edited by

      My pfSense box performance is awful and I just cant seem to find a good explanation.  Hoping someone can point me in the right direction.

      Hardware: HP Proliant DL360G6.  4xCPU, 4GB RAM, 256G HDD
      ISP: Comcast Business Class @ 100MB
      Users: ~250

      System is configured pretty vanilla.  OpenVPN is configured for admin access, about 5 inbound rules, no proxys or filters running.

      Here is the symptoms of what I am seeing:

      • Client going through the firewall is averaging around 13MB
      • Client bypassing the firewall directly into the Comcast router is averaging 98MB
      • Client running iperf is getting ~600MB to the pfSense box on both the LAN and WAN sides

      Most of the articles I find on performance point to network cards but I am getting great performance running iperf tests.  Below is my top system activity which doesnt seem to show hardly any load on the system.  All the advanced network settings are out of the box defaults.

      I have plenty of horsepower, network performance seems good, the Internet connection itself seems fine.  Any suggestions on where to look next?

      _last pid: 39659;  load averages:  0.02,  0.05,  0.06  up 56+07:32:34    16:58:08
      138 processes: 5 running, 106 sleeping, 2 zombie, 25 waiting

      Mem: 76M Active, 79M Inact, 121M Wired, 552K Cache, 112M Buf, 3213M Free
      Swap: 8192M Total, 8192M Free

      PID USERNAME  PRI NICE  SIZE    RES STATE  C  TIME  WCPU COMMAND
        11 root      171 ki31    0K    32K CPU1    1 1337.0 100.00% [idle{idle: cpu1}]
        11 root      171 ki31    0K    32K CPU0    0 1318.6 100.00% [idle{idle: cpu0}]
        11 root      171 ki31    0K    32K RUN    2 1315.1 100.00% [idle{idle: cpu2}]
        11 root      171 ki31    0K    32K CPU3    3 1293.9 99.85% [idle{idle: cpu3}]
        12 root      -68    -    0K  200K WAIT    0  17.3H  2.59% [intr{irq257: bce0}]
        12 root      -68    -    0K  200K WAIT    0 756:53  1.76% [intr{irq258: bce1}]_

      1 Reply Last reply Reply Quote 0
      • F
        firewalluser last edited by

        What CPU do you have?

        Intel chips can under perform when the L2 cache gets its wrong with out of branch execution and some other code instructions. Whether that would equate to the drop in performance that you see I dont know.

        What if you rolled things back a stage, like see how things are before OpenVPN is installed, or try a real default install to see if you see the same performance drops.

        You might spot which config change caused the problem, ie have you changed the MTU's as an example?

        Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

        Asch Conformity, mainly the blind leading the blind.

        1 Reply Last reply Reply Quote 0
        • A
          adfischer last edited by

          Interesting on the L2 cache.  These CPUs also have L3.  Not sure I can disable that but its worth a shot to try.

          CPU Type Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
          4 CPUs: 1 package(s) x 4 core(s)

          I did not roll back OpenVPN yet but I did stop the services and test with no luck.  In fact I shut down every service I could and still poor performance.

          Have not changed any network settings, including MTUs.

          I do plan on going back to default and testing but have to wait for a downtime window that I can get in there and do it.

          Thanks for the advice.

          1 Reply Last reply Reply Quote 0
          • S
            Supermule Banned last edited by

            Have you checked for eny interface speed mismatch settings??

            1 Reply Last reply Reply Quote 0
            • A
              adfischer last edited by

              Yes.  Both interfaces are 1G / FD.  I thought of that but what is interesting is when I run a iperf test between a client and the server I get great throughput.

              1 Reply Last reply Reply Quote 0
              • stephenw10
                stephenw10 Netgate Administrator last edited by

                My first check with that level of throttling would be an interface mismatch too. Check the Status: Interfaces: page for errors or collisions.

                Read this: https://doc.pfsense.org/index.php/Low_Throughput_Troubleshooting

                You're seeing good bandwidth bewteen a client and the LAN interface so the throttling is presumed to be on the LAN side. You can check that by running a download test on the firewall itself though. At the command line:

                fetch -o /dev/null http://cachefly.cachefly.net/10mb.test

                You have listed everything in Bytes (B) but I assume some of that is bits (b). That can confuse matters, a lot!  ;)

                Steve

                1 Reply Last reply Reply Quote 0
                • A
                  adfischer last edited by

                  Thanks for the reply.  I thought about the interface mismatch as well and have checked that.  Both are good:

                  WAN interface (bce0):
                  Media 1000baseT <full-duplex,flowcontrol,rxpause,txpause>In/out packets 25154099/13742928 (29.89 GB/2.68 GB)
                  In/out packets (pass) 25154099/13742928 (29.89 GB/2.68 GB)
                  In/out packets (block) 44073/479 (4.46 MB/55 KB)
                  In/out errors 0/0
                  Collisions 0

                  LAN interface (bce1):
                  Media 1000baseT <full-duplex,flowcontrol,master,rxpause,txpause>In/out packets 12789344/24380268 (2.35 GB/29.48 GB)
                  In/out packets (pass) 12789344/24380268 (2.35 GB/29.48 GB)
                  In/out packets (block) 19664/2 (2.13 MB/152 bytes)
                  In/out errors 0/0
                  Collisions 0

                  I ran the download on the firewall and the result was 1246 kBps.  If I convert that right I am at 9.734375 Mbps.

                  Sorry about the B vs b, you are right big difference.  I have been going through the troubleshooting & tuning document.  Unfortunately I cant reboot at the moment so I will have to wait and see.

                  I have confirmed I dont have any traffic shaping on.  One thing I have noticed is the issue only seems to be on incoming traffic.  The line is 100 Mbps down and 20 Mbps up.  I can hit the upstream limit every time, the downstream is only getting ~ 10-13 Mbps.</full-duplex,flowcontrol,master,rxpause,txpause></full-duplex,flowcontrol,rxpause,txpause>

                  1 Reply Last reply Reply Quote 0
                  • S
                    Supermule Banned last edited by

                    Can you set it to 1000 Auto on both ends and no flowcontrol and rx/tx off??

                    And try again….

                    1 Reply Last reply Reply Quote 0
                    • A
                      adfischer last edited by

                      Ok, I set both sides to:

                      1000baseT <full-duplex>and set the switch to the same.  No change in the result.</full-duplex>

                      1 Reply Last reply Reply Quote 0
                      • S
                        Supermule Banned last edited by

                        Can you set it to full auto pls. On all 3 attached NIC's :)

                        1 Reply Last reply Reply Quote 0
                        • bmeeks
                          bmeeks last edited by

                          @Supermule:

                          Can you set it to full auto pls. On all 3 attached NIC's :)

                          Try what Supermule suggested – "auto" instead of hard-coding the duplex.  1000BaseT links expect the duplex settings to be "auto".  It wants to auto-negotiate with the endpoint.  If you hard-code, one side can silently fall back to half-duplex on you.

                          Bill

                          1 Reply Last reply Reply Quote 0
                          • stephenw10
                            stephenw10 Netgate Administrator last edited by

                            What do you have upstream of the WAN interface? Some sort of Comcast CPE box I assume. Do you have access to it? Is it showing any errors?

                            That download result from cachefly is really only valid if you prove you can download the test file far faster with a direct connection. Though 10Mbps is about what you're seeing at the clients.  ;)

                            Steve

                            1 Reply Last reply Reply Quote 0
                            • A
                              adfischer last edited by

                              Thanks for the suggestion.  I have tried every mode on the NICs with no change in the results (it is a dual port NIC).  I have also been playing with the advanced network setting and trying different combinations.  I am really beginning to believe this is probably related to the NIC driver itself based on some more research I have been doing.  I have spent so much time on this I am about to just pull the trigger and buy one preconfigured from pfSense so I know it works and has support.  I usually dont want to give up and want to find the answer but I am spending way too much time on this and its taking away from my other projects.  Any downside to the preconfigured pfSense boxes you are aware of?

                              1 Reply Last reply Reply Quote 0
                              • S
                                Supermule Banned last edited by

                                Pretty sure its a Broadcom chip. Try use dual or quad port INtel's.

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post