Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Slow upload speeds on HP Z2 G9 PFSense Box

    Scheduled Pinned Locked Moved General pfSense Questions
    74 Posts 3 Posters 4.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • BearB
      Bear @stephenw10
      last edited by

      @stephenw10

      I've got the numbers from the i5-14500 /w the X520, flow control off.

      With - P -R

      # iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 -R
      [ ID]   Interval         Transfer     Bitrate         Retr
      [SUM]   0.00-10.01  sec  5.20 GBytes  4.46 Gbits/sec  10655       sender
      [SUM]   0.00-10.00  sec  5.07 GBytes  4.35 Gbits/sec              receiver
      

      With -P

      # iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120
      [ ID]   Interval         Transfer     Bitrate         Retr
      [SUM]   0.00-10.01  sec  1.61 GBytes  1.38 Gbits/sec  27815      sender
      [SUM]   0.00-10.02  sec  1.60 GBytes  1.37 Gbits/sec             receiver
      

      Note that with Speed Test, I initially see speeds of 1.6-1.7Gbit that quickly slow down to 400-650Mbits. This does not happen with the 6100.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Hmm, so that looks like the same result as a client behind the 6100.

        Does a client behind the i5 still see a much lower limit on iperf? Like ~400Mbps?

        If so that implies a LAN side issue with traffic going into pfSense.

        BearB 1 Reply Last reply Reply Quote 0
        • BearB
          Bear @stephenw10
          last edited by

          @stephenw10 Yes, a client behind the i5 sees a much lower limit. Though I'm not sure where to go with this from here. Same cables, same switch between both units. Same config even, since it was a restore.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Hmm, well that seems to confirm some issue on the LAN side then.

            Still no input errors on the LAN interface?

            Can you try connecting a client to the LAN directly to rule out some low level issue between the LAN and switch?

            BearB 1 Reply Last reply Reply Quote 0
            • BearB
              Bear @stephenw10
              last edited by

              @stephenw10 I've tried that. Still getting the same speeds.

              BearB 1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                So to be clear the client behind the i5 sees ~400Mbps both for iperf and speedtest.net? Whether or not via the switch. And no errors are shown on the NIC.

                Hmm. What about an iperf test between the client and pfSense directly? Does that also get throttled for traffic going into the LAN NIC?

                1 Reply Last reply Reply Quote 0
                • BearB
                  Bear @Bear
                  last edited by

                  @Bear To put a finer point on it, there is no "real" LAN side besides physical, since I'm running a filtered bridge. The LAN side is a filtered port on the same NIC that connects to a Netgear L3 multigig switch.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Ah, yes I forgot about that. From a packets-in packets-out perspective it makes no difference. It still feels like the internal NIC causing the issue.

                    However I think it would be worth testing a more basic routing setup if you can just to be sure it still shows the same issue. Hard to see why the 6100 would behave any differently but the fact you're running a bridged setup is at least unusual.

                    BearB 1 Reply Last reply Reply Quote 0
                    • BearB
                      Bear @stephenw10
                      last edited by

                      @stephenw10

                      I’ve already tried two completely different Intel NICs. The i7-14500 has more than enough power to run in filtering bridge mode since the 6100 is doing it as well using the same config.

                      I’ve gathered a bunch of data already - I can’t reconfigure the network as I’m running a /25 of public IPs with lots of firewall rules.

                      Is there a potential issue with the software and more recent Intel hardware here? Again, I’m running as plain and supported a piece of hardware as I can, save for it being a much later model Intel system.

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        You replaced the i5 with an i7? Or is that the client you're testing from? Not that it should matter.

                        Bridging has always been somewhat fragile in pfSense/FreeBSD and can create some unexpected traffic scenarios. It would be good to rule that out entirely if you can. But, yes, I agree that it seems unlikely here since the 6100 passes it OK.

                        BearB 2 Replies Last reply Reply Quote 0
                        • BearB
                          Bear @stephenw10
                          last edited by

                          @stephenw10 Sorry, it’s an i5-14500.

                          1 Reply Last reply Reply Quote 0
                          • BearB
                            Bear @stephenw10
                            last edited by

                            @stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

                            You replaced the i5 with an i7? Or is that the client you're testing from? Not that it should matter.

                            Bridging has always been somewhat fragile in pfSense/FreeBSD and can create some unexpected traffic scenarios. It would be good to rule that out entirely if you can. But, yes, I agree that it seems unlikely here since the 6100 passes it OK.

                            Any other suggestions for diagnostics here? I'm just about at my wit's end - This is a relatively high end workstation with ECC RAM, and otherwise all standardized components.

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Did you try an iperf test between an internal client and pfSense directly?

                              If it is some low level issue I'd expect to see the same issue there for the client sending. Though in that scenario it does cross the bridge differently.

                              You could disabling filtering entirely. If the issue remains that proves it's a driver/hardware issue rather than something in pf.

                              BearB 1 Reply Last reply Reply Quote 0
                              • BearB
                                Bear @stephenw10
                                last edited by Bear

                                @stephenw10 I'll try running iperf3 in server mode later tonight/tomorrow and see what my Mac Studio client (my control) gets to it.

                                The other data I've got is, I've got 2 of these HP Z2 G9s as nodes with the exact same NIC running Proxmox VE (Spicy Debian) and I have none of these upload speed issues with either the command prompt or from within LXCs and VMs.

                                If I did attempt a reinstall, just to give it a clear slate, will my Netgate ID/registration remain the same?

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  Yes the NDI will remain unchanged. You could install 24.11 directly again.

                                  BearB 1 Reply Last reply Reply Quote 0
                                  • BearB
                                    Bear @stephenw10
                                    last edited by

                                    @stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

                                    Yes the NDI will remain unchanged. You could install 24.11 directly again.

                                    As a control, I ran iperf3 on the 6100 and used my Mac to see what I'd get.

                                    # iperf3 -c (router IP)  -P 120
                                    [ ID]   Interval         Transfer     Bitrate         Retr
                                    [SUM]   0.00-10.00  sec  2.48 GBytes  2.13 Gbits/sec        sender
                                    [SUM]   0.00-10.02  sec  2.45 GBytes  2.10 Gbits/sec        receiver
                                    
                                    # iperf3 -c (router IP)  -P 120 -R
                                    [ ID]   Interval         Transfer     Bitrate         Retr
                                    [SUM]   0.00-10.07  sec  3.61 GBytes  3.08 Gbits/sec  29586  sender
                                    [SUM]   0.00-10.00  sec  3.48 GBytes  2.99 Gbits/sec         receiver
                                    

                                    I'm getting better performance through the 6100 than I am hitting it directly.

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Good, that's what I'd expect to see. At those speeds you're probably seeing iperf using 100% of one CPU core. iperf is deliberately single threaded. And pfSense is optimised for routing not serving.

                                      As a side note using 120 streams is probably counter productive. You usually won't see any increase beyond the available number of NIC queues. So 8 for the ix NICs in the 6100.

                                      On the i5 one CPU core is capable of far higher iperf values and the remaining cores are capable of pushing it.

                                      BearB 1 Reply Last reply Reply Quote 0
                                      • BearB
                                        Bear @stephenw10
                                        last edited by

                                        @stephenw10

                                        Looks like you're correct. 6100:

                                        # iperf3 -c (router IP)  -P 8
                                        [SUM]   0.00-10.00  sec  2.48 GBytes  2.13 Gbits/sec            sender
                                        [SUM]   0.00-10.03  sec  2.47 GBytes  2.12 Gbits/sec            receiver
                                        
                                        # iperf3 -c (router IP)  -P 8 -R
                                        [ ID]   Interval         Transfer     Bitrate         Retr
                                        [SUM]   0.00-10.01  sec  3.68 GBytes  3.16 Gbits/sec   14     sender
                                        [SUM]   0.00-10.00  sec  3.68 GBytes  3.16 Gbits/sec            receiver
                                        

                                        I'll try to get i5 numbers in the next day or so. Each core on that is more powerful than the entire Atom CPU, so I'd expect to see higher numbers, unless there's a LAN or bridging issue...hopefully this'll help give us that data.

                                        stephenw10S 1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator @Bear
                                          last edited by

                                          @Bear said in Slow upload speeds on HP Z2 G9 PFSense Box:

                                          Each core on that is more powerful than the entire Atom CPU

                                          Ha, yup. So it would be interesting to see what the limiting factor is there. Unknown throttling aside.

                                          BearB 1 Reply Last reply Reply Quote 0
                                          • BearB
                                            Bear @stephenw10
                                            last edited by

                                            @stephenw10

                                            Running with the i5-14500...

                                            # iperf3 -c (router IP)  -P 8
                                            [ ID]   Interval         Transfer     Bitrate         Retr
                                            [SUM]   0.00-10.00  sec  8.71 GBytes  7.48 Gbits/sec        sender
                                            [SUM]   0.00-10.00  sec  8.70 GBytes  7.47 Gbits/sec        receiver
                                            
                                            # iperf3 -c (router IP)  -P 8 -R
                                            [ ID]   Interval         Transfer     Bitrate         Retr
                                            [SUM]   0.00-10.01  sec  11.0 GBytes  9.40 Gbits/sec  167    sender
                                            [SUM]   0.00-10.01  sec  10.9 GBytes  9.39 Gbits/sec            receiver
                                            

                                            Bear in mind, the RG is connected to the same NIC that the "LAN" side of the bridge is. Just the second port.

                                            Any other thoughts/suggestions?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.