Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    10GB lan speeds

    Scheduled Pinned Locked Moved
    Virtualization
    5
    29
    3.2k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • kiokomanK
      kiokoman LAYER 8
      last edited by kiokoman

      MTU ?
      was it the same? should be 9000 for 10gbe on all interfaces

      ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
      Please do not use chat/PM to ask for help
      we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
      Don't forget to Upvote with the 👍 button for any post you find to be helpful.

      1 Reply Last reply Reply Quote 0
      • 1
        12Sulljo
        last edited by

        It was at 1500 on everything then I set every lan interface, and my windows 10 pc to 9000. But now my PC to unraid transfer speed has gone from 52MB/s to 18

        1 Reply Last reply Reply Quote 0
        • NollipfSenseN
          NollipfSense @12Sulljo
          last edited by

          @12Sulljo Stuff like, what pfSense version, hardware, RAM, NIC, and packages installed to help diagnose a throughput decline...seems that you are making progress, are you?

          pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
          pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

          1 1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by stephenw10

            Yeah, so your load is not spreading across the cores well at all. Particularly on the bxe NIC.

            How are those NICs assigned? bxo0 and vmx0 seem to be doing all the work there.

            If you setup multiple connections do you see more queues loaded in the top output?

            You probably need this:

            MULTIPLE QUEUES
     The vmx driver supports multiple transmit and receive queues.  Multiple
     queues are	only supported by certain VMware products, such	as ESXi.  The
     number of queues allocated	depends	on the presence	of MSI-X, the number
     of	configured CPUs, and the tunables listed below.	 FreeBSD does not
     enable MSI-X support on VMware by default.	 The
     hw.pci.honor_msi_blacklist	tunable	must be	disabled to enable MSI-X sup-
     port.

            You definitely would in VMWare.

            Try running vmstat -i to see how many queues are created.

            The bxe driver looks to be creating the expected 4 default queues but using only one of them. That could be simply because of how you're testing.

            Steve

            1 1 Reply Last reply Reply Quote 0
            • 1
              12Sulljo @stephenw10
              last edited by

              @stephenw10 vmstat -i.PNG

              1 Reply Last reply Reply Quote 0
              • 1
                12Sulljo @NollipfSense
                last edited by 12Sulljo

                @NollipfSense said in 10GB lan speeds:

                @12Sulljo Stuff like, what pfSense version, hardware, RAM, NIC, and packages installed to help diagnose a throughput decline...seems that you are making progress, are you?

                pfSense version 2.4.4-RELEASE-p3

                hardware: unraid VM with 8 threads of a Ryzen 2700 which is passed through using vmx3net, the card then goes into a dell force 10 s55 switch with all ports on a single vlan, the back of the switch has 4 sfp+ ports, one of those connects to my workstation which uses a 2700x and a Supermicro AOC-STGN-i2S rev 2.0 Dual SFP+ Intel 82599 10GbE Controller NIC

                The other ethernet ports on the switch connect to my wifi and other devices.

                RAM 8GB

                pfSense NIC: Dell Y40PH broadcom 10GB dual port pci-e sfp+ nic

                packages installed: none yet

                1 Reply Last reply Reply Quote 0
                • 1
                  12Sulljo
                  last edited by

                  interface status.PNG

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Ok so you only have one queue for the vmx NIC like I suspected. Add that loader tunable I posted to /boot/loader.conf.local and reboot.

                    What's in bridge0? But more to the point why do you have a bridge? That's probably killing your throughput.

                    What's on igb0? The interrupt rate is quite high there.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • 1
                      12Sulljo
                      last edited by

                      igb0 is the wan

                      I setup the bridge because it was recommended to me on an unraid forum on how to have the dockers and everything else on my server be visible to the rest of my physical lan network.

                      before setting that up my server was unable to communicate with the rest of the network, only pfsense and the rest of the physical network was able to communicate.

                      I'm still new at this stuff, I'm not sure what you mean by adding the loader tunable, all I see on this post is something called multiple queues but I'm not sure how I would turn that on.

                      Thank you all for this help by the way, I'm eager to understand where my mistakes are so I can learn from them.

                      DerelictD 1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate @12Sulljo
                        last edited by

                        @12Sulljo said in 10GB lan speeds:

                        I setup the bridge because it was recommended to me on an unraid forum on how to have the dockers and everything else on my server be visible to the rest of my physical lan network.

                        That would probably be a bridge on unraid, not on your firewall.

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          The options for vmx are detailed here: https://www.freebsd.org/cgi/man.cgi?query=vmx

                          But you want to create the file /boot/loader.conf.local (if it doesn't exist) and add to it the line:
                          hw.pci.honor_msi_blacklist=0

                          You can do that from the webgui in Diag > Edit File.

                          Then when you reboot the vmx driver will enable MSI and you will get multiple queues on the NIC. That should allow you to use multiple cores and get far better throughput for multiple connections. Though you may still be restricted somewhere else.

                          Steve

                          1 1 Reply Last reply Reply Quote 0
                          • 1
                            12Sulljo @stephenw10
                            last edited by

                            @stephenw10 ok so I added it but before I do anything should I remove the usb one?

                            pf.PNG

                            stephenw10S 1 Reply Last reply Reply Quote 0
                            • 1
                              12Sulljo
                              last edited by

                              Also, how can my vmx and physical lan be on the same subnet if they aren't in a bridge?

                              1 Reply Last reply Reply Quote 0
                              • DerelictD
                                Derelict LAYER 8 Netgate
                                last edited by

                                Use a switch?

                                Chattanooga, Tennessee, USA
                                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                1 Reply Last reply Reply Quote 0
                                • 1
                                  12Sulljo
                                  last edited by

                                  I have a switch, How does a virtual nic which communicates between unraid and a VM of pfsense use a switch?

                                  1 Reply Last reply Reply Quote 0
                                  • DerelictD
                                    Derelict LAYER 8 Netgate
                                    last edited by

                                    Like I said before. you do not need a pfSense bridge to communicate across your virtual environment. You need a bridge in the virtual environment.

                                    Chattanooga, Tennessee, USA
                                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                    1 Reply Last reply Reply Quote 0
                                    • 1
                                      12Sulljo
                                      last edited by

                                      ok, but then how does unraid communicate with the physical environment without assigning an ip on a different subnet?

                                      1 Reply Last reply Reply Quote 0
                                      • DerelictD
                                        Derelict LAYER 8 Netgate
                                        last edited by

                                        you connect your hypervisor to the bridge by bridging in a hypervisor physical NIC.

                                        Chattanooga, Tennessee, USA
                                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                        1 1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator @12Sulljo
                                          last edited by

                                          @12Sulljo

                                          That line should should go in /boot/loader.conf.local to avoid being overwritten at upgrade. You will probably need to create that.

                                          Don't remove the default values from loader.conf.

                                          Steve

                                          1 1 Reply Last reply Reply Quote 0
                                          • 1
                                            12Sulljo @stephenw10
                                            last edited by

                                            @stephenw10 the line increased my speed to 54MB/s and only two of my threads are running at 100%

                                            stephenw10S 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post