Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Inter vlan routing speed

    Scheduled Pinned Locked Moved General pfSense Questions
    11 Posts 5 Posters 2.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V
      voxel
      last edited by

      I have some issues with throughput for inter vlan routing. Currently I get around 20 MBps from my 1 Gbps link but was hoping for more.

      Everything works as it should but not sure if the low speed is due to my hardware or some configuration error in pfsense.

      Everything is setup as "router on a stick" as i want each vlan to pass through the firewall.

      2.4.5-RELEASE-p1 with below hardware,
      Intel(R) Atom(TM) CPU E3845 @ 1.91GHz
      Current: 498 MHz, Max: 1909 MHz
      4 CPUs: 1 package(s) x 4 core(s)
      AES-NI CPU Crypto: Yes (active)
      82583V Gigabit Network Connection

      All hardware offloading settings are stock.

      Are these results expected or should my hardware be able to route at higher throughput?

      JKnottJ M 2 Replies Last reply Reply Quote 0
      • JKnottJ
        JKnott @voxel
        last edited by

        @voxel

        One thing to bear in mind with "router on a stick" is you're cutting your bandwidth, between VLANs, in half as the packets travel over the same cable twice. Also is that actually MB/s or Mb/s?

        PfSense running on Qotom mini PC
        i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
        UniFi AC-Lite access point

        I haven't lost my mind. It's around here...somewhere...

        V 1 Reply Last reply Reply Quote 0
        • V
          voxel @JKnott
          last edited by

          @jknott
          Interface says full duplex so shouldnt it still be able to handle 1Gbps between vlans?

          Yeah, its MB/s (megabytes). Inside the vlan i can fully saturate the link.

          JKnottJ 1 Reply Last reply Reply Quote 0
          • JKnottJ
            JKnott @voxel
            last edited by

            @voxel

            Yes, it is full duplex, but every packet has to be sent twice over the same link, once in each direction. Don't forget, a computer attached to a network would normally be able to send and receive at the same time. If it was a straight connection, the data wouldn't be sent twice.

            Data rates are normally measured in b/s, while file transfers in B/s. 20 MB = 160 Mb then each packet has to be sent twice, which will cause a slowdown. Also, how fast is the hardware you're running pfsense on. I have a 500/20 connection from my ISP and with the previous computer I used, I would generally get low 500s down. When that computer died, I replaced it with the one described in my sig. Here is a recent speedtest. I saw an immediate performance increase, so the old computer was limiting what I could get.
            Bottom line, a Gb NIC doesn't guarantee Gb throughput, if the rest of the hardware isn't up to it.

            PfSense running on Qotom mini PC
            i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
            UniFi AC-Lite access point

            I haven't lost my mind. It's around here...somewhere...

            V 1 Reply Last reply Reply Quote 0
            • M
              marvosa @voxel
              last edited by

              @voxel Yeah, router on a stick leans towards security, but the trade-off is performance. You'll be very hard-pressed to see anything approaching wire-speed with a router on a stick design.

              Without assessing any other variables, if you have a want (or need) to firewall your VLANS, the only option I see adding a significant performance boost is adding a 2nd NIC and moving any VLANs where you want inter-vlan performance boosted to the 2nd NIC.

              A performance-based design change option would be moving your VLANs to an L3 switch. Your inter-vlan traffic would then be handled by the L3 switch @ wire speed, but then you lose the ability to firewall your VLANs. Personally, I always lean towards performance, but your priorities may be different.

              Another observation that probably isn't the main issue, but may be a contributing factor is the fact that your CPU is running at 1/4th its speed... I would verify that PowerD is enabled and set to either Hiadaptive or Maximum.

              V 1 Reply Last reply Reply Quote 1
              • V
                voxel @marvosa
                last edited by

                @marvosa
                I have more nics so I'll try that approach and see if it makes any difference.

                Yeah, I've considered using an L3 switch instead but as the network is small and the only reason i use vlans is from a security perspective i might as well just skip vlans entirely.

                PowerD is enabled and set to Hiadaptive but will try maximu as well.
                During transfer the CPU peaks at about 30% but not sure at what Hz.

                1 Reply Last reply Reply Quote 0
                • V
                  voxel @JKnott
                  last edited by

                  @jknott
                  Ok, i thought that theoretically full duplex would mean that "router on a stick" would still be able to route 1Gbps. Will try more nics and see if that increases performance.

                  Harware is stated in original post. Quad core Intel(R) Atom(TM) CPU E3845 @ 1.91GHz.

                  H 1 Reply Last reply Reply Quote 0
                  • H
                    heper @voxel
                    last edited by

                    @voxel

                    Whatever others might claim: it's perfectly possible to get near 930MB/s inter-vlan traffic over a single gbe interface in a lab/home setup.
                    Source: I've been doing that on multiple sites for over a decade

                    It's either the CPU that cannot keep up or buggy drivers or it needs nic tuning

                    1 Reply Last reply Reply Quote 1
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Yeah, it depends how you're testing. In a UDP test you can send and receive at close to line rate and see that as throughput. In, much more typical, TCP test though all the ACK traffic and and any other traffic the other way removes available bandwidth for the test traffic.

                      I would expect to see significantly better than 160Mbps in an iperf test there with that CPU.

                      Check the output of top -aSH whilst testing. Make sure you are not pinning a single CPU core.

                      Steve

                      V 1 Reply Last reply Reply Quote 1
                      • V
                        voxel @stephenw10
                        last edited by

                        @stephenw10
                        My tests have been just plain file copy. From within the vlan they perform near max but from any external vlan they drop to 160 Mbps.

                        Will check cpu utilization and test if dedicated nics make any difference.

                        Is it possible I've made any configuration error with the vlans and the routing that could interfere?

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Unlikely, it's just forwarding in and out between two directly connected subnets.

                          Some MTU mismatch could cause that sort of problem.

                          Steve

                          1 Reply Last reply Reply Quote 1
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.