Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Inter-subnet routing throughput lower than expected

    Scheduled Pinned Locked Moved Hardware
    3 Posts 1 Posters 1.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      Rural
      last edited by

      I have a fairly decent rack-mount server that I'm testing pfSense on. It has an Intel Xeon E5506 processor (2.13GHz, 4 cores), two onboard Intel GigE NICs and a dual interface PCI-Express GigE card. This machine isn't ridiculously fast, but it isn't a slouch either.

      From a virgin pfSense install, I have the WAN on em0, and the LAN on a LAGG of em1, em2, and em3. I've also got VLAN2 and VLAN3 on the same LAGG, OPT1 on VLAN2, OPT2 on VLAN3, and OPT1 and OPT2 are on different subnets (ie. not bridged, which would be silly). The only firewall rules I've added are allow any-any rules for OPT1 and OPT2. Everything else was just left as the defaults.

      The performance I'm seeing is disappointing. Between VLAN2 and VLAN3, a 30 second IPerf gets about 228 Mb/s. On the same VLAN (ie. pfSense isn't involved), I see 280 to 300 Mb/s. The workstations are fairly old machines, but doing an IPerf to themselves (ie. loopback), they get more than 700 Mb/s. So the workstation NICs are constraining the test to 280-300 Mb/s and pfSense must be the bottleneck for the inter-VLAN performance (228 Mb/s).

      This same box did much better as just a plain Arch Linux box and using Vyatta. Well enough that I didn't seen any point in benchmarking (but I'll be setting up the machine under Arch for testing tomorrow, just to be sure).

      I see lots of folk on the forums achieving 2-6 Gb/s performance on hardware that isn't much different than what I've got. I certainly see folk achieving similar throughput to what I'm seeing on relatively modest hardware. What might they be doing that I'm not? Or maybe it would be more correct to ask what my pfSense box is doing that there's isn't?

      1 Reply Last reply Reply Quote 0
      • R
        Rural
        last edited by

        Boo-ya!

        So I struggled through getting an Arch router set up on exactly the same hardware and the same network configuration. Then I went on a wild goose chase as to why I couldn't ping the Windows workstations I was using for testing (Windows firewall was disallowing the pings). ;)

        I did some IPerfs and was shocked to see exactly the same performance figures as under pfSense. My heart didn't know whether to swell or shrink at this point. Either my problem was worse than I thought or it was something simple and I could use pfSense for my routing needs.

        It was something simple.

        So I hook up my relatively new laptop to the same network and run an IPerf to the Arch box. It gets 938 Mb/s. "What!", I thinks to myself. Running IPerf to a machine on the other side of the Arch router's NAT wall, it gets 924 Mb/s. Awesome!

        Seems like the problem is just the workstations I was using for testing have really horrible NICs.

        1 Reply Last reply Reply Quote 0
        • R
          Rural
          last edited by

          Put pfSense back on the hardware (just swapped out a hot-swap drive) and did some similar tests between more modern machines than the problem workstations. Unsurprisingly, it cruises fairly well.

          Nice! Now I can start playing with some features that would be a bear to set up under plain old Arch.

          1 Reply Last reply Reply Quote 0
          • First post
            Last post
          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.