Inter-subnet routing throughput lower than expected

[Rural]

I have a fairly decent rack-mount server that I'm testing pfSense on. It has an Intel Xeon E5506 processor (2.13GHz, 4 cores), two onboard Intel GigE NICs and a dual interface PCI-Express GigE card. This machine isn't ridiculously fast, but it isn't a slouch either.

From a virgin pfSense install, I have the WAN on em0, and the LAN on a LAGG of em1, em2, and em3. I've also got VLAN2 and VLAN3 on the same LAGG, OPT1 on VLAN2, OPT2 on VLAN3, and OPT1 and OPT2 are on different subnets (ie. not bridged, which would be silly). The only firewall rules I've added are allow any-any rules for OPT1 and OPT2. Everything else was just left as the defaults.

The performance I'm seeing is disappointing. Between VLAN2 and VLAN3, a 30 second IPerf gets about 228 Mb/s. On the same VLAN (ie. pfSense isn't involved), I see 280 to 300 Mb/s. The workstations are fairly old machines, but doing an IPerf to themselves (ie. loopback), they get more than 700 Mb/s. So the workstation NICs are constraining the test to 280-300 Mb/s and pfSense must be the bottleneck for the inter-VLAN performance (228 Mb/s).

This same box did much better as just a plain Arch Linux box and using Vyatta. Well enough that I didn't seen any point in benchmarking (but I'll be setting up the machine under Arch for testing tomorrow, just to be sure).

I see lots of folk on the forums achieving 2-6 Gb/s performance on hardware that isn't much different than what I've got. I certainly see folk achieving similar throughput to what I'm seeing on relatively modest hardware. What might they be doing that I'm not? Or maybe it would be more correct to ask what my pfSense box is doing that there's isn't?

[Rural]

Boo-ya!

So I struggled through getting an Arch router set up on exactly the same hardware and the same network configuration. Then I went on a wild goose chase as to why I couldn't ping the Windows workstations I was using for testing (Windows firewall was disallowing the pings). ;)

I did some IPerfs and was shocked to see exactly the same performance figures as under pfSense. My heart didn't know whether to swell or shrink at this point. Either my problem was worse than I thought or it was something simple and I could use pfSense for my routing needs.

It was something simple.

So I hook up my relatively new laptop to the same network and run an IPerf to the Arch box. It gets 938 Mb/s. "What!", I thinks to myself. Running IPerf to a machine on the other side of the Arch router's NAT wall, it gets 924 Mb/s. Awesome!

Seems like the problem is just the workstations I was using for testing have really horrible NICs.

[Rural]

Put pfSense back on the hardware (just swapped out a hot-swap drive) and did some similar tests between more modern machines than the problem workstations. Unsurprisingly, it cruises fairly well.

Nice! Now I can start playing with some features that would be a bear to set up under plain old Arch.