I wanted to follow up on this thread from a couple years and share some updated 10Gbit performance statistics using the latest version of pfSense (2.5.0 at the time of this writing). Overall, I have to say that I'm quite impressed and seeing 10 - 25% increases in performance (packet throughput) compared to when I posted this back in 2018. The testing setup is essentially the same (i.e. the only change I have made hardware wise is switch out the 2 port Chelsio T520 with its bigger brother, the 4 port Chelsio T540):
Host 1: i7 4790K based machine with 32GB RAM, Intel X550 NIC running Debian Linux 10.8
Host 2: i5 7600 based machine with 16GB RAM, Intel X550 NIC, running Debian Linux 10.8
Switch: Ubiquiti ES-16-XG
pfSense: Supermicro 5018D-F8NT server with 16GB RAM, and additional Chelsio T540-SO-CR SFP+ add-on card.
Host 1 and Host 2 are on separate network networks segments (let's call them VLAN 1 and VLAN 2), and VLAN 1 is allowed to talk to VLAN 2 across the firewall without restrictions. Snort is active on both VLAN 1 and VLAN 2.
Even with Snort enabled, I'm now seeing 1.3 - 1.5 million packets of throughput across the firewall when running a Flent RRUL test. The average is probably closer to 1.35 - 1.40 million packets. I like the Flent RRUL test because it is full duplex, i.e. tests upload and download at the same time (4 parallel RX streams and 4 parallel TX streams, tested for 60 seconds):
https://flent.org/tests.html#the-realtime-response-under-load-rrul-test
Flent RRUL Test Results:
Please ignore the avg ping values, these don't appear to be accurate latency calculations by the test.
One of the top test results:
avg median
Ping (ms) ICMP : 7.14 4.53 ms
Ping (ms) UDP BE : 545.44 4.05 ms
Ping (ms) UDP BK : 516.01 5.16 ms
Ping (ms) UDP EF : 743.00 2.83 ms
Ping (ms) avg : 601.48 4.44 ms
TCP download BE : 1714.08 1775.34 Mbits/s
TCP download BK : 2416.47 2488.45 Mbits/s
TCP download CS5 : 2377.02 2407.92 Mbits/s
TCP download EF : 2289.63 2323.54 Mbits/s
TCP download avg : 2199.30 2223.70 Mbits/s
TCP download sum : 8797.20 8894.63 Mbits/s
TCP totals : 17689.26 17899.85 Mbits/s
TCP upload BE : 2318.72 2407.06 Mbits/s
TCP upload BK : 1867.99 1952.64 Mbits/s
TCP upload CS5 : 2375.49 2423.72 Mbits/s
TCP upload EF : 2329.86 2427.93 Mbits/s
TCP upload avg : 2223.01 2255.39 Mbits/s
TCP upload sum : 8892.06 9019.27 Mbits/s
Closer to average:
avg median
Ping (ms) ICMP : 3.02 1.79 ms
Ping (ms) UDP BE : 693.13 2.92 ms
Ping (ms) UDP BK : 693.84 2.49 ms
Ping (ms) UDP EF : 701.94 2.65 ms
Ping (ms) avg : 696.30 2.71 ms
TCP download BE : 1371.36 1323.14 Mbits/s
TCP download BK : 2508.64 2556.88 Mbits/s
TCP download CS5 : 2356.60 2475.18 Mbits/s
TCP download EF : 1318.73 1310.40 Mbits/s
TCP download avg : 1888.83 1929.42 Mbits/s
TCP download sum : 7555.33 7717.70 Mbits/s
TCP totals : 16483.28 16708.92 Mbits/s
TCP upload BE : 1882.59 2066.45 Mbits/s
TCP upload BK : 2427.06 2475.99 Mbits/s
TCP upload CS5 : 2195.50 2269.01 Mbits/s
TCP upload EF : 2422.80 2523.43 Mbits/s
TCP upload avg : 2231.99 2265.72 Mbits/s
TCP upload sum : 8927.95 9062.89 Mbits/s
Overall, I'm very happy with these results and it shows me that there is a lot of life left in this Xeon-D hardware (purchased back in 2017), if / when multi-gigabit internet service is available.