Multi combined 10gbe and 1gbe bridged LAN with dual WAN

digital_janitor

Hello community. Long time lurker and information sponge, but first post.

I have long since had a dream of not spending $1000 on a 10gbe switch. I have been working on and off for some time trying to get a decent performing bridge working on several flavors of Linux, and several manufactures of 10g cards. Pfsense is by far my favorite, and I'm happy to say I've achieved my goal using it.

My specific goal was to get 4x10g +1x1g ports bridged for a flat LAN configuration, and then a dual WAN setup. Using the 1g port as an uplink to a 24 port managed switch, it effectively gives me the equivalent of a 24 port gigabit switch with 4 10 gig links. This allows me to build a home "core" 10g infrastructure of NAS, ESX clusters, etc. Plus 10g to my desktop just for overkill, and hook everything else up to the 24 port gig switch. I also wanted to do it as cheaply as possible.

I'm making this post because I have tried numerous combinations of hardware and Linux distributions, with mostly lackluster results. Tons of driver issues, and even when the cards do work, terrible performance. I hope my pain and effort might help someone else down the road. I am assuming a general working knowledge of pfsense.

Disclaimer, your mileage may vary, I make no guarantees you won't shoot your eye out, or burn down your house or whatever, so use the following information at your own risk.

Hardware: Asus Sabertooth x58 mb. 6 gig ram. Xeon W3550. 2x Chelsio S320E-LP-CR Dual-Port 10GBe PCI-E Adapter w/ SFPs. 1x INTEL - PRO/1000 PT QUAD PORT SERVER ADAPTER PCI-E (EXPI9404PT). I had the motherboard and ram already. I ebayed the xeon for around 50 bucks, paid 80 a piece for the Chelsio cards, and 50 for the Intel card.

Software: PFSENSE 2.1.3-RELEASE (amd64)

10g cards in PCIEx8, 1g card in PCIEx4

Install pfsense. Select em0 as WAN, em2 as LAN (for my intel card em0 is the top port, em3 is bottom. Walk through the initial web config, and complete installation. Enable SSH. (This assumes LAN gets a static address with DHCP server enabled.)

If the Chelsio cards do not load, and you get the message: cxgb0: Could not find firmware image cxgb_t3fw.ko repeatedly, do the following:

Download the compiled FreeBSD 8.3 ko file from http://files.nl.pfsense.org/jimp/ko-8.3/amd64/cxgb_t3fw.ko. (Thanks to whoever did this)
SCP the file to your pfsense box to /boot/kernel/.
chmod 555 /boot/kernel/cxgb_t3fw.ko
vi /boot/loader.conf
add if_cxgb_load="YES" on it's own line if it doesn't exist.
reboot

(Hit apply when necessary)
Go to interfaces, assign. Add interfaces OPT1-OPT5. This should give you a config of WAN:em0 LAN:em2 OPT1:em1 OPT2:cxgb0 OPT3:cxgb1 OPT4:cxgb2 OPT5:cxgb3 (depending on your hardware this may vary.)
Go to interfaces, and select each interface and enable it.
Go to System, advanced, system tunables, change net.link.bridge.pfil_member to 0(from default (1)).
Go to Firewall, rules, add a rule for each interface LAN, OPT2, OPT3, OPT4, OPT5; Pass rule, all defaults except Protocol:Any. To be clear, that will be 5 rules, all identicle, on each interface.
Go to interfaces, assign, bridges.
Create BRIDGE0, use OPT4 and OPT5 to create the bridge.
Go to interfaces, assign. Add interface OPT6, this should be BRIDGE0.
Go to Firewall, rules, add a rule for OPT6, Pass rule, all defaults except Protocol:Any.

Disclaimer for the next few steps; I'm pretty sure they are right, but I'm writing this from memory. Sometimes you have to remove an interface from a bridge in order for pfsense to allow you to manipulate it the way you want, so if it's wrong, I apologize, try a couple combos, it's pretty straight forward, just make sure you can always get to the webgui, or console, I may have had to disable the DHCP server at one point)

Go to interfaces, assign, bridges. Edit Bridge0 to include LAN, OPT2, OPT3, OPT4, OPT5.
Go to interfaces, assign. Change LAN through OPT6 interfaces to the following (WITHOUT HITTING APPLY BETWEEN THE SELECTIONS): LAN:BRIDGE0, OPT1:em1, OPT2:em2, OPT3:cxgb0, OPT4:cxgb1, OPT5:cxgb2, OPT6, cxgb3. (now apply)
Go to interfaces, assign, bridges, and remove LAN from the bridge, and make sure OPT2,OPT3,OPT4,OPT5,OPT6 are all part of the bridge.
You can re-enable the DHCP server on LAN at this point if it was disabled.

I am going to forgo the dual WAN setup instructions as there are others out there. For the purposes here em1/OPT1 was reserved for the second WAN connection. em2/OPT2 is reserved for an uplink to a multiport 1gig switch.

Placing point to point hosts on all 4 10g links I was able to get between 7 and 9.50 gbps between all the hosts using iperf, received a DHCP address, and maintained a flat WAN.

I do have some tuning parameters I have been playing with, I may post them later if there ends up being any interest in this novel.

Please do your testing without added packages. Bandwidthd, darkstat, ntop, rrdgraphs can all drastically effect performance.

q54e3w

Thanks for documenting this. Ive just replicated your setup on my Supermicro Rangeley board with a Intel x520 & i350 quad card and am seeing dreadful performance, in the region of 600mbps! :( Can you share your tunables and loader config files please to see if they shed any light on why my setup is so slow?
thx

I'm seeing 900Mbps on a 2 core rangely over IPSec using 2.2.

@irj972:

Thanks for documenting this. Ive just replicated your setup on my Supermicro Rangeley board with a Intel x520 & i350 quad card and am seeing dreadful performance, in the region of 600mbps! :( Can you share your tunables and loader config files please to see if they shed any light on why my setup is so slow?
thx

@gonzopancho:

I'm seeing 900Mbps on a 2 core rangely over IPSec using 2.2.

Don't think you two are on the same page ;)

@digital_janitor: thanks for the post.
@irj972: https://redmine.pfsense.org/issues/3092

Oh, and I know exactly where that 600Mbps comes from. ::)

q54e3w

Thx for the link re lro!

Ive been spending some time optimising my FreeNAS system…finally getting somewhere but have a weird pfSense issue which is a bit beyond me at the moment which is holding me back.

pfSense sits at close to 0% CPU utilisation for two minutes and then jumps into life, 6gbps shows 75% interrupts on one core of the Rangeley c2758 but Im beaten why it takes two minutes to spring into life....its repeatable too. Ill keep debugging and see what I find.
Hopefully something I overlooked. PS. I managed to get FreeNAS running at 9+ gbps with Intel x520 directly connected to my Mac & PC so hopefully some of that knowledge and settings are transferable to pfSense.


j@freenas:~ % iperf -c 192.168.124.126 -i 1 -t 10000
------------------------------------------------------------
Client connecting to 192.168.124.126, TCP port 5001
TCP window size: 16.0 MByte (default)
------------------------------------------------------------
[  3] local 192.168.124.143 port 33619 connected with 192.168.124.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  21.0 MBytes   176 Mbits/sec
[  3]  1.0- 2.0 sec   384 KBytes  3.15 Mbits/sec
[  3]  2.0- 3.0 sec   512 KBytes  4.19 Mbits/sec
[  3]  3.0- 4.0 sec   384 KBytes  3.15 Mbits/sec
[  3]  4.0- 5.0 sec   512 KBytes  4.19 Mbits/sec
[  3]  5.0- 6.0 sec   512 KBytes  4.19 Mbits/sec
[  3]  6.0- 7.0 sec   512 KBytes  4.19 Mbits/sec
[  3]  7.0- 8.0 sec   640 KBytes  5.24 Mbits/sec
[  3]  8.0- 9.0 sec   384 KBytes  3.15 Mbits/sec
[  3]  9.0-10.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 10.0-11.0 sec   384 KBytes  3.15 Mbits/sec
[  3] 11.0-12.0 sec   384 KBytes  3.15 Mbits/sec
[  3] 12.0-13.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 13.0-14.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 14.0-15.0 sec   384 KBytes  3.15 Mbits/sec
[  3] 15.0-16.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 16.0-17.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 17.0-18.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 18.0-19.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 19.0-20.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 20.0-21.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 21.0-22.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 22.0-23.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 23.0-24.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 24.0-25.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 25.0-26.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 26.0-27.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 27.0-28.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 28.0-29.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 29.0-30.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 30.0-31.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 31.0-32.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 32.0-33.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 33.0-34.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 34.0-35.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 35.0-36.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 36.0-37.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 37.0-38.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 38.0-39.0 sec   384 KBytes  3.15 Mbits/sec
[  3] 39.0-40.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 40.0-41.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 41.0-42.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 42.0-43.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 43.0-44.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 44.0-45.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 45.0-46.0 sec  1.00 MBytes  8.39 Mbits/sec
[  3] 46.0-47.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 47.0-48.0 sec  1.00 MBytes  8.39 Mbits/sec
[  3] 48.0-49.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 49.0-50.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 50.0-51.0 sec  1.00 MBytes  8.39 Mbits/sec
[  3] 51.0-52.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 52.0-53.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 53.0-54.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 54.0-55.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 55.0-56.0 sec  1.00 MBytes  8.39 Mbits/sec
[  3] 56.0-57.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 57.0-58.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 58.0-59.0 sec  1.00 MBytes  8.39 Mbits/sec
[  3] 59.0-60.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 60.0-61.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 61.0-62.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 62.0-63.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 63.0-64.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 64.0-65.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 65.0-66.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 66.0-67.0 sec  1.00 MBytes  8.39 Mbits/sec
[  3] 67.0-68.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 68.0-69.0 sec   896 KBytes  7.34 Mbits/sec
[  3] 69.0-70.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 70.0-71.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 71.0-72.0 sec  1.25 MBytes  10.5 Mbits/sec
[  3] 72.0-73.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 73.0-74.0 sec   768 KBytes  6.29 Mbits/sec
[  3] 74.0-75.0 sec  1.00 MBytes  8.39 Mbits/sec
[  3] 75.0-76.0 sec  1.25 MBytes  10.5 Mbits/sec
[  3] 76.0-77.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 77.0-78.0 sec  1.38 MBytes  11.5 Mbits/sec
[  3] 78.0-79.0 sec  1.38 MBytes  11.5 Mbits/sec
[  3] 79.0-80.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 80.0-81.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 81.0-82.0 sec  1.50 MBytes  12.6 Mbits/sec
[  3] 82.0-83.0 sec  1.25 MBytes  10.5 Mbits/sec
[  3] 83.0-84.0 sec  1.25 MBytes  10.5 Mbits/sec
[  3] 84.0-85.0 sec  1.25 MBytes  10.5 Mbits/sec
[  3] 85.0-86.0 sec  1.50 MBytes  12.6 Mbits/sec
[  3] 86.0-87.0 sec  1.50 MBytes  12.6 Mbits/sec
[  3] 87.0-88.0 sec  1.38 MBytes  11.5 Mbits/sec
[  3] 88.0-89.0 sec  1.25 MBytes  10.5 Mbits/sec
[  3] 89.0-90.0 sec  1.50 MBytes  12.6 Mbits/sec
[  3] 90.0-91.0 sec  1.88 MBytes  15.7 Mbits/sec
[  3] 91.0-92.0 sec  2.12 MBytes  17.8 Mbits/sec
[  3] 92.0-93.0 sec  1.62 MBytes  13.6 Mbits/sec
[  3] 93.0-94.0 sec  1.12 MBytes  9.44 Mbits/sec
[  3] 94.0-95.0 sec  1.62 MBytes  13.6 Mbits/sec
[  3] 95.0-96.0 sec  2.12 MBytes  17.8 Mbits/sec
[  3] 96.0-97.0 sec  2.12 MBytes  17.8 Mbits/sec
[  3] 97.0-98.0 sec  2.12 MBytes  17.8 Mbits/sec
[  3] 98.0-99.0 sec  1.38 MBytes  11.5 Mbits/sec
[  3] 99.0-100.0 sec  1.88 MBytes  15.7 Mbits/sec
[  3] 100.0-101.0 sec  1.62 MBytes  13.6 Mbits/sec
[  3] 101.0-102.0 sec  2.00 MBytes  16.8 Mbits/sec
[  3] 102.0-103.0 sec  2.75 MBytes  23.1 Mbits/sec
[  3] 103.0-104.0 sec  2.12 MBytes  17.8 Mbits/sec
[  3] 104.0-105.0 sec  2.00 MBytes  16.8 Mbits/sec
[  3] 105.0-106.0 sec  1.75 MBytes  14.7 Mbits/sec
[  3] 106.0-107.0 sec  1.38 MBytes  11.5 Mbits/sec
[  3] 107.0-108.0 sec  2.38 MBytes  19.9 Mbits/sec
[  3] 108.0-109.0 sec  2.00 MBytes  16.8 Mbits/sec
[  3] 109.0-110.0 sec  2.00 MBytes  16.8 Mbits/sec
[  3] 110.0-111.0 sec  2.75 MBytes  23.1 Mbits/sec
[  3] 111.0-112.0 sec  2.38 MBytes  19.9 Mbits/sec
[  3] 112.0-113.0 sec  2.38 MBytes  19.9 Mbits/sec
[  3] 113.0-114.0 sec  2.00 MBytes  16.8 Mbits/sec
[  3] 114.0-115.0 sec  2.50 MBytes  21.0 Mbits/sec
[  3] 115.0-116.0 sec  1.50 MBytes  12.6 Mbits/sec
[  3] 116.0-117.0 sec  1.50 MBytes  12.6 Mbits/sec
[  3] 117.0-118.0 sec  3.12 MBytes  26.2 Mbits/sec
[  3] 118.0-119.0 sec  1.62 MBytes  13.6 Mbits/sec
[  3] 119.0-120.0 sec  2.12 MBytes  17.8 Mbits/sec
[  3] 120.0-121.0 sec  2.12 MBytes  17.8 Mbits/sec
[  3] 121.0-122.0 sec  2.00 MBytes  16.8 Mbits/sec
[  3] 122.0-123.0 sec  2.88 MBytes  24.1 Mbits/sec
[  3] 123.0-124.0 sec  2.75 MBytes  23.1 Mbits/sec
[  3] 124.0-125.0 sec  1.88 MBytes  15.7 Mbits/sec
[  3] 125.0-126.0 sec  2.50 MBytes  21.0 Mbits/sec
[  3] 126.0-127.0 sec  2.75 MBytes  23.1 Mbits/sec
[  3] 127.0-128.0 sec  5.12 MBytes  43.0 Mbits/sec
[  3] 128.0-129.0 sec  2.50 MBytes  21.0 Mbits/sec
[  3] 129.0-130.0 sec  8.75 MBytes  73.4 Mbits/sec
[  3] 130.0-131.0 sec  6.50 MBytes  54.5 Mbits/sec
[  3] 131.0-132.0 sec   123 MBytes  1.03 Gbits/sec
[  3] 132.0-133.0 sec   303 MBytes  2.54 Gbits/sec
[  3] 133.0-134.0 sec   390 MBytes  3.27 Gbits/sec
[  3] 134.0-135.0 sec   401 MBytes  3.36 Gbits/sec
[  3] 135.0-136.0 sec   407 MBytes  3.42 Gbits/sec
[  3] 136.0-137.0 sec   431 MBytes  3.62 Gbits/sec
[  3] 137.0-138.0 sec   475 MBytes  3.99 Gbits/sec
[  3] 138.0-139.0 sec   506 MBytes  4.25 Gbits/sec
[  3] 139.0-140.0 sec   532 MBytes  4.46 Gbits/sec
[  3] 140.0-141.0 sec   549 MBytes  4.61 Gbits/sec
[  3] 141.0-142.0 sec   552 MBytes  4.63 Gbits/sec
[  3] 142.0-143.0 sec   557 MBytes  4.67 Gbits/sec
[  3] 143.0-144.0 sec   551 MBytes  4.62 Gbits/sec
[  3] 144.0-145.0 sec   556 MBytes  4.66 Gbits/sec
[  3] 145.0-146.0 sec   558 MBytes  4.68 Gbits/sec
[  3] 146.0-147.0 sec   567 MBytes  4.75 Gbits/sec
[  3] 147.0-148.0 sec   580 MBytes  4.87 Gbits/sec
[  3] 148.0-149.0 sec   574 MBytes  4.82 Gbits/sec
[  3] 149.0-150.0 sec   577 MBytes  4.84 Gbits/sec
[  3] 150.0-151.0 sec   578 MBytes  4.85 Gbits/sec
[  3] 151.0-152.0 sec   579 MBytes  4.86 Gbits/sec
[  3] 152.0-153.0 sec   587 MBytes  4.93 Gbits/sec
[  3] 153.0-154.0 sec   577 MBytes  4.84 Gbits/sec
[  3] 154.0-155.0 sec   573 MBytes  4.80 Gbits/sec
[  3] 155.0-156.0 sec   579 MBytes  4.86 Gbits/sec
[  3] 156.0-157.0 sec   586 MBytes  4.92 Gbits/sec
[  3] 157.0-158.0 sec   590 MBytes  4.95 Gbits/sec
[  3] 158.0-159.0 sec   592 MBytes  4.96 Gbits/sec
[  3] 159.0-160.0 sec   596 MBytes  5.00 Gbits/sec
[  3] 160.0-161.0 sec   601 MBytes  5.04 Gbits/sec
[  3] 161.0-162.0 sec   649 MBytes  5.44 Gbits/sec
[  3] 162.0-163.0 sec   615 MBytes  5.16 Gbits/sec
[  3] 163.0-164.0 sec   627 MBytes  5.26 Gbits/sec
[  3] 164.0-165.0 sec   640 MBytes  5.37 Gbits/sec
[  3] 165.0-166.0 sec   646 MBytes  5.42 Gbits/sec
[  3] 166.0-167.0 sec   663 MBytes  5.56 Gbits/sec
[  3] 167.0-168.0 sec   669 MBytes  5.62 Gbits/sec
[  3] 168.0-169.0 sec   671 MBytes  5.63 Gbits/sec
[  3] 169.0-170.0 sec   673 MBytes  5.65 Gbits/sec
[  3] 170.0-171.0 sec   661 MBytes  5.54 Gbits/sec
[  3] 171.0-172.0 sec   673 MBytes  5.65 Gbits/sec
[  3] 172.0-173.0 sec   674 MBytes  5.66 Gbits/sec
[  3] 173.0-174.0 sec   672 MBytes  5.64 Gbits/sec
[  3] 174.0-175.0 sec   678 MBytes  5.68 Gbits/sec
[  3] 175.0-176.0 sec   681 MBytes  5.71 Gbits/sec
[  3] 176.0-177.0 sec   671 MBytes  5.63 Gbits/sec
[  3] 177.0-178.0 sec   676 MBytes  5.67 Gbits/sec
[  3] 178.0-179.0 sec   682 MBytes  5.72 Gbits/sec
[  3] 179.0-180.0 sec   684 MBytes  5.74 Gbits/sec
[  3] 180.0-181.0 sec   686 MBytes  5.76 Gbits/sec
[  3] 181.0-182.0 sec   690 MBytes  5.79 Gbits/sec
[  3] 182.0-183.0 sec   697 MBytes  5.85 Gbits/sec
[  3] 183.0-184.0 sec   697 MBytes  5.85 Gbits/sec
[  3] 184.0-185.0 sec   700 MBytes  5.88 Gbits/sec
[  3] 185.0-186.0 sec   700 MBytes  5.87 Gbits/sec
[  3] 186.0-187.0 sec   698 MBytes  5.85 Gbits/sec
[  3] 187.0-188.0 sec   695 MBytes  5.83 Gbits/sec
[  3] 188.0-189.0 sec   706 MBytes  5.92 Gbits/sec
[  3] 189.0-190.0 sec   703 MBytes  5.90 Gbits/sec
[  3] 190.0-191.0 sec   702 MBytes  5.89 Gbits/sec
[  3] 191.0-192.0 sec   700 MBytes  5.87 Gbits/sec
[  3] 192.0-193.0 sec   705 MBytes  5.91 Gbits/sec
[  3] 193.0-194.0 sec   706 MBytes  5.92 Gbits/sec
[  3] 194.0-195.0 sec   707 MBytes  5.93 Gbits/sec
[  3] 195.0-196.0 sec   710 MBytes  5.95 Gbits/sec
[  3] 196.0-197.0 sec   711 MBytes  5.96 Gbits/sec
[  3] 197.0-198.0 sec   711 MBytes  5.97 Gbits/sec
[  3] 198.0-199.0 sec   703 MBytes  5.90 Gbits/sec
[  3] 199.0-200.0 sec   718 MBytes  6.02 Gbits/sec
[  3] 200.0-201.0 sec   720 MBytes  6.04 Gbits/sec
[  3] 201.0-202.0 sec   720 MBytes  6.04 Gbits/sec
[  3] 202.0-203.0 sec   721 MBytes  6.05 Gbits/sec
[  3] 203.0-204.0 sec   718 MBytes  6.03 Gbits/sec
[  3] 204.0-205.0 sec   718 MBytes  6.03 Gbits/sec
[  3] 205.0-206.0 sec   718 MBytes  6.03 Gbits/sec
[  3] 206.0-207.0 sec   719 MBytes  6.03 Gbits/sec
[  3] 207.0-208.0 sec   718 MBytes  6.02 Gbits/sec
[  3] 208.0-209.0 sec   715 MBytes  6.00 Gbits/sec
[  3] 209.0-210.0 sec   718 MBytes  6.02 Gbits/sec
[  3] 210.0-211.0 sec   727 MBytes  6.10 Gbits/sec
[  3] 211.0-212.0 sec   720 MBytes  6.04 Gbits/sec
[  3] 212.0-213.0 sec   722 MBytes  6.05 Gbits/sec
[  3] 213.0-214.0 sec   727 MBytes  6.10 Gbits/sec
[  3] 214.0-215.0 sec   732 MBytes  6.14 Gbits/sec
[  3] 215.0-216.0 sec   728 MBytes  6.10 Gbits/sec
[  3] 216.0-217.0 sec   728 MBytes  6.11 Gbits/sec
[  3] 217.0-218.0 sec   727 MBytes  6.10 Gbits/sec
...
...
... at 6gbps until stopped.

heper

https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

hope it helps

is old, and somewhat out of date.

heper

@gonzopancho:

is old, and somewhat out of date.

i'm sure it is within your power to make it "new and shiny" and somewhat up-to-date :))