Just got a Protectli FW4C!
-
Thanks! I'll try forcing NAT-T later tonight when there's less traffic on my residential service (Frontier Gig Fiber). Speedtest and every other test seems to be very highly variable from this site during the day; it's more consistent late at night when, presumably, fewer people are using it.
In late evenings my home office (FW4C) will typically speedtest 900/700, whereas right now it's testing 600/600.
My main office (MBT-2220) is on commercial fiber, and it's more consistently at 700/650.
Both lines are nominally 1000/1000. I've put a different device at the edge of my office network and speedtested 922/885, so the MBT-2220 is limited in some respect just for speedtest.
I enabled port-forwarding through both pfense routers, and iperf from a host behind my FW4C to a host port-forwarded through the MBT-2220 reports:
./iperf3 -w 1M -c <main.office.public.ip> Connecting to host <main.office.public.ip>, port 5201 [ 6] local 192.168.1.230 port 56528 connected to <main.office.public.ip> port 5201 [ ID] Interval Transfer Bandwidth [ 6] 0.00-1.00 sec 44.4 MBytes 373 Mbits/sec [ 6] 1.00-2.00 sec 52.9 MBytes 444 Mbits/sec [ 6] 2.00-3.00 sec 53.5 MBytes 449 Mbits/sec [ 6] 3.00-4.00 sec 53.0 MBytes 444 Mbits/sec [ 6] 4.00-5.00 sec 52.7 MBytes 442 Mbits/sec [ 6] 5.00-6.00 sec 53.5 MBytes 449 Mbits/sec [ 6] 6.00-7.00 sec 53.6 MBytes 449 Mbits/sec [ 6] 7.00-8.00 sec 53.5 MBytes 449 Mbits/sec [ 6] 8.00-9.00 sec 53.4 MBytes 448 Mbits/sec [ 6] 9.00-10.00 sec 53.5 MBytes 449 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 6] 0.00-10.00 sec 524 MBytes 440 Mbits/sec sender [ 6] 0.00-10.00 sec 524 MBytes 440 Mbits/sec receiverThe reverse, iperf from a host behind my MBT-2220 to a host port-forwarded through the FW4C reports:
iperf3 -w 2M -c <home.office.public.ip> Connecting to host <home.office.public.ip>, port 5201 [ 4] local 192.168.0.13 port 49174 connected to <home.office.public.ip> port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 43.4 MBytes 363 Mbits/sec [ 4] 1.00-2.00 sec 45.1 MBytes 378 Mbits/sec [ 4] 2.00-3.00 sec 45.4 MBytes 381 Mbits/sec [ 4] 3.00-4.00 sec 45.2 MBytes 380 Mbits/sec [ 4] 4.00-5.00 sec 45.4 MBytes 380 Mbits/sec [ 4] 5.00-6.00 sec 44.2 MBytes 372 Mbits/sec [ 4] 6.00-7.01 sec 37.9 MBytes 316 Mbits/sec [ 4] 7.01-8.00 sec 23.1 MBytes 195 Mbits/sec [ 4] 8.00-9.01 sec 17.0 MBytes 142 Mbits/sec [ 4] 9.01-10.01 sec 15.6 MBytes 131 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.01 sec 362 MBytes 304 Mbits/sec sender [ 4] 0.00-10.01 sec 361 MBytes 302 Mbits/sec receiveralthough it varies a lot from run to run.
-
Well its above 100Mbps at least so it's not something restricting all traffic in the path. It may still be ESP traffic though.
I would also test setting some MSS values on the tunnel. If you are seeing packet fragmentation it can really hurt throughput.
Steve
-
For what its worth i did have a similar issue like yours with IPsec throughput. Moving to NAT-T and having packets encapsulated with UDP helped alot. There was something in the path not liking ESP and clearly reducing my speed because of it.
-
@stephenw10 said in Just got a Protectli FW4C!:
Well its above 100Mbps at least so it's not something restricting all traffic in the path. It may still be ESP traffic though.
I would also test setting some MSS values on the tunnel. If you are seeing packet fragmentation it can really hurt throughput.
Steve
@michmoor said in Just got a Protectli FW4C!:
For what its worth i did have a similar issue like yours with IPsec throughput. Moving to NAT-T and having packets encapsulated with UDP helped alot. There was something in the path not liking ESP and clearly reducing my speed because of it.
Thanks for both of your suggestions.
I turned on MSS clamping with a max value of 1392, and my best throughput did increase from ~160 Mbps up to ~220:
[ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 262 MBytes 220 Mbits/sec sender [ 4] 0.00-10.00 sec 259 MBytes 217 Mbits/sec receiverSwitching NAT-T from Auto to Force and back again did not change the results.
So it's getting better, but inch by inch.
-
You might try a much lower value just to check. I have seen IPSec tunnels that require MSS as low as 1100 to prevent fragmentation. Though not over a route as short as 10ms.
-
@stephenw10 For good measure i would test another protocol like wireguard if you can. Curious if the low performance follows.
-
@stephenw10 said in Just got a Protectli FW4C!:
You might try a much lower value just to check. I have seen IPSec tunnels that require MSS as low as 1100 to prevent fragmentation. Though not over a route as short as 10ms.
Ok, I'll try that tonight. Does the MSS have to be set on both sides of the tunnel? And does the tunnel have to be disconnected and reconnected in order for the new value to take effect?
-
@michmoor said in Just got a Protectli FW4C!:
@stephenw10 For good measure i would test another protocol like wireguard if you can. Curious if the low performance follows.
The problem with WG is that I don't have a baseline, and Protectli doesn't, either. So if I get some performance number, I won't know if it's higher, lower, or exactly as expected.
I also was not successful in setting it up last time I tried.
Whereas for IPSec, we have a Netgate person letting us know that I'm way under expectations.
But WG testing would be useful down the road, once I have IPSec established and optimized.
-
It should only need to be set on one side but it doesn't hurt to se it on both.
-
@thewaterbug Not sure it was asked but what Phase 2 parameters are you using?
-
Both Phase 1 and Phase 2 are AES-GCM-128, SHA256, and DH14.
-
@thewaterbug Ahhh theres one more setting that helped out a lot for me. PowerD settings. Enable and set to either Maximum or HiAdaptative.
When i was running OPNsense on a Protectli a year ago i had problems with poor performance on Wireguard. The recommendation was to enable this. Once i did that things moved a lot better.
-
AES-GCM doesn't require a hash for authentication, that's one of the reasons it's faster. You can remove that. It should just ignore it already though.
-
Ah yes. It was selected before, when I was using AES-CBC to work around the SG-1100/SafeXcel problem, and once I deselected AES-CBC and selected AES-GCM, the hash just stayed selected.
-
I'm already set to HiAdaptive on both sides. It doesn't make a difference in my test results.
-
@stephenw10 said in Just got a Protectli FW4C!:
AES-GCM doesn't require a hash for authentication, that's one of the reasons it's faster. You can remove that. It should just ignore it already though.
Is this true for both Phase 1 and Phase 2? If yes, I'm curious as to why the Phase 1 setup has a selector for Hash if AES-GCM is chosen as the encryption:
-
It is true but it doesn't really matter at phase 1. The phase 2 config is what actually governs the traffic over the tunnel once it's established.
-
While I'm mulling over how to improve throughput on the MBT-2220 side, I thought I'd put the two FW4C units on the bench and try them out, side-by-side, with only 6' of cabling between them, <<<< 1 ms ping, and no other traffic:
The best I could achieve was 626 Mbps over a 10 hour period.
Things that puzzle me:
- Throughput seems to vary from run to run, despite there being very few variables in the setup.
- There is no internet traffic, no routing outside of the two units, and not even a switch (I have the two WAN ports connected with a cable at 2500BaseT).
- Sometimes a 10 second run will achieve ~720 Mbps
- Sometimes a 10 second run will achieve only ~300 Mbps
- CPU utilization on both sender and receiver get no higher than 80%, and core temps no higher than 61ºC, but I'm still getting significantly less than the ~980 Mbps reported by Protectli.
Things I fiddled with that made no improvement:
- NAT-T
- MSS Clamping
- Connecting the WAN ports through a 1000BaseT switch.
- This reduced throughput by maybe 5 Mbps, but that might be just sampling error.
- Unchecked all the "Disable . . . " checkboxes in System > Advanced > Networking > Network Interface
- iperf simultaneous connections, e.g. "-P 2" or "-P 4". No improvement, and significant degradation at > 4.
- iperf TCP window size, e.g. "-w 2M" or "-w 4M". No improvement.
- iperf direction, e.g. "-R". Performance is the same, and just as variable, in both directions.
Are there another tunables that might improve things in this type of lab scenario?
My real goal is to maximize application throughput in the real world, where I have 2 ISPs, 8 miles, and 10 msec of ping between my two locations, but first I want to optimize in the lab to see what's possible.
- Throughput seems to vary from run to run, despite there being very few variables in the setup.
-
@thewaterbug If you do just an iPerf test without VPN, what do you get?
I gotta be honest with you, I got a Protectli 2 port and 6 port. Inter-vlan at a house i can get around 970Mbps. Over a IPsec vlan where the remote site is capped at 200/10 i can saturate that link no issue. Local speedtests at each site i can cap the connection no issue.Could it be possible that your FW4C is a dud?
-
I didn't test through a port forward, and I took one of the units out of the lab, but I can do that test on Monday. I might also be using iperf incorrectly.
Right now I've returned one of the FW4C units into service at my house, with the tunnel up between it and the MBT-2220 at the main office, and I'm watching a Veeam Backup Copy Job in progress.
This tunnel will iperf from FW4C-->MBT-2220 at ~200 Mbps, and will iperf the other way MBT-2220-->FW4C at ~135 Mbps, but this Veeam copy job is showing throughput in the "slow" direction (MBT-2220-->FW4C) of up to 250 - 300 Mbps, which is several times the iperf speed, and a lot closer to what Netgate has suggested the MBT-2220 is capable of:
So it's possible that I'm not using iperf optimally. When I iperf across the LAN, with no tunnel or port forwarding, I get 940 Mbps between two machines with just:
./iperf3 -son the server and
./iperf3 -c iperf.server.ip.addressor
./iperf3 -c iperf.server.ip.address -Ron the client, so I've been using that same command across the tunnel. I've experimented with -P and -w as noted above, but there might be other knobs I can turn to improve the throughput.
Because I'm not building an iperf tunnel; I'm building a tunnel to do real work, like backup copy jobs, so I need to measure it using the correct metrics.
