SG-1100 Throughput Test

testgate

@testgate said in SG-1100 Throughput Test:

I understand some time has passed on this thread, but I also see the slower speeds mentioned for this firewall.

Using iperf3 client-to-server through a cheap 1G switch I consistently get about 930 Mbits/sec. Using the same cables, client, and server through the SG-1100 LAN-to-WAN I consistently get about 445 Mbits/sec. The results are slightly slower if I use the built-in iperf3 package within pfSense as the server with the client on the LAN link. The netgate is running 2.4.4_3.

To specify the lab: a laptop is configured with a static IP in the WAN IP scope, is directly attached to the WAN port with a Cat6 cable, and listens with iperf3 as server. A laptop is configured via DHCP in the LAN IP scope, is attached to the LAN port with a Cat6 cable, and runs iperf3 as a client. Outside (WAN) laptop command is “iperf3 -p 5001 -s”. Inside (LAN) laptop command is “iperf3 -p 5001 -c <static IP of the WAN connected laptop>”.

This is a fresh install of pfSense with no special sauce. The LAN→WAN firewall rule is an IPv4* any any.

Now the really crazy part happens when I run the netgate built-in iperf3 package as a client. Running the same laptop as server on the LAN link, the netgate consistently gets about 865 Mbits/sec! Reverse this with the built-in iperf3 as server and the same laptop as client on the LAN link, back to mid 445 Mbits/sec. Huh?!?

stephenw10

Hmm, what happens if you keep the server on the SG-1100 but run the client with the reverse option -R?

Or run the client on the SG-1100 with reverse?

The default in iperf is to have the client send traffic to the server (which always seemed an odd decision to me!). So in that test you are seeing nearly 900Mbps when it's sending but less than half that when it's receiving.

Running either client with -R reverses the traffic but keeps the states opening the same way. So swapping it determines if it's the way the firewall opens states or the traffic direction.

Steve

testgate

@stephenw10

Thanks for your reply. Funny timing, I found that switch a few hours ago! Same results. It simply seems that the connection speed is asynchronous. The below is viewed from over the console cable to the SG-1100. The other testing device is a laptop connected to the LAN port. The text wrapped, but the second command had the -R.

I did a full factory reset earlier and had similar results.

Also of potential interest, I setup different scopes on LAN and OPT and tested across them as well as to/from WAN. Tests involving WAN from either LAN or OPT have the same asynchronous results. Tests between OPT and LAN were synchronous, but always below 500 Mbits/sec.

The only other thing I noticed strange is that it seems the netgate is tagging outbound packets with TOS 7. You can even see this activity with simply ping responses from the netgate. Strange.

I really wouldn’t ordinarily notice or care much, but this installation is going into a FIOS delivered service very close to 1Gbps. So the SG-1100 throughput will matter.

In the below capture, the laptop connected directly to LAN was running "iperf -p 5001 -s".

stephenw10

Hmm, so still slower receiving.

You might try disabling pf entirely (pfctl -d) and testing between LAN and OPT. You should see close to line rate under those conditions unless there is some thing very wrong in the setup.

I will say though that your unlikely to see much over 500Mbps from WAN to LAN, even after tuning, with firewalling and NAT in place. If you need close to 1Gbps you should upgrade to the SG-3100.

Steve

johnpoz

@testgate said in SG-1100 Throughput Test:

tagging outbound packets with TOS 7

Where you seeing that - I just looked, and no that is not happening..

Please post your sniff showing that.

testgate

@stephenw10 Thanks again. Here goes:

Followed the below process for LAN<->WAN test:

Reset SG-1100 to factory default via console cable. Set interface(s) IP address via console WAN (mvneta0.4090) IP 10.10.10.1/24, LAN (mvnet0.4091) left default IP – 192.168.1.1 and default DHCP scope. Connected laptop1 to LAN, logged into Web GUI, completed the wizard leaving all defaults, but set new admin password. Laptop2 connected to WAN and set IP 10.10.10.150/24.

On laptop2, ran command “iperf3 -p 5001 -s”
On laptop1, ran command “iperf3 -p 5001 -c 10.10.10.150”
On laptop1, ran command “iperf3 -p 5001 -c 10.10.10.150 -R”
Result screen capture below:

Followed the below process for LAN<-> OPT test:

Executed “pfctl -d” command as root via console. Configured OPT (mvneta0.4092) via Web GUI for IP 192.168.100.1/24. Moved laptop2 to OPT and set IP 192.168.100.150/24.

On laptop2, ran command “iperf3 -p 5001 -s”
On laptop1, ran command “iperf3 -p 5001 -c 192.168.100.150”
On laptop1, ran command “iperf3 -p 5001 -c 192.168.100.150 -R”
Result screen capture below:

Followed the below process for laptop1<->laptop2 test:

Set laptop1 IP 192.168.100.200. Connected laptop1 directly to laptop2.

On laptop1, ran command “iperf3 -p 5001 -c 192.168.100.150”
On laptop1, ran command “iperf3 -p 5001 -c 192.168.100.150 -R”
Result screen capture below:

Taking Steve’s advice, I ordered an SG-3100. We have several of these in production and have had good results. The SG-1100 was/is an experiment. Interesting results.

testgate

@johnpoz Apologies, I didn’t save the capture where I saw this and could not replicate it again after a factory reset.

stephenw10

The SG-3100 is definitely a better option here.

However when you disable pf using 'pfctl -d' it will be enabled again by making any changes in the gui that apply firewall or NAT changes. (unless it's disabled in the GUI). I suspect in the test where you added opt and tested from LAN to OPT it had become enabled again. Better to disable it immediately before the test.

Steve

testgate

@stephenw10 I think you are correct on both points. Thanks for that, I appreciate your input. The asynchronous results of LAN-WAN testing still baffles me.

stephenw10

As far as I've been able to determine it's due the way NAT states are opened. It you test between interfaces that are not NATing you will see it's a lot closer or completely symmetric. So between LAN and OPT or with pf disabled or with static routes on the WAN side devices opening states directly to the LAN subnet.

Steve

testgate

@stephenw10 Confirmed, my order of operation had inadvertently re-enabled pf. Rerunning the test LAN to OPT with the order you specified gave back results not very dissimilar to my laptop-to-laptop run.

Will I not experience this same issue with the 3100? If not, is that due to the differences in HW architecture?

stephenw10

You may see some asymmetry in throughput but not as extreme. You should see close to Gigabit download (other variables allowing!).

Steve

testgate

@stephenw10 SG-3100 arrived today. Same LAN-WAN testing results below. LAN box on 192.168.1.100, WAN box on 10.10.10.10. 100% vanilla install fresh out of box. Thanks for all your help.