PFSense on fitlet-XA10-LAN - Decent throughput for a $350 platform?
-
You may find the system tune settings here worth trying:
https://ashbyte.com/ashbyte/wiki/pfSense/Tuning
I'd start with system tunables:
net.inet.tcp.syncookies=0
net.inet.raw.maxdgram=16384
net.inet.raw.recvspace=16384
net.inet.tcp.tcbhashsize=1024
kern.ipc.maxsockets=51200
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_inc=32768
net.inet.tcp.sendbuf_inc=32768Let us know how you make out..I'll update the Quotom build thread accordingly :-)
-
You may find the system tune settings here worth trying:
https://ashbyte.com/ashbyte/wiki/pfSense/Tuning
I'd start with system tunables:
net.inet.tcp.syncookies=0
net.inet.raw.maxdgram=16384
net.inet.raw.recvspace=16384
net.inet.tcp.tcbhashsize=1024
kern.ipc.maxsockets=51200
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_inc=32768
net.inet.tcp.sendbuf_inc=32768Let us know how you make out..I'll update the Quotom build thread accordingly :-)
I would like to know what negative effects these settings may cause. In the beginning of that page, there's a warning that some of the settings break IPSec. I'm not sure which ones break IPSec, but I need IPSec. If I change some of these settings now, could I experience some issues in the future with some other protocols?
-
You may find the system tune settings here worth trying:
https://ashbyte.com/ashbyte/wiki/pfSense/Tuning
I'd start with system tunables:
net.inet.tcp.syncookies=0
net.inet.raw.maxdgram=16384
net.inet.raw.recvspace=16384
net.inet.tcp.tcbhashsize=1024
kern.ipc.maxsockets=51200
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_inc=32768
net.inet.tcp.sendbuf_inc=32768Let us know how you make out..I'll update the Quotom build thread accordingly :-)
I would like to know what negative effects these settings may cause. In the beginning of that page, there's a warning that some of the settings break IPSec. I'm not sure which ones break IPSec, but I need IPSec. If I change some of these settings now, could I experience some issues in the future with some other protocols?
I am pretty sure that the fastforwarding setting broke IPSec.
The settings in the quoted post all seem safe.
-
I've tested these settings for tuning pfSense performance.
The best performance I get from the fitlet is:
WAN-to-LANiperf3 -c 192.168.160.100 -i3 -fm -P3 -R Connecting to host 192.168.160.100, port 5201 Reverse mode, remote host 192.168.160.100 is sending [ 4] local 192.168.200.30 port 54482 connected to 192.168.160.100 port 5201 [ 6] local 192.168.200.30 port 54483 connected to 192.168.160.100 port 5201 [ 8] local 192.168.200.30 port 54484 connected to 192.168.160.100 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-3.00 sec 38.8 MBytes 109 Mbits/sec [ 6] 0.00-3.00 sec 163 MBytes 455 Mbits/sec [ 8] 0.00-3.00 sec 38.0 MBytes 106 Mbits/sec [SUM] 0.00-3.00 sec 239 MBytes 669 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 3.00-6.00 sec 36.0 MBytes 101 Mbits/sec [ 6] 3.00-6.00 sec 168 MBytes 469 Mbits/sec [ 8] 3.00-6.00 sec 35.3 MBytes 98.7 Mbits/sec [SUM] 3.00-6.00 sec 239 MBytes 668 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 6.00-9.00 sec 41.1 MBytes 115 Mbits/sec [ 6] 6.00-9.00 sec 157 MBytes 440 Mbits/sec [ 8] 6.00-9.00 sec 40.5 MBytes 113 Mbits/sec [SUM] 6.00-9.00 sec 239 MBytes 668 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 9.00-10.00 sec 16.9 MBytes 142 Mbits/sec [ 6] 9.00-10.00 sec 46.0 MBytes 386 Mbits/sec [ 8] 9.00-10.00 sec 16.7 MBytes 140 Mbits/sec [SUM] 9.00-10.00 sec 79.6 MBytes 668 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 133 MBytes 112 Mbits/sec sender [ 4] 0.00-10.00 sec 133 MBytes 112 Mbits/sec receiver [ 6] 0.00-10.00 sec 535 MBytes 448 Mbits/sec sender [ 6] 0.00-10.00 sec 534 MBytes 448 Mbits/sec receiver [ 8] 0.00-10.00 sec 131 MBytes 110 Mbits/sec sender [ 8] 0.00-10.00 sec 131 MBytes 110 Mbits/sec receiver [SUM] 0.00-10.00 sec 799 MBytes 670 Mbits/sec sender [SUM] 0.00-10.00 sec 798 MBytes 669 Mbits/sec receiver iperf Done.LAN-to-WAN:
[~] # iperf3 -c 192.168.160.100 -i3 -fm -P3 Connecting to host 192.168.160.100, port 5201 [ 4] local 192.168.200.30 port 54487 connected to 192.168.160.100 port 5201 [ 6] local 192.168.200.30 port 54488 connected to 192.168.160.100 port 5201 [ 8] local 192.168.200.30 port 54489 connected to 192.168.160.100 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-3.00 sec 53.9 MBytes 151 Mbits/sec [ 6] 0.00-3.00 sec 74.9 MBytes 209 Mbits/sec [ 8] 0.00-3.00 sec 112 MBytes 315 Mbits/sec [SUM] 0.00-3.00 sec 241 MBytes 675 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 3.00-6.00 sec 61.7 MBytes 172 Mbits/sec [ 6] 3.00-6.00 sec 78.2 MBytes 219 Mbits/sec [ 8] 3.00-6.00 sec 104 MBytes 290 Mbits/sec [SUM] 3.00-6.00 sec 244 MBytes 681 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 6.00-9.00 sec 110 MBytes 307 Mbits/sec [ 6] 6.00-9.00 sec 61.4 MBytes 172 Mbits/sec [ 8] 6.00-9.00 sec 65.6 MBytes 184 Mbits/sec [SUM] 6.00-9.00 sec 237 MBytes 662 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 9.00-10.00 sec 20.6 MBytes 173 Mbits/sec [ 6] 9.00-10.00 sec 20.1 MBytes 168 Mbits/sec [ 8] 9.00-10.00 sec 38.8 MBytes 325 Mbits/sec [SUM] 9.00-10.00 sec 79.5 MBytes 667 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 246 MBytes 206 Mbits/sec sender [ 4] 0.00-10.00 sec 245 MBytes 206 Mbits/sec receiver [ 6] 0.00-10.00 sec 235 MBytes 197 Mbits/sec sender [ 6] 0.00-10.00 sec 234 MBytes 196 Mbits/sec receiver [ 8] 0.00-10.00 sec 321 MBytes 269 Mbits/sec sender [ 8] 0.00-10.00 sec 320 MBytes 268 Mbits/sec receiver [SUM] 0.00-10.00 sec 801 MBytes 672 Mbits/sec sender [SUM] 0.00-10.00 sec 799 MBytes 670 Mbits/sec receiver iperf Done.The System Tunables I used to achieve these settings are:
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_max=16777216All other settings either don't affect the throughput or lower it about 5-10%
The values for the parameters specified in the suggested tunable that are equal to 1024 and 16384 are lower than the default values in my installation of pfSense. So, I've tried those lower values suggested, but later removed the tunable to go back to the default values as there was no throughput increase in using the suggested values over the default values.
The last two values on the list:
net.inet.tcp.recvbuf_inc=32768
net.inet.tcp.sendbuf_inc=32768are larger than the default values for these parameters, but did not improve the throughput (and in my opinion, they actually decreased the throughput), so I tried the suggested values, but then went back to the defaults in my pfSense installation.
-
My L2TP over IPSec throughput is around 140 Mbps both LAN-to-WAN and WAN-to-LAN
-
I've finally transitioned the fitlet-XA10-LAN to production. My Internet connection is 90 Mbps / 12 Mbps (from Comcast Xfinity), which the fitlet-XA10-LAN platform can handle with ease. All of my previous tests with iPerf3 were in the lab environment just to see what the limitations the fitlet-XA10-LAN platform are, and they seem to be at around 670 Mbps throughput LAN-to-WAN and WAN-to-LAN (with NAT and Firewall enabled) and an IPSec Phase 1 policy enabled. I noticed that when I disable an IPSec Phase 1 policy, the throughput improves marginally.
When establishing a VPN connection to the pfSense running on fitlet-XA10-LAN, I was able to achieve a throughput of 140 Mbps via an L2TP over IPSec tunnel in the lab environment (both WAN-to-LAN and LAN-to-WAN) Because my Internet connection is lower than 140 Mbps, I can't test the limits of the VPN throughput on a live Internet connection yet - until I get a better Internet bandwidth.
In my opinion, the fitlet-XA10-LAN, which is made in Israel, is a solid fanless platform that performs quite well for a very small footprint. For example, when I run a bandwidth test via speediest.net, (with my Internet offering capped at 90 Mbps download bandwidth), the CPU utilization does not exceed 15%, and the ping RTT is at 11 ms to this server on the Internet via Wi-Fi from my Mac Mini (to a server located about 30 miles away). My topology is MacMini > Asus RT-N66U (AP Mode) > Cisco 3560CG (L3 switch)> fitlet-XA10-LAN (running pfSense) > Comcast Cable Modem.
Additionally, the Fitlet comes with a serial port to which the console output can be directed, so that one doesn't have to connect a monitor and keyboard to it in order to access pfSense via console. Additionally, BIOS can be accessed via this serial port's console as well. In other words, this serial port provides a complete appliance-like experience unlike other fanless boxes that lack a serial port.
–---------------
Whether one should consider this platform depends on what the future of one's Internet pipe is. If you think you will be getting Google Fiber (or similar offering), with a very affordable symmetric 1 Gbps Internet pipe, you should skip any hardware platform now that cannot deliver a throughput close to 1 Gbps and instead invest into something that can push at least 1 Gbps. If you think you will stay on cable (like Comcast Xfinity), then fitlet-XA10-LAN is a good choice IMHO because it will probably take another decade before Comcast will offer a symmetrical 1 Gbps for $70 like what Google Fiber is offering today.
A well-discussed Chinese-made fanless quad-core Celeron (with no AES hardware support) system costs $260. Fitlet-XA10-LAN (which has a quad-core AMD CPU with hardware support for AES) comes with 4 Gigabit NICs and sells for $315 barebones. There is not much reason to have more than 16 GB of SSD and 4 GB of RAM in this box to run pfSense, so it's feasible to get the complete system for under $370. Fitlet comes with a 5-year warranty, and it's made in Israel by a company that's known for embedded systems that they supply to industrial and military sectors. They also offer a no-questions-asked return policy directly to the manufacturer - Compulab - (once your return period with the reseller runs out). The downsides of the fitlet-XA10-LAN system is that today one can buy a Check Point 620, which provides 750 Mbps throughput and 150 Mbps VPN throughput for under $300, but Check Point 620 is supposedly End Of Sale and its stock is diminishing with resellers.
If you want to go with a system that can push 1 Gbps (with NAT and Firewall enabled), Check Point 750 can do it. Additionally, Check Point 750 can do 500 Mbps of VPN throughput, but it costs around $600. From the information I've been able to gather so far, if you want to build a system for pfSense that can provide 1 Gbps of throughput (with NAT and Firewall enabled), you will end up paying between $600 and $700 for the hardware, so again it seems that Check Point matches pfSense-based hardware on the price-to-performance ratio at least in this SOHO/SMB segment.
One more thing - it may be a good idea to do inter-VLAN routing in a Layer 3 switch rather than in pfSense. There's an argument whether one should have a Layer 3 switch in a SOHO or SMB environment vs having a Layer 2 switch and doing inter-VLAN routing in a router. In my opinion, now that we are approaching Gigabit bandwidth offerings from providers like Google Fiber, L3 switching between LAN subnets (aka Inter-VLAN routing) at the firewall (or Internet router) will reduce the throughout for the traffic from/to the Internet. A decent Layer 3 managed switch can be had for about $100, so those who have multiple subnets (VLANs) on the LAN and whose Internet bandwidth is close to the hardware limitations of their firewall should consider moving the Layer 3 boundary from the firewall (Internet router) to the Layer 3 switch. Additionally, decent Layer 3 switches provide backplane throughput equal to the combined bandwidth of all of its ports, so the switching fabric should not be a bottleneck for inter-VLAN routing in such switches.
-
PfSense has been running for over a year on the Fitlet. The uptime as of today is 414 days. Not a single hang or any need to reboot in 414 days. That’s pretty impressive, so I’m happy I got this system built over a year ago.
The system is running version 2-3-2-RELEASE.
-
PfSense has been running for over a year on the Fitlet. The uptime as of today is 414 days. Not a single hang or any need to reboot in 414 days. That’s pretty impressive, so I’m happy I got this system built over a year ago.
The system is running version 2-3-2-RELEASE.
I think it's time to install some security updates ;D You are waaaaay behind.
-
The auto-update is showing 2.3.3_1
Is this an intermediate image that’s required to update from 2.3.2?I’ve read the release notes and it seems that I should be able to update directly from 2.3.2 to 2.4.0.
Why doesn’t the auto-update show 2.4.0?
-
The auto-update is showing 2.3.3_1
Is this an intermediate image that’s required to update from 2.3.2?I’ve read the release notes and it seems that I should be able to update directly from 2.3.2 to 2.4.0.
Why doesn’t the auto-update show 2.4.0?
Probably because of the auto-updater settings. You may need the intermediate version from there before you can continue. Direct upgrades are possible, but you often have to supply the direct upgrade image. This can also be done from the interface and also from the SSH console.