Home+lab feedback

khegel

A local ISP recently installed fiber in my city and during the installation informed me that they would be giving me 10gbps symmetric service for the same price as the 1gig service I signed up for. The ONT device they provided also did routing and basic NAT, but it was very under powered to handle a connection that fast.

Requirements

I only have a half depth rack in my laundry room so I needed a small 1U half depth server
I want to be able to route IMIX at line speed
IPsec around half line speed
NAT and basic ACLs
Not too high power consumption because electricity is very expensive where I live

I ended up buying a used Netgate XG-1537 at a consignment auction from a startup that closed their office.

Installation
Installation was very easy following the provided documentation. I only ran into two issues:

There is a bug in anaconda which crashes the installation when block devices have identical UUIDs which is the case in software RAID1 that is configured on the boot NVMe drives on the Netgate appliance. To solve this I changed the UUID of one of the drives in the mirrored array and the installer was able to continue.
The text based installer is not able to configure software raid. I solved this by adding a grub boot option to start a vnc server with the GUI installer.

Using the GUI installer I was then able to configure RAID1 on the two drives similar to how it was previously set up.

Configuration
I used the configuration from the Zero to Ping document minus the DNS and DHCP portion because I have a pi-hole on my network handling DNS and DHCP. As soon as my ISP changed my ONT to bridged mode and the WAN interface got a public IP address everything started working immediately.

Performance
My internet connection is much more stable now. I don't have access to another network with a 10g port to do any sort of complex testing but I am able to transfer around 8gbps. The only issue I have is with a Windows 10 client and having tcp autotuninglevel set to normal. When this is set I get very high burst speeds but throughput is very erratic. When this feature is turned off, I get slower sustained transfer speeds but they are very consistent. I'm not familiar enough with networking to figure out how this TCP window scaling is working on Windows or how to troubleshoot but I have no issues on linux servers.

Overall I am very pleased with how easy tnsr was to set up and configure. My expertise is not in networking but I was able to understand the configuration syntax and the documentation was very thorough and helpful. I did not have time to learn how to set up VPP and DPDK myself and this product saved me a lot of time in that regard. I do wish there was a fully open source version similar to pfsense as I don't think I would like to spend money just for home use.