TNSR for my homelab.
-
Hi.
Since @gabacho4 wrote up such an excellent post about his experience (TNSR adventures on my home network) I figured I would borrow his format for my own feedback, why reinvent the wheel if someone else did it better?Preface
I want to share my experiences with installing TNSR in my homelab. I am a software engineer with a homelab rack that I use for development environments, self hosted cloud storage, hosting the odd dedicated game server for friends, etc.
I'm the type of person who loves experimenting with new tech, learning new things, and have a deep appreciation for really well engineered and high performance products and technology.
So, in the interest of future-proofing my home routing setup for high performance internet down the line, as well as being able to route between local vlans with high throughput, trying out TNSR was a natural choice for me.Background
My internet connection is 1000/1000 which is directly connected to my ISP's switch over single mode fiber.
The previous setup before TNSR was a Ubiquiti EdgeRouter-8 Pro and it worked great for providing performant internet access, however, given that I have 100GbE and 40GbE switches in my local network, it naturally comes up short as soon as I want to set up vlans and route between them without needing to maintain routing and ACLs on all of my switches.
This lead me to experiment with different software-based router OSes, such as VyOS, but I quickly found myself unable to completely saturate single TCP 40Gbit connections across vlans because of CPU overhead.
Enter TNSR, here is a software-router OS promising up to 100Gbit routing on off-the-shelf commodity hardware, are my problems now solved?Requirements
- Manage firewalling and routing across Vlans and WAN from a single point/CLI.
- High performance, bursts up to 40Gbit when necessary.
Implementation
- Installing TNSR on a self built bare-metal server was straight forward and simple, the only issue I ran into was that since I had integrated networking as well as 3 separate NICs, it wasn't clear which device in the list was the integrated NIC during setup of the management interface.
- Set up my devices for use with the DPDK dataplane, and configured vlans and IP ranges.
- Set up Kea DHCP for all vlans, with static IP reservations.
- Set up Unbound DNS with forwarding to my AD DNS server for my Domain.
- Configure Static NAT for mapping of external ports to internal IPs and ports for servers in the DMZ vlan.
- Set up ACL for Local <-> WAN, this took some trial and error, and re-reading of the documentation to properly understand the interaction between the ACL and NAT, as well as stateful outbound connections.
- Set up Guest Vlan ACL.
Results
- TNSR provides internet connectivity to all my local devices/homelab rack.
- TNSR firewalls traffic to the Internet and between Vlan.
- NAT configuration provides connectivity for WAN to reach local servers and services.
- TNSR routes traffic for single TCP connections across vlans, completely saturating 40Gbit rate.
Considerations
Since TNSR does not support 6rd IPv6(at least yet), and I don't want to set up a separate box for IPv6 connectivity, I am currently foregoing IPv6 connectivity.Conclusion
I now have an extremely powerful router for my home network/lab which has performance to spare, which allows me to manage all routing between Vlans, as well as firewalling from a single point, without any appreciable loss in performance.
Given that I have a smattering of Arista, Mellanox, and Ubiquiti Edgeswitches, managing routing and ACLs on 3 different command-line interfaces is rather cumbersome, so a single point of management greatly simplifies maintenance for me.
When putting the box through it's paces during testing, I was effectively forced to conclude that I was not capable of generating enough traffic to actually stress the box without a lot of effort, and will never naturally generate traffic anywhere near what the box is capable of handling.
The lowest MTU I tried was 512, and I was still able to saturate 40gbit with 8~Mpps on a single one-way connection, with relatively low and very comfortable system load. That is with out of the box dataplane settings, without tuning workers and rx/tx queues. To give a point of reference, the most I had been able to get out of the box previously across multiple connections and NICs, in aggregate, was 3 Mpps. -
@tsteine Thank you very much for sharing your experience and the details of your journey. Glad to hear that TNSR addressed your requirements and is performing so well.
-
Great write-up of your TNSR experience and packet pushing needs. You're definitely moving data in a big way. I have a 1000/1000 fiber connection at my home in the US that I anxiously look forward to abusing (not EULA
) one day. As for your formatting, they do say that "imitation is the sincerest form of flattery." I'm geeking out knowing that I'm not the only home user of TNSR. -
I absolutely believe there is a market for TNSR among networking enthusiasts looking for high performance routing at home.
In all fairness, the amount of throughput for TNSR is usually limited by my internet connection's bandwidth, the high throughput tasks across vlans will be DMZ servers accessing the storage, or me moving files to my storage boxes from my workstation, etc.
Most of the really high throughput stuff will be local to the vlan it originates from, and not having to get elevated to the router and pass to a different vlan.
99% time, the throughput is less than 1gbit, but having the ability to push a monstrous amount of packets and throughput when the need arises is any networking enthusiast's dream. The ease of not having to manage firewalling and routing on the switches also cannot be overstated in my opinion.
There is something immensely satisfying with having rock solid networking performance and seeing your file transfers, backup tasks, vm migrations, etc running at several gigabytes per second.
-
@tsteine said in TNSR for my homelab.:
There is something immensely satisfying with having rock solid networking performance and seeing your file transfers, backup tasks, vm migrations, etc running at several gigabytes per second.
You must be my doppelganger...
-
@tsteine looks like you're famous.
https://twitter.com/NetgateUSA/status/1278742734031982592?s=19
-
Now it's my turn to geek out
-
@audian said in TNSR for my homelab.:
@tsteine Muito obrigado por compartilhar sua experiência e os detalhes de sua jornada. Fico feliz em saber que o TNSR atendeu aos seus requisitos e está funcionando tão bem.
@tsteine Sou grato por você partilhar sua experiência e os detalhes de sua carreira. Fico alegre em saber que o TNSR compre os seus requisitos potenciais e está funcionando de forma excelente.
-
One huge weakness of TNSR that I've encountered is that it does not support dhcp relay.
Even though I am a home user, I do not want a DHCP server running on any of my switches or routers. Following the same thought process, I do not want a full Unbound DNS server (any dns server, frankly) running on my router.
It's nice to have the option, but it creates unnecessary vulnerabilities and increases your overall attack surface.
In order to enable DHCP relay, you require a managed switch with a related SVI to relay DHCP requests. This is a huge weakness in my book. Even though TNSR is dumb fast, it does not have many of the basic (crucial) features that other routers have.
-
This post is deleted! -
This post is deleted!
