@tsteine said in TNSR for my homelab.:
Hi.
Since @gabacho4 wrote up such an excellent post about his experience (TNSR adventures on my home network) I figured I would borrow his format for my own feedback, why reinvent the wheel if someone else did it better?
Preface
I want to share my experiences with installing TNSR in my homelab. I am a software engineer with a homelab rack that I use for development environments, self hosted cloud storage, hosting the odd dedicated game server for friends, etc.
I'm the type of person who loves experimenting with new tech, learning new things, and have a deep appreciation for really well engineered and high performance products and technology.
So, in the interest of future-proofing my home routing setup for high performance internet down the line, as well as being able to route between local vlans with high throughput, trying out TNSR was a natural choice for me.
Background
My internet connection is 1000/1000 which is directly connected to my ISP's switch over single mode fiber.
The previous setup before TNSR was a Ubiquiti EdgeRouter-8 Pro and it worked great for providing performant internet access, however, given that I have 100GbE and 40GbE switches in my local network, it naturally comes up short as soon as I want to set up vlans and route between them without needing to maintain routing and ACLs on all of my switches.
This lead me to experiment with different software-based router OSes, such as VyOS, but I quickly found myself unable to completely saturate single TCP 40Gbit connections across vlans because of CPU overhead.
Enter TNSR, here is a software-router OS promising up to 100Gbit routing on off-the-shelf commodity hardware, are my problems now solved?
Requirements
Manage firewalling and routing across Vlans and WAN from a single point/CLI.
High performance, bursts up to 40Gbit when necessary.
Implementation
Installing TNSR on a self built bare-metal server was straight forward and simple, the only issue I ran into was that since I had integrated networking as well as 3 separate NICs, it wasn't clear which device in the list was the integrated NIC during setup of the management interface.
Set up my devices for use with the DPDK dataplane, and configured vlans and IP ranges.
Set up Kea DHCP for all vlans, with static IP reservations.
Set up Unbound DNS with forwarding to my AD DNS server for my Domain.
Configure Static NAT for mapping of external ports to internal IPs and ports for servers in the DMZ vlan.
Set up ACL for Local <-> WAN, this took some trial and error, and re-reading of the documentation to properly understand the interaction between the ACL and NAT, as well as stateful outbound connections.
Set up Guest Vlan ACL.
Results
TNSR provides internet connectivity to all my local devices/homelab rack.
TNSR firewalls traffic to the Internet and between Vlan.
NAT configuration provides connectivity for WAN to reach local servers and services.
TNSR routes traffic for single TCP connections across vlans, completely saturating 40Gbit rate.
Considerations
Since TNSR does not support 6rd IPv6(at least yet), and I don't want to set up a separate box for IPv6 connectivity, I am currently foregoing IPv6 connectivity.
Conclusion
I now have an extremely powerful router for my home network/lab which has performance to spare, which allows me to manage all routing between Vlans, as well as firewalling from a single point, without any appreciable loss in performance.
Given that I have a smattering of Arista, Mellanox, and Ubiquiti Edgeswitches, managing routing and ACLs on 3 different command-line interfaces is rather cumbersome, so a single point of management greatly simplifies maintenance for me.
When putting the box through it's paces during testing, I was effectively forced to conclude that I was not capable of generating enough traffic to actually stress the box without a lot of effort, and will never naturally generate traffic anywhere near what the box is capable of handling.
The lowest MTU I tried was 512, and I was still able to saturate 40gbit with 8~Mpps on a single one-way connection, with relatively low and very comfortable system load. That is with out of the box dataplane settings, without tuning workers and rx/tx queues. To give a point of reference, the most I had been able to get out of the box previously across multiple connections and NICs, in aggregate, was 3 Mpps.
Thanks for the post, it was very helpful.
