Netgate 6100 too slow to route 6gbps internet??

[gabe-a]

@keyser
Thanks. Although no stranger to the commandline, I'm definitely starting from near zero when it comes to networking/hardware things.

The only things I'm planning to connect to the 6gbps network are Linux workstations. There's also a "free" 1gbps ethernet service (on a different static xfinity IP) also included that I'm using for them now and any "household" things (TVs, laptops, wifi, etc), routed through a totally separate $300 router that works very well. (In fact that router has 2.5gbps ports that really do function at their advertised speeds, making it almost equivalent to this Netgate router in performance despite being a third of the price!).

Regardless, I can confidently answer that the Linux boxes can indeed be configured with software firewalls before connecting. In fact I've been using the direct SFP connection from the aria switch -> juniper switch -> PC already and reaching very close to 6gbps in normal internet use. (The same PC is also independently connected to the 2.5gbps network but completely disregards the 1gbps internet stream they all use, and they also cannot connect to the internet through the one PC that is connected directly to the 6gbps -- they all share the inferior 1gbit behind a normal consumer router + firewall).

Public IPs -- my ISP just gives me one single static IP address that's public facing. I have no choice but to get a router to split internet delivered via that IP address across the other machines via localnet (otherwise I'd have been happy with a passive switch...but a switch can't give out separate local IP addresses that can independently send/receive traffic. Hence, a switch is not a router).

NAT -- no, I really have no idea what a NAT is, but with the limited aid of wikipedia I see that it's likely I can answer yes to your question, as the static IP I'm assigned comes directly from washington over fiber optic and gets transformed through 2 separate boxes, the aria and the juniper mentioned previously. What pops out the end is a glorious 6gbps signal whose IP address does not look like it's filtered through middleware as it literally from the city that serves the fiber network, not from some local sub-NAT redistribution point (hey, sorry if I'm mixing up terms or concepts, but you get what I mean...I hope?).

[stephenw10]

The 6100 has two console ports, an RJ-45 port, as commonly found on switches and routers, but also a USB port that you can connect to any host that has a USB port. It is supplied with a cable for connecting to USB-A. You can also connect to USB-C using an adapter or a USB-C to USB-micro-B cable.
https://docs.netgate.com/pfsense/en/latest/solutions/netgate-6100/connect-to-console.html

Steve

[gabe-a]

@stephenw10
Well that helps. Although I'm not a fan of how the documentation is modular to the point of actually being less helpful (why isn't there one manual per product line? It's not like there are tons of product lines here).

1. Downloaded the ISO (you'll have to request this in the netgate shop)
2. Burnt it using balenaEtcher to a usb stick
3. Plugged the USB stick into the router
4. Plugged the micro-USB into the router, big end into my computer
5. Ran sudo screen /dev/ttyUSB0 115200
6. Pressed enter again to get stuff to actually appear (otherwise screen shows a blank screen...bad)
7. Ran reboot
8. On reboot, pressed F2
9. Selected the second option in the boot menu (grub something)
10. Followed along here from step 7: https://docs.netgate.com/tnsr/en/latest/installation/index.html
11. Got stuck at step 19's note (the instructions simply don't work):

user:~$ sudo -u tnsr clixon_cli
tnsr# nacm group admin
CLI syntax error: "nacm group admin": Unknown command

CLI syntax error: "nacm group admin": Unknown command

Fail. I think these instructions need an update, as the entire set up to 10 aren't included anywhere and you can't expect people to just "know" them. And the hopelessly spread-around [dis]information on nacm is atrocious, and clearly does not work. Do I or do I not need to run that failed command? If so, how? This doesn't bode well for actually setting up interfaces later...

[dennypage]

@gabe-a said in Netgate 6100 too slow to route 6gbps internet??:

Pressed enter again to get stuff to actually appear (otherwise screen shows a blank screen...bad)

Yes, this is to be expected. The OS has no way to know if you are connected to the serial port or not. It's inherent in the nature of a serial connection.

@gabe-a said in Netgate 6100 too slow to route 6gbps internet??:

user:~$ sudo -u tnsr clixon_cli
tnsr# nacm group admin
CLI syntax error: "nacm group admin": Unknown command

CLI syntax error: "nacm group admin": Unknown command

I believe that the command you are trying to execute (nacm group admin) requires that you enter config mode first. At the tnsr# prompt enter "configure" before proceeding.

This section may be of use to you: Zero-to-Ping: Getting Started.

If you've managed network equipment such as a Cisco router before, these commands are all pretty intuitive. If you haven't, there's a learning curve.

[gabe-a]

@gabe-a

11 (repeated). Multiple pages later, an explanation for #19 is given. Yes, unbelievably even the linked page on nacm introduction and subsequent examples fails to convey the fundamentals of usage See that weird parenthesis? The one that says "(config)" in it? Well it turns out you have to type the word "configuration" once you've done sudo -u tnsr clixon_cli. I know. I know... so many huge gaps. You have to read chapters ahead and go back again to actually follow the instructions.

12. dataplane dpdk dev 0000:03:00.0 network name WAN
13. dataplane dpdk dev 0000:03:00.1 network name LAN
14. configure terminal
    Follow the ACL instructions here: https://docs.netgate.com/tnsr/en/latest/ztp/index.html
    QUESTION: why? Why do we need ACL? What happens if we don't use it? Does it slow things down? Will the following steps work without it?
15. Actually do the IP setup for your internet. Use 2 addresses after the static IP that comcast gives you (that's because they give you the switch's IP, and then the gateway is that plus one, and then the actual IP address is 1 after that. So if it ends in .21, you have to use .23. I know, also not documented by xfinity. Anyway:

tnsr# configure terminal
tnsr(config)# interface WAN
tnsr(config-interface)# ip address XX.XXX.XX.XX/30
tnsr(config-interface)# ipv6 address 2001:XXX:XXX::2/48
tnsr(config-interface)# enable
tnsr(config-interface)# exit
tnsr(config)# interface LAN
tnsr(config-interface)# ip address 10.2.0.1/24
tnsr(config-interface)# ipv6 address 2001:db8:1::1/64
tnsr(config-interface)# enable
tnsr(config-interface)# exit

Yes, carpel tunnel after all this repetitive copying/pasting/typing is a real risk. This is really, truly cumbersome and terrible.
16. NAT setup. Same question -- why? What? And how come we have to set this up? In pfsense I didn't have to touch this? Does this slow things down? What if I want to do simple port forwarding later?

user tnsr(config)# nat global-options nat44 forwarding true
user tnsr(config)# nat global-options nat44 endpoint-dependent true
user tnsr(config)# nat global-options nat44 enabled true
user tnsr(config)# nat pool interface WAN
user tnsr(config)# interface WAN
user tnsr(config-interface)# ip nat outside
user tnsr(config-interface)# exit
user tnsr(config)# interface LAN
user tnsr(config-interface)# ip nat inside
user tnsr(config-interface)# exit
user tnsr(config)# dhcp4 server
user tnsr(config-kea-dhcp4)# description LAN DHCP Server
user tnsr(config-kea-dhcp4)# interface listen LAN
user tnsr(config-kea-dhcp4)# lease lfc-interval 3600
user tnsr(config-kea-dhcp4)# subnet 10.2.0.1/24
user tnsr(config-kea-subnet4)# pool 10.2.0.100-10.2.0.245
user tnsr(config-kea-subnet4-pool)# exit
user tnsr(config-kea-subnet4)# interface LAN
user tnsr(config-kea-subnet4)# option domain-name-servers
user tnsr(config-kea-subnet4-opt)# data 10.2.0.1
user tnsr(config-kea-subnet4-opt)# exit
user tnsr(config-kea-subnet4)# option routers
user tnsr(config-kea-subnet4-opt)# data 10.2.0.1
user tnsr(config-kea-subnet4-opt)# exit
user tnsr(config-kea-subnet4)# exit
user tnsr(config-kea-dhcp4)# exit
user tnsr(config)# dhcp4 enable
user tnsr(config)# unbound server
user tnsr(config-unbound)# interface 127.0.0.1
user tnsr(config-unbound)# interface 10.2.0.1
user tnsr(config-unbound)# access-control 10.2.0.1/24 allow
user tnsr(config-unbound)# forward-zone .
user tnsr(config-unbound-fwd-zone)# nameserver address 8.8.8.8
user tnsr(config-unbound-fwd-zone)# nameserver address 8.8.4.4
user tnsr(config-unbound-fwd-zone)# exit
user tnsr(config-unbound)# exit
user tnsr(config)# unbound enable
user tnsr(config)# system dns-resolver dataplane
user tnsr(config-dns-resolver)# server localhost 127.0.0.1
user tnsr(config-dns-resolver)# exit
user tnsr(config)# system dns-resolver host
user tnsr(config-dns-resolver)# server g1 8.8.8.8
user tnsr(config-dns-resolver)# server g2 8.8.4.4
user tnsr(config-dns-resolver)# exit
user tnsr(config)# configuration copy running startup
user tnsr(config)# configuration save running backup.xml

[gabe-a]

@dennypage
Things that are "inherent in the nature of X", where X is absolutely intimitely essential to following the directions... should be explicitly said in the directions.
Nobody I've ever seen in my whole life has ever touched a serial connection. That's 90s tech. Why would you assume this is common knowledge? It's not on wikipedia, it's not anywhere I searched. I tried to restart my system multiple times before pounding my keyboard in frustration before stuff popped up.

Why isn't "configure" in the logical order of the setup document? Again, gaps.

I'm trying to work through this. I have no idea what examples are applicable to my use case and what aren't.

I'm -- literally -- trying to set up something that takes 30 seconds in any other respectable router. Just want to enter my static IP (that's how comcast gives them for gigabit pro), the gateway, the subnet, and then plug a switch into the other hole to get my internet & networking.

Why the literally hundreds of lines of inscrutable text? Why is this considered the replacement for pfsense? Why can't the pfsense interface just hook into this nasty thing and do this for us?

[gabe-a]

@gabe-a
Okay, sorry for the double post, and I just had a realization. It just ocurred to me you're probably wondering (in shock and horror) why this rando came on getting frustrated about setting up your "business firewall" and what could have possessed such a purchase in the first place.

Let me take a step back for context. When comcast set up my gigabit pro (that's their new 6gbps fiber internet), they literally told me this: buy a netgate router, the one with 2 10gbps holes, plug this cable into one of the holes (the internet), and plug your 10G switch (it's a trendnet ultra-cheap) into the other. Boom -- 10 gigabit ethernet network plus 6gbps internet, easy as can be. Just like all the other consumer routers for normal cable internet. And since almost all desktops/workstations these days have 10gbe ports, well... seemed like a no-brainer.

This is THE router recommended for THE fiber optic 10gbe connection in the united states, to my knowledge. I am extremely vocal so I immediately came on here to complain and try to make it work, but I'm sure others have not and are suffering or canceling.

If it wasn't clear before, I'm really not liking this tnsr business. Really user-unfriendly way to get internet out of a router. Who the heck has experience managing "Cisco routers" among those who just want fast xfinity at home? Not I!

[Patch]

@gabe-a You have been given access to a 6G internet service at a good price.
Do you actually need or would notice fully WAN speed in practice?
The reason I ask is that speed is not widely available to consumers so not broadly supported with consumer gear.
If you instead just use what readily works (such as pfsense) you may find that is all you need at the moment.

PS
I agree a pfsense front end to tnsr back end would be awesome.

[gabe-a]

@patch

user tnsr(config)# route table ipv4-VRF:0
user tnsr(config-route-table)# route 0.0.0.0/0
user tnsr(config-rttbl4-next-hop)# next-hop 0 via XX.XXX.XX.23
user tnsr(config-rttbl4-next-hop)# exit
user tnsr(config-route-table)# exit

Forgot the above. Wow, this documentation is really, really poor, I'm so very sorry to say. It's so poor because it's fragmented and nonsensical.

The above was required to simply add a gateway. Yes, really!

I'm exhausted and drained.

It works now. https://www.speedtest.net/result/13827218769
Properly serving the network over DHCP. Now I have to test if the workstations can see each other (seeing the internet is only half the battle). Still also have to figure out port forwarding. Probably some setting in the NAT thingy.

[gabe-a]

@patch That's actually a good question. Yes, I routinely download and upload to AWS, very large datasets of multiple terabytes each. Sometimes I need to pull in dozens of these on my RAID of m2 SSDs and distribute across my workstations. Each workstation has >=1TB of RAM and 56 to 128 cores for parallel processing. Network I/O (internal and to the net) was the bottleneck.

I know what you're thinking: still excessive.

But here's the catch. It was either this (6gbps) or 200mbps over aging coaxial. And the price difference is non-trivial here! I just want to get what I paid for (out of the router and out of the service).

Service was easy (although a few construction crews tore up the basement and had to close the street and "splice" cable from far away, which was a huge inconvenience to my neighbors). This router was a real battle.

[gabacho4]

@gabe-a what are you upload speeds supposed to be. 6 gig down and 700 meg up is kinda weird imho but so is Comcast sometimes.

Couple thoughts for you, none of which are meant to be mean.

1. Having a 6 gig internet connection puts you in a different category of network needs for performance. You’ll never find a consumer device (at least now) capable of giving you those speeds. This means you will have to move toward more commercial-strength gear for routing. TNSR is in that class. There are other options that come to mind but they are also command line driven OR require you to pay insane subscription fees or fees to enable the full potential of your hardware. Switching, as you’ve seen, is much easier to do at speed because of ASICS and because a switch doing L2 switching requires far less power than the router which is doing L3 and other. If you were to buy a cheap switch that can also do L3 (or try to at least) you’d find that the switch would get clobbered.

2. As a result of said reality, the terminal with a serial connection is absolutely something normal and standard for configuring enterprise type gear. In fact, once you’ve gotten used to it, most people tend to prefer it because it’s faster and can give you more info than a GUI. It does come with a learning curve but gets familiar once you’ve used it a bit. Documentation is also your best friend.

3. I’ve been using TNSR for quite some time and didn’t find the documentation to be bad. Confusing if you’re not used to this type of setup? Sure. But their zero to ping worked outstandingly for me and I remember it being made very clear when you have to go into configure mode and all. You can also tell by the type of prompt you have in the cli. Anything that sets config parameters in TNSR requires configuration mode. Most things that touch on the host configuration and access to Linux internals is done via sudo from user mode.

4. I expect you didn’t think, or have any interest, in becoming a sys admin. Given the storage and device setup you have you are already far beyond the average user. All you were missing was a multi gig internet service with an enterprise router! I think you could come to love it, learn in the process, and fully utilize your service.

[gabe-a]

@gabacho4
Thanks, this is the furthest thing from mean I've ever seen. Supportive, encouraging, sensible maybe, but mean? Sorry, nope, not you.

Speeds are supposed to be 6 up, 6 down, and they are. Upload speeds over 1gbps has known issues with speedtest.net + some of their test servers. Other servers show 2gig up, others more or less. It's not concerning because I actually scp'd some big files up there and they drive really close to 6gbps up so it's fine -- speedtest is just better recognized for download speeds (their mobile site doesn't even test upload). Same happens with direct connection to the PC (without a router).

I agree with most of these points and use the linux commandline quite a bit. The problem is more about the software not having sensible defaults and, failing that, presets/guides for simple configuration. The zero to ping instructions did not work for me because:

1. Inadequate explanation of what was going on and what was truly optional (and why/how to skip them), including speed and security implications of each piece. At one point there is this big piece about setting up nameservers, after which there's a blurb saying "oh that was optional btw haha" or similar.
2. They did not include info or links or instructions for setting up a static IP (I believe all fiber connections in the US currently provide one)
3. They did not include setting up a gateway, which is required to route the traffic to the ISP
4. The enumeration of the devices and ports is non-obvious. They aren't labeled the same thing they are in pfsense or on the actual device.
5. Comcast literally recommended it. Maybe it was just that one tech, but that recommendation means something, and xfinity is indeed consumer facing, not business/enterprise, so expectations were set
6. It's hard to imagine typing literally hundreds of lines is faster than 3 clicks and pasting in IP addresses. I agree commandline is faster for some things. But just pumping my internet from one hole to another, enabling sensible default routing and firewall rules? That should be trivial!
7. Hard to understand why commercial-grade things would be more complicated. Don't the bigwigs like big shiny polished things? Simplicity being the ultimate sophistication and whatnot?

Thinking of making a shell script that automates this for people. Simple picture guide for the initial install, then a shell script you can literally paste into the terminal. It'll ask your IP, subnet, gateway. That's it. Plug in your internet on the left, switch on the right, and go. Y'know, like regular routers. :)

[dennypage]

@gabe-a said in Netgate 6100 too slow to route 6gbps internet??:

Nobody I've ever seen in my whole life has ever touched a serial connection. That's 90s tech. Why would you assume this is common knowledge? It's not on wikipedia, it's not anywhere I searched.

It's common knowledge to an IT professional person that has experience with configuration of advanced networking hardware. This is the kind of background that someone configuring tnsr would normally have.

If you don't have this background, I would recommend staying with pfSense for the time being, even if it is a bit slower. It will save you a lot of frustration in the end.

[Patch]

@dennypage Given he has a 6G symmetric service which required building work to run, he is paying full rate for the service, he is regularly saturated that service, and now has it working. I would have thought tnsr was a good fit even if it means a painful learning curve or employing a consultant occasionally.

I don't know where Netgate wants to take the tnsr technology, however if they want wider market penetration, perhaps the pain described here could by used to reduce the pain for future customers.

[gabacho4]

@gabe-a I think it’s freaking awesome that you were able to take performance from 2 gig to 6 gig by switching software in your router. Dayum. Definitely understand some of your pain factor though a lot of it comes easier as you get more exposure. TNSR wasn’t meant to be a plug and play software option but perhaps there could be ways to make it more accessible. Netgate has been good about listening and responding to its users over the years that I’ve been using their hardware/software. I’m still geeking out over the very significant performance improvement.

[MarinSNB]

So then, out of curiosity, since TNSR doesn’t have any firewall capabilities, do you then hook up another pfsense appliance to it if for firewall purposes? What would that setup look like? Thanks!

[gabacho4]

@marinsnb TNSR uses ACLs for firewalling. They work well and are stateful if you make the rule so. I think an enterprise would normally put something else in front but for pfsense you’d need to have some truly beefy hardware to get the 6 gig or greater NAT/firewall performance. For a home connection, OPs setup works just fine.

[MarinSNB]

@gabacho4 thanks so much!

[Patch]

@gabe-a said in Netgate 6100 too slow to route 6gbps internet??:

why isn't there one manual per product line?

Perhaps you are looking for this link
https://docs.netgate.com/manuals/tnsr/en/latest/tnsr-product-manual.pdf

Which is given at the early in the online documentation
https://docs.netgate.com/tnsr/en/latest/

[stephenw10]

There is a specific docs page for the 6100 with TNSR
https://docs.netgate.com/tnsr/en/latest/platforms/netgate-6100/index.html

But it is modular. The reinstall instructions link to the main page.

TNSR is not intended to be a replacement for pfSense. It's a commercial/enterprise router and the docs are written with that in mind. But there's always room for improvement.

@gabe-a said in Netgate 6100 too slow to route 6gbps internet??:

When comcast set up my gigabit pro (that's their new 6gbps fiber internet), they literally told me this: buy a netgate router, the one with 2 10gbps holes

That's very interesting. Did they actually recommend the 6100 directly?

Steve