Netgate 6100 too slow to route 6gbps internet??

gabe-a

@patch That's actually a good question. Yes, I routinely download and upload to AWS, very large datasets of multiple terabytes each. Sometimes I need to pull in dozens of these on my RAID of m2 SSDs and distribute across my workstations. Each workstation has >=1TB of RAM and 56 to 128 cores for parallel processing. Network I/O (internal and to the net) was the bottleneck.

I know what you're thinking: still excessive.

But here's the catch. It was either this (6gbps) or 200mbps over aging coaxial. And the price difference is non-trivial here! I just want to get what I paid for (out of the router and out of the service).

Service was easy (although a few construction crews tore up the basement and had to close the street and "splice" cable from far away, which was a huge inconvenience to my neighbors). This router was a real battle.

gabacho4

@gabe-a what are you upload speeds supposed to be. 6 gig down and 700 meg up is kinda weird imho but so is Comcast sometimes.

Couple thoughts for you, none of which are meant to be mean.

Having a 6 gig internet connection puts you in a different category of network needs for performance. You’ll never find a consumer device (at least now) capable of giving you those speeds. This means you will have to move toward more commercial-strength gear for routing. TNSR is in that class. There are other options that come to mind but they are also command line driven OR require you to pay insane subscription fees or fees to enable the full potential of your hardware. Switching, as you’ve seen, is much easier to do at speed because of ASICS and because a switch doing L2 switching requires far less power than the router which is doing L3 and other. If you were to buy a cheap switch that can also do L3 (or try to at least) you’d find that the switch would get clobbered.
As a result of said reality, the terminal with a serial connection is absolutely something normal and standard for configuring enterprise type gear. In fact, once you’ve gotten used to it, most people tend to prefer it because it’s faster and can give you more info than a GUI. It does come with a learning curve but gets familiar once you’ve used it a bit. Documentation is also your best friend.
I’ve been using TNSR for quite some time and didn’t find the documentation to be bad. Confusing if you’re not used to this type of setup? Sure. But their zero to ping worked outstandingly for me and I remember it being made very clear when you have to go into configure mode and all. You can also tell by the type of prompt you have in the cli. Anything that sets config parameters in TNSR requires configuration mode. Most things that touch on the host configuration and access to Linux internals is done via sudo from user mode.
I expect you didn’t think, or have any interest, in becoming a sys admin. Given the storage and device setup you have you are already far beyond the average user. All you were missing was a multi gig internet service with an enterprise router! I think you could come to love it, learn in the process, and fully utilize your service.

gabe-a

@gabacho4
Thanks, this is the furthest thing from mean I've ever seen. Supportive, encouraging, sensible maybe, but mean? Sorry, nope, not you.

Speeds are supposed to be 6 up, 6 down, and they are. Upload speeds over 1gbps has known issues with speedtest.net + some of their test servers. Other servers show 2gig up, others more or less. It's not concerning because I actually scp'd some big files up there and they drive really close to 6gbps up so it's fine -- speedtest is just better recognized for download speeds (their mobile site doesn't even test upload). Same happens with direct connection to the PC (without a router).

I agree with most of these points and use the linux commandline quite a bit. The problem is more about the software not having sensible defaults and, failing that, presets/guides for simple configuration. The zero to ping instructions did not work for me because:

Inadequate explanation of what was going on and what was truly optional (and why/how to skip them), including speed and security implications of each piece. At one point there is this big piece about setting up nameservers, after which there's a blurb saying "oh that was optional btw haha" or similar.
They did not include info or links or instructions for setting up a static IP (I believe all fiber connections in the US currently provide one)
They did not include setting up a gateway, which is required to route the traffic to the ISP
The enumeration of the devices and ports is non-obvious. They aren't labeled the same thing they are in pfsense or on the actual device.
Comcast literally recommended it. Maybe it was just that one tech, but that recommendation means something, and xfinity is indeed consumer facing, not business/enterprise, so expectations were set
It's hard to imagine typing literally hundreds of lines is faster than 3 clicks and pasting in IP addresses. I agree commandline is faster for some things. But just pumping my internet from one hole to another, enabling sensible default routing and firewall rules? That should be trivial!
Hard to understand why commercial-grade things would be more complicated. Don't the bigwigs like big shiny polished things? Simplicity being the ultimate sophistication and whatnot?

Thinking of making a shell script that automates this for people. Simple picture guide for the initial install, then a shell script you can literally paste into the terminal. It'll ask your IP, subnet, gateway. That's it. Plug in your internet on the left, switch on the right, and go. Y'know, like regular routers. :)

dennypage

@gabe-a said in Netgate 6100 too slow to route 6gbps internet??:

Nobody I've ever seen in my whole life has ever touched a serial connection. That's 90s tech. Why would you assume this is common knowledge? It's not on wikipedia, it's not anywhere I searched.

It's common knowledge to an IT professional person that has experience with configuration of advanced networking hardware. This is the kind of background that someone configuring tnsr would normally have.

If you don't have this background, I would recommend staying with pfSense for the time being, even if it is a bit slower. It will save you a lot of frustration in the end.

Patch

@dennypage Given he has a 6G symmetric service which required building work to run, he is paying full rate for the service, he is regularly saturated that service, and now has it working. I would have thought tnsr was a good fit even if it means a painful learning curve or employing a consultant occasionally.

I don't know where Netgate wants to take the tnsr technology, however if they want wider market penetration, perhaps the pain described here could by used to reduce the pain for future customers.

gabacho4

@gabe-a I think it’s freaking awesome that you were able to take performance from 2 gig to 6 gig by switching software in your router. Dayum. Definitely understand some of your pain factor though a lot of it comes easier as you get more exposure. TNSR wasn’t meant to be a plug and play software option but perhaps there could be ways to make it more accessible. Netgate has been good about listening and responding to its users over the years that I’ve been using their hardware/software. I’m still geeking out over the very significant performance improvement.

MarinSNB

So then, out of curiosity, since TNSR doesn’t have any firewall capabilities, do you then hook up another pfsense appliance to it if for firewall purposes? What would that setup look like? Thanks!

gabacho4

@marinsnb TNSR uses ACLs for firewalling. They work well and are stateful if you make the rule so. I think an enterprise would normally put something else in front but for pfsense you’d need to have some truly beefy hardware to get the 6 gig or greater NAT/firewall performance. For a home connection, OPs setup works just fine.

MarinSNB

@gabacho4 thanks so much!

Patch

@gabe-a said in Netgate 6100 too slow to route 6gbps internet??:

why isn't there one manual per product line?

Perhaps you are looking for this link
https://docs.netgate.com/manuals/tnsr/en/latest/tnsr-product-manual.pdf

Which is given at the early in the online documentation
https://docs.netgate.com/tnsr/en/latest/

stephenw10

There is a specific docs page for the 6100 with TNSR
https://docs.netgate.com/tnsr/en/latest/platforms/netgate-6100/index.html

But it is modular. The reinstall instructions link to the main page.

TNSR is not intended to be a replacement for pfSense. It's a commercial/enterprise router and the docs are written with that in mind. But there's always room for improvement.

@gabe-a said in Netgate 6100 too slow to route 6gbps internet??:

When comcast set up my gigabit pro (that's their new 6gbps fiber internet), they literally told me this: buy a netgate router, the one with 2 10gbps holes

That's very interesting. Did they actually recommend the 6100 directly?

Steve

gabacho4

@stephenw10 I’m not super surprised personally. How many other consumer accessible routers are out there with 10 gig ports on them? Unfortunately the tech didn’t really understand performance related factors. Just because you have 10 gig ports doesn’t mean you have 10 gig throughput with other features enabled. A lesson OP unfortunately had to learn on the tech’s behalf.

DefenderLLC

@gabacho4 said in Netgate 6100 too slow to route 6gbps internet??:

@stephenw10 I’m not super surprised personally. How many other consumer accessible routers are out there with 10 gig ports on them? Unfortunately the tech didn’t really understand performance related factors. Just because you have 10 gig ports doesn’t mean you have 10 gig throughput with other features enabled. A lesson OP unfortunately had to learn on the tech’s behalf.

At least the tech recommended the right brand. :)

stephenw10

Yes. It would be interesting to know where that recommendation came from. And if it was for the 6100 specifically or, say, the 1537 which would pass that with pfSense.

gabe-a

Trying to update (because the glitchy thing doesn't send the gateway IP during its DHCP broadcast, meaning I have to manually configure it on my hosts -- buggy system!!), and I saw the bug was fixed in the latest tnsr release.

But for some completely inane reason, the device that literally serves internet... doesn't get internet. I ping google.com and it says ping: google.com: Temporary failure in name resolution

It only works in the clixon_cli, but that's useless to me since I need to install OS updates.

Can't believe I'm asking this, but how can I literally enable internet connectivity... on the box that serves everything else internet connectivity? Like, why guys...

[edit] Oh but of course, it can't use the actual internet it routes (yes, that's sarcasm -- why the heck can't it?). It needs to be fed internet through a different hole, the one you used when you first set it up. Which of course can't be the actual fiber connection, but an ethernet connection routed from a different router. If I didn't have 2 separate connections in this house and an old router, I swear, this would literally be impossible. This is so awful, guys -- I don't understand how this system is great. So many opportunities to make it easy, but it's so restricted in so many inane ways, for no conceivable reason. Just need a router... period. I don't care about all these "extras" which just cause trouble and configuration nightmares.

gabe-a

@stephenw10 said in Netgate 6100 too slow to route 6gbps internet??:

Yes. It would be interesting to know where that recommendation came from. And if it was for the 6100 specifically or, say, the 1537 which would pass that with pfSense.

There's no conceivable way they'd recommend a rack-looking thing at a price point well over $1000 just to route my internet. I don't remember specifically and I don't think they said specifically besides the cheapest one with the 10G support, which is what I found (and it still wasn't cheap).

The problem is also that tnsr is marketed for 'home use'. Why do this if it's clearly not cut out for anything other than hours-long full-time-job-level config? The point of tech and software is automation, making things easier for people not harder. This does the opposite so fails at its fundamental mission.

Still in process of trying to update the router and I expect extreme breakage every step of the process, like every step before this. Expect me back here begging for troubleshooting help. BTW did you guys know tnsr literally doesn't support querying ifconfig to get what ports on this thing are connected? I had to try each port in the hopes of discovering the one that could receive internet. When I type ifconfig, it says to install net-tools. And when I try that it says it's not available. Like... there's a certain level of unpolished product where things go a little beyond "rough around the edges."

And having to bring down host interfaces, bring them back up, and then manually type "addroute" to the gateway IP... that's simply broken behavior on tnsr's part. If it serves DHCP, it should serve the gateway. The whole network is unreliable because the thing can't keep its connection up -- each time the DHCP lease renews, the gateway is magically lost on all the hosts and I had to set them up again. Hoping the upgrade will fix it. Seriously, has nobody tested the most fundamental feature of the router, the actual gateway? The instructions were buried so deep how to even set up a gateway, so it's no surprise its behavior is broken out of the [net]gate, but still... so much jank.

gabe-a

Of course, the update is broken too.

gabe@gabe10g:~$ sudo ./update-tnsr-newbase.sh 
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-security InRelease
Ign:5 https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ InRelease
Err:6 https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ Release
  Could not load client certificate (/etc/pki/tls/tnsr/certs/tnsr-updates.crt, SslCert option) or key (/etc/pki/tls/tnsr/private/tnsr-updates.key, SslKey option): Error while reading file. [IP: 208.123.73.205 443]
Reading package lists... Done                              
E: The repository 'https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

So...jank...so...bad. I sound like a broken record, don't I? ;)

In fact, now just running

gabe@gabe10g:~$ sudo apt update
Ign:1 https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Err:3 https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ Release
  Could not load client certificate (/etc/pki/tls/tnsr/certs/tnsr-updates.crt, SslCert option) or key (/etc/pki/tls/tnsr/private/tnsr-updates.key, SslKey option): Error wh
ile reading file. [IP: 208.123.73.205 443]
Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:6 http://archive.ubuntu.com/ubuntu focal-security InRelease
Reading package lists... Done
E: The repository 'https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Is also broken. So fragile. Literally followed the instructions word for word and they broke core ubuntu. Yep, tnsr is totally a finished product. /shot

Your amateur update script broke the whole distro. Shame on you -- do customers pay for this? There's no excuse for literally breaking a system with your script-kiddie tnsr trigger-happy overwriting of linux system files without knowing what you're doing.

I tried to circumvent the security issue by adding sudo apt update --allow-unauthenticated --allow-insecure-repositories but that still yields

Err:7 https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ Packages
  Could not load client certificate (/etc/pki/tls/tnsr/certs/tnsr-updates.crt, SslCert option) or key (/etc/pki/tls/tnsr/private/tnsr-updates.key, SslKey option): Error wh
ile reading file. [IP: 208.123.73.205 443]
Ign:8 https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ Translation-en_US
Ign:9 https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ Translation-en
Reading package lists... Done
W: The repository 'https://tnsr-deb-hw.netgate.com/20.04/main/x86_64 ./ Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://tnsr-deb-hw.netgate.com/20.04/main/x86_64/./Packages  Could not load client certificate (/etc/pki/tls/tnsr/certs/tnsr-updates.crt, SslCert optio
n) or key (/etc/pki/tls/tnsr/private/tnsr-updates.key, SslKey option): Error while reading file. [IP: 208.123.73.205 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.

So I guess your insecure broken system has permanently ruined updates for me, opening me up to OS-level insecurity. Great job.

gabe-a

@gabe-a A hint in the series of breakage:

when trying to install the certificate (why it can't do this automatically is again, horrible design):
Nov 2 02:42:23: Backend downcall: application invalid-value key tnsr-updates does not exist

Nobody ever gave me a certificate and I never had to mess with it before. Why is this crap all of a sudden coming up to cause problems? Why can't your software, for once, just actually work? There are so many bad practices here it's staggering.

Please tell me what I actually need to do to fix your 6100 running your 22.06 tnsr system and let it update properly. It should not be this hard. It should not be. This is truly, absolutely, incontrovertibly, bad software design. It simply is.

gabacho4

@gabe-a did you purchase a license? If you’re running the home+lab version:

How do I update my Home+Lab installation?

The Home+Lab version of TNSR can't be updated in-place. To upgrade to the newest version, simply order the new release and run a fresh installation. There is a documented path here to migrate most configuration settings from the prior installation.

For a more seamless software upgrade process we recommend you upgrade to TNSR with TAC Pro or TNSR with TAC Enterprise, which includes access to rolling updates and 24/7 expert TAC support.

Derelict

@gabe-a The private key must be installed before installing the matching certificate.

Follow these instructions to the letter:

https://docs.netgate.com/tnsr/en/latest/updating/index.html

Note that if you are running Home+Lab you are not entitled to an update certificate.

You can "purchase" ($0) Home+Lab again and you will be offered the 22.10 ISO file you can use to reinstall.