SG-3100 Slow Throughput

[johnpoz]

There you go - that is some screaming performance to be honest with 949 being pretty much max speed in theory, etc.

[sean.allen]

@johnpoz I have had the SG-3100 for a year, but my fiber line just got installed 10 days ago. Before that I had AT&T bonded 50Mb DSL that ran at about 40Mb with this SG-3100. That exact same config, when running through the new AT&T RG in "IP Passthrough mode" (poor-mans bridging) yielded sub-100Mb speed. I then implemented the netgraph-based RG bypass hoping the issue was double-natting or something else with the RG - but it didn't change the results. I haven't touched shaping/limiting in probably a year. Definitely not since I added fiber to the house. And I think we just saw that none of that still exists in my config.

[sean.allen]

@grimson In checking the interface statistics, hopefully correctly, I find that all are at 1000baseT or above with no errors or drops.

mvneta1 is the four-port switched LAN group
mvneta2 is the routed NIC that is plugged into the ONT
ngeth0 is the interface created by netgraph that tags traffic VLAN 0 so AT&T will work with it without the RG

netstat -i shows zero errors or drops across all interfaces:

Apologies if that doesn't get the interface statistics you were talking about. The "basic steps" I've seen here are "buy a new switch, sniff the traffic, look for anything out of place" and "factory reset your SG-3100" - the reasons why they've been suggested don't make sense to me based on evidence. I'm not trying to argue with the superior hive mind here, I'm just trying to offer logic that says "there are no shapers or limiters in effect, how can it be that, so what does starting from scratch do?" and "I can buy a smart switch, but I can't read a packetcap or know what is out of place - and what are we hoping to learn as the problem persists whether I go through the AT&T RG or bypass it." Factory reset will take forever and I will likely screw something up trying to recreate everything I've done over the past five years to this install (across two hardware platforms for pfSense) without screwing it up. Certs, OpenVPN server and client, firewall rules, packages (and their configs), etc. If that comes across whiney, please forgive me. Not my intent. I appreciate any and all specific help that is being offered. Even when it comes with snark.

@Derelict Got it. I think we've ruled out limiters/shapers in my config. ifconfig -a shows 1000BaseT or 2500BaseT full duplex. I'm reading your advice as replacing the WAN side of my network with a system running iperf and test. That makes sense. It isolates the SG-3100. If that's what was suggested prior by someone else, and I was too thick to see it, sorry. I was hoping to leave the ISP in the mix for convenience since it was proven to be 900Mb when going client->RG->ONT->speedtest server

[Derelict]

You can back up the configuration, reset to defaults, and do a quick static WAN and test like I did there. Then maybe do a very basic configuration for the RG bypass and test that.

You're 2 minutes from being back where you were by simply restoring the configuration.

The object here is to isolate whether the hardware is bad or if it's something else in your environment.

AT&T is not doing anybody any favors with this arrangement. They are basically telling their customers to See Figure 1.

[Grimson]

@sean-allen said in SG-3100 Slow Throughput:

Factory reset will take forever and I will likely screw something up trying to recreate everything I've done over the past five years to this install (across two hardware platforms for pfSense) without screwing it up.

Bullshit. Take a config backup, install a fresh 2.4.4 image, do the basic setup, test throughput. If you then get full speed you know something in your config is messed up, in that case restore the config section by section and check each time until it breaks again. If not restore the config backup and continue searching. That shouldn't take more than an hour.

Also if that config is five years old and spans multiple hardware platforms you might have at one point or the other added/changed system tunables that could negatively affect the performance on your current hardware. Also you experimented with fq_codel which before 2.4.4 required intervention in system files/the command line, there might still be remnant effects of that. So we need a known good baseline to start diagnosing from and that is a clean installation with a basic setup.

As it stands now your like a whining kid, expecting us to solve your problems while you are unwilling to put work into it. This is wasting the good will and time of the community.

[Derelict]

Yeah. While you're down a fresh install of 2.4.4 certainly wouldn't hurt. Only takes a few minutes and then you know.

https://www.netgate.com/docs/pfsense/solutions/sg-3100/reinstall-pfsense.html

[Derelict]

And that ngeth is completely untested by anyone here as far as I know. It is not part of pfSense. I don't think anyone here has to suffer Uverse for their internet.

Any side effects or other issues are unknown. Anecdotal evidence suggests it is working for some people.

[sean.allen]

@derelict Ok. Interestingly, I had that exact URL up already (fresh install on SG-3100). Isolate the hardware, then isolate the bypass. Makes sense.

And totally agree on AT&T. Miserable and very AT&T of them.

ngeth0 is untested here, I agree. However, I was seeing the same slowdown issues when it was not in the mix and I was simply using mvneta2 and WAN and attaching the to AT&T RG in IP Passthrough. And I know of at least one other person that implemented this same exact bypass, on an SG-3100, and is seeing no slowdown.

@Grimson Are there any hacks to "restore the config section by section" - or just take really good notes on what you have and type it back in until things go bad?

Great points on rot being in the config over that time span. I don't recall doing tunables (except yesterday adding net.inet.ip.fastforwarding to try and speed up OpenVPN - but that was after these general speed problems). Thanks for the great reasoning on cruft that could be in my config over time.

I have kids. I hate it when they whine. Truly loathe it. However, I really love it when they ask questions, push back, and try to learn why they're being asked to do something. Shows me they're thinking - trying to learn so they don't need as much help in the future. And who knows, maybe someone else seeing/hearing the conversation that is afraid to ask the question (perhaps for fear of being flamed) learns from it, too.

Regardless, I sincerely appreciate y'all taking the time to help. I apologize that my approach rubs some the wrong way. Just trying to learn. I'm off to rebuild. Not afraid of the work, just want to know why I'm doing it.

[Grimson]

@sean-allen said in SG-3100 Slow Throughput:

@Grimson Are there any hacks to "restore the config section by section" - or just take really good notes on what you have and type it back in until things go bad?

After 5 years you never looked at the backup and restore page of pfSense?

[sean.allen]

@grimson Uhm, shit, can we forget I said that part? Wow, I just earned every bit of your frustration/angst right there. Yep, I've been there several times. And double yep, I totally forgot that you could do it in sections...making my objection to rebuilding seem petty. Got it.

Mea culpa. Quite sorry.

It was worthwhile and helpful to know all the different bits of cruft that can be cleared out with this approach, but the process is nothing. I've already downloaded and created a USB drive of the fresh 2.4.4 image. Off I go.

[aus]

FWIW, I'm running a Dell R210 II via Xeon E31220 @ 3.10GHz on pfSense 2.4.4. Here's a recent speedtest from a server within my LAN:

SpeedTest++ version 1.14
Speedtest.net command line interface
Info: https://github.com/taganaka/SpeedTest
Author: Francesco Laurita <francesco.laurita@gmail.com>

IP: 
Finding fastest server... 7727 Servers online
............
Server: speedtest: 2 ms
Ping: 2 ms.
Jitter: 0 ms.
Determine line type (2) ........................
Fiber / Lan line type detected: profile selected fiber

Testing download speed (32) ....................................................................................................................................................................................................................................................................
Download: 954.57 Mbit/s
Testing upload speed (12) .................................................................................................................................................................................................................................................................................................................................................................................................
Upload: 799.39 Mbit/s

I've seen a few other reports of performance differences between pfatt.sh, IP-Passthrough and no bypass. In the past, I haven't been convinced the problem is with pfatt.sh due to a variety of discrepancies with reported testing methodologies.

That being said, I've never been able to push my upload past ~820 Mbit/s with pfatt.sh. It's very possible there is a subtle issue here. Unfortunately, there are a lot of moving pieces between AT&T, speed test methodology, pfSense, configurations, and hardware. Troubleshooting requires downtime, and like you, I signed a 99.999% uptime SLA with my family.

I'll keep following this thread. Curious to see how your testing goes.

[sean.allen]

Well...that sucked. It was faaar from "simply rebuild, then restore section by section" - but I did land on a much better result. A sincere thank you @torred @Grimson @Derelict @johnpoz @gsmornot @aus for your help. I spent a bunch of time in the config.xml file comparing my old config to a clean new one. Amazing how much rot develops over five years trying to learn pfSense and eek out better speed from AT&T.

I am now between 600-700Mb down and 850-935Mb up. That is not a typo. My upload screams past my download. I can finally host that p0rn server I've always wanted to. Kidding aside, anything jump out as a reason for that difference? BTW - speed testing is a non-deterministic pile of poo.

Side note: OpenVPN client performance on a gig line with a SG-3100 is thoroughly disappointing. Did a bunch of reading on that and no matter the link speed, seems that people are maxing the 3100 out at 100-150Mb. I thought about trying to set up IPSec instead, but I've had about as much fun as I can take right now.

When I recover, I am probably going to rebuild this from complete scratch. Manually reenter everything - no restore. Kill cruft. Trying to figure out how to do that while also pounding some bourbon. What could go wrong?

Any advice on upload outpacing download or OpenVPN client performance is appreciated.

Y'all rock.

[Derelict]

Great to hear. Without seeing what you changed, no. I don't have any ideas what it could have been.

OpenVPN is just....slow. It spends more time context switching between user and kernel modes that it does doing anything else.

[Rico]

OpenVPN VS IPsec forever and a day Flexibility VS Speed.

-Rico

[sean.allen]

@rico Interesting. You'd sacrifice 80-90% of the links speed to get the flexibility OpenVPN offers? That really says something...like I'm going to hate it if I try IPSec.

[Rico]

I keep my fingers crossed for Multicore Support in OpenVPN 2.5
In the meantime you can run OpenVPN and IPsec peaceful together and do some testing, this should not be any Problem.

-Rico

[gsmornot]

@sean-allen said in SG-3100 Slow Throughput:

@rico Interesting. You'd sacrifice 80-90% of the links speed to get the flexibility OpenVPN offers? That really says something...like I'm going to hate it if I try IPSec.

It may appear to be 80-90% because 100Mb of 1000Mb but in reality IPSEC on the 3100 is only going to do @300. So yea, you’re giving up 66% in speed but only compared 300Mb. In my use, primarily mobile, I like OpenVPN for it “stay connectedness” vs IPSEC which can be less resilient to connection changes. OpenVPN vs IPSEC security I will let others speak on.