SG-3100 Slow Throughput

[torred]

I've had the SG-3100 for about a year now and it has been pretty solid, until the last month or so. I have AT&T 1GB service, but when connecting through the pfsense, I cannot get more than ~40Mbit/s down and ~70Mbit/s up. I've been mucking with settings for the last week and I've finally hit a wall. I'm hoping someone may see something I've over looked. (I don't recall making any changes that would affect this, other than updating to the latest versions. I was using remote logging for a while, but have that disabled now for testing.)

Setup
Internet<-->ATT Residential Gateway<--->(WAN port)SG 3100(LAN port)<-->LAN
Version 2.4.4-RELEASE (arm)
built on Thu Sep 20 09:33:19 EDT 2018
FreeBSD 11.2-RELEASE-p3

Tests
Directly connected to the RG - 650+ down / 400.0+ up
Directly connected to the SG-3100 - 43.2 / 70.0

Interfaces:

"Relevant" Settings:
MBuf clusters
kern.ipc.nmbclusters=65536
Note: I had it set to 1M, just lowered it to 65K to see if anything changed

TSO
net.inet.tcp.tso=0
Note: This keeps turning back on... I have it set to 0 in /boot/loader.conf, but upon boot it's back to 1

I have openvpn (server) running, but nothing is connected.
I don't have anything else running (no squid, no suricata).

Any help what-so-ever would be greatly appreciated.

Edit:
LAN Speed seems to be fine, iperf3 between the pfsense and a computer on the LAN is 705 Mbit/s

[gsmornot]

@torred
I don’t know. I have ATT gig service and SG-3100 and see 910 down, 940 up. I see the same using the provided gateway or bypassing the gateway. I have no idea what would cause you to see such low speeds. Your speeds through the gateway are off a bit too. Good luck.

[johnpoz]

The 2 sg3100 I have in production see their full internet speed as well.. Its not gig.. But way above the 40/70 your seeing... That screams something really wrong - maybe duplex mismatch?

So directly connect you only see about 1/2 of what your paying for - and you don't think that is a problem? If your paying for gig - ou should see high 800's if not low 900's at min.. If not I would be looking to what is wrong there.

You should not need to do any tweaking like your doing.. It sure and the hell is not going to make 40mbps jump to 600..

[torred]

@johnpoz said in SG-3100 Slow Throughput:

So directly connect you only see about 1/2 of what your paying for - and you don't think that is a problem?

I chatted with AT&T about this, it would seem that 600Mbit is "within tolerance" for their 1Gb service, smh.

I agree that these tweaks shouldn't make such a huge difference, so should I be contacting Netgate about a possible hardware issue?

[johnpoz]

what about the crappy 400 up? Is that in their tolerance range as well... They should advertise it as you "MIGHT" get close to gig with our service.. But prob not ;) If you get 400 then we say its good! So Piss off and send us your money!!

How exactly are you testing this? Take your isp device out of the equation... run say iperf..

iperf server --- wan pfsense lan --- iperf client..

What do you see then? This should be damn close to this

ifperf server ---- iperf client

If no pfsense and iperf serv and client are good, and with pfsense and tested same cables its BAD... like what your seeing then yeah I would be contacting support abut bad hardware..

What speeds were you seeing with it before you switched to ATT?

[torred]

So good idea removing the RG from the equation, here's what I came up with...

In summary, TCP looked "good" (~700Mbit - ~900Mbit), regardless of pfsense WAN or LAN ports used (also good if pfsense was not in the path, e.g. just across my switches). Sooo....hardware on the SG-3100 is good...hardware on RG is "good" (suspect)...but as soon as I connect the two they hate me.

I'm going to do some packet captures and see if I can gleam anything from those. If I find something interesting I'll reply back. If I don't reply in 3 days, I've thrown all of my equipment away and moved to the middle of Montana to start my life as a hermit.

Same cables used for all tests (Cat 5E)

--------------------- WAN-to-LAN tests (No ATT RG) -------------------

Laptop(iperf client)->(WAN port)pfsense (iperf server)
TCP: 737 Mbit/s
Reverse TCP: 809 MBit/s

Laptop(iperf client)->(WAN port-NAT rule)pfsense(LAN port 1)->[2x Netgear ProSafe Switches]->Internal server(iperf server)
TCP: 863 Mbit/s
Reverse TCP: 759 MBit/s

--------------------- LAN-to-LAN tests (No ATT RG) -------------------

Laptop(iperf client)->(LAN port 4)pfsense (iperf server)
TCP: 679 Mbit/s
Reverse TCP: 700 MBit/s

Laptop(iperf client)->(LAN port 4)pfsense(LAN port 1)->[2x Netgear ProSafe Switches]->Internal server(iperf server)
TCP: 909 Mbit/s
Reverse TCP: 795 Mbit/s

(Test without pfsense)
Laptop(iperf client)->Netgear Switch->Internal Server (iperf server)
TCP: 899 Mbit/s
Reverse TCP: 949 Mbit/s

--------------------- LAN-to-Internet tests -------------------

(SG-3100 and ATT RG)
Laptop(iperf client)->[2x Netgear ProSafe Switches]->(LAN port 1)pfsense(WAN)->(LAN Port 1)ATT RG->Internet VPS(iperf server)
TCP: 14.4 Mbit/s (this would be me uploading) -- Likely an issue with my VPS, it throttles uploads
Reverse TCP: 43.7 Mbit/s (this would be me downloading)

Same thing, except to http://speedtest.att.com/speedtest/
Upload: 47 MBit/s
Download: 76.6 MBit/s

(Just ATT RG, no SG-3100)
(I used the same cable that was between the SG-3100 and the RG)
Laptop(iperf client)->(LAN Port 1)ATT RG->Internet VPS(iperf server)
TCP: 14.8 Mbit/s (this would be me uploading) -- Likely an issue with my VPS, it throttles uploads
Reverse TCP: 21.6 - 455 Mbit/s (this would be me downloading) (Why the huge difference??? IDK.. sometimes low, sometimes 200s sometimes 400s... over 10 tests)

Same thing, except to http://speedtest.att.com/speedtest/
Upload: 124 - 459 MBit/s
Download: 268 - 828 MBit/s (Again, all over the place)

@johnpoz said in SG-3100 Slow Throughput:

What speeds were you seeing with it before you switched to ATT?

I've had AT&T for 3 years. When I got the SG-3100 back in October 2017 (as soon as it was released), my speed was good (600-700+). I've only noticed the slow down in the last month or two.

[johnpoz]

@torred said in SG-3100 Slow Throughput:

pfsense (iperf server)

That is going to show you low results compared to routing THRU pfsense.. You need 2 boxes.. Do not use pfsense as client or server in your iperf testing.

[torred]

@johnpoz I had only done that as one part of the test, as you can see, I did test through it in other tests.

I honestly do not know what is going on. I did a factory reset on the SG-3100 with the same results. Except now I'm experiencing a multitude of other failures.

I've removed my pfsense, and am just using the ATT RG and everything works perfectly.

Thanks for your help @johnpoz, I'll be contacting Netgate Support and see if they can help me out.

[torred]

Alright, if anyone happens to read all the way down here, I never figured out what the problem between the two was, but I ended up bypassing the AT&T RG by doing this:
https://github.com/aus/pfatt

It was fairly easy to compile the needed ng_etf.ko kernel module for armv6:

1. Get a FreeBSD 11.2 amd64 VM going (for pfSense 2.4.4) -- make sure to include src when installing
2. Get a shell, and do this:
   $ cd /usr/src
   $ make kernel-toolchain TARGET_ARCH=armv6
   # Wait about an hour
   $ make buildenv TARGET_ARCH=armv6 BUILDENV_SHELL=/bin/sh
   $ cd /usr/src/sys/modules/netgraph/etf
   $ make
3. You now have your own compile netgraph etf module, follow the rest of the guide.
4. I used the OPT1 (mvneta0) port for the RG, and the WAN (mvneta2) port for the ONT

Everything works fine now.