SG-2440 Upload Speed Limited After a Few Minuites

steve1515

I'm having a strange issue with my SG-2440 appliance running 21.05.2-RELEASE.

I recently upgraded my Internet service to one that has 20Mbps upload and have noticed I only get my full upload speed for a few minutes after pfSense boot up or unplugging/re-connecting the WAN cable. After a few minutes my speed tests start only showing about 9-9.5Mbps consistently. If I reboot pfSense, I'm back at 20Mbps, but only for a few minutes before it goes back to ~9Mbps.

If I connect a PC directly to my modem, I get 20-25Mbps upload consistently.

To make things even stranger, my download speed of 200Mbps seems unaffected though. This leads me to believe pfSense on the SG-2440 can easily do 20Mbps if it can handle 200Mbps.

I've swapped out the WAN cable to a new one and also disabled the hardware checksum offload option which was previously enabled. I'm still getting the same problem though.

I'm not sure where to look next to troubleshoot this. Has anyone ever seen this happen or know what to look at or try next?

stephenw10

Indeed the SG-2440 can easily pass 200Mbps, this is not a hardware limitation.

What sort of WAN connection is it? What is the 2440 actually connected to?

It sounds more like a traffic shaping or policing issue. Something is kicking in to deliberately limit your upload. But since that doesn't happen when directly connected maybe it's linking differently, with flow-control perhaps?

Check Status > Interfaces page for errors on the WAN and it's link speed.

Steve

steve1515

The WAN connection is through a Comcast cable connection with a static IP.

I've never enabled any kind of traffic shaping on pfSense, but is there a way to see for sure if it's kicking in?

The WAN status shows no errors that I can see:

WAN Interface (wan, igb0)

Status
    up 
MAC Address
    00:08:a2:xx:xx:xx
IPv4 Address
    x.x.x.x
Subnet mask IPv4
    255.255.255.252 
Gateway IPv4
    x.x.x.x
IPv6 Link Local
    fe80::x:x:x:x%igb0 
MTU
    1500 
Media
    1000baseT <full-duplex> 
In/out packets
    18500032/6658690 (22.89 GiB/1.07 GiB) 
In/out packets (pass)
    18500032/6658690 (22.89 GiB/1.07 GiB) 
In/out packets (block)
    9689/0 (1.76 MiB/0 B) 
In/out errors
    0/0 
Collisions
    0

stephenw10

Ok, so looks to be linked without flow control though igb should be enabled for it if the other side supports it.
Are you able to see how the device connected directly links to the modem?

Check Firewall > Traffic Shaper, 'By Queue' and 'Limiters' tabs. Both should be empty if there is no traffic shaping in play.

Steve

steve1515

The modem interface is pretty limited, unfortunately. It does say the link is 1000Mbps though.

Under Queue and Limiters I have nothing listed. Is it possible that one is active but not listed somehow?

stephenw10

No, if nothing is shown there then there is shaping in pfSense.

The flow control status on the link from the modem to a laptop (or however you tested) is what I'd check.

Other than that you might spoofing the MAC address in case they are shaping you upstream for some reason.

Steve

steve1515

I did some more testing...

Changed WAN MAC address on pfSense with no effect on upload issue.
Connected laptop again to check and it's still getting 20Mbps upload. I don't see any connection issues on the laptop either. It's connected to the modem at 1Gbps full-duplex. I don't see any indication that flow control is enabled.
I disabled pfBlockerNG just to make sure that wasn't the problem. It wasn't... still no effect on the upload issue.

Any other ideas to try out or things to investigate?

stephenw10

Hmm, nothing immediately springs to mind. I suppose Comcast can tell you nothing useful?

You could try applying some shaping so the upload never goes above, say, 18Mbps. See if that then stops it getting clamped.

Steve

steve1515

I don't think I can go to Comcast unless I can prove it isn't my pfSense box since the laptop plugged in works.

I just added a limiter at 5Mbps to verify it worked and then I did one at 15Mbps and it was still limited at 9.5Mbps.

Hopefully, I can think of something else to try. This seems really strange... It's almost as if it's limited at 10Mbps somehow.

bPsdTZpW

@steve1515 I suspect a bad cable, though you've already swapped cables once. The NICs might say 1000Mbps, but actually operate at only 10Mbps.

stephenw10

Yeah, certainly worth trying a different cable. I would also try assigning WAN to a different port.

mer

@steve1515
The cable you used to connect the PC to the cable modem; try using that specific one on the WAN of SG2440.

steve1515

I did some more testing today. Here's what I tried.

I tried connecting the pfSense WAN with the cable that worked with the laptop. I even tried it plugged into a different port in my modem. (The modem has a 4-port switch built in.) No change. I was still getting 9.5Mbps.
I tried plugging in the cable that I was using with pfSense into the laptop... Basically plugged into the same port on the modem and same cable... Only change was I used the laptop instead of pfSense. This worked well and I had the full 20Mbps upload for repeated tests I did for at least 10 minutes. I think this eliminates any cabling problems.
I next tried using another port (OPT2) as the WAN port. (All I did was swap the two in Interfaces --> Assignments and reboot after switching the cable over to OPT2.) This has been giving me consistent 20Mpbs upload for over 20 minutes. This kind of seems like a bad NIC in the pfSense, but only when uploading...

I'll keep monitoring the speed just to see what happens. Part of me thinks it will break again right as I click to submit this post.

Has anyone ever had a bad NIC like this or know what could cause it to happen? Are there any software tools that could confirm a bad NIC?

steve1515

Spoke to soon...

It's back at 9Mbps upload.

I guess it's not the NIC.

bPsdTZpW

I'm scratching my head here. Maybe try a different laptop, or a different NIC (e.g., USB-based) on the same laptop? I have (rarely) seen NICs that don't work well with each other, but work OK with other NICs.

stephenw10

Yes, it's hard to know what would cause that sort of dynamic throughput. The only time I can recall seeing anything like that is with providers that specify an allowed throughput and police it rather than set it. Then they clamp you down to some much lower level if you go over it.

Try putting a switch between the modem and 2440 just to rule out some link negotiation issue if you can.

Steve

steve1515

Some more testing results...

Note: I've put the WAN cable back into the WAN port instead of the OPT2 port.

I've now tried with a different laptop (this one has Realtek NIC vs the other laptop that had an Intel NIC... after about 2hrs, the laptop still shows 20Mbps upload every time. Same results as before.
I've put a switch in between the modem and the pfSense WAN. This seemed ok for about 1hr, but then it went back to 9Mbps upload.
Now that there's a switch in between the devices, I thought I'd try rebooting the pfSense and see what happens. My thoughts were that before when rebooting either device (modem or pfSense), the link gets dropped and either device could be doing some reset in software, etc. This time, with the switch, the link would stay up on the non-rebooted device. So, after a reboot of the pfSense, I'm back up to 20Mbps. (It's only been about 30 minutes of course, so this might not last.)
I've also been doing some iperf tests from the command line of my pfsense box. I'm noticing that when I'm getting full bandwidth, the 'Retr' (Retransmitted TCP packets) column indicates zero 99% of the time. When I'm being limited to 9Mbps, I get many more occasional re-transmits in the iperf test. These are usually like 1-5, so it's not much, but it seems to come up in almost all tests when my upload is limited. When the upload is back up at 20Mbps, I'm constantly able to get all zeros except for the occasional test. I'm not sure if that's an indicator of anything though???

So, I'm very much scratching my head on this one also...
Anyone have any other ideas?

steve1515

Quick addition to the above post...

Item #3... As expected, it eventually went back to 9Mbps. This time I rebooted the modem. An interesting result was found! Because the pfSense never lost the link (because of the switch), it never triggered any kind of reset. As soon as the modem came back up, I was still limited to 9Mbps.

This kind of leads me to believe there is something going on with pfSense where it is limiting upload for some reason somewhere. But where???

mer

@stephenw10 is there any benefit to having @steve1515 post hist config? Anything enabled should be in there, no?
Filter rules with rate limiting?
Some other package rate limiting?

gabacho4

Have you tried just taking a backup of your config and then resetting the pfsense router and run a very vanilla install to see if your upload speed changes?