Uploading files from any machine KILLS latency for everyone on the network

azcire · Feb 29, 2012, 6:27 PM

We've had a mysterious issue for a long time now that I was finally able to replicate. On ANY machine, when I upload files on FileZilla to any FTP server, or upload a big file to senduit.com or similar (basically, any activity that maxes out our upload), the entire network's latency increases to like 500ms or worse when pinging google. This negatively affects our VOIP quality and causes us to scramble to figure out which workstation is raping the network.

This doesn't seem like it should be happening though. How can I protect against this?

Specs:

1.2.3-RELEASE
PC Engines / Netgate ALIX embedded Firewall
3158/10000 states
259 /1920 MBUF Usage
41% Memory usage
11% Disk usage

stephenw10 · Feb 29, 2012, 7:13 PM

Why do you think this wouldn't happen? If you have used all your upstream bandwidth it's bound to cause trouble for other traffic using the connection.
To prevent this happening you need to use some form of traffic shaping:
http://doc.pfsense.org/index.php/Traffic_Shaping_Guide#PFSense_2.0_Traffic_Shaping_Guide_.28Work_In_progress.29

Steve

azcire · Feb 29, 2012, 7:19 PM

Logically, it's unreasonable to expect that anytime anyone is uploading a file on a corporate network that it would hose latency for EVERYONE. There has to be an easy way to solve for this. This does not happen when directly connected to the cable modem, and it does not happen on my home network with a crappy DD-WRT router. Some more clues…

With traffic shaping OFF, and upload bandwidth saturated, there are no issues (average 78ms ping time).

With traffic shaping ON, and upload bandwidth saturated, there ARE issues (average 350ms ping time).

Here are my traffic shaping settings:

http://imgur.com/a/dfpE8

What's wrong?

stephenw10 · Feb 29, 2012, 7:19 PM

Ah OK. It does seem at least unacceptable!
What is your connection speed? The ALIX has limited computing power, is it running 100% CPU trying to do traffic shaping?

Steve

azcire · Feb 29, 2012, 7:29 PM

Advertised connection speed is 50mb down, 5mb up. CPU does not appear to be maxing out (or even spiking) when I saturate upload, according to top via SSH.

Edit: I read it wrong, I was just looking at "user". Here's the pic from top during an upload with traffic shaping on:

http://screencast.com/t/ijcnt1mV

And here it is with traffic shaping off:

http://screencast.com/t/AgKU0zXf

So, idle is 0% with traffic shaping on and idle is 62% with traffic shaping off. "Nice" and "system" do spike. And if idle is at 0% I assume that means CPU is 100% utilized.

stephenw10 · Feb 29, 2012, 7:34 PM

Yes, I'd say you've found the problem.
I'm no expert on traffic shaping so someone else can almost certainly give you better advise. Less queues?
Different type of shaping? :-\

Steve

dhatz · Feb 29, 2012, 7:47 PM

@azcire:

Logically, it's unreasonable to expect that anytime anyone is uploading a file on a corporate network that it would hose latency for EVERYONE. There has to be an easy way to solve for this. This does not happen when directly connected to the cable modem, and it does not happen on my home network with a crappy DD-WRT router. Some more clues…

The behavior you describe is in fact very common, and would be fully expected if one doesn't use traffic shaping. It is however very strange that you say it doesn't happen when you're connected to the cable modem directly… are you sure about that?

azcire · Feb 29, 2012, 7:58 PM

@dhatz:

@azcire:

Logically, it's unreasonable to expect that anytime anyone is uploading a file on a corporate network that it would hose latency for EVERYONE. There has to be an easy way to solve for this. This does not happen when directly connected to the cable modem, and it does not happen on my home network with a crappy DD-WRT router. Some more clues…

The behavior you describe is in fact very common, and would be fully expected if one doesn't use traffic shaping. It is however very strange that you say it doesn't happen when you're connected to the cable modem directly… are you sure about that?

Ya. On top of that I showed screenshots both with and without traffic shaping turned on. It's clear that it's the actual traffic shaping and resulting CPU usage that is causing this latency problem.

Here's what we're dealing with:

cmb · Feb 29, 2012, 9:01 PM

It's normal without shaping to have significant increases in latency when your download or upload especially is pegged, just a fact of networking. Not as much as with shaping because the ICMP falls in lower priority queues which by the nature of shaping get delayed more.

Ping response time has no relation to latency of VoIP traffic or even latency in general when shaping, you're not queuing ICMP in a high priority queue, hence it falls into the default which will (and must) add latency when congestion occurs in order to get higher priority traffic out. Without shaping it's FIFO, so your VoIP is getting delayed behind the ICMP that gets out more quickly.

azcire · Feb 29, 2012, 9:47 PM

@cmb:

It's normal without shaping to have significant increases in latency when your download or upload especially is pegged, just a fact of networking. Not as much as with shaping because the ICMP falls in lower priority queues which by the nature of shaping get delayed more.

Ping response time has no relation to latency of VoIP traffic or even latency in general when shaping, you're not queuing ICMP in a high priority queue, hence it falls into the default which will (and must) add latency when congestion occurs in order to get higher priority traffic out. Without shaping it's FIFO, so your VoIP is getting delayed behind the ICMP that gets out more quickly.

True, bad ping times don't necessarily mean VOIP traffic is suffering. However, in this case, when ping times go to crap on my network, VOIP quality does suffer significantly. People experience a delay on calls and customers start cutting in and out. All evidence seems to point to the CPU being overloaded, and ALL traffic suffering as a result.

Also interesting is when shaping is off, I get 75mbit down on speed tests. With traffic shaping on, the most I've ever seen is 25mbit down.

chpalmer · Mar 1, 2012, 1:02 AM

I was able to control my voip qos issues by using the soproxd package which I actually installed for another reason. The "qos" settings (Ill call them) seem to have made a huge difference for my office. May or may not be an option for you.

But for reference… http://cable-dsl.navasgroup.com/#Asymmetry

Are you using a cable gateway (has own router) device at the office location? Or just a bridge?