Speed Limited to about 28 Mbps?

hackin8 · Dec 27, 2012, 3:34 PM

I am running a simple dualWAN- Single LAN pfs 2.0.1 setup which works perfectly apart from one aspect. (Simple load balancing, no shaping / limiting)

I have two PPPoE connections - one ADSL (6Mbps/512Kbps) and one fttc (BT Infinity 2 - 80Mbps/20Mbps).

Problem is apparent speed limiting when using pfs - maximum download speed obtained (speedtest or BT own tester) is about 28-30Mbps.
If I connect PC (win7-64bit) direct to VDSL2 modem using PPPoE I get 77Mbps. (UL with both is around 18Mbps)

I have tried connecting PC and PPPoE directly to pfs server (No additional switches) and this makes no difference.
I have also tried different pfs server (higher power CPU/more memory) - again with no difference.

Have tried using only one WAN rather than load balancing - again no difference.

It seems no matter what I do I am limited to 28-30Mbps. It doesn't appear to be capped - still some variation within a test / between tests. Only indication is that latency does increase (12ms>50ms) when running speed test.

Network cards are 10/100 all running at 100 full duplex.

Any suggestions on what could be the problem - or how to narrow it down - much appreciated.

TIA

dimkyson · Dec 27, 2012, 5:11 PM

Any packages installed? Squid, HAVP or something else? How do you test the bandwidth?

hackin8 · Dec 27, 2012, 5:38 PM

No packages installed - plain vanilla pfs. Tried routing set through loadbalancing and direct to interface - no difference.

Testing using speedtest.net and speedchecker.bt.com - both show 75-77 with direct PC>PPPoE and 28-30 with PC>pfs>PPPoE

Also tried from pfs using fetch -o /dev/null http://cachefly.cachefly.net/100mb.test (and similar dl from other test sites) These give up to 20Mbps - but believe that is their limitation - as same for direct and >pfs. (If anyone knows a truly unlimited URL let me know!!)

stephenw10 · Dec 27, 2012, 8:42 PM

Since you are in the UK I recommend using the download files at:
http://www.thinkbroadband.com/download/

Using those I can max out my two connections, 20Mb ADSL and 40Mb FTTC.
Using pfSense setup to load balance these I see around 57Mb at speedtest.net, provided it's the right time of day and the client machine I'm using is capable of it. I spent many hours trying to trace a fault that turned out to be the client machine not up to the task (P4, 2.8GHz, WinXP).

Can you get any better if you remove load balancing? Try policy routing your client machine to the FTTC connection.

Other than that I would look for an connection mismatch between the fiber modem and the pfSense WAN NIC.

Are you still running that P2 350? What is the cpu loading like?

Steve

hackin8 · Dec 27, 2012, 9:20 PM

I had tried dl from thinkbroadband - similar results - around 20-22 Mbps for single stream - both from LAN (through pfs) and from within pfs (cli - using fetch).

Have already tried policy routing - no difference (as expected - same as fetch from cli which effectively doesn't use pfs)

Am still using PII-350, loading is <1, typically less than 0.5, still plenty free RAM. As mentioned I also tried using higher spec machine (Dual Core P5 with 4GB ram) but saw same problem - at the exact same speed.

I tend to think it is a miss-match somehow between WAN NIC and VDSL. (LAN NIC doesn't come into it when "fetching". Am using gigabit NICs - will have a play around with some other makes of 10/100 NIC's to see if any different.

Only other possibility relates to MTU:-

BT's implementation of fttc is compatible with MTU up to 1500 as per RFC 4638 - so direct connection to PC will use this. If I understand the various threads - pfs (and BSD itself, or at least the NIC drivers) will only use MTU of 1492 (RFC 2516) - even if changed through GUI I believe it ignores the change and stays at 1492? I am not sure if this would make so much difference? As there seems no way around it I will focus on the NIC compatibility for the moment.

One other question - I thought speedtest.net was single threaded - how is it possible for you to get 57Mbps with load balancing - you can't be bonding ADSL and FTTC so MLPP is not possible? Or do you mean you are running two instances of speedtest - one through each WAN simultaneously?

Thanks for the suggestions

stephenw10 · Dec 27, 2012, 11:58 PM

I thought that too but it appears that Speedtest is capable of summing multiple simultaneous downloads. It is able to use both my connections in a round robin load balanced config.
As you say the fact that using far higher specced machine doesn't seem to help matters seems to point at an interface mismatch. Otherwise I might have called into question the capability of a P2 350 to keep up. Is this a new setup?

Have you tried something simple like a different cable to the modem?

Steve

hackin8 · Dec 28, 2012, 12:55 AM

So far haven't found any difference changing out NICs - but maybe found one issue?

With apologies - I am totally out of my depth with the next question:

On checking BIOS I see all three network cards are sharing IRQ11 - together with VGA card. Two NICs are PCI - third one is "built in" - so prob PCI as well.

Verified through BSD, vmstat -i gives

interrupt total rate
irq0: clk 12101263 998
irq1: atkbd0 6 0
irq6: fdc0 13 0
irq7: ppc0 1 0
irq8: rtc 1548825 127
irq11: fxp0 re0 rl+ 1018163 84
irq14: ata0 27969 2
Total 14696240 1213

Remembering its an old BIOS (no pnp):

Does this matter? Could this slow down throughput between LAN/WAN?
Should I change two PCI NICS to IRQ9 and 10 in BIOS?
If I do - do I need to reset / reconfig BSD / pfs?

Thanks :)

cmb · Dec 28, 2012, 5:45 AM

Having multiple NICs on the same IRQ has caused performance issues for others in the past in some cases. If you can get the BIOS to assign them differently, you should be good. No changes needed other than the BIOS, what it's doing will be automatically picked up.

Dec 28, 2012, 6:05 AM

Have you tried to run the 'iperf' package, as a server and use a PC as a client?

On my atom system, I can see between 250 to 350 Mbps on the LAN.

stephenw10 · Dec 28, 2012, 11:07 AM

You could free up some IRQs by disabling stuff in the bios. I notice you have the floppy disk controller and parallel port controller using IRQs but not doing anything.

Steve

hackin8 · Dec 28, 2012, 4:06 PM

Changed interrupts (didn't need to disable anything - used 9 and 10 that were already free). Didn't make any difference to speed but will have to wait and see whether it stops the (very occasional) hardware failure I asked about in another thread. Interestingly - now the LAN interface is completely clean - where before it was showing a few collisions.

Still no success with speed - despite trying different network cards. So ATM my belief is that it may still relate to inability to set maximum MTU - although it still seems surprising it has so much effect.

Am planning to try iperf - to see whether the pfs setup I have is at least capable of higher speeds. Will take a little while to setup and test - will report back later.

Thanks again

hackin8 · Dec 29, 2012, 4:21 PM

Managed to run iperf, results as follows:

Direct LAN PC1-PC2 (No pfs) 94Mbps - as expected on 10/100 NICS
PC1-pfs-PC2 50 Mbps - very consistent +/- 0.5Mbps (Using PC2 effectively as WAN)

Then using speedtest.net (Java and file DL)

PC1-PPPoE 77Mbps
PC1-pfs-PPPoE 24-28 Mbps

So - pfs is certainly capable of higher speeds - ran iperf continuously for 10 mins at 50Mbps and no sign of overloading / dropped packets etc

Seems to me that problem must relate to PPPoE implementation and/or MTU? pfs and/or BSD?

I am now out of ideas - will have to live with 28 Mbps unless any further suggestions / ideas :)

stephenw10 · Dec 29, 2012, 9:54 PM

Check the WAN NIC is set to auto negotiate the media type and that it is negotiating correctly. I would guess you have 100Mb half duplex connection perhaps.

Steve

hackin8 · Dec 29, 2012, 10:06 PM

Have double checked - It is set to auto - currently running at 100 full duplex.

hackin8 · Jan 2, 2013, 11:51 PM

Final (?) request.

Based upon testing so far it seems the only thing left is the MTU. I gather (from other threads) that the MTU for a PPPoE link is fixed at 1492 - even if you change it in pfs it will stay at 1492. Ideally this should be at 1500 for the BT fttc connection - or at least that is what the direct PC-PPPoE connection uses and gets 77Mbps.

Is there anyway to force the issue through (manually) changing MTU settings in BSD / pfs? This would seem to be the only possible way to increase my speed to closer to the actual limit?

Thanks in advance for any suggestions.

stephenw10 · Jan 3, 2013, 12:50 AM

I don't believe it has anything to do with mtu. I'm using the default pfsense pppoe settings and have no problem getting full speed from my fttc connection.
Are you sure it was no better using faster hardware?

Steve

hackin8 · Jan 3, 2013, 1:15 AM

I am positive that faster hardware made no difference - still seemed to limit at the same speed. Also I have managed to get twice the speed through this setup using the same NIC's etc - but connecting to a PC instead of via PPPoE

I have settled on MTU simply because I can't think of anything else to try!! What VDSL modem do you use for the fttc - maybe I could try changing it? (Although I have tried two different models from BT)

Again - suggestions welcome.

wallabybob · Jan 3, 2013, 1:28 AM

@hackin8:

Thanks in advance for any suggestions.

On Status -> Interfaces do any of the interfaces show non-zero error counts?
Do the error counts change after running a speedtest?

What is the output of pfSense shell command```
ifconfig -i ; netstat -s -p ip ; netstat -s -p tcp


If you have a Linux or Unix system downstream of pfSense on which you can run the speed test please post the output of shell command```
ifconfig -i ; netstat -s -p ip ; netstat -s -p tcp
```executed on that system.

Perhaps your pfSense is running out of puff and occasionally dropping packets forcing a TCP timeout and consequent loss of throughput.

hackin8 · Feb 18, 2013, 5:44 PM

Just double checked. When I connect PC directly to VDSL modem using PPPoE I get high speed transfer (76/19) - but on checking the mtu is 1480 (netsh interface ipv4 show subinterfaces) compared to the "normal" WIN7 LAN connection setting of 1500.

Checking pfS (route get domain) shows mtu of 1492. Even if I change it in interface advance settings - it stays at 1492. Is there anyway to change it to 1480 to see if this makes any difference?

Thanks

stephenw10 · Feb 19, 2013, 1:02 AM

Have you upgraded to 2.0.2? I see there were some MTU fixes that update.

Steve