High latency and packet loss HELP!! [not solved]

[tripplex]

RIG:

IBM thinkcenter MT-M 6087
core 2 duo  E4400 @ 2.00GHz
3GB ram DDR2
80GB HDD
HP NC364T PCI Express Quad Port Nic
and a Onboard intel Nic

Packages Installed:
freeRadius3

pfsense fresh install and reconfigured.

Whats causing this, I just can't figure it out. I am wondering if its the HP NC364T PCI Express Quad Port Nic, the pcie slot being too slow or ISP Buffer bloat?


[Derelict]

What is going on on your network at the time?

[tripplex]

I have load balance setup amongst the 3 wan, I did a speed test and it reported high latency. I just don't know whats wrong. I just finsh reinstalling pfsense and reconfigured captive portal and freeRadius3. also configured loadbalancing.

[Derelict]

With that description nobody else will either.

All of your WANs are private addresses. What is in front of them?

[tripplex]

the ISP Modem which are arris tg862 modems  they have NAT enabled so that's the reason for the private ip addresses. I also tried setting the modems in bridge mode which changed nothing, problem still occurs. Thought it was pfsense causing the issues so a reinstalled it but to my surprise it didn't resolve anything.

[Derelict]

Why do they have NAT enabled?

What does it do when you don't load balance and just use one WAN?

What kind of speeds are you expecting on each WAN?

Does it do that on upload or upload and download?

It is nearly impossible to apply download shaping in that situation because it is impossible for the LAN interface (where download shaping must occur) to know which WAN is passing traffic/saturated and which one is not.

[tripplex]

Why do they have NAT enabled?  Tried it in bridge mode didn't make any difference problem still occurs.

What does it do when you don't load balance and just use one WAN? The problem still occurs.

What kind of speeds are you expecting on each WAN? 20mbps connection on all

Does it do that on upload or upload and download? download only

I can increase the latency threshold in the gateway setting to avoid it have latency issues but i feel it isn't accurate to do that to the wan interfaces.

[Derelict]

I can't think of any way to shape that. Sorry.

All I can say is if it is the same in bridge mode and in NAT mode, I would choose bridge mode. Won't help - just sayin'.

Maybe go to 2.4 and put limiters on each WAN. (You have to go to 2.4 because there is a NAT+Limiter bug that isn't fixed until then).

[johnpoz]

"20mbps connection on all"

So these are the same ISP just 3 different connections via cable?  So your trying to understand when you PEG your pipe via a speed test why dpinger reports high latency to and from google dns since that is what you have for your monitoring IP and whatever that 65.183 address is..

If you Fill your pipe - then yes the pings used by dpinger are going to show latency and loss.. To get around that you would have to shape so you do not saturate your pipe..  And I am curious if this is the same ISP and cable you would be sharing the connection, etc.

[tripplex]

@johnpoz:

To get around that you would have to shape so you do not saturate your pipe.

How can i shape it in 2.3.4_p1?

[Derelict]

With three WANs into one LAN I don't think you can.

Not downloads anyway.

No decent ISP would buffer like that sending to you.

[johnpoz]

I don't use 2.3.4 any more I am on 2.4, according to Derelict there is a bug in the nat+limiter.. So go to 2.4  its going to be Release on monday or something I think ;)

So for example here is my gateway while running speed test..  Just using the fq_codel that is available in 2.4

So you notice the ping times do go up for my cable connection wan_dhcp, and the tunnel from HE response goes up as well - this is expected since its going over the same wan.. But do not see any loss and latency is only slightly increased..

[Derelict]

https://redmine.pfsense.org/issues/4326

That generally means you can use limiters on LAN but not WAN, because there is usually NAT on WAN, and often not on LAN.

[tripplex]

@johnpoz:

I don't use 2.3.4 any more I am on 2.4, according to Derelict there is a bug in the nat+limiter.. So go to 2.4  its going to be Release on monday or something I think ;)

So for example here is my gateway while running speed test..  Just using the fq_codel that is available in 2.4

So you notice the ping times do go up for my cable connection wan_dhcp, and the tunnel from HE response goes up as well - this is expected since its going over the same wan.. But do not see any loss and latency is only slightly increased..

What are the settings for fq_codel and where should i set it? can you send screen shot and details how to accomplish this. I am going to update to 2.4 from 2.3.4_p1 is that ok?

[johnpoz]

see the fq_codel thread… All the info you need is there..

https://forum.pfsense.org/index.php?topic=126637.0

But sure can send you screenshots if you want.  Also just use the cron job to put the info back in on upgrade vs the file edit mode..  See the thread all info needed is in there.

[tripplex]

can i just update to 2.4 from 2.3.4_p1? Will it cause any issues?

getting this while updating to 2.4.0:
pkg: cached package pear-Net_Growl-2.7.0: size mismatch, cannot continue
Failed

[johnpoz]

Normal advice for packages when updating is to remove them, update then put your packages back..

What package is using growl?  I just looked through the package manager and do not see that listed in either the packages I have installed or available packages.

Well I do see it in pfsense..

[2.4.0-RC][root@pfsense.local.lan]/root: pkg info pear-Net_Growl-2.7.0         
pear-Net_Growl-2.7.0
Name          : pear-Net_Growl
Version        : 2.7.0
Installed on  : Fri Mar 31 07:43:42 2017 CDT
Origin        : net/pear-Net_Growl
Architecture  : FreeBSD:11:amd64
Prefix        : /usr/local
Categories    : net pear
Licenses      : BSD2CLAUSE
Maintainer    : garga@FreeBSD.org
WWW            : https://pear.php.net/package/Net_Growl/
Comment        : Send notifications to Growl
Annotations    :
        repo_type      : binary
        repository    : pfSense
Flat size      : 1.03MiB
Description    :
Growl is a MACOSX application that listen to notifications sent by
applications and displays them on the desktop using different display
styles. Net_Growl offers the possibility to send notifications to Growl
from your PHP application through network communication using UDP.

WWW: https://pear.php.net/package/Net_Growl/
[2.4.0-RC][root@pfsense.local.lan]/root:

[Derelict]

Codel won't do any good if the buffering is in the ISP network, which it must be because pfSense won't buffer bloat like that without significant effort.

There is no way to shape on pfSense LAN because he is trying to load balance 3 20Mbit connections into 60Mbit and there is no way for the shaper to know which WAN is saturated when sending out LAN.

He needs to limit downloads on each WAN interface to something that will reduce the buffer bloat in his crappy crappy ISP network.

The only way I can think of to do that on each WAN independently is a limiter on each WAN. That requires 2.4.

It is unknown if 2.4 will introduce something that doesn't work for you. Probably not. 2.4.0-RELEASE should be real soon now.

If you have a 2.3.4_1 installer ready and a current backup you can get back to where you are in about 10 minutes.

[Derelict]

@tripplex:

can i just update to 2.4 from 2.3.4_p1? Will it cause any issues?

getting this while updating to 2.4.0:
pkg: cached package pear-Net_Growl-2.7.0: size mismatch, cannot continue
Failed

What, exactly, did you do there to initiate that upgrade?

[johnpoz]

Yeah I fail to understand such a odd ball setup?  If you want 60mbps - why not just get a pipe that big?  So you have 3 cable providers in your area but nothing fatter than 20mbps?

I just ordered 500/50 to be installed on wed! ;)  Big jump from my 75/10 connection and cheaper too.  Sad part is my pfsense VM can not route that that speed, so until I update my vm host or pull trigger on sg3100.. Which wife would kill me on cost of either option right at the moment… So I had to order a unifi usg :( cheapest option that I could find that could handle the speed.. But looks like openvpn and ipv6 tunnels going to be a PITA vs the easy clickity clickity of pfsense..  And didn't tell her about that cost, just that needed new modem to handle the new speed..  So prob in for some shit when she sees the amazon bill ;) hehehehe

[tripplex]

ok i got it updated 2.4.0 now i am going to set the limiters on the wan.

Is the attached settings correct? because it isn't working to limit the wan speed.


[johnpoz]

no dude no.. Did you not read the thread I pointed too.

[tripplex]

there are so much things there i can't find what i needed gonna read through it all.

[Derelict]

Remove the shaper. You need to use limiters.

[tripplex]

after creating the limiter where shall i put it on the wan tab of the firewall rule and how is it suppose to be configured i.e. the firewall rule with limiter?

[Derelict]

On a floating match rule on WAN out.


[Derelict]

I used 18/5 just to demonstrate this:

[2.4.0-RC][root@pfSense-b]/root: speedtest-cli
Retrieving speedtest.net configuration…
Testing from Cox ()...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by LV.Net (Las Vegas, NV) [13.99 km]: 18.701 ms
Testing download speed…............................................................................
.Download: 17.23 Mbit/s
Testing upload speed...
Upload: 5.15 Mbit/s

[tripplex]

Its working but no more latency issue when i drop the bandwidth to 10 mbps but i still get packet loss and eventually the interface goes offline.

Am i right to feel this is a hardware problem?

[tripplex]

Its happening badly when the speed test tries to upload. the latency is in the thousands.

I have another pc can install pfsense to a hp z200 workstation. Should i do that?

[Derelict]

It's not pfSense. It's your ISP.

Edit the gateways and tell them to always be up while you test. That way the dpinger alarms won't take interfaces down and the interfaces will stay up.


[johnpoz]

also are you trying to test through the load balancer?  Some tests can create multiple sessions to try and fill the pipe vs using 1 session..  I think your going to have a problem with 3 different wans and shaping if your trying to load balance.

And I am curious if there is any shared connectivity here?  So these are 3 different isp with 3 different lines or are you just running 3 different modems from the same cable company?

[Harvy66]

Upload bufferbloat is easy to fix. Set FariQ traffic shaping on all 3 WAN interfaces, set your bandwidth, and preferably set your default queue to have Codel enabled.

[tripplex]

@johnpoz:

also are you trying to test through the load balancer?  Some tests can create multiple sessions to try and fill the pipe vs using 1 session..  I think your going to have a problem with 3 different wans and shaping if your trying to load balance.

And I am curious if there is any shared connectivity here?  So these are 3 different isp with 3 different lines or are you just running 3 different modems from the same cable company?

3 different lines with modems  from the same ISP.

I just increased the latency threshold on the wan interfaces. working good so far , that's what i did in the past before reinstalling pfsense.

[johnpoz]

"3 different lines with modems  from the same ISP."

For what possible reason??  Makes zero sense - from same isp, so if they have issues all 3 of your connections going to have problems.  Your telling me the fastest connection they can give you is 20.. On cable?  So you got 3 of them?

[tripplex]

There 50mbps is the next step from 20 mbps but the price for that bandwidth is so high i couldn't afford it but three 20mbps line is almost the same cost, so i went with that option. Seems now i am paying the price for that.  :-[ :-\

Thanks to everyone who help on this thread. And I installed pfsense on the hp z200 workstation and it had the same problem as the IBM so its definitely the ISP as [b]Derelict said.

[johnpoz]

With cost of 3 modems.. That you are renting or you bought..  Sorry but I don't so how this was ever a viable option for anything..  And could not make sense cost wise.. What going to save 5 bucks or something for the 3 lines vs the 50mbps line?

Live with the 20.. If that is not enough then get the 50.. You can not just magically put a bunch of lines together and think its going to act like 1 fat pipe.. It just doesn't work that way no matter how many people want it too..

Yes load sharing is some cool shit.. Works when you have multiple users.. How many different users do you have trying to use this connection setup?

[chpalmer]

Two of our people using the tg862 both had different issues.  Have you tried any other modems?

http://forums.xfinity.com/t5/Your-Home-Network/Arris-TG862G-CT-Bandwidth-ping-and-packet-loss-all-over-the/td-p/1894021

http://www.dslreports.com/forum/r29241569-Equip-That-tg862g-is-a-joke-of-a-modem

[tripplex]

@johnpoz:

With cost of 3 modems.. That you are renting or you bought..  Sorry but I don't so how this was ever a viable option for anything..  And could not make sense cost wise.. What going to save 5 bucks or something for the 3 lines vs the 50mbps line?

Live with the 20.. If that is not enough then get the 50.. You can not just magically put a bunch of lines together and think its going to act like 1 fat pipe.. It just doesn't work that way no matter how many people want it too..

Yes load sharing is some cool shit.. Works when you have multiple users.. How many different users do you have trying to use this connection setup?

around 100 users

[tripplex]

@chpalmer:

Two of our people using the tg862 both had different issues.  Have you tried any other modems?

http://forums.xfinity.com/t5/Your-Home-Network/Arris-TG862G-CT-Bandwidth-ping-and-packet-loss-all-over-the/td-p/1894021

http://www.dslreports.com/forum/r29241569-Equip-That-tg862g-is-a-joke-of-a-modem

no i didn't.