Packet Loss and Latency/Jitter on PPPoE Interface.

[marrold]

Hi all, first post here so be gentle :)

I have just had BT Infinity (FTTC) installed here in the UK. I've always had Jitter and packet loss on the line, but put it down to ADSL. Now Im using a new modem and FTTC, Im still seeing similar issues, and it looks as though its down to pfSense/PPPoE.

This is using a Technicolor TG582 router. No packet loss and minimal 'smoke'

Using pfSense with ECI B-FOCuS V-2FUb/R Rev.B FTTC modem, connected via PPPoE. Note the blue, its packet loss. There is also a lot of smoke/jitter

My other WAN interface's smokeping graph looks as it should, so it doesn't look as though its an issue with pfSense, rather the PPPoE interface.

The Hardware is a 1.8 Ghz P4 with 256MB RAM, all I can think is the processor is too slow to deal with the PPPoE correctly, hence the jitter and lost packets, but my CPU utilization is usually very low.

Any ideas?

Thanks for your help

[stephenw10]

Hi, I'm using a 1.2 GHz P4 with BT Infinity 40Mbps. I also have an adsl connection on the same box. The FTTC connection is always far more stable even though, in my case, both technologies are using almost identical cable lengths. See attached RRD graphs from pfSense.
You are presumably using a box behind pfSense to get those stats?

Steve


[marrold]

Thanks for your input and sharing your graphs.

The graphs are external to pfSense / my LAN, it's a tool called smoke ping, pinging my public IP's.

Here is my pfSense RRD graph for the infinity connection-

[stephenw10]

Hmm, well your connection is clearly not as stable as mine, see my 8Hour graph below.
Is that typical do you think? The ping spikes do not co-inside with time slots I might expect high usage but perhaps you are running external backups?
I notice that graph is labled WAN2VF, are you running more than one WAN? (Edit: I see you already said you are! What connection type is it?) Are you running anything CPU hungry like VPNs?
What does your CPU usage graph look like over the same period? Mine (see below) shows almost nothing.

Steve


[marrold]

The latency spikes don't seem to correlate with CPU usage, and I don't have any backups or regular usage .

I do indeed have two WAN's, one is Virgin Media 120/12. the Other is BT Infinity 68/18.

Thanks for your input.


[stephenw10]

So are you not using PPPoE to connect to Virgin?

Steve

[marrold]

No, Virgins modem provides a public IP via DHCP on my WAN1 interface. I dont have any of the jitter/packet loss issues with my Virgin Media connection.

For the sake of testing, Ive setup another pfSense box on a slightly more modern CPU (And a different NIC) and Im still seeing a lot of jitter on the connection. The odd thing is, if I remove pfSense from the equation and just use the Technicolor, it's stable with no jitter.

It looks as though its an issue with pfSense, or the combination of pfSense and my connection.

Can you confirm your MTU for me?

Cheers

[senser]

OT: Why don't you guys just copy your rrd-graph images? No need to take screenshots from them. :)

[marrold]

@senser:

OT: Why don't you guys just copy your rrd-graph images? No need to take screenshots from them. :)

I think I did it because it's not possible to copy graphs from smoke ping, thanks for pointing that out though.

[stephenw10]

My PPP settings are all default, I never had to do any tuning, so it's 1492.

$ ifconfig pppoe1
pppoe1: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492
	inet6 fe80::290:7fff:****:****%pppoe1 prefixlen 64 scopeid 0x13 
	inet 81.152.***.*** --> 217.32.***.*** netmask 0xffffffff 
	nd6 options=43<performnud,accept_rtadv></performnud,accept_rtadv></up,pointopoint,running,noarp,simplex,multicast>

There's nothing to suggest that my connection wouldn't look the same as yours if I tested it externally. I am lucky enough to live very close to the exchange so my connection is close to optimal. Though with FTTC that shouldn't be such as issue as long as your green cabinet is relatively close.
I would guess that pfSense does not prioritise ping response in anyway and perhaps the Technicolor router does. That doesn't explain why your Cable connection is so much better though.  :-\

I note that in your original comparison graphs from smoke ping you are not comparing like with like. The pfSense graph is several days where as the Technicolor graph is 40mins. Do similar time spans still show disparity?

@senser:

OT: Why don't you guys just copy your rrd-graph images? No need to take screenshots from them. :)

Good question. Didn't think of it I guess, too used to taking a screen shot to show settings etc.   ::)

Steve

[marrold]

@stephenw10:

I note that in your original comparison graphs from smoke ping you are not comparing like with like. The pfSense graph is several days where as the Technicolor graph is 40mins. Do similar time spans still show disparity?

I only tried the Technicolor TG582 for about an hour, hence why its a smaller section. I appreciate its not a great comparison, but there is definitely something weird going on with PPPoE.

I investigated the possibility that ICMP traffic may be being prioritized differently, but I ran some iperf UDP tests, and there is definitely packet loss going on.

Thanks

[stephenw10]

Hmm, not sure what to say about the packet loss.
I'm running a smoke ping test against my FTTC connection from here: http://www.dslreports.com/smokeping. I'll report back when it starts giving results.

Steve

[marrold]

Any luck Stephen?

[stephenw10]

Ooops, totally forgot about that! Doesn't seem to be showing anything now. Started again. The graph should appear here:
http://www.dslreports.com/r3/smokeping.cgi?target=network.e6c1442a6598dd4e26a0fa816f137954&r=33

Steve

[marrold]

Thanks for your help. Do you mind if I add you IP / Domain to my monitoring? You can PM me if you wish, but I understand if you would rather not.

[stephenw10]

Moving this back to the forum where it could help others.
Your tests results seem to show this isn't an inherent pfSense/FreeBSD problem. Do you have the HG612 vdsl modem?

Steve

[marrold]

For clarity for other users, here are the graphs.

My own, showing lots of packet loss/jitter-

stephenw10's graph, showing no packet loss and a steady connection-

I have the newer ECI  B-FOCuS V-2FUb/R Rev.B VDSL2 FTTC modem, so I am unable to 'hack' it to get the line stats, but I intend on ordering a HG612 next week. I've asked my ISP (which funnily enough I work for) for the line stats, but our upstream provider is taking their time.

So far, it remains a mystery.

[stephenw10]

@marrold:

I've asked my ISP (which funnily enough I work for)

Ha, no excuses then.  ;D
Interesting that your base ping time is lower than mine by quite a bit. Is your smoke ping test box within the ISPs network?

Steve

[marrold]

Unfortunately we are essentially a reseller so I am still relying on a third-part for the info.

The smokeping box is indeed in the ISP's network, only a few hops away from the Broadband End Point (The 'LNS' I believe, I have to admit Im not an expert on the finer details). I have monitored another colleagues FTTC connection using the Technicolor TG582, and again, his connection looks fine.

I will push for the line stats tomorrow. Until then, there isn't much else I can try.

I find it bizarre the problem has been present on both the ADSL and VDSL service, as the wiring, modems, filters, and pfSense box have all been changed with no effect, hence my first hunch was it must of been pfSense's PPPoE implementation, but it doesn't look like that is the case.

[stephenw10]

I assume you aren't seeing any errors or collisions on the WAN NIC? Can you switch the pppoe connection to another NIC? Do you have another box you can load pfSense onto for a test?

Steve

[stephenw10]

Just re-reading this: http://forum.pfsense.org/index.php/topic,64786.msg354161.html#msg354161
Definitely check for a duplex mismatch between the modem and pfSense box.

Steve

[marrold]

@stephenw10:

I assume you aren't seeing any errors or collisions on the WAN NIC? Can you switch the pppoe connection to another NIC? Do you have another box you can load pfSense onto for a test? Steve

There are currently 0 errors/collision on the WAN NIC. I've had the same issues with other NIC's / boxes with a variety of NICs

@stephenw10:

Just re-reading this: http://forum.pfsense.org/index.php/topic,64786.msg354161.html#msg354161
Definitely check for a duplex mismatch between the modem and pfSense box.
Steve

I have forced the speed and duplex on the pfSense box but I am unable to force it on my OR modem unfortunately.

[stephenw10]

You should only force the speed/duplex if the modem-pfSense connection is not negotiating correctly. Was it negotiating to full speed full duplex? It seems this is not the first time it's happened with that modem:
http://forum.mikrotik.com/viewtopic.php?f=3&t=70166

Try putting a switch between the modem and pfSense box. Return everything to auto-negotiation.

Steve

[marrold]

pfSense was auto-negotiating correctly, so I shall return it back to normal. The OR modem is currently hooked up to a switch, and then VLAN'd to the pfSense box, but I still get the same latency/jitter when connecting directly.

Im still waiting on the line stats from the upstream provider, I really need to get a 'hacked' box.

[stephenw10]

It looks like there are some hacking options for that box too but they involve soldering so it's really non-reversible. BT will know if they come round to fix it and charge you accordingly!
Since it's the modem that seems to be suspect in negotiating the connection speed/duplex can you find out if it's doing so correctly with the switch?
If it was just a line problem why does it not happen with the SOHO router you have? I suspect the line stats will show you nothing.  :-\

Steve

[marrold]

I have the newer hardware revision which has limited hacking options unfortunately. The Huawei HG612 seems to the most established hacked box, so I don't mind grabbing one off Ebay for modifications, as you say, BT will charge if I damage their supplied equipment.

Speed and Duplex is being correctly negotiated by the switch.

I also suspect the line stats will be inconclusive, but Its literally my last hope! I could do with taking my pfSense box to another location with FTTC, but they are few and far between in my area. I was the very first customer in my cabinet!

Thanks again for your help, even though it doesnt look as though a resoloution is in sight.

[marrold]

So, I tested some more scenarios, using the same hardware (except the iMac)-

pfSense

This graph shows the jitter and latency Im plagued with when using pfSense.

m0n0wall (BSD based, similar to pfSense)-

Jitter and Latency still present.

Ubuntu 12.04

Looks perfect!

iMac

again, looks perfect.

Ive not got any closer to solving this one, its a mystery at present.

[stephenw10]

So that's Ubuntu and m0n0wall running on the same hardware? Hmm. It seems like it must be a FreeBSD driver issue. You already tried moving the PPPoE connection to a different NIC I recall. They are Intel NICs anyway so it should be fine.  :-\

Steve

[marrold]

My original graphs are using a physical box. More recently I have been using virtual machines on an idle host. I know this adds an extra element of latency/something to go wrong, but I can replicate the same issue on physical hardware, so it seems to work for testing this case, and minimizes disruption to my network.

I've added a recent graph using pfSense within a Virtual Machine above for clarity. Ubuntu and m0n0wall was also virtualized.

I may try raw FreeBSD next, but Ive not used it before.

[stephenw10]

Using VMs you are effectively using the same NIC/driver combination for every case but I guess that includes pfSense. Interesting that m0n0wall shows less latency. It's based on FreeBSD 8.2 last time I checked. pfSense 2.0.x is build on 8.1 and 2.1RC on 8.3 so all different versions. You could try an older pfSense, 1.2.3 was built on FreeBSD 7.3 (I think). It can only support one PPPoE session though so limited. You could try PC-BSD which is easy to setup. Various versions built on various FreeBSD versions are available. I agree though that testing a VM of FreeBSD 8.3 is probably the best test you could do. I've no idea how to setup a PPPoE session directly in FreeBSD though.  ::)

Steve