Packet Loss and Latency/Jitter on PPPoE Interface.

[marrold]

Hi all, first post here so be gentle :)

I have just had BT Infinity (FTTC) installed here in the UK. I've always had Jitter and packet loss on the line, but put it down to ADSL. Now Im using a new modem and FTTC, Im still seeing similar issues, and it looks as though its down to pfSense/PPPoE.

This is using a Technicolor TG582 router. No packet loss and minimal 'smoke'

Using pfSense with ECI B-FOCuS V-2FUb/R Rev.B FTTC modem, connected via PPPoE. Note the blue, its packet loss. There is also a lot of smoke/jitter

My other WAN interface's smokeping graph looks as it should, so it doesn't look as though its an issue with pfSense, rather the PPPoE interface.

The Hardware is a 1.8 Ghz P4 with 256MB RAM, all I can think is the processor is too slow to deal with the PPPoE correctly, hence the jitter and lost packets, but my CPU utilization is usually very low.

Any ideas?

Thanks for your help

[stephenw10]

Hi, I'm using a 1.2 GHz P4 with BT Infinity 40Mbps. I also have an adsl connection on the same box. The FTTC connection is always far more stable even though, in my case, both technologies are using almost identical cable lengths. See attached RRD graphs from pfSense.
You are presumably using a box behind pfSense to get those stats?

Steve


[marrold]

Thanks for your input and sharing your graphs.

The graphs are external to pfSense / my LAN, it's a tool called smoke ping, pinging my public IP's.

Here is my pfSense RRD graph for the infinity connection-

[stephenw10]

Hmm, well your connection is clearly not as stable as mine, see my 8Hour graph below.
Is that typical do you think? The ping spikes do not co-inside with time slots I might expect high usage but perhaps you are running external backups?
I notice that graph is labled WAN2VF, are you running more than one WAN? (Edit: I see you already said you are! What connection type is it?) Are you running anything CPU hungry like VPNs?
What does your CPU usage graph look like over the same period? Mine (see below) shows almost nothing.

Steve


[marrold]

The latency spikes don't seem to correlate with CPU usage, and I don't have any backups or regular usage .

I do indeed have two WAN's, one is Virgin Media 120/12. the Other is BT Infinity 68/18.

Thanks for your input.


[stephenw10]

So are you not using PPPoE to connect to Virgin?

Steve

[marrold]

No, Virgins modem provides a public IP via DHCP on my WAN1 interface. I dont have any of the jitter/packet loss issues with my Virgin Media connection.

For the sake of testing, Ive setup another pfSense box on a slightly more modern CPU (And a different NIC) and Im still seeing a lot of jitter on the connection. The odd thing is, if I remove pfSense from the equation and just use the Technicolor, it's stable with no jitter.

It looks as though its an issue with pfSense, or the combination of pfSense and my connection.

Can you confirm your MTU for me?

Cheers

[senser]

OT: Why don't you guys just copy your rrd-graph images? No need to take screenshots from them. :)

[marrold]

@senser:

OT: Why don't you guys just copy your rrd-graph images? No need to take screenshots from them. :)

I think I did it because it's not possible to copy graphs from smoke ping, thanks for pointing that out though.

[stephenw10]

My PPP settings are all default, I never had to do any tuning, so it's 1492.

$ ifconfig pppoe1
pppoe1: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492
	inet6 fe80::290:7fff:****:****%pppoe1 prefixlen 64 scopeid 0x13 
	inet 81.152.***.*** --> 217.32.***.*** netmask 0xffffffff 
	nd6 options=43<performnud,accept_rtadv></performnud,accept_rtadv></up,pointopoint,running,noarp,simplex,multicast>

There's nothing to suggest that my connection wouldn't look the same as yours if I tested it externally. I am lucky enough to live very close to the exchange so my connection is close to optimal. Though with FTTC that shouldn't be such as issue as long as your green cabinet is relatively close.
I would guess that pfSense does not prioritise ping response in anyway and perhaps the Technicolor router does. That doesn't explain why your Cable connection is so much better though.  :-\

I note that in your original comparison graphs from smoke ping you are not comparing like with like. The pfSense graph is several days where as the Technicolor graph is 40mins. Do similar time spans still show disparity?

@senser:

OT: Why don't you guys just copy your rrd-graph images? No need to take screenshots from them. :)

Good question. Didn't think of it I guess, too used to taking a screen shot to show settings etc.   ::)

Steve

[marrold]

@stephenw10:

I note that in your original comparison graphs from smoke ping you are not comparing like with like. The pfSense graph is several days where as the Technicolor graph is 40mins. Do similar time spans still show disparity?

I only tried the Technicolor TG582 for about an hour, hence why its a smaller section. I appreciate its not a great comparison, but there is definitely something weird going on with PPPoE.

I investigated the possibility that ICMP traffic may be being prioritized differently, but I ran some iperf UDP tests, and there is definitely packet loss going on.

Thanks

[stephenw10]

Hmm, not sure what to say about the packet loss.
I'm running a smoke ping test against my FTTC connection from here: http://www.dslreports.com/smokeping. I'll report back when it starts giving results.

Steve

[marrold]

Any luck Stephen?

[stephenw10]

Ooops, totally forgot about that! Doesn't seem to be showing anything now. Started again. The graph should appear here:
http://www.dslreports.com/r3/smokeping.cgi?target=network.e6c1442a6598dd4e26a0fa816f137954&r=33

Steve

[marrold]

Thanks for your help. Do you mind if I add you IP / Domain to my monitoring? You can PM me if you wish, but I understand if you would rather not.

[stephenw10]

Moving this back to the forum where it could help others.
Your tests results seem to show this isn't an inherent pfSense/FreeBSD problem. Do you have the HG612 vdsl modem?

Steve

[marrold]

For clarity for other users, here are the graphs.

My own, showing lots of packet loss/jitter-

stephenw10's graph, showing no packet loss and a steady connection-

I have the newer ECI  B-FOCuS V-2FUb/R Rev.B VDSL2 FTTC modem, so I am unable to 'hack' it to get the line stats, but I intend on ordering a HG612 next week. I've asked my ISP (which funnily enough I work for) for the line stats, but our upstream provider is taking their time.

So far, it remains a mystery.

[stephenw10]

@marrold:

I've asked my ISP (which funnily enough I work for)

Ha, no excuses then.  ;D
Interesting that your base ping time is lower than mine by quite a bit. Is your smoke ping test box within the ISPs network?

Steve

[marrold]

Unfortunately we are essentially a reseller so I am still relying on a third-part for the info.

The smokeping box is indeed in the ISP's network, only a few hops away from the Broadband End Point (The 'LNS' I believe, I have to admit Im not an expert on the finer details). I have monitored another colleagues FTTC connection using the Technicolor TG582, and again, his connection looks fine.

I will push for the line stats tomorrow. Until then, there isn't much else I can try.

I find it bizarre the problem has been present on both the ADSL and VDSL service, as the wiring, modems, filters, and pfSense box have all been changed with no effect, hence my first hunch was it must of been pfSense's PPPoE implementation, but it doesn't look like that is the case.

[stephenw10]

I assume you aren't seeing any errors or collisions on the WAN NIC? Can you switch the pppoe connection to another NIC? Do you have another box you can load pfSense onto for a test?

Steve