Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Packet Loss and Latency/Jitter on PPPoE Interface.

    General pfSense Questions
    3
    30
    9476
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      marrold last edited by

      Hi all, first post here so be gentle :)

      I have just had BT Infinity (FTTC) installed here in the UK. I've always had Jitter and packet loss on the line, but put it down to ADSL. Now Im using a new modem and FTTC, Im still seeing similar issues, and it looks as though its down to pfSense/PPPoE.

      This is using a Technicolor TG582 router. No packet loss and minimal 'smoke'

      Using pfSense with ECI B-FOCuS V-2FUb/R Rev.B FTTC modem, connected via PPPoE. Note the blue, its packet loss. There is also a lot of smoke/jitter

      My other WAN interface's smokeping graph looks as it should, so it doesn't look as though its an issue with pfSense, rather the PPPoE interface.

      The Hardware is a 1.8 Ghz P4 with 256MB RAM, all I can think is the processor is too slow to deal with the PPPoE correctly, hence the jitter and lost packets, but my CPU utilization is usually very low.

      Any ideas?

      Thanks for your help

      1 Reply Last reply Reply Quote 0
      • stephenw10
        stephenw10 Netgate Administrator last edited by

        Hi, I'm using a 1.2 GHz P4 with BT Infinity 40Mbps. I also have an adsl connection on the same box. The FTTC connection is always far more stable even though, in my case, both technologies are using almost identical cable lengths. See attached RRD graphs from pfSense.
        You are presumably using a box behind pfSense to get those stats?

        Steve

        ![fttc connection.jpg](/public/imported_attachments/1/fttc connection.jpg)
        ![fttc connection.jpg_thumb](/public/imported_attachments/1/fttc connection.jpg_thumb)
        ![adsl connection.jpg](/public/imported_attachments/1/adsl connection.jpg)
        ![adsl connection.jpg_thumb](/public/imported_attachments/1/adsl connection.jpg_thumb)

        1 Reply Last reply Reply Quote 0
        • M
          marrold last edited by

          Thanks for your input and sharing your graphs.

          The graphs are external to pfSense / my LAN, it's a tool called smoke ping, pinging my public IP's.

          Here is my pfSense RRD graph for the infinity connection-

          1 Reply Last reply Reply Quote 0
          • stephenw10
            stephenw10 Netgate Administrator last edited by

            Hmm, well your connection is clearly not as stable as mine, see my 8Hour graph below.
            Is that typical do you think? The ping spikes do not co-inside with time slots I might expect high usage but perhaps you are running external backups?
            I notice that graph is labled WAN2VF, are you running more than one WAN? (Edit: I see you already said you are! What connection type is it?) Are you running anything CPU hungry like VPNs?
            What does your CPU usage graph look like over the same period? Mine (see below) shows almost nothing.

            Steve

            ![fttc connection2.jpg](/public/imported_attachments/1/fttc connection2.jpg)
            ![fttc connection2.jpg_thumb](/public/imported_attachments/1/fttc connection2.jpg_thumb)
            ![cpu usage1.jpg](/public/imported_attachments/1/cpu usage1.jpg)
            ![cpu usage1.jpg_thumb](/public/imported_attachments/1/cpu usage1.jpg_thumb)

            1 Reply Last reply Reply Quote 0
            • M
              marrold last edited by

              The latency spikes don't seem to correlate with CPU usage, and I don't have any backups or regular usage .

              I do indeed have two WAN's, one is Virgin Media 120/12. the Other is BT Infinity 68/18.

              Thanks for your input.

              ![Screen Shot 2013-08-06 at 9.11.36 AM.png](/public/imported_attachments/1/Screen Shot 2013-08-06 at 9.11.36 AM.png)
              ![Screen Shot 2013-08-06 at 9.11.36 AM.png_thumb](/public/imported_attachments/1/Screen Shot 2013-08-06 at 9.11.36 AM.png_thumb)
              ![Screen Shot 2013-08-06 at 9.13.44 AM.png](/public/imported_attachments/1/Screen Shot 2013-08-06 at 9.13.44 AM.png)
              ![Screen Shot 2013-08-06 at 9.13.44 AM.png_thumb](/public/imported_attachments/1/Screen Shot 2013-08-06 at 9.13.44 AM.png_thumb)

              1 Reply Last reply Reply Quote 0
              • stephenw10
                stephenw10 Netgate Administrator last edited by

                So are you not using PPPoE to connect to Virgin?

                Steve

                1 Reply Last reply Reply Quote 0
                • M
                  marrold last edited by

                  No, Virgins modem provides a public IP via DHCP on my WAN1 interface. I dont have any of the jitter/packet loss issues with my Virgin Media connection.

                  For the sake of testing, Ive setup another pfSense box on a slightly more modern CPU (And a different NIC) and Im still seeing a lot of jitter on the connection. The odd thing is, if I remove pfSense from the equation and just use the Technicolor, it's stable with no jitter.

                  It looks as though its an issue with pfSense, or the combination of pfSense and my connection.

                  Can you confirm your MTU for me?

                  Cheers

                  1 Reply Last reply Reply Quote 0
                  • S
                    senser last edited by

                    OT: Why don't you guys just copy your rrd-graph images? No need to take screenshots from them. :)

                    We use the mighty pf, we cannot be fooled.

                    1 Reply Last reply Reply Quote 0
                    • M
                      marrold last edited by

                      @senser:

                      OT: Why don't you guys just copy your rrd-graph images? No need to take screenshots from them. :)

                      I think I did it because it's not possible to copy graphs from smoke ping, thanks for pointing that out though.

                      1 Reply Last reply Reply Quote 0
                      • stephenw10
                        stephenw10 Netgate Administrator last edited by

                        My PPP settings are all default, I never had to do any tuning, so it's 1492.

                        $ ifconfig pppoe1
                        pppoe1: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492
                        	inet6 fe80::290:7fff:****:****%pppoe1 prefixlen 64 scopeid 0x13 
                        	inet 81.152.***.*** --> 217.32.***.*** netmask 0xffffffff 
                        	nd6 options=43<performnud,accept_rtadv></performnud,accept_rtadv></up,pointopoint,running,noarp,simplex,multicast>
                        

                        There's nothing to suggest that my connection wouldn't look the same as yours if I tested it externally. I am lucky enough to live very close to the exchange so my connection is close to optimal. Though with FTTC that shouldn't be such as issue as long as your green cabinet is relatively close.
                        I would guess that pfSense does not prioritise ping response in anyway and perhaps the Technicolor router does. That doesn't explain why your Cable connection is so much better though.  :-\

                        I note that in your original comparison graphs from smoke ping you are not comparing like with like. The pfSense graph is several days where as the Technicolor graph is 40mins. Do similar time spans still show disparity?

                        @senser:

                        OT: Why don't you guys just copy your rrd-graph images? No need to take screenshots from them. :)

                        Good question. Didn't think of it I guess, too used to taking a screen shot to show settings etc.   ::)

                        Steve

                        1 Reply Last reply Reply Quote 0
                        • M
                          marrold last edited by

                          @stephenw10:

                          I note that in your original comparison graphs from smoke ping you are not comparing like with like. The pfSense graph is several days where as the Technicolor graph is 40mins. Do similar time spans still show disparity?

                          I only tried the Technicolor TG582 for about an hour, hence why its a smaller section. I appreciate its not a great comparison, but there is definitely something weird going on with PPPoE.

                          I investigated the possibility that ICMP traffic may be being prioritized differently, but I ran some iperf UDP tests, and there is definitely packet loss going on.

                          Thanks

                          1 Reply Last reply Reply Quote 0
                          • stephenw10
                            stephenw10 Netgate Administrator last edited by

                            Hmm, not sure what to say about the packet loss.
                            I'm running a smoke ping test against my FTTC connection from here: http://www.dslreports.com/smokeping. I'll report back when it starts giving results.

                            Steve

                            1 Reply Last reply Reply Quote 0
                            • M
                              marrold last edited by

                              Any luck Stephen?

                              1 Reply Last reply Reply Quote 0
                              • stephenw10
                                stephenw10 Netgate Administrator last edited by

                                Ooops, totally forgot about that! Doesn't seem to be showing anything now. Started again. The graph should appear here:
                                http://www.dslreports.com/r3/smokeping.cgi?target=network.e6c1442a6598dd4e26a0fa816f137954&r=33

                                Steve

                                1 Reply Last reply Reply Quote 0
                                • M
                                  marrold last edited by

                                  Thanks for your help. Do you mind if I add you IP / Domain to my monitoring? You can PM me if you wish, but I understand if you would rather not.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10
                                    stephenw10 Netgate Administrator last edited by

                                    Moving this back to the forum where it could help others.
                                    Your tests results seem to show this isn't an inherent pfSense/FreeBSD problem. Do you have the HG612 vdsl modem?

                                    Steve

                                    1 Reply Last reply Reply Quote 0
                                    • M
                                      marrold last edited by

                                      For clarity for other users, here are the graphs.

                                      My own, showing lots of packet loss/jitter-

                                      stephenw10's graph, showing no packet loss and a steady connection-

                                      I have the newer ECI  B-FOCuS V-2FUb/R Rev.B VDSL2 FTTC modem, so I am unable to 'hack' it to get the line stats, but I intend on ordering a HG612 next week. I've asked my ISP (which funnily enough I work for) for the line stats, but our upstream provider is taking their time.

                                      So far, it remains a mystery.

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10
                                        stephenw10 Netgate Administrator last edited by

                                        @marrold:

                                        I've asked my ISP (which funnily enough I work for)

                                        Ha, no excuses then.  ;D
                                        Interesting that your base ping time is lower than mine by quite a bit. Is your smoke ping test box within the ISPs network?

                                        Steve

                                        1 Reply Last reply Reply Quote 0
                                        • M
                                          marrold last edited by

                                          Unfortunately we are essentially a reseller so I am still relying on a third-part for the info.

                                          The smokeping box is indeed in the ISP's network, only a few hops away from the Broadband End Point (The 'LNS' I believe, I have to admit Im not an expert on the finer details). I have monitored another colleagues FTTC connection using the Technicolor TG582, and again, his connection looks fine.

                                          I will push for the line stats tomorrow. Until then, there isn't much else I can try.

                                          I find it bizarre the problem has been present on both the ADSL and VDSL service, as the wiring, modems, filters, and pfSense box have all been changed with no effect, hence my first hunch was it must of been pfSense's PPPoE implementation, but it doesn't look like that is the case.

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10
                                            stephenw10 Netgate Administrator last edited by

                                            I assume you aren't seeing any errors or collisions on the WAN NIC? Can you switch the pppoe connection to another NIC? Do you have another box you can load pfSense onto for a test?

                                            Steve

                                            1 Reply Last reply Reply Quote 0
                                            • stephenw10
                                              stephenw10 Netgate Administrator last edited by

                                              Just re-reading this: http://forum.pfsense.org/index.php/topic,64786.msg354161.html#msg354161
                                              Definitely check for a duplex mismatch between the modem and pfSense box.

                                              Steve

                                              1 Reply Last reply Reply Quote 0
                                              • M
                                                marrold last edited by

                                                @stephenw10:

                                                I assume you aren't seeing any errors or collisions on the WAN NIC? Can you switch the pppoe connection to another NIC? Do you have another box you can load pfSense onto for a test? Steve

                                                There are currently 0 errors/collision on the WAN NIC. I've had the same issues with other NIC's / boxes with a variety of NICs

                                                @stephenw10:

                                                Just re-reading this: http://forum.pfsense.org/index.php/topic,64786.msg354161.html#msg354161
                                                Definitely check for a duplex mismatch between the modem and pfSense box.
                                                Steve

                                                I have forced the speed and duplex on the pfSense box but I am unable to force it on my OR modem unfortunately.

                                                1 Reply Last reply Reply Quote 0
                                                • stephenw10
                                                  stephenw10 Netgate Administrator last edited by

                                                  You should only force the speed/duplex if the modem-pfSense connection is not negotiating correctly. Was it negotiating to full speed full duplex? It seems this is not the first time it's happened with that modem:
                                                  http://forum.mikrotik.com/viewtopic.php?f=3&t=70166

                                                  Try putting a switch between the modem and pfSense box. Return everything to auto-negotiation.

                                                  Steve

                                                  1 Reply Last reply Reply Quote 0
                                                  • M
                                                    marrold last edited by

                                                    pfSense was auto-negotiating correctly, so I shall return it back to normal. The OR modem is currently hooked up to a switch, and then VLAN'd to the pfSense box, but I still get the same latency/jitter when connecting directly.

                                                    Im still waiting on the line stats from the upstream provider, I really need to get a 'hacked' box.

                                                    1 Reply Last reply Reply Quote 0
                                                    • stephenw10
                                                      stephenw10 Netgate Administrator last edited by

                                                      It looks like there are some hacking options for that box too but they involve soldering so it's really non-reversible. BT will know if they come round to fix it and charge you accordingly!
                                                      Since it's the modem that seems to be suspect in negotiating the connection speed/duplex can you find out if it's doing so correctly with the switch?
                                                      If it was just a line problem why does it not happen with the SOHO router you have? I suspect the line stats will show you nothing.  :-\

                                                      Steve

                                                      1 Reply Last reply Reply Quote 0
                                                      • M
                                                        marrold last edited by

                                                        I have the newer hardware revision which has limited hacking options unfortunately. The Huawei HG612 seems to the most established hacked box, so I don't mind grabbing one off Ebay for modifications, as you say, BT will charge if I damage their supplied equipment.

                                                        Speed and Duplex is being correctly negotiated by the switch.

                                                        I also suspect the line stats will be inconclusive, but Its literally my last hope! I could do with taking my pfSense box to another location with FTTC, but they are few and far between in my area. I was the very first customer in my cabinet!

                                                        Thanks again for your help, even though it doesnt look as though a resoloution is in sight.

                                                        1 Reply Last reply Reply Quote 0
                                                        • M
                                                          marrold last edited by

                                                          So, I tested some more scenarios, using the same hardware (except the iMac)-

                                                          pfSense

                                                          This graph shows the jitter and latency Im plagued with when using pfSense.

                                                          m0n0wall (BSD based, similar to pfSense)-

                                                          Jitter and Latency still present.

                                                          Ubuntu 12.04

                                                          Looks perfect!

                                                          iMac

                                                          again, looks perfect.

                                                          Ive not got any closer to solving this one, its a mystery at present.

                                                          1 Reply Last reply Reply Quote 0
                                                          • stephenw10
                                                            stephenw10 Netgate Administrator last edited by

                                                            So that's Ubuntu and m0n0wall running on the same hardware? Hmm. It seems like it must be a FreeBSD driver issue. You already tried moving the PPPoE connection to a different NIC I recall. They are Intel NICs anyway so it should be fine.  :-\

                                                            Steve

                                                            1 Reply Last reply Reply Quote 0
                                                            • M
                                                              marrold last edited by

                                                              My original graphs are using a physical box. More recently I have been using virtual machines on an idle host. I know this adds an extra element of latency/something to go wrong, but I can replicate the same issue on physical hardware, so it seems to work for testing this case, and minimizes disruption to my network.

                                                              I've added a recent graph using pfSense within a Virtual Machine above for clarity. Ubuntu and m0n0wall was also virtualized.

                                                              I may try raw FreeBSD next, but Ive not used it before.

                                                              1 Reply Last reply Reply Quote 0
                                                              • stephenw10
                                                                stephenw10 Netgate Administrator last edited by

                                                                Using VMs you are effectively using the same NIC/driver combination for every case but I guess that includes pfSense. Interesting that m0n0wall shows less latency. It's based on FreeBSD 8.2 last time I checked. pfSense 2.0.x is build on 8.1 and 2.1RC on 8.3 so all different versions. You could try an older pfSense, 1.2.3 was built on FreeBSD 7.3 (I think). It can only support one PPPoE session though so limited. You could try PC-BSD which is easy to setup. Various versions built on various FreeBSD versions are available. I agree though that testing a VM of FreeBSD 8.3 is probably the best test you could do. I've no idea how to setup a PPPoE session directly in FreeBSD though.  ::)

                                                                Steve

                                                                1 Reply Last reply Reply Quote 0
                                                                • First post
                                                                  Last post