Issues with High Latency on PPPOE Reconnect

stephenw10

Curious. You could try enabling lacp debugging.
sysctl net.link.lagg.lacp.debug=1

Are those em NICs all the same type?

Steve

Flole

No they are not all the same, em2 is a different one (it's the onboard NIC). The other ones are 82571EB.

Now that I know what causes the issue I will check if I still have one of those computers somewhere. I need a machine for testing all the stuff.

Flole

Could this patch solve the issue? https://svnweb.freebsd.org/base?view=revision&revision=336313

Looks like there are known Problems with that chipset causing some hangs, looks like exactly what I've experienced.

stephenw10

TSO should be disabled by default in pfSense anyway in System > Advanced > Networking. But check the ifconfig output for the parent NICs to be sure.

Steve

Flole

I have it enabled, is it possible that thats what caused all the issues? I assumed that for Intel NICs that works fine. Do I understand it correctly that the patch is basically disabling the TSO, so there's no point in having it enabled on my installation?

stephenw10

It does appear to. I would definitely try that.

Steve

Flole

Unfortunately i can't post the ifconfig output as that constantly triggers the spam filter. On em2 there's nothing about tso, on em0 and em1 theres tso4. Also em2 is still unplugged, not sure if that matters.

stephenw10

It may be correctly disabled on the NIC but passed through when it's added to the lagg group. Does lagg0 show TSO4?

Steve

Flole

I have tried globally disabling TSO, but that did not solve the issue. After a reboot the issue was back, pulling the cable did not help, only removing the interface from the LAGG.

As this seems to be an issue with the specific Intel NIC, maybe someone can get one of these and look into the issue.

stephenw10

Are you seeing entries in the ppp log that look like this?:

Nov 21 16:46:35 pfsense ppp: [opt4_link0] Link: reconnection attempt 24
Nov 21 16:46:35 pfsense ppp: [opt4_link0] PPPoE: can't connect "[26]:"->"mpd98452-0" and "[17e]:"->"left": No such file or directory
Nov 21 16:46:35 pfsense ppp: [opt4_link0] can't remove hook mpd98452-0 from node "[26]:": No such file or directory
Nov 21 16:46:35 pfsense ppp: [opt4_link0] Link: DOWN event
Nov 21 16:46:35 pfsense ppp: [opt4_link0] LCP: Down event
Nov 21 16:46:35 pfsense ppp: [opt4_link0] Link: reconnection attempt 25 in 4 seconds
Nov 21 16:46:35 pfsense ppp: [wan_link0] Link: reconnection attempt 33
Nov 21 16:46:35 pfsense ppp: [wan_link0] PPPoE: can't connect "[1b]:"->"mpd17126-0" and "[4e]:"->"left": No such file or directory
Nov 21 16:46:35 pfsense ppp: [wan_link0] can't remove hook mpd17126-0 from node "[1b]:": No such file or directory
Nov 21 16:46:35 pfsense ppp: [wan_link0] Link: DOWN event
Nov 21 16:46:35 pfsense ppp: [wan_link0] LCP: Down event
Nov 21 16:46:35 pfsense ppp: [wan_link0] Link: reconnection attempt 34 in 2 seconds

And entries in the System log like this:

Nov 21 16:45:17 pfsense kernel: vlan0: changing name to 'lagg0.101'
Nov 21 16:45:21 pfsense kernel: vlan1: changing name to 'lagg0.102'
Nov 21 16:45:21 pfsense kernel: vlan2: changing name to 'lagg0.103'

There are a number of open bugs that seem likely to be related if so. Specifically:
https://redmine.pfsense.org/issues/9148

Steve

Flole

I've had that quite a few times in the logs, but also after taking out that problematic interface. Nothing thats specific to my problem here.

I also had it quite a few times that the ppp interface just didn't want to reconnect, only a reboot helped in that case.

And also I'm having some trouble with states staying during the reconnect and IP Change, but that's all nothing new and not part of the issue here.

stephenw10

I think there may be some underlying issue here that is causing this. We are looking into it.

Steve

stephenw10

You may want to try this patch:
https://github.com/pfsense/pfsense/commit/433a8e71f3b68c39634e11b62d8bf3d9e8ec878c.patch

You can apply that using the system patches package. It will be in 2.4.4p1 when that is released otherwise.

It seems to have corrected all the issues I was seeing with PPPoE but they weren't identical to yours.

Steve

Flole

I highly doubt that it will fix all the issues, as there is also that radvd issue which only comes up with that 82579LM NIC..... I also got another Intel GBit NIC with an 82572GI Chip on today for testing, during next reboot I will apply the 2.4.4p1 and put in that new NIC to test.

Flole

I just applied the 2.4.4p1 and had already 2 issues: For some reason the pppoe interface got stuck somehow, in the logs it was outputting those "deleting xxx: no such files or directory" lines but it just didn't get a connection. Clicking on "Connect" on the interfaces tab did the trick. I assume the pppoe0 interface needed an "ifconfig pppoe0 up" or the equivalent that is being done when I click the connect button.

The other issue I am having is dpinger showing my WAN and PPPOE DHCPv6 Gateways as "Pending", ~~even though they are being used by the clients properly and they do work as expected the dashboard shows them as pending~~. Any idea what's going on there? Correction: They do not work properly, it switched to the IPv6 Tunnel which does work properly.

I haven't been brave enough to put my Intel NIC with the issues into the LACP yet, I would first like to get the dpinger issue sorted out.

stephenw10

If you restart the dpinger service does it show success?

Can you ping the gateway manually from the WAN IP?

My own ISP changed it's PPPoE gateway to a box that doesn't respond to ping a while back.

Steve

Flole

I somehow had something wrong with the Interfaces that caused it to crash, reconnecting WAN and PPPOE fixed it.

I will try with the problematic onboard NIC later, the new NIC which is a

em3@pci0:2:0:0: class=0x020000 card=0x10838086 chip=0x10b98086 rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82572EI Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet

works perfectly fine aswell.