pfsense 2.4.4 fails all pppoe's after disabling one

Tokk · Nov 19, 2018, 11:43 AM

Ran into the same issue when also updating track interface on LAN, just as @breakaway described. I've been bitten by it once before when setting up a new Ipsec VTI. It did not reconnect and was stuck, had to reboot the router to get things working smoothly again.

v8star · Nov 21, 2018, 11:44 AM

hi everybody, i have the same problem on my pfsense 2.4.4 vm on top of esx. I have 5 pppoe wans on a vlan interface (one left always disabled): yesterday after disabling one of them (for stuck modem) the others three went down and i lose remote access. Neither bringing them up manually (when i came home) won't stay up. Rebooting bring back all in a operative state.
The ppp logs are very similar to OP. I never had in the past this issue but i don't know if is a problem of v. 2.4.4 or the combination between ppp and vlans. With the purpose of adding a new ppp wan connection without reboot the firewall, i migrate to vlans: before i had a virtual interface for every single ppp/wan interface.

stephenw10 · Nov 22, 2018, 12:06 AM

Hmm, I am seeing this now although I did not initially after upgrading. My PPPoE interfaces usually stay up for months.

What exactly did you do to trigger this?

Whilst testing something else I re-saved a WAN connection and it took down both PPPoE connections with the same errors logged:

Nov 21 16:46:14	ppp		[opt4_link0] Link: reconnection attempt 17 in 2 seconds
Nov 21 16:46:14	ppp		[opt4_link0] LCP: Down event
Nov 21 16:46:14	ppp		[opt4_link0] Link: DOWN event
Nov 21 16:46:14	ppp		[opt4_link0] can't remove hook mpd98452-0 from node "[26]:": No such file or directory
Nov 21 16:46:14	ppp		[opt4_link0] PPPoE: can't connect "[26]:"->"mpd98452-0" and "[17e]:"->"left": No such file or directory
Nov 21 16:46:14	ppp		[opt4_link0] Link: reconnection attempt 16
Nov 21 16:46:13	ppp		[wan_link0] Link: reconnection attempt 25 in 1 seconds
Nov 21 16:46:13	ppp		[wan_link0] LCP: Down event
Nov 21 16:46:13	ppp		[wan_link0] Link: DOWN event
Nov 21 16:46:13	ppp		[wan_link0] can't remove hook mpd17126-0 from node "[1b]:": No such file or directory
Nov 21 16:46:13	ppp		[wan_link0] PPPoE: can't connect "[1b]:"->"mpd17126-0" and "[4e]:"->"left": No such file or directory
Nov 21 16:46:13	ppp		[wan_link0] Link: reconnection attempt 24

However I was able to bring them back up by simply clicking connect in Status > Interfaces.
I am also running over VLANs.

Steve

breakaway · Nov 22, 2018, 12:59 AM

@stephenw10 said in pfsense 2.4.4 fails all pppoe's after disabling one:

Hmm, I am seeing this now although I did not initially after upgrading. My PPPoE interfaces usually stay up for months.

What exactly did you do to trigger this?

I just click “save” on my LAN interface, then click apply. Don’t even have to change any settings.

Yes, going to interfaces > connect fixes it, but what if you need to make a change on a remote pfsense.

netblues · Nov 22, 2018, 3:45 AM

I can confirm that this is exactly the case. Hitting reconnect brings it back.
However, on a system with multiple ppp connections, vpn over them etc, a simple change somewhere brings down many things. And obviously, if you are remote, over ppp, things get very very serious.
And bye bye to ppp interfaces staying up for months too :(
It doesn't matter the nic make, if it is physical or virtual, it happens over all.
What hits me is how few people are using ppp over vlans.
Can we expect a ticket? A quick fix?

albert_lightware · Nov 22, 2018, 3:21 PM

Another trouble I found is one ppoe connection type won't even reconnect if I just reboot or press connect button on interfaces menu.
I have to rebuild the connection to get connectivity.
Remove the pppoe config, remove the vlan interface config and create again the vlan for the interface, and the pppoe settings.
In my case is Vodafone Spain, vlan 100 and ppoe session over the vlan100.

I think there are some issues with the vlans and pppoe, they have to investigate that on this release 2.4.4 this wasn't affecting previous versions.

stephenw10 · Nov 22, 2018, 4:37 PM

I opened a bug to track this. https://redmine.pfsense.org/issues/9148

stephenw10 · Nov 22, 2018, 4:50 PM

When you see this do you also see logged:

Nov 21 16:45:21 pfsense kernel: vlan1: changing name to 'lagg0.102'

Or similar?

I actually see the connection come back up successfully in the system log and then fail. I think think this is actually a symptom of something else.

netblues · Nov 23, 2018, 7:04 AM

@stephenw10 indeed.. this is in system log

Nov 22 05:36:26 lb kernel: vlan2: changinn name to 're0.103'
v 22 05:36:32 lb php-fpm[347]: /interfaces.php: MONITOR: VDSL_OTE1_PPP_PPPOE is down, omitting from routing group Default_Gateway_Group_ipv4 80.106.125.100|2.86.52.251|VDS
L_OTE1_PPP_PPPOE|14.683ms|0.578ms|90%|down

and then keeps spitting errors in ppp log like above

breakaway · Nov 23, 2018, 7:07 AM

This post is deleted!

v8star · Nov 26, 2018, 1:04 PM

i've just discovered that adding a new vlan leaving untouched all others vlan used or not by pppoe breaks all pppoe connections. very sad.

netblues · Nov 26, 2018, 10:32 PM

Yep, thats true.
Adding a vlan brings down ALL pppoe connections.
Easy to replicate.

breakaway · Nov 27, 2018, 12:35 AM

@v8star said in pfsense 2.4.4 fails all pppoe's after disabling one:

i've just discovered that adding a new vlan leaving untouched all others vlan used or not by pppoe breaks all pppoe connections. very sad.

I've got a single WAN - PPPoE on VLAN. I Just tried to replicate this.

If by "adding a VLAN" you mean going to Interfaces > Assign > VLAN > fill out the form > then click "Add" - I just did this, and it did NOT take my PPPoE down.

I then went to interfaces > Assignments, selected the VLAN I just added as an interface from the "Available network ports:" dropdown, clicked "Add" then "Save" - that still did NOT take my PPPoE down.

Finally, I went to interfaces > OPT2 (That was the interface name assigned), then ticked the "Enable Interface", Leaving both "IPv4 config type" and "IPv6 config type" as "NONE" - and that did it. I could no longer access internet. The WebGUI was unresponsive. Fortunately, about 30-45 seconds later it came back - PPPoE was up but it had obviously disconnected and reconnected.

Lastly, I went to disable the interface (unticked the box next to "Enable Interface") - this completely killed my internet. PPPoE went down and shows as disconnected. Here's whats in the logs:

Nov 26 12:23:44	ppp		[wan_link0] Link: reconnection attempt 27 in 3 seconds
Nov 26 12:23:44	ppp		[wan_link0] LCP: Down event
Nov 26 12:23:44	ppp		[wan_link0] Link: DOWN event
Nov 26 12:23:44	ppp		[wan_link0] can't remove hook mpd63022-0 from node "[52]:": No such file or directory
Nov 26 12:23:44	ppp		[wan_link0] PPPoE: can't connect "[52]:"->"mpd63022-0" and "[8d]:"->"left": No such file or directory
Nov 26 12:23:44	ppp		[wan_link0] Link: reconnection attempt 26
Nov 26 12:23:42	ppp		[wan_link0] Link: reconnection attempt 26 in 2 seconds
Nov 26 12:23:42	ppp		[wan_link0] LCP: Down event
Nov 26 12:23:42	ppp		[wan_link0] Link: DOWN event
Nov 26 12:23:42	ppp		[wan_link0] can't remove hook mpd63022-0 from node "[52]:": No such file or directory
Nov 26 12:23:42	ppp		[wan_link0] PPPoE: can't connect "[52]:"->"mpd63022-0" and "[8d]:"->"left": No such file or directory
Nov 26 12:23:42	ppp		[wan_link0] Link: reconnection attempt 25
Nov 26 12:23:39	ppp		[wan_link0] Link: reconnection attempt 25 in 3 seconds
Nov 26 12:23:39	ppp		[wan_link0] LCP: Down event
Nov 26 12:23:39	ppp		[wan_link0] Link: DOWN event
Nov 26 12:23:39	ppp		[wan_link0] can't remove hook mpd63022-0 from node "[52]:": No such file or directory
Nov 26 12:23:39	ppp		[wan_link0] PPPoE: can't connect "[52]:"->"mpd63022-0" and "[8d]:"->"left": No such file or directory
Nov 26 12:23:39	ppp		[wan_link0] Link: reconnection attempt 24
Nov 26 12:23:37	ppp		[wan_link0] Link: reconnection attempt 24 in 2 seconds
Nov 26 12:23:37	ppp		[wan_link0] LCP: Down event
Nov 26 12:23:37	ppp		[wan_link0] Link: DOWN event
Nov 26 12:23:37	ppp		[wan_link0] can't remove hook mpd63022-0 from node "[52]:": No such file or directory
Nov 26 12:23:37	ppp		[wan_link0] PPPoE: can't connect "[52]:"->"mpd63022-0" and "[8d]:"->"left": No such file or directory
Nov 26 12:23:37	ppp		[wan_link0] Link: reconnection attempt 23
Nov 26 12:23:35	ppp		[wan_link0] Link: reconnection attempt 23 in 2 seconds
Nov 26 12:23:35	ppp		[wan_link0] LCP: Down event
Nov 26 12:23:35	ppp		[wan_link0] Link: DOWN event
Nov 26 12:23:35	ppp		[wan_link0] can't remove hook mpd63022-0 from node "[52]:": No such file or directory
Nov 26 12:23:35	ppp		[wan_link0] PPPoE: can't connect "[52]:"->"mpd63022-0" and "[8d]:"->"left": No such file or directory
Nov 26 12:23:35	ppp		[wan_link0] Link: reconnection attempt 22
Nov 26 12:23:32	ppp		[wan_link0] Link: reconnection attempt 22 in 2 seconds
Nov 26 12:23:32	ppp		[wan_link0] LCP: Down event
Nov 26 12:23:32	ppp		[wan_link0] Link: DOWN event
Nov 26 12:23:32	ppp		[wan_link0] can't remove hook mpd63022-0 from node "[52]:": No such file or directory
Nov 26 12:23:32	ppp		[wan_link0] PPPoE: can't connect "[52]:"->"mpd63022-0" and "[8d]:"->"left": No such file or directory
Nov 26 12:23:32	ppp		[wan_link0] Link: reconnection attempt 21
Nov 26 12:23:29	ppp		[wan_link0] Link: reconnection attempt 21 in 3 seconds
Nov 26 12:23:29	ppp		[wan_link0] LCP: Down event
Nov 26 12:23:29	ppp		[wan_link0] Link: DOWN event
Nov 26 12:23:29	ppp		[wan_link0] can't remove hook mpd63022-0 from node "[52]:": No such file or directory
Nov 26 12:23:29	ppp		[wan_link0] PPPoE: can't connect "[52]:"->"mpd63022-0" and "[8d]:"->"left": No such file or directory
Nov 26 12:23:29	ppp		[wan_link0] Link: reconnection attempt 20
Nov 26 12:23:27	ppp		[wan_link0] Link: reconnection attempt 20 in 2 seconds
Nov 26 12:23:27	ppp		[wan_link0] LCP: Down event
Nov 26 12:23:27	ppp		[wan_link0] Link: DOWN event
Nov 26 12:23:27	ppp		[wan_link0] can't remove hook mpd63022-0 from node "[52]:": No such file or directory
Nov 26 12:23:27	ppp		[wan_link0] PPPoE: can't connect "[52]:"->"mpd63022-0" and "[8d]:"->"left": No such file or directory
Nov 26 12:23:27	ppp		[wan_link0] Link: reconnection attempt 19
Nov 26 12:23:26	ppp		[wan_link0] Link: reconnection attempt 19 in 1 seconds
Nov 26 12:23:26	ppp		[wan_link0] LCP: Down event

I had to go to Status > Interfaces and manually hit "connect" - now everything is back to normal.

My pfSense spec

Intel(R) Core(TM) i3-4100U CPU @ 1.80GHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
8GB RAM
80GB SATA SSD
4 x igb0 Interfaces (only 2 in use - one is my WAN and one is my LAN)

Installed Packages

NUT
OpenVPN Client Export
pfBlockerNG
net-snmp
iPerf

Everything is the latest version at time of writing.

stephenw10 · Nov 27, 2018, 8:23 PM

Ok, try this patch if you are able. It should work against 2.4.4 and 2.4.5snapshots:
https://github.com/pfsense/pfsense/commit/433a8e71f3b68c39634e11b62d8bf3d9e8ec878c.patch

You can apply that via the system patches package. It resolved the issues I was seeing. It will be in 2.4.4p1.

Steve

netblues · Nov 28, 2018, 6:05 AM

@stephenw10 said in pfsense 2.4.4 fails all pppoe's after disabling one:

https://github.com/pfsense/pfsense/commit/433a8e71f3b68c39634e11b62d8bf3d9e8ec878c.patch

I can confirm, the patch fixes it. (appplied to a 2.4.4 system, updated from 2.4.3).
I will move it into other boxes on the first available maintenance window

Thanks

carobell · Dec 17, 2018, 3:24 PM

Made my own thread as I did not find this on my first search for a solution (https://forum.netgate.com/topic/138802/lan-interface-makes-wan-not-receive-address-from-isp)

2.4.4_1 took care of all my problems.

gek · Jul 9, 2019, 6:19 AM

It's reproducing in 2.4.4-RELEASE-p3 , difference is errors logged in /var/log/system.log instead of /var/log/ppp.log

Jul  9 08:23:49 pfSense ppp: [wan_link0] Link: reconnection attempt 184 in 4 seconds
Jul  9 08:23:54 pfSense ppp: [wan_link0] Link: reconnection attempt 184
Jul  9 08:23:54 pfSense ppp: [wan_link0] PPPoE: can't connect "[12]:"->"mpd9301-0" and "[10]:"->"left": No such file or directory
Jul  9 08:23:54 pfSense ppp: [wan_link0] can't remove hook mpd9301-0 from node "[12]:": No such file or directory
Jul  9 08:23:54 pfSense ppp: [wan_link0] Link: DOWN event
Jul  9 08:23:54 pfSense ppp: [wan_link0] LCP: Down event

netblues · Jul 9, 2019, 7:29 AM

If I recall correctly fix is to be included in the 2.5 release.
So apply the patch above.
It causes no harm on 2.4.4p3 and certainly fixes the issue.

stephenw10 · Jul 9, 2019, 11:17 AM

That patch is already in 2.4.4p3. You can only revert it with the patches packages, and you don't want to do that!

Ar you running 2.4.4p3 clean? Did it just start happening?

I've not seen this myself or any other reports of it since that patch went in.

Steve

gek · Jul 10, 2019, 8:51 AM

@netblues said in pfsense 2.4.4 fails all pppoe's after disabling one:

If I recall correctly fix is to be included in the 2.5 release.
So apply the patch above.
It causes no harm on 2.4.4p3 and certainly fixes the issue.

At first i've upgraded to 2.4.4-RELEASE-p3 . Problem haven't gone. Then i've applied that patch above yesterday (by System_Patches plugin) . Today morning i've lost my pfsense again. In the ppp.log i see now:

Jul 10 08:25:21 pfSense ppp: [wan_link0] Link: reconnection attempt 96 in 1 seconds
Jul 10 08:25:22 pfSense ppp: [wan_link0] Link: reconnection attempt 96
Jul 10 08:25:22 pfSense ppp: [wan_link0] PPPoE: can't connect "[13]:"->"mpd9379-0" and "[11]:"->"left": No such file or directory
Jul 10 08:25:22 pfSense ppp: [wan_link0] can't remove hook mpd9379-0 from node "[13]:": No such file or directory
Jul 10 08:25:22 pfSense ppp: [wan_link0] Link: DOWN event
Jul 10 08:25:22 pfSense ppp: [wan_link0] LCP: Down event
Jul 10 08:25:22 pfSense ppp: [wan_link0] Link: reconnection attempt 97 in 4 seconds
Jul 10 08:25:26 pfSense ppp: [wan_link0] Link: reconnection attempt 97
Jul 10 08:25:26 pfSense ppp: [wan_link0] PPPoE: can't connect "[13]:"->"mpd9379-0" and "[11]:"->"left": No such file or directory
Jul 10 08:25:26 pfSense ppp: [wan_link0] can't remove hook mpd9379-0 from node "[13]:": No such file or directory
Jul 10 08:25:26 pfSense ppp: [wan_link0] Link: DOWN event

https://redmine.pfsense.org/issues/9148