PPPoE WAN fails to reconnect after link loss

vegbrasil

Hi,

I'm using pfSense 2.4.3 with em Intel network interfaces and my WAN is PPPoE based. When the connection drops for any reason, pfSense does not reconnect until a full system reboot is done.

The same WAN in a domestic router (e.g. TP-Link) reconnects just fine. I found a similar issue on Redmine but it is from six years ago.

Observing the logs, I could not understand why this is happening.

Here's two logs:
Log #1: PPPoE connects OK in the system startup
Log #2: PPPoE fails to reconnect indefinitely
(I've changed the IPs to $_WAN_PUBLIC_IP_$ and $_ISP_GATEWAY_$ on both logs)

Could anyone help me on this issue?

Thanks!

stephenw10

I assume you have redacted the IPs from those logs?
That does mean we can't assess whether or not they are reasonable.

This line appears to be key [wan] IFACE: Adding IPv4 address to pppoe0 failed

Does the IP it's trying to add conflict? The previous connection still exist somehow maybe?

Do you see any other errors in the system log at that point?

Steve

vegbrasil

@stephenw10

Hi Steve, thanks for your time.

Yes, I've redacted the IPs replacing them with $_WAN_PUBLIC_IP_$ where my public IP was logged and $_ISP_GATEWAY_$ for my ISP's gateway. Please note that those two IPs are the only ones logged and they stay the same on both logs (#1 and #2).

About the conflict, I think that is possible. Since the ISP is giving the same public IP to this PPPoE every time. Here's something I found in the system log:

Jun 28 15:28:17     ppp         [wan] IPCP: rec'd Configure Request #2 (Ack-Sent)
Jun 28 15:28:17     ppp         [wan] IPADDR $_ISP_GATEWAY_$
Jun 28 15:28:17     ppp         [wan] $_ISP_GATEWAY_$ is OK
Jun 28 15:28:17     ppp         [wan] IPCP: SendConfigAck #2
Jun 28 15:28:17     ppp         [wan] IPADDR $_ISP_GATEWAY_$
Jun 28 15:28:17     ppp         [wan] IPCP: rec'd Configure Ack #7 (Ack-Sent)
Jun 28 15:28:17     ppp         [wan] IPADDR $_WAN_PUBLIC_IP_$
Jun 28 15:28:17     ppp         [wan] PRIDNS 8.8.8.8
Jun 28 15:28:17     ppp         [wan] SECDNS 8.8.4.4
Jun 28 15:28:17     ppp         [wan] IPCP: state change Ack-Sent --> Opened
Jun 28 15:28:17     ppp         [wan] IPCP: LayerUp
Jun 28 15:28:17     ppp         [wan] $_WAN_PUBLIC_IP_$ -> $_ISP_GATEWAY_$
Jun 28 15:28:17     ppp         [wan] IFACE: Adding IPv4 address to pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): File exists
Jun 28 15:28:17     check_reload_status         Rewriting resolv.conf
Jun 28 15:28:18     php-fpm     61165   /rc.newwanipv6: rc.newwanipv6: Info: starting on pppoe0.
Jun 28 15:28:18     php-fpm     61165   /rc.newwanipv6: rc.newwanipv6: No IPv6 address found for interface WAN [wan].
Jun 28 15:28:18     check_reload_status         rc.newwanip starting pppoe0
Jun 28 15:28:19     php-fpm     3065    /rc.newwanip: rc.newwanip: Info: starting on pppoe0.
Jun 28 15:28:19     php-fpm     3065    /rc.newwanip: rc.newwanip: on (IP address: ) (interface: WAN[wan]) (real interface: pppoe0).
Jun 28 15:28:19     php-fpm     3065    /rc.newwanip: rc.newwanip: Failed to update wan IP, restarting...
Jun 28 15:28:19     check_reload_status         Configuring interface wan
Jun 28 15:28:20     ppp         Multi-link PPP daemon for FreeBSD
Jun 28 15:28:20     ppp         process 95029 started, version 5.8 (nobody@pfSense_v2_4_3_amd64-pfSense_v2_4_3-job-08 16:18 16-Mar-2018)
Jun 28 15:28:20     ppp         waiting for process 13192 to die...
Jun 28 15:28:20     ppp         caught fatal signal TERM
Jun 28 15:28:20     ppp         [wan] IFACE: Close event
Jun 28 15:28:20     ppp         [wan] IPCP: Close event
Jun 28 15:28:20     ppp         [wan] IPCP: state change Opened --> Closing
Jun 28 15:28:20     ppp         [wan] IPCP: SendTerminateReq #8
Jun 28 15:28:20     ppp         [wan] IPCP: LayerDown
Jun 28 15:28:20     check_reload_status         Rewriting resolv.conf
Jun 28 15:28:20     ppp         [wan] IFACE: Removing IPv4 address from pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address

Does this help at all? This keeps repeating until a full system reboot.

Thanks a lot!

stephenw10

That just looks like the ppp output that's logged in both places.

Hmm, this appears to be something specific to your ISP. I have PPPoE WANs and don't see this. Perhaps some timing issue.

Is that intertelecom.ua ? Someone else might have hit it.

Steve

vegbrasil

@stephenw10 It's not intertelecom.ua. We're using a local FTTH ISP.

The weird thing is that the issue only happens with pfSense. Routers with DD-WRT or OpenWRT can reconnect just fine.

Is it possible to set a delay between reconnection attempts?

stephenw10

Are you spoofing the WAN MAC? What NIC driver is the WAN using?

Steve

vegbrasil

@stephenw10

Are you spoofing the WAN MAC?

Yes! Could it be the cause?

What NIC driver is the WAN using?

It's a Intel PRO/1000, using the em(4) drivers.

Thanks!

stephenw10

I did see a similar report that spoofing the MAC broke PPPoE in some cases.
I wouldn't expect that to be required for PPPoE, can you test it without spoofing?

Also where are you spoofing it? On the PPPoE interface or on the parent interface?
I could imagine an issue passing that through to the parent perhaps.

Steve

vegbrasil

@stephenw10

I wouldn't expect that to be required for PPPoE, can you test it without spoofing?

I can but it's not easy... Probably next weekend - needs to be in office off-hours and my ISP needs to authorize the correct MAC addr. I'll do and return with feedback.

Also where are you spoofing it? On the PPPoE interface or on the parent interface?

I'm spoofing directly on the physical interface (em0). I don't see the option to spoof on the pppoe0 interface.

Thanks a lot!

stephenw10

Great. If that does work we can open a bug report to look into it.

Interesting though, I see the MAC spoof option here on PPPoE interfaces in 2.4.3p1:

Steve

vegbrasil

@stephenw10 Sorry for the delay. Unfortunately, the problem persists.

Here's what I've done since then:

Updated to 2.4.3p1;
Fixed the authorized MAC address with my ISP. There's no spoofing anymore;
Recreated the PPPoE interface from scratch;
Tried another network port in the same machine;
Used PPPoE with the same ISP but different layer 1 topology. Reconnection works fine when using their wireless uplink. The problem only happens on the FTTH connection;

Is there anything else I can do?

Thanks!

stephenw10

Hmm, the logs still show the same thing? Can we see a current PPP log showing the failure?

Steve

vegbrasil

@stephenw10 Here's the updated logs. For safety, I've replaced the WAN IP with $_WAN_PUBLIC_IP_$ and my ISP's gateway with $_ISP_GATEWAY_$ .

Log #1 - Connection works after system restart

Log #2 - Fails to reconnect if the connection dropped for any reason

This line keeps appearing:

IFACE: Adding IPv4 address to pppoe2 failed(IGNORING for now. This should be only for PPPoE friendly!): File exists

Thanks!

Joxxxx

Hello, did you find out what was the problem ? Because I have the exact same problem with the exact same errors after upgrading to 2.4.4 .

[opt1] IFACE: Adding IPv4 address to pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): File exists

[opt1] IFACE: Removing IPv4 address from pppoe0 failed(IGNORING for now. This should be only for PPPoE friendly!): Can't assign requested address

Tried everything, nothing helps. The connection works on VyOS and directly on the PC.

vegbrasil

@joxxxx I have yet to find a solution. For me, the (re)connection works fine on other OSes and devices as well.

Let me know you find out anything. Unfortunately I'm considering migrating to something else.

Joxxxx

@vegbrasil After countless hours, I found out that the pppoe layer of pfsense 2.4.4 is bugged ! Something changed and because of that, the connection cannot be made with the above errors. Tried it also on a clean 2.4.4 install.

Re-installed 2.4.3 and all works again without any problems ! The thing is that I cannot install any packages anymore because it wants me to update to 2.4.4...

Btw, tried the dev 2.4.5 version and the bug is still the same.

I hope this will be fixed because it mus affect a lot of people...

stephenw10

Hmm, there must be something different about your connection. I have two WANs here, both are PPPoE. The only issue I saw was when the ISP(s) had an outage. However the logs showed it as very broken, nothing like what you're seeing.
If PPPoE was broken in general there would be hundreds of posts about it by now.

Are you able to try a different NIC type?

Steve

vegbrasil

@stephenw10 I've been using pfSense with the same FTTH PPPoE connection and two different NICs models:

Realtek NICs (re): crazy bugs, completely unusable - expected, I guess;
Intel PRO/1000 NICs (em): Problem of the initial post: first connection is fine, re-connection never happens. If we forget this problem, everything is working great and I can max-out the connection (150 Mbps).

Upgraded and tried clean install between 2.3.5 and all 2.4.x versions.

Thanks!

stephenw10

Hmm, yes, this seems like it must be some other issue since you were seeing it in 2.4.3 also.

Steve

Joxxxx

I'm using a quad ethernet intel pro and never got any problems since v 2.0 or something like that on vmware and the vmx3 driver.

After analysing a bit the problem, could it be that I have this problem because pfsense sees his local ip address as the FTTH gateway ?
Maybe something changed in pf 2.4.4 in the pppoe connection that forbids the use of it ? (This could explain the two errors I get).