PPPoE WAN not connecting since upgrade from pfsense 2.6.0 to 2.7.0-RELEASE

[mpcjames]

Hello

We have an issue with one VDSL connection which will no longer connect since the upgrade from pfsense 2.6 to 2.7 (which was completed last night).
This connection has previously worked fine in pfsense for the past 2 years with the same hardware etc.

The connection is a standard VDSL/FTTC connection (80mb/20mb) as provided to many premises in the UK via Openreach (BT). The connection has a standard Openreach VDSL modem which presents the connection as RJ45 ethernet (from the port on the wall originally coming in as RJ11/Phoneline etc). All that is required to connect to this type of a connection is a PPPoE Username/Password. The IP, Gateway etc are then assigned automatically by the provider.

Since the upgrade to 2.7, the connection shows an uptime of 2secs in the pfsense interfaces, and the drops again. The PPP logs show the following:

Sep 6 11:55:27	ppp	13544	[wan_link0] LCP: rec'd Terminate Ack #4 (Closing)
Sep 6 11:55:27	ppp	13544	[wan_link0] LCP: state change Closing --> Closed
Sep 6 11:55:27	ppp	13544	[wan_link0] LCP: LayerFinish
Sep 6 11:55:27	ppp	13544	[wan_link0] Link: DOWN event
Sep 6 11:55:27	ppp	13544	[wan_link0] Link: giving up after 0 reconnection attempts
Sep 6 11:55:27	ppp	13544	[wan_link0] LCP: Close event
Sep 6 11:55:27	ppp	13544	[wan_link0] LCP: Down event
Sep 6 11:55:27	ppp	13544	[wan_link0] LCP: state change Closed --> Initial
Sep 6 11:55:28	ppp	42771	waiting for process 13544 to die...
Sep 6 11:55:29	ppp	42771	waiting for process 13544 to die...
Sep 6 11:55:29	ppp	13544	[wan] Bundle: Shutdown
Sep 6 11:55:29	ppp	13544	[wan_link0] Link: Shutdown
Sep 6 11:55:29	ppp	13544	process 13544 terminated
Sep 6 11:55:30	ppp	42771	web: web is not running
Sep 6 11:55:30	ppp	42771	[wan] Bundle: Interface ng1 created
Sep 6 11:55:30	ppp	42771	[wan_link0] Link: OPEN event
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: Open event
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: state change Initial --> Starting
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: LayerStart
Sep 6 11:55:30	ppp	42771	[wan_link0] PPPoE: Connecting to 'none'
Sep 6 11:55:30	ppp	42771	PPPoE: rec'd ACNAME "acc-aln9.yv"
Sep 6 11:55:30	ppp	42771	[wan_link0] PPPoE: connection successful
Sep 6 11:55:30	ppp	42771	[wan_link0] Link: UP event
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: Up event
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: state change Starting --> Req-Sent
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: SendConfigReq #1
Sep 6 11:55:30	ppp	42771	[wan_link0] PROTOCOMP
Sep 6 11:55:30	ppp	42771	[wan_link0] MRU 1492
Sep 6 11:55:30	ppp	42771	[wan_link0] MAGICNUM 0xb75ba742
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: rec'd Configure Request #133 (Req-Sent)
Sep 6 11:55:30	ppp	42771	[wan_link0] MRU 1492
Sep 6 11:55:30	ppp	42771	[wan_link0] AUTHPROTO CHAP MD5
Sep 6 11:55:30	ppp	42771	[wan_link0] MAGICNUM 0x054ef983
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: SendConfigAck #133
Sep 6 11:55:30	ppp	42771	[wan_link0] MRU 1492
Sep 6 11:55:30	ppp	42771	[wan_link0] AUTHPROTO CHAP MD5
Sep 6 11:55:30	ppp	42771	[wan_link0] MAGICNUM 0x054ef983
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: state change Req-Sent --> Ack-Sent
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: rec'd Configure Reject #1 (Ack-Sent)
Sep 6 11:55:30	ppp	42771	[wan_link0] PROTOCOMP
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: SendConfigReq #2
Sep 6 11:55:30	ppp	42771	[wan_link0] MRU 1492
Sep 6 11:55:30	ppp	42771	[wan_link0] MAGICNUM 0xb75ba742
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: rec'd Configure Ack #2 (Ack-Sent)
Sep 6 11:55:30	ppp	42771	[wan_link0] MRU 1492
Sep 6 11:55:30	ppp	42771	[wan_link0] MAGICNUM 0xb75ba742
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: state change Ack-Sent --> Opened
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: auth: peer wants CHAP, I want nothing
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: LayerUp
Sep 6 11:55:30	ppp	42771	[wan_link0] CHAP: rec'd CHALLENGE #1 len: 79
Sep 6 11:55:30	ppp	42771	[wan_link0] Name: "acc-aln9.yv"
Sep 6 11:55:30	ppp	42771	[wan_link0] CHAP: Using authname "na2111288@broadbandcomplete"
Sep 6 11:55:30	ppp	42771	[wan_link0] CHAP: sending RESPONSE #1 len: 48
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: rec'd Configure Request #1 (Opened)
Sep 6 11:55:30	ppp	42771	[wan_link0] AUTHPROTO CHAP MD5
Sep 6 11:55:30	ppp	42771	[wan_link0] MAGICNUM 0xf32272f3
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: LayerDown
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: SendConfigReq #3
Sep 6 11:55:30	ppp	42771	[wan_link0] MRU 1492
Sep 6 11:55:30	ppp	42771	[wan_link0] MAGICNUM 0xb75ba742
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: SendConfigAck #1
Sep 6 11:55:30	ppp	42771	[wan_link0] AUTHPROTO CHAP MD5
Sep 6 11:55:30	ppp	42771	[wan_link0] MAGICNUM 0xf32272f3
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: state change Opened --> Ack-Sent
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: rec'd Configure Ack #3 (Ack-Sent)
Sep 6 11:55:30	ppp	42771	[wan_link0] MRU 1492
Sep 6 11:55:30	ppp	42771	[wan_link0] MAGICNUM 0xb75ba742
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: state change Ack-Sent --> Opened
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: auth: peer wants CHAP, I want nothing
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: LayerUp
Sep 6 11:55:30	ppp	42771	[wan_link0] CHAP: rec'd CHALLENGE #2 len: 34
Sep 6 11:55:30	ppp	42771	[wan_link0] Name: "lns1.gs2.scp1"
Sep 6 11:55:30	ppp	42771	[wan_link0] CHAP: Using authname "na2111288@broadbandcomplete"
Sep 6 11:55:30	ppp	42771	[wan_link0] CHAP: sending RESPONSE #2 len: 48
Sep 6 11:55:30	ppp	42771	[wan_link0] CHAP: rec'd SUCCESS #2 len: 4
Sep 6 11:55:30	ppp	42771	[wan_link0] LCP: authorization successful
Sep 6 11:55:30	ppp	42771	[wan_link0] Link: Matched action 'bundle "wan" ""'
Sep 6 11:55:30	ppp	42771	[wan_link0] Link: Join bundle "wan"
Sep 6 11:55:30	ppp	42771	[wan] Bundle: Status update: up 1 link, total bandwidth 64000 bps
Sep 6 11:55:30	ppp	42771	[wan] IPCP: Open event
Sep 6 11:55:30	ppp	42771	[wan] IPCP: state change Initial --> Starting
Sep 6 11:55:30	ppp	42771	[wan] IPCP: LayerStart
Sep 6 11:55:30	ppp	42771	[wan] IPCP: Up event
Sep 6 11:55:30	ppp	42771	[wan] IPCP: state change Starting --> Req-Sent
Sep 6 11:55:30	ppp	42771	[wan] IPCP: SendConfigReq #1
Sep 6 11:55:30	ppp	42771	[wan] IPADDR 0.0.0.0
Sep 6 11:55:30	ppp	42771	[wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Sep 6 11:55:30	ppp	42771	[wan] IPCP: rec'd Configure Request #1 (Req-Sent)
Sep 6 11:55:30	ppp	42771	[wan] IPADDR 10.75.66.1
Sep 6 11:55:30	ppp	42771	[wan] 10.75.66.1 is OK
Sep 6 11:55:30	ppp	42771	[wan] IPCP: SendConfigAck #1
Sep 6 11:55:30	ppp	42771	[wan] IPADDR 10.75.66.1
Sep 6 11:55:30	ppp	42771	[wan] IPCP: state change Req-Sent --> Ack-Sent
Sep 6 11:55:30	ppp	42771	[wan] IPCP: rec'd Configure Reject #1 (Ack-Sent)
Sep 6 11:55:30	ppp	42771	[wan] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Sep 6 11:55:30	ppp	42771	[wan] IPCP: SendConfigReq #2
Sep 6 11:55:30	ppp	42771	[wan] IPADDR 0.0.0.0
Sep 6 11:55:30	ppp	42771	[wan] IPCP: rec'd Configure Nak #2 (Ack-Sent)
Sep 6 11:55:30	ppp	42771	[wan] IPADDR 81.168.68.61
Sep 6 11:55:30	ppp	42771	[wan] 81.168.68.61 is OK
Sep 6 11:55:30	ppp	42771	[wan] IPCP: SendConfigReq #3
Sep 6 11:55:30	ppp	42771	[wan] IPADDR 81.168.68.61
Sep 6 11:55:30	ppp	42771	[wan] IPCP: rec'd Configure Ack #3 (Ack-Sent)
Sep 6 11:55:30	ppp	42771	[wan] IPADDR 81.168.68.61
Sep 6 11:55:30	ppp	42771	[wan] IPCP: state change Ack-Sent --> Opened
Sep 6 11:55:30	ppp	42771	[wan] IPCP: LayerUp
Sep 6 11:55:30	ppp	42771	[wan] 81.168.68.61 -> 10.75.66.1
Sep 6 11:55:30	ppp	42771	[wan] IFACE: Up event
Sep 6 11:55:30	ppp	42771	[wan] IFACE: Rename interface ng1 to pppoe1
Sep 6 11:55:30	ppp	42771	[wan] IFACE: Add description "FTTC"
Sep 6 11:55:32	ppp	68796	Multi-link PPP daemon for FreeBSD
Sep 6 11:55:32	ppp	68796	process 68796 started, version 5.9
Sep 6 11:55:32	ppp	42771	caught fatal signal TERM
Sep 6 11:55:32	ppp	42771	[wan] IFACE: Close event
Sep 6 11:55:32	ppp	42771	[wan] IPCP: Close event
Sep 6 11:55:32	ppp	42771	[wan] IPCP: state change Opened --> Closing
Sep 6 11:55:32	ppp	42771	[wan] IPCP: SendTerminateReq #4
Sep 6 11:55:32	ppp	68796	waiting for process 42771 to die...
Sep 6 11:55:32	ppp	42771	[wan] IPCP: LayerDown

We have verified that the Openreach modem is still working by directly connecting a Windows laptop via RJ45 ethernet to it, and configuring a PPPoE connection in Windows as normal. This connects successfully and gets the correct WAN IP and can browse internet etc.

The Openreach modem's do not have any configurable settings or interface to change parameters. They serve the same purpose as other VDSL modems such as the Draytek 130 or Draytek 167 etc, whereby they add the VLAND ID 101 (required for all UK FTTC connections running over Openreach standard network) automatically, meaning that you do not need to specify the VLAN on your Router etc.

Like I said, this worked fine before the upgrade to 2.7.
I've found the following articles which seem to mention something about blank VLANs on PPP connections, but I'm getting a little lost and they're all relatively old.
https://redmine.pfsense.org/issues/12821

https://redmine.pfsense.org/issues/9148

https://www.reddit.com/r/PFSENSE/comments/15i0gn0/pppoe_connection_does_not_work_after_upgrade_to/

I have tried a full turn off and turn on of all equipment.
Also have removed the whole PPP connection from pfsense interfaces and reconfigured it again from stretch. Essentially:

1. add the interface
2. pick PPPoE
3. enter the username/password for the connection
4. and Save.

The same logs repeat over and over with the connection attempting and then dropping.

For reference, the interfaces used are:

igb0 - this problem WAN
igb1 - our LAN
igb2 - Another WAN connection (FTTP) which is also a PPPoE connection and this is working fine, but it's fibre to an ONT so a little different (and much faster)

Any thoughts are greatly appreciated, thank you.

[stephenw10]

It's because you have two competing mpd processes both trying to use the same NIC. When the new one starts it signals the old one to die. Repeat.

The question is what is triggering the new process to start? And why don't you see it on the other link?

I also have dual PPPoE links in the UK and do not see that in 2.7.

If you manually disconnect the PPPoE interface and then restart it does it do the same thing?

Do you have IPv6 configured on either link?

Steve

[stephenw10]

Oh do you have a VIP on the interface?
https://redmine.pfsense.org/issues/14434

[mpcjames]

Hi @stephenw10
Thanks for the fast response.

Manually disconnecting the PPPoE interface makes no difference.

IPv6 is disabled on both WAN links.

Wow, it's the VIP causing it!
I never thought to look there, and it looks like VIPs remain even after you delete and then re-create the interface.
Our connection does actually have 2 useable IPs (which I had completely forgotten about). As soon as I deleted the 2x VIPs from pfsense the gateway came online.

Although I had searched, I had never came across the bug #14434 and corresponding forum post.

Our ISP provides 2 useable addresses:
1.1.1.61
1.1.1.62

The first address gets assigned to the interface automatically as part of the PPPoE connection.
And then (I'm assuming) I'll need to add the second address as a /32 VIP if I intend to route traffic through/to it. But perhaps that won't work until bug #14434 is resolved.

Thanks again.

[stephenw10]

Yup, that's a bug. I'm not aware of a workaround there but I don't have more than IP on either link here to test with. You might try adding the VIP on localhost for example.

[RobbieTT]

@mpcjames
Just to help with some of your early questions.

You are correct that the Openreach VLAN is added by the modem or ONT. For generic modems this may have to be added manually, or not, as some UK-specific firmware loads do this for you (eg on some Draytek units with BT-approved firmware).

The MTU for your WAN/pppoe0 link (shown as MRU 1492 in your stats above) should be set at 1500 - ie the standard packet size. The actual physical interface connection between your pfSense router to the Openreach modem or ONT should be set at 1508 MTU, to allow for the extra 8-bytes of the PPPoE wrapper:

You will see the somewhat bogus PPPoE MTU 1492 mentioned a lot on English-speaking forums as they tend to be dominated by those from the US, where they do things differently. The 1492 setting has become somewhat of an internet lore but is incorrect for many other countries, including the UK.

I'm on the pfSense Plus side of the house where there have also been a number of PPPoE niggles, one of which is the multiple attempts to achieve a PPPoE link, rather than the expected single attempt. This can muddy the waters when doing any testing. For reasons unexplained the latest 23.09 dev firmware is more likely to make a PPPoE connection at first try. So there is hope that things are getting better for UK-style connections.

I hope this adds some UK-orientated clarity!

️