Todays (14/6) build broke PPPOE-internet connection (FATAL !!)

louis2

Hello,

A couple of hours ago I updated to the latest build. After doing so the PPPOE interface was broken, which is a fatal problem (no internet).

I did manage to get the system up and running again, by going back to the previous kernel:

make sure you have direct access to the computer promt
reboot e.g. via pfSense menu 5
wait for the system to reboot
during reboot there is very very short (to short) a menu choose 3 (or ?)
type ^boot kernel.old^
That did the job for me!

During the update process (DO NOT START THE UPDATE IF YOU ARE USING PPPOE!!) My PPPOE is implemented using EM0

I noticed:

FreeBSD 12.1-STABLE 06bf794ac10(devel-12) pfSense amd64
at the update screen a message "Please select the branch from which to update the system firmware." however no option to do that!
the "well kown" php error "[14-Jun-2020 12:02:42 Europe/Amsterdam] PHP Warning: Invalid argument supplied for foreach() in /etc/rc.dyndns.update on line 52"
1. error 1
  Jun 14 12:01:00 pfSense kernel: ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
  Jun 14 12:01:00 pfSense kernel: ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
Jun 14 12:01:00 pfSense kernel: [ath_hal] loaded
Jun 14 12:01:00 pfSense kernel: module_register_init: MOD_LOAD (vesa, 0xffffffff813cb3a0, 0) error 19
Jun 14 12:02:41 pfSense check_reload_status[414]: updating dyndns wan
Jun 14 12:02:39 pfSense php-fpm[401]: /interfaces.php: Unbound /var/unbound/root.key file is corrupt, removing and recreating.

Perhaps it works with your config, however ^Be warned!^

Louis

louis2

Hello,

I need to add, that booting from kernel.old is only a "one time thing"! So next tike you reboot the problem is there again.

Additionally you have to do the following

mv /boot/kernel /boot/kernel.bad
cp /boot/kernel /boot/kernel.old to /boot/kernel /boot/kernel
mv /boot/kernel /boot/kernel.old to mv /boot/kernel /boot/kernel.ppoe_ok

Have a look at https://www.freebsd.org/doc/handbook/kernelconfig-trouble.html

The resulting system is not optimal I think, but better than a not working firewall

Louis

w0w

Tried all snapshots in sequence from 14/6 and PPPoE is working.

louis2

Thanks for testing,

but I do have a problem .... do not know why . Have been using the PPP settings for years ...

I did have a look in the config file and copied some parts. I do not think the problem is there, never the less some remarks

The WAN gui and the xml does not to be in line ....

GUI

Request a IPv6 prefix/information through the IPv4 connectivity link = yes
Request only IPV6-prefix = yes
DHCP Prefix = 48
Send an IPv6 prefix hint to indicate the desired prefix size for delegation = yes

That does not seem to match the XML !!??

In advanced there is a field "Prefix Interface" ..... really do not know what the purpose is. Since you have to select someting there, I did use PCLAN in the past using the WAN interface now. What ever both do work!

At this moment I am runnig the kernel from a few days ago, and will wait with updating again up to I understand the problem and the php bug is fixed.

Louis

20200615 pfSenseConfig_WANpart.txt

w0w

Ok, I see your configuration is much complicated then mine, I don't have IPv6 active on PPPoE, no VLAN also. What was the build number working for you before you got this problem?

louis2

Sorry,

I do not know exactly. build from 3 june was OK, but I think I was runnig the 10 june version.

The actual version I am running is that version "to a certain extend"

By the way I swiched back to PCLAN as Prefix interface, since it was not working, during a test. But not sure that was related.

Perhaps I will try to update ones again. I do not like this "mixed" install. I did download some USB-packages, just to be able to recover, if things go terrible wrong.

Louis

louis2

Puff,

I did some tests:

I did a clean install (disk formatted) from the today version pfSense-CE-memstick-2.5.0-DEVELOPMENT-amd64-20200615-0050.img.gz
==> FAILED NO PPPOE
Then I did a clean install (disk formatted) of pfSense-CE-memstick-2.5.0-DEVELOPMENT-amd64-20200611-1250.img.gz
==> That one was still working

So something whent wrong between 20200611 and 20200614 release and it is not fixed in the 20200615 release

As said, FATAL if you are using PPPOE

Louis

w0w

https://redmine.pfsense.org/issues/10597#change-46684 the only thing I see during this period.
Do you have some errors in PPP (Status/System Logs/PPP) connection log when pf can not connect to the internet? It would be good if you can provide both logs for working and non working snapshots.

louis2

Oeps!

I did not realize that there was a separate log for that Sorry!

Hereby the actual log (working connection).

Redoing the test with the not working connection, is significant work ...

Problem is that as far as I know, there is no option for a "A-side" and a "B-side" both having a config (system-version + config) in pfSense

Neither there is a good option to do a fresh install including an existing config. I did a post about that an hour ago triggered by the tests I did today (and same work/issues I had in the past)
https://forum.netgate.com/category/38/general-pfsense-questions

Louis
20200615 pfSensePPPOE_Log_WorkingConnection_SW20200611.txt

louis2

Puff,

I did decide to go back again to todays version in order to fetch the PPPOE log from the not working system. Attached.

At the bottum of the log there is a message "Jun 15 19:27:58 pfSense ppp[27687]: web: web is not running" !!??

20200615 pfSensePPPOE_Log_NotWorkingConnection_SW20200615.txt

Louis

PS I tryed to do an install with a config conform https://docs.netgate.com/pfsense/en/latest/backup/automatically-restore-during-install.html ..... but that did not work (the usbstick did have a gpt table perhaps was that the problem, whatever stupid)

louis2

I just saw the link to the bug report .... and that the solution was merged 5 days ago ......

so that must be a coincidence

Louis

w0w

Hmm... I see you have configured service name as
<provider>INTERNET</provider>
Do your ISP really need that? What if you just remove this? I mean in GUI, leave it empty in WAN config or go to Interfaces/Interface Assignments select PPPs tab, remove exciting service name "internet" and tick "Configure NULL service name". Not sure if it helps, most of ISPs just ignores everything in this field, but...

louis2

Hum,

I am not going to do that test,

Because it is noncense, you can easely try it your self and it is very very likely that the problem is caused by the patch installed a couple of days ago.

Louis

w0w

@louis2
You can then revert this patch manually and test again, if we are talking about "Setting host-uniq for PPPoE" feature patch.
I’m not sure that this patch is the cause of the error, quite possibly some other changes in the kernel. As I said it does not broke PPPoE connection on my firewall and I have very similar connection log, nothing unusual.
Or... just wait for Netgate guys to answer, I hope somebody looking into this thread.

louis2

To be sure that the developer is informed, I did ad a comment to the patch (Feature #10597 Setting host-uniq for PPPoE)

Louis

jimp

Are you using the host-uniq setting?

There should be no change to the PPPoE config if you are not using that.

I have a couple 2.5.0 systems using PPPoE and they all work, even on current snapshots, and one of them is PPPoE on a VLAN like yours.

What is in /var/etc/mpd_wan.conf ? (you can redact private info, but don't erase it)

From the look of your log it's just a timeout, so it could possibly not actually be a PPPoE problem at all but something with the underlying interface.

louis2

Jim,

Hereby the requested file, as uploaded from my actual system. So the 11/6 version. No reason to assume it differs from the not working kernel.

mpd_wan.conf.txt

For info all my physical interfaces (or the lagg) are carrying vlan's (12 in total). Internet itself is also arriving via a vlan (via em0 vlan6).

Note that the PPPOE interface settings did not change during years.

The FW is implemented on a relative fast intel-pentium-pc, having plenty of ram and starting from a small SSD.

The config I am running now is exactly the config I using for the not working instance.

I added the upper-part of my config files (up to where the rule definitions start).

config-pfSense.lan-20200616181031_PW-removed_UpperPart.xml

The config I am running now is exactly the config I using for the not working instance.

Note, that the logs are allready uploaded, but from your reaction I conclude that You allready noticed.

Louis

louis2

I forgot to explicitly answer your first question.

Are you using the host-uniq setting? The answer is NO.

This is also not possible unless you install the gui patch or have been editing the xml. Could it be that the problem is that there is no entry for that in the config file!!??

Louis

jimp

Nothing stands out there, either. But it's definitely not related to the host-uniq change since there is no trace of it in your config or the generated MPD config.

There were a lot of other PRs merged on the 10th, all those changes should have been in the snapshot from the 11th. Nothing substantial changed after that in the base system, just syntax fixes.

louis2

After lots of test and trys I found the problem, I think.

In the WAN interface there is a field MTU, that field was in my case always empty ..... that does not work any longer, at least not in my case!

Setting MTU to 1508 did the job for me. So simple ..... if you know what the problem is ....

Question is why is suddenly required to fill that field.... !!??

However, I am glad I found the problem!

Louis