Pfsense 2.7.0 -> 2.7.2 can't start GUI

stephenw10

Hmm, is it otherwise working as expected?

The only other time I've seen this myself was with an old bug but that stopped the firewall passing traffic.

thekorn

@stephenw10 yeah it's passing & blocking traffic as usual, DHCP and bind are running just fine, SSH is up and working, even open VPN is hunky dory. Just the entire web server process is DOA after upgrade.

(Wireguard was broken which is what triggered the reboot as an attempted diagnosis/fix. Still busted. :P )

thekorn

@stephenw10 Looks like the upgrade from 2.7.0 to 2.7.2 really trashed something, but don't know what.

On a hunch, I ssh'd in and edited /cf/conf/config.xml , and changed the protocol line from https to http. Rebooted. (Only changed the one line!)

Web gui is up!

...but you can't log in. Or rather, you can log in, but logging in just brings you right back to the login screen. It doesn't complain about bad credentials (unless you blow the password, of course). If you're logged in on the console you'll see a successful login, just... the web gui is screw-oooo00000000000ed.

So looks like the 2.7.0 -> 2.7.2 upgrade really torched something. I reverted the change via ssh and yup, web gui won't start again.

I previously saved the current config file and am about to restore my christmas config, see if that gets me anywhere. I'll keep this config file around just for giggles for a bit just in case.

thekorn

@stephenw10 Well, restoring my christmas config backup worked! Web GUI is now up and running on 2.7.2 with https.

I ran a diff on the christmas backup vs the 2.7.2-upgrade backup file. Didn't see anything dramatic, except that the borked upgrade config file's lines look like they were sent through a url sanitizer???

left is christmas config, right is borked 2.7.2 upgrade config:

<                       <descr><![CDATA[Allows a pre-configured OpenVPN Windows Client or Mac OS X's Viscosity configuration bundle to be exported directly from pfSense.]]></descr>
<                       <version>1.6_8</version>
---
>                       <descr><![CDATA[Allows a pre-configured OpenVPN Windows Client or Mac OS X&#039;s Viscosity configuration bundle to be exported directly from pfSense.]]></descr>
>                       <version>1.9.2</version>

(That's just a random example I grabbed from the diff file, but it happens ALL OVER the config file wherever the original had an apostrophe.)

I'm imagining that if I dig deep enough into the config file, some encryption key happened to have an ' in it and that was replaced by ' someplace that it shouldn't have, and that's what killed everything. But it's 2AM, I'm tried, and I'll continue after some sleep.

thekorn

Ha, the forum munged my comment.

I'm imagining that the config file has a ' replaced by #039; someplace that it really shouldn't, and that's what trashed the whole thing.

thekorn

@stephenw10 Well whaddya know, restoring my christmas backup (with the apostrophes not replaced by #039;s) fixed my wireguard not running problem, too!

I'm going to bed.

stephenw10

Hmm, if it really is missing something required for https I'd expect some more useful error output trying to start it.

I would just install 2.7.2 clean from there though. Otherwise you'll never really know the install is good.

stephenw10

Were all the changes inside CDATA sections?

thekorn

@stephenw10 For the ' to #039; swaps, yes, confirmed they were always confined to CDATA sections.

But there were other changes in the diff as well. Scanning the diff, I see differences in...

versions of various things (makes sense)
dhcpdata (makes sense)
this section, which I'm thinking applies to the wan (and a few more like it)

<                               <target></target>
<                               <targetip></targetip>
<                               <targetip_subnet></targetip_subnet>
---
>                               <target>wanip</target>
1772a1769
>                               <target_subnet></target_subnet>

Probably the most troublesome, the entire <sshdata> section is missing! (...whaaaaa?)

Also checked if the xml was well-formed, both configs it is so that's not a problem. (...dang?)

stephenw10

Hmm, interesting. Hard to imagine hat might have caused that.

Are the config versions shown correct?
https://docs.netgate.com/pfsense/en/latest/releases/versions.html#pfsense-ce-software

thekorn

@stephenw10 yes sir they are!

zep

I just did a 2.7.0 to 2.7.2 upgrade
The GUI is not loading for me
The sympton seems to be similar to what "thekorn" experienced

Is there a fix?

I see the mention of installing 2.7.2 fresh

Is that the only option?

stephenw10

Check the config. Has it been corrupted in the same way they saw?

thekorn

@zep The ultimate fix for me was to restore a config from before the upgrade. Been running 100% since then! (Luckily I had one on hand!)

Check your config history (diagnostics, backup & restore, then config history). If you're lucky maybe there's still one from before your upgrade in there.

edit: oh duh, www not running, can't check that. I'm sure it exists somewhere in the file system, but don't know pfsense well enough to know where. Maybe someone else does?

stephenw10

Backup configs are stored locally in /conf/backup

If you have ACB enabled they are also stored there.

zep

@stephenw10
Thanks for the info

So there were some backups in the /conf/backup folder

I copied a backup config file
from /conf/backup/config-1694658441.xml
to /cf/conf/config.xml

I rebooted the router
The router is still working (ie still routing)
But, I still can not log into the Web Gui

I do see some &#39 entries, in the config files (both the current file and the backup file)
But they are in places I would expect

My sons name is Jason. I call him J. One DHCP reservation is for J's server (xml files shows it as J&#39s server)

There was also a backup folder under the /cf/conf (so /cf/conf/backup)
I tried a backup file from there as well
No luck
Same issue
The router comes up and works (ie still routing)
But, I still can not log into the Web Gui

Any other suggestions?
I can do a fresh install if need be (I would do it on a fresh server, just to keep the old one on hand for now)

Thanks

stephenw10

Ok so what exactly are you seeing?

Do you see nginx running in the output of ps -aux?

Any errors in the nginx logs or main system logs?

zep

@stephenw10

So
The PFSense web page would not load
There would be an hour glass on the browser for a few seconds (15 - 30 seconds)
Then a browsers error message, indicating page took too long to respond / did not load

This was from MS edge

This morning I decided to try Chrome
The PFSense web page loaded fine from Chrome

I tried accessing the PFSense web page from MS Edge on a 2nd PC
The PFSense web page loaded fine from the 2nd PC using MS edge

I ended up clearing the cache on my broken MS edge PC
The PFSense web - GUI page now loads fine from MS edge for me

Looks like something happened to my MS edge during the upgrade

The PF Sense upgrade seams fine
Just my MS Edge browser issue

I know try another browser, is just about trouble shooting 101
But I was in MS edge to start the upgrade
I upgraded PF Sense
Then Edge no longer loaded the PF sense web page - GUI

All seams fine now (post clearing my MS edge cache)

Thanks for the help

stephenw10

Ah, nice. Yup easily done when it's worked fine previously.

thekorn

@stephenw10 The saga continues!

So I went to enable the new DHCP server, and... all of a sudden the WWW interface died again!

Went in through ssh, and sure enough,

cat /conf/config.xml | grep "#039"

I get hits on #039s, and the WWW interface is dead.

So I checked a config file I made a few days ago, and there are STILL #039s in it. Uh-oh.

I then scp'd over my old christmas config (2023-12-29). and checked it for #039s, none found.

cat /conf/config-pfSense.housenet-20231229092319.xml | grep "#039"

No hits, good. Then I copied christmas config file to config.xml (original is still in place) and check AGAIN for #039s. None found. Reboot. A-OK, web gui comes up.

SSH back in...

cat /conf/config.xml | grep "#039"

SURE ENOUGH, the #039s are back! Reboot again, and now WWW is dead again!

Something is actively corrupting my config file after the first reboot! (And as a result the www process is failing.)

Is there a way to roll back to 2.7.0 just to double-check that this wasn't happening on that version? Or do I just have to completely re-image it with 2.7.0?