Pfsense 2.7.0 -> 2.7.2 can't start GUI

stephenw10

Does it have a WAN IP? Is DHCP failing there? Seeing 0.0.0.0 there usually implies that.

If not is some other interface stuck on 0.0.0.0?

thekorn

@stephenw10 No, all interfaces have the IP addresses they should...


 WAN (wan)       -> igb3       -> v4/DHCP4: 107.192.xxx..yyy/22
                                  v6/DHCP6: 2600:1700:ea1:360:a236:xxxx:xxxx:xxxx/64
 LAN (lan)       -> igb0       ->
 WIREGUARDINBOUND (opt1) -> tun_wg0    -> v4: 172.18.xx.xx/24
 HOUSE_VLAN5 (opt2) -> igb0.5     -> v4: 172.18.xx.xx/24
                                  v6/t6: 2600:1700:ea1:36f:a236:xxxx:xxxx:xxxx/64
 CAMERAS_VLAN25 (opt3) -> igb0.25    -> v4: 172.18.xx.x/24
 OPENVPNUDP (opt4) -> ovpns1     -> v4: 172.18.xx.x/25
 OPENVPNTCP (opt5) -> ovpns2     -> v4: 172.18.xx.x/24
 INTERNET_ONLY_VLAN15 (opt6) -> igb0.15    -> v4: 172.18.xx.x/24
                                  v6/t6: 2600:1700:ea1:36e:a236:xxxx:xxxx:xxxx/64
 172_18_1_X__NOVLANTAGS (opt7) -> re0        -> v4: 172.18.x.x/24

(x's are munged, obviously)

opt7 isn't plugged in, it's a redundant oh sh*t interface. ;)

stephenw10

Hmm, and none of that changed since 2.7.0?

thekorn

@stephenw10 Nope, those haven't changed in probably two years now!

stephenw10

Hmm, is it otherwise working as expected?

The only other time I've seen this myself was with an old bug but that stopped the firewall passing traffic.

thekorn

@stephenw10 yeah it's passing & blocking traffic as usual, DHCP and bind are running just fine, SSH is up and working, even open VPN is hunky dory. Just the entire web server process is DOA after upgrade.

(Wireguard was broken which is what triggered the reboot as an attempted diagnosis/fix. Still busted. :P )

thekorn

@stephenw10 Looks like the upgrade from 2.7.0 to 2.7.2 really trashed something, but don't know what.

On a hunch, I ssh'd in and edited /cf/conf/config.xml , and changed the protocol line from https to http. Rebooted. (Only changed the one line!)

Web gui is up!

...but you can't log in. Or rather, you can log in, but logging in just brings you right back to the login screen. It doesn't complain about bad credentials (unless you blow the password, of course). If you're logged in on the console you'll see a successful login, just... the web gui is screw-oooo00000000000ed.

So looks like the 2.7.0 -> 2.7.2 upgrade really torched something. I reverted the change via ssh and yup, web gui won't start again.

I previously saved the current config file and am about to restore my christmas config, see if that gets me anywhere. I'll keep this config file around just for giggles for a bit just in case.

thekorn

@stephenw10 Well, restoring my christmas config backup worked! Web GUI is now up and running on 2.7.2 with https.

I ran a diff on the christmas backup vs the 2.7.2-upgrade backup file. Didn't see anything dramatic, except that the borked upgrade config file's lines look like they were sent through a url sanitizer???

left is christmas config, right is borked 2.7.2 upgrade config:

<                       <descr><![CDATA[Allows a pre-configured OpenVPN Windows Client or Mac OS X's Viscosity configuration bundle to be exported directly from pfSense.]]></descr>
<                       <version>1.6_8</version>
---
>                       <descr><![CDATA[Allows a pre-configured OpenVPN Windows Client or Mac OS X&#039;s Viscosity configuration bundle to be exported directly from pfSense.]]></descr>
>                       <version>1.9.2</version>

(That's just a random example I grabbed from the diff file, but it happens ALL OVER the config file wherever the original had an apostrophe.)

I'm imagining that if I dig deep enough into the config file, some encryption key happened to have an ' in it and that was replaced by ' someplace that it shouldn't have, and that's what killed everything. But it's 2AM, I'm tried, and I'll continue after some sleep.

thekorn

Ha, the forum munged my comment.

I'm imagining that the config file has a ' replaced by #039; someplace that it really shouldn't, and that's what trashed the whole thing.

thekorn

@stephenw10 Well whaddya know, restoring my christmas backup (with the apostrophes not replaced by #039;s) fixed my wireguard not running problem, too!

I'm going to bed.

stephenw10

Hmm, if it really is missing something required for https I'd expect some more useful error output trying to start it.

I would just install 2.7.2 clean from there though. Otherwise you'll never really know the install is good.

stephenw10

Were all the changes inside CDATA sections?

thekorn

@stephenw10 For the ' to #039; swaps, yes, confirmed they were always confined to CDATA sections.

But there were other changes in the diff as well. Scanning the diff, I see differences in...

versions of various things (makes sense)
dhcpdata (makes sense)
this section, which I'm thinking applies to the wan (and a few more like it)

<                               <target></target>
<                               <targetip></targetip>
<                               <targetip_subnet></targetip_subnet>
---
>                               <target>wanip</target>
1772a1769
>                               <target_subnet></target_subnet>

Probably the most troublesome, the entire <sshdata> section is missing! (...whaaaaa?)

Also checked if the xml was well-formed, both configs it is so that's not a problem. (...dang?)

stephenw10

Hmm, interesting. Hard to imagine hat might have caused that.

Are the config versions shown correct?
https://docs.netgate.com/pfsense/en/latest/releases/versions.html#pfsense-ce-software

thekorn

@stephenw10 yes sir they are!

zep

I just did a 2.7.0 to 2.7.2 upgrade
The GUI is not loading for me
The sympton seems to be similar to what "thekorn" experienced

Is there a fix?

I see the mention of installing 2.7.2 fresh

Is that the only option?

stephenw10

Check the config. Has it been corrupted in the same way they saw?

thekorn

@zep The ultimate fix for me was to restore a config from before the upgrade. Been running 100% since then! (Luckily I had one on hand!)

Check your config history (diagnostics, backup & restore, then config history). If you're lucky maybe there's still one from before your upgrade in there.

edit: oh duh, www not running, can't check that. I'm sure it exists somewhere in the file system, but don't know pfsense well enough to know where. Maybe someone else does?

stephenw10

Backup configs are stored locally in /conf/backup

If you have ACB enabled they are also stored there.

zep

@stephenw10
Thanks for the info

So there were some backups in the /conf/backup folder

I copied a backup config file
from /conf/backup/config-1694658441.xml
to /cf/conf/config.xml

I rebooted the router
The router is still working (ie still routing)
But, I still can not log into the Web Gui

I do see some &#39 entries, in the config files (both the current file and the backup file)
But they are in places I would expect

My sons name is Jason. I call him J. One DHCP reservation is for J's server (xml files shows it as J&#39s server)

There was also a backup folder under the /cf/conf (so /cf/conf/backup)
I tried a backup file from there as well
No luck
Same issue
The router comes up and works (ie still routing)
But, I still can not log into the Web Gui

Any other suggestions?
I can do a fresh install if need be (I would do it on a fresh server, just to keep the old one on hand for now)

Thanks