Pfsense 2.7.0 -> 2.7.2 can't start GUI

thekorn

Hello,

upgraded my pfsense 2.7.0 box to 2.7.2. Rebooted, except now the web server refuses to start.

I can ssh in (and su). Pfsense appears to be running as traffic is going through, just the www interface seems to be completely dead.

Where/how do I begin diagnosing/repairing this?

Thanks in advance!

(I was trying to diagnose a wireguard problem and one recommended solution was to reboot, which is what started the dominos falling.)

I was able to ssh in and grab a copy of the config file at /cf/conf/config.xml and put it in a safe place, so if nuking and reinstalling is necessary it's not a horrendous thought, but obviously would like to avoid that if possible!

Thanks in advance!

stephenw10

If you run option 11 from the console menu do you see error logged in the system or nginx logs?

thekorn

@stephenw10 Restart webconfigurator says...

nginx: 2024/01/18 16:34:53 [emerg] 98789#101489: bind() to 0.0.0.0:443 failed (48: Address already in use)

(a bunch of times)

Message from syslogd@pfSense at Jan 18 16:34:53 ...
nginx: 2024/01/18 16:34:53 [emerg] 98789#101489: still could not bind()
 done.

Web server still down after it gives up.

stephenw10

Does it have a WAN IP? Is DHCP failing there? Seeing 0.0.0.0 there usually implies that.

If not is some other interface stuck on 0.0.0.0?

thekorn

@stephenw10 No, all interfaces have the IP addresses they should...


 WAN (wan)       -> igb3       -> v4/DHCP4: 107.192.xxx..yyy/22
                                  v6/DHCP6: 2600:1700:ea1:360:a236:xxxx:xxxx:xxxx/64
 LAN (lan)       -> igb0       ->
 WIREGUARDINBOUND (opt1) -> tun_wg0    -> v4: 172.18.xx.xx/24
 HOUSE_VLAN5 (opt2) -> igb0.5     -> v4: 172.18.xx.xx/24
                                  v6/t6: 2600:1700:ea1:36f:a236:xxxx:xxxx:xxxx/64
 CAMERAS_VLAN25 (opt3) -> igb0.25    -> v4: 172.18.xx.x/24
 OPENVPNUDP (opt4) -> ovpns1     -> v4: 172.18.xx.x/25
 OPENVPNTCP (opt5) -> ovpns2     -> v4: 172.18.xx.x/24
 INTERNET_ONLY_VLAN15 (opt6) -> igb0.15    -> v4: 172.18.xx.x/24
                                  v6/t6: 2600:1700:ea1:36e:a236:xxxx:xxxx:xxxx/64
 172_18_1_X__NOVLANTAGS (opt7) -> re0        -> v4: 172.18.x.x/24

(x's are munged, obviously)

opt7 isn't plugged in, it's a redundant oh sh*t interface. ;)

stephenw10

Hmm, and none of that changed since 2.7.0?

thekorn

@stephenw10 Nope, those haven't changed in probably two years now!

stephenw10

Hmm, is it otherwise working as expected?

The only other time I've seen this myself was with an old bug but that stopped the firewall passing traffic.

thekorn

@stephenw10 yeah it's passing & blocking traffic as usual, DHCP and bind are running just fine, SSH is up and working, even open VPN is hunky dory. Just the entire web server process is DOA after upgrade.

(Wireguard was broken which is what triggered the reboot as an attempted diagnosis/fix. Still busted. :P )

thekorn

@stephenw10 Looks like the upgrade from 2.7.0 to 2.7.2 really trashed something, but don't know what.

On a hunch, I ssh'd in and edited /cf/conf/config.xml , and changed the protocol line from https to http. Rebooted. (Only changed the one line!)

Web gui is up!

...but you can't log in. Or rather, you can log in, but logging in just brings you right back to the login screen. It doesn't complain about bad credentials (unless you blow the password, of course). If you're logged in on the console you'll see a successful login, just... the web gui is screw-oooo00000000000ed.

So looks like the 2.7.0 -> 2.7.2 upgrade really torched something. I reverted the change via ssh and yup, web gui won't start again.

I previously saved the current config file and am about to restore my christmas config, see if that gets me anywhere. I'll keep this config file around just for giggles for a bit just in case.

thekorn

@stephenw10 Well, restoring my christmas config backup worked! Web GUI is now up and running on 2.7.2 with https.

I ran a diff on the christmas backup vs the 2.7.2-upgrade backup file. Didn't see anything dramatic, except that the borked upgrade config file's lines look like they were sent through a url sanitizer???

left is christmas config, right is borked 2.7.2 upgrade config:

<                       <descr><![CDATA[Allows a pre-configured OpenVPN Windows Client or Mac OS X's Viscosity configuration bundle to be exported directly from pfSense.]]></descr>
<                       <version>1.6_8</version>
---
>                       <descr><![CDATA[Allows a pre-configured OpenVPN Windows Client or Mac OS X&#039;s Viscosity configuration bundle to be exported directly from pfSense.]]></descr>
>                       <version>1.9.2</version>

(That's just a random example I grabbed from the diff file, but it happens ALL OVER the config file wherever the original had an apostrophe.)

I'm imagining that if I dig deep enough into the config file, some encryption key happened to have an ' in it and that was replaced by ' someplace that it shouldn't have, and that's what killed everything. But it's 2AM, I'm tried, and I'll continue after some sleep.

thekorn

Ha, the forum munged my comment.

I'm imagining that the config file has a ' replaced by #039; someplace that it really shouldn't, and that's what trashed the whole thing.

thekorn

@stephenw10 Well whaddya know, restoring my christmas backup (with the apostrophes not replaced by #039;s) fixed my wireguard not running problem, too!

I'm going to bed.

stephenw10

Hmm, if it really is missing something required for https I'd expect some more useful error output trying to start it.

I would just install 2.7.2 clean from there though. Otherwise you'll never really know the install is good.

stephenw10

Were all the changes inside CDATA sections?

thekorn

@stephenw10 For the ' to #039; swaps, yes, confirmed they were always confined to CDATA sections.

But there were other changes in the diff as well. Scanning the diff, I see differences in...

versions of various things (makes sense)
dhcpdata (makes sense)
this section, which I'm thinking applies to the wan (and a few more like it)

<                               <target></target>
<                               <targetip></targetip>
<                               <targetip_subnet></targetip_subnet>
---
>                               <target>wanip</target>
1772a1769
>                               <target_subnet></target_subnet>

Probably the most troublesome, the entire <sshdata> section is missing! (...whaaaaa?)

Also checked if the xml was well-formed, both configs it is so that's not a problem. (...dang?)

stephenw10

Hmm, interesting. Hard to imagine hat might have caused that.

Are the config versions shown correct?
https://docs.netgate.com/pfsense/en/latest/releases/versions.html#pfsense-ce-software

thekorn

@stephenw10 yes sir they are!

zep

I just did a 2.7.0 to 2.7.2 upgrade
The GUI is not loading for me
The sympton seems to be similar to what "thekorn" experienced

Is there a fix?

I see the mention of installing 2.7.2 fresh

Is that the only option?

stephenw10

Check the config. Has it been corrupted in the same way they saw?