2.1 -> 2.1.2 Upgrade Issue

marvosa

My first install several years ago was v1.2.3 and have been successful with upgrading each subsequent release until I tried upgrading v2.1 to v2.1.2 (skipping 2.1.1).

The process itself went smooth (no errors), but after the system came back up I could access PFsense, but I had no internet access throughout the LAN including on PFsense. Pinging anywhere on the net prompted a "Destination host unreachable." reply from my PFsense IP. Pinging anywhere on the net from PFsense prompted "no route to host" or similar response. I double checked that my gateway, DNS, etc all came through, which they did. I also disabled and re-enabled my WAN interface thinking that may help, but no go. Everything appeared to be identical to my working, pre-upgrade v2.1 config, so I was up in arms. The last thing I checked was my routing table… everything looked normal except there was no default route... I knew this had the be the issue, but I'm not savvy enough at the CLI to add it in manually.

At this point, I knew I had a backup of my config, so I decided to format and re-install fresh v2.1.2. Went thru the wizard, put in all my static WAN info and bam... I have internet! So, I decided to restore my config thinking a fresh install fixed it, but after the config is restored, PFsense reboots, comes back up and boom... NO internet! Back to "Destination host unreachable" on my pings again.

I thinking it has to be the gateway, so I go to the WAN interface and even though the correct gateway is listed I decide to add it again with a different name thinking this must be some sort of glitch, but it get an error saying my gateway already exists. After see that error, I was going to delete the gateway and re-add it... again thinking it was some sort of glitch. So, I went to System-> Routing -> and edited my gateway to verify settings before I blew it away and noticed the "default gateway" flag was unchecked... hmmm... I checked it, saved the config... and .. bam... we have internet!

So, long story short, I was down for 2 hours after going from v2.1 to v2.1.2 and the issue was the default gateway flag on my WAN interface got stripped off during the upgrade. Restoring a config backup on top of a fresh install did not resolve the issue, so I have a few questions:

Is this a known issue that I missed in the changelog or release notes?
Has anyone else had this issue?
Is this a possible bug in the upgrade process, a possible a bug in the backup/restore process?
Possible bug isolated to upgrades from v2.1 to v2.1.2?

cmb

The default gateway flag is never touched by anything other than the user getting in and checking or unchecking it. One of two probable causes:

You had multiple gateway entries with the same name, and they were configured inconsistently. One a default, one not. Pre-2.1 you could configure duplicate gateway names on interfaces.php, input validation prohibits that there now same as System>Routing. The duplicate would be hidden in the XML unless you deleted the one in the GUI, then deleted again to get rid of both to make sure that gateway name is completely gone, and re-add it. 2.1 and newer, input validation should prevent that misconfiguration from being possible.
you unchecked it at some point, it didn't actually get removed until after a reboot, so you didn't notice until the post-upgrade reboot.

marvosa

The default gateway flag is never touched by anything other than the user getting in and checking or unchecking it.

I believe you, but if we assume the following:

I didn't touch it
The upgrade process didn't touch it
The backup/restore process didn't touch it

How else do we explain what happened? On to probable causes:

You had multiple gateway entries with the same name, and they were configured inconsistently. One a default, one not. Pre-2.1 you could configure duplicate gateway names on interfaces.php, input validation prohibits that there now same as System>Routing. The duplicate would be hidden in the XML unless you deleted the one in the GUI, then deleted again to get rid of both to make sure that gateway name is completely gone, and re-add it. 2.1 and newer, input validation should prevent that misconfiguration from being possible.

Sounds like a perfectly rational explanation and I'm not ruling it out as the culprit, but if this were true wouldn't there be traces of this somewhere (GUI/XML)? I would like to get this validated because there are no duplicates in the GUI.

you unchecked it at some point, it didn't actually get removed until after a reboot, so you didn't notice until the post-upgrade reboot.

I can say with 100% certainty that this did not happen. I wasn't anywhere near that menu (System -> Routing) pre-upgrade. Everything was running perfect and I was on 195 days of rock solid operation before the v2.1.2 upgrade. The chain of events went as follows:

Received email communication that v2.1.2 was available on 4/10
Noticed v2.1.2 was available for upgrade in the GUI this morning (4/12)
Backed up my config with the default options (Diagnostics -> Backup/Restore -> Selected "Download configuration")
From the dashboard, clicked on the update link, then pressed the button stating "Invoke Auto Upgrade" or something to that effect (I don't remember the exact verbiage)
The upgrade files were downloaded and installed followed by a PFsense reboot
Upon successful boot, no internet access (default gateway flag stripped from WAN)

If it's proven that this was a simple case of an inconsistent configuration, I'll be the first to fess up but my opinion…. something changed because I've followed the exact same procedure with all previous upgrades and they've always gone without a hitch. The only difference with this upgrade vs. the others is that I didn't upgrade sequentially… from v2.1, I skipped v2.1.1 and went straight to v2.1.2.

The other strange thing I noticed was that my OpenVPN Export Package did not show a version number (expected to see 1.2.5), which lead me to believe it didn't get re-installed correctly during the upgrade, but it seemed so minor that I didn't even mention it in my OP because of other postings stating just to re-install it manually. Co-incidence? shrug.... probably unrelated, but thought I would mention it.

phil.davis

The other strange thing I noticed was that my OpenVPN Export Package did not show a version number (expected to see 1.2.5)

The System->Packages screen tries to find the online version numbers, and show the installed version and the online available version (if different). If you have no internet connectivity, then after timing out it just shows that column blank.
The installed version is there in the local config file, so it could be displayed - I guess that behavior could be fixed.

cmb

@marvosa:

Sounds like a perfectly rational explanation and I'm not ruling it out as the culprit, but if this were true wouldn't there be traces of this somewhere (GUI/XML)? I would like to get this validated because there are no duplicates in the GUI.

It only shows one per name in the GUI. Open up a backup file in a text editor and search for <gateways>and you'll see what's there in full.</gateways>

marvosa

Here is the "gateways" section:

	 <gateways><gateway_item><interface>opt1</interface>
			<gateway>192.168.65.254</gateway>
			<name>AspinwallGW</name>
			 <weight><interval><ipprotocol>inet</ipprotocol></interval></weight></gateway_item> 
		 <gateway_item><interface>wan</interface>
			<gateway>X.159.198.129</gateway>
			<name>CharterBusiness</name>
			 <weight><interval><descr><ipprotocol>inet</ipprotocol></descr></interval></weight></gateway_item> 
		 <gateway_item><interface>wan</interface>
			<gateway>X.181.155.193</gateway>
			<name>CharterBusiness_LJG2</name>
			 <weight><interval><descr><ipprotocol>inet</ipprotocol></descr></interval></weight></gateway_item> 
		 <gateway_item><interface>lan</interface>
			<gateway>192.168.50.2</gateway>
			<name>Untangle_GW</name>
			<weight>1</weight>
			<ipprotocol>inet</ipprotocol>
			 <interval></interval></gateway_item> 
		 <gateway_item><interface>opt2</interface>
			<gateway>172.20.1.1</gateway>
			<name>OPT2GW</name>
			 <weight><ipprotocol>inet</ipprotocol>
			 <interval><defaultgw></defaultgw></interval></weight></gateway_item></gateways>

A couple things:

My WAN default gateway is X.181.155.193 and named "CharterBusiness_LJG2"
That "opt2" interface was just entered as a test and was deleted months ago.
I used to see that opt2 interface IP (172.20.1.1) in my logs, but couldn't for the life of me figure out where it was coming from since it wasn't configured in my system anywhere…. now I know.... it's there behind the scenes.