Upgraded from 2.1.5 to 2.2.4 on CARP Backup, never came back up…

jasonlitka

So I upgraded my CARP backup from 2.1.5 to 2.2.4 last night and it never came back up properly. The link light is off and I can't ping the LAN interface (obviously) which explains why the Web UI isn't available. I've logged in remotely using the IPMI card and I don't see anything out of order on the console. This is a 10Gbe ix port with a genuine Intel optic in it. I tried adding "hw.ix.unsupported_sfp=1" to /boot/loader.conf.local and restarting, just in case, but that didn't help.

The SFP+ module and fiber patch are fine, because I moved them to the primary box still running 2.1.5 and they came up instantly, but when plugged into the backup box they stay dark.

What should I be looking at here?

cmb

'clog /var/log/system.log' at console, anything related to the ix NIC?

The usual things here most often the cause of a 10G NIC not working.
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

jasonlitka

clog /var/log/system.log | grep ix

Not much for me to go on.

![pfsense-2 ix.png](/public/imported_attachments/1/pfsense-2 ix.png)
![pfsense-2 ix.png_thumb](/public/imported_attachments/1/pfsense-2 ix.png_thumb)

jasonlitka

I suppose the other thing work mentioning is that during startup it sits at "Configuring firewall" for a rather long time (by my standards), about a minute. Not sure if that's related.

![pfsense-2 fw.PNG](/public/imported_attachments/1/pfsense-2 fw.PNG)
![pfsense-2 fw.PNG_thumb](/public/imported_attachments/1/pfsense-2 fw.PNG_thumb)

jasonlitka

No other comments? The optics are glowing so they're getting power but there's no link-up. I'm seeing a lot of people reporting this issue for FreeBSD when in a KVM container but not on bare metal.

Any idea how to roll back from the command line using the full backup I made before the upgrade? The restore recent config menu option only seems to allow me to restore the config, not the entire system.

EDIT 1: Found /etc/rc.restore_full_backup, going to give that a try.

EDIT 2: Yeah, that got me back up and running under 2.1.5, though I did need to recycle a few ports courtesy of BPDU Guard. This is the second time I've had issues upgrading a box from 2.1.x to 2.2.x. I think I'm going to go demo a pair of ASAs before I come back and try again.

jjoaquina

Same thing here. Our production firewall died (v2.1.4) after a disk failure so we had to use a backup box. It was a nice opportunity to install a 2.2.4 from scratch after the disk change and later download the rules xml backup, but after all done, the boot screen stops for almost 10 minutes on the "Configuring Firewall" and when it comes alive my tables are all empty. I have a 6 interface box routing between 2 different LANs and 4 WANs and many IPSEC VPNs, so I have a huge rule and table lists. It's not an option loading each item manually to understand what is going on. I end the day giving up on the 2.2.4 and installing the 2.1.4 on that box. It works, not the latest kernel, not the latest code, but it simply works. I’m willing to give another shot if someone enlightens me on what can be going wrong. It seems the aliases are not filling the tables properly so the rules just don’t work.
Thanks

jasonlitka

@jjoaquina:

Same thing here. Our production firewall died (v2.1.4) after a disk failure so we had to use a backup box. It was a nice opportunity to install a 2.2.4 from scratch after the disk change and later download the rules xml backup, but after all done, the boot screen stops for almost 10 minutes on the "Configuring Firewall" and when it comes alive my tables are all empty. I have a 6 interface box routing between 2 different LANs and 4 WANs and many IPSEC VPNs, so I have a huge rule and table lists. It's not an option loading each item manually to understand what is going on. I end the day giving up on the 2.2.4 and installing the 2.1.4 on that box. It works, not the latest kernel, not the latest code, but it simply works. I’m willing to give another shot if someone enlightens me on what can be going wrong. It seems the aliases are not filling the tables properly so the rules just don’t work.
Thanks

Did you ever get your system working?

I just tried to get to 2.2.5, twice, and both times left me with no carrier on the ix interface. For the second time around I tried swapping all the interfaces over to ix0 from ix1, dropping all the traffic shaping rules, and eliminating all packages other than AutoConfigBackup and still no luck.

I suspect the issue is relating to the vLAN interfaces but there's little I can do about those since I need them there.

EDIT 1: It's the vLANs on the ix interfaces. If I remove them then the upgrade works. I can add them back after the update completed too so it's something with the upgrade process.

EDIT 2: May have spoken too soon… Just tried disabling CARP on the master to test 2.2.5 on the backup and I end up with no internet access.

EDIT 3: Rebooting brought back the no carrier. Awesome.

EDIT 4: Moved everything over to 1Gbe interfaces (igb) and while the system comes back after a reboot, I still get no internet access. No logged firewall data with an any/any log rule at the top of LAN and 0 states active. I've wasted enough time here. I'm going home.

EDIT 5: One more... I can ping from the firewall to anywhere outside or inside when on the proper interface, but not outside when using LAN. This would appear to be none of the NAT rules loading. Any ideas on how to troubleshoot that?

jasonlitka

Opened a support ticket. Turns out that the issue was related to pfBlocker not being properly uninstalled during the upgrade and leaving behind a broken alias/rules. Once the config was cleaned up the Firewall/NAT rules loaded properly and the box worked. I am now in the process of upgrading the primary as well.

No idea on the ix interfaces with vLANs though. That is still broken for me.