To 23.01 or not ? that is the question :)

chudak

Finally, 23.01+ is ready!

Gathering success and failure feedback for a new point pfSense Plus 22.05 release.

Please add your comment!

SteveITS

chudak

@chudak said in To 23.01 or not ? that is the question :):

Finally, 23.01+ is ready!

Gathering success and failure feedback for a new point pfSense Plus 22.05 release.

Please add your comment!

Of course!

stephenw10

What hardware are you running?

Check you're not affected by any of the known issues:
https://docs.netgate.com/pfsense/en/latest/releases/23-01.html#errata-known-hardware-issues

I've upgraded hundreds of things and am pretty confident.

Steve

AndyRH

I am waiting to hit 180 days up on the previous version, you know, to make sure it is stable... Then I will test 23.01.

3 more days. (works out to be the weekend)

lpfw

@chudak

Nothing but problems. Don't do it.

tman222

I upgraded two systems from 22.05 to 23.01 successfully about 24 hours ago - no major issues.

System one is a slightly older Xeon D-1518 based Supermicro unit - when it came back up it had a message for a crash log. Looking at the log though it was essentially empty. No further errors since (after subsequent reboots or otherwise). System two is a newer 10th gen i3 custom build - no errors or crash logs were present after upgrading.

Overall, the upgrades went smoothly. Just ran into a couple minor things on system two:

For some reason after the upgrade the system had trouble accessing an NTP server through a VPN tunnel. No problems before on 22.05 or older versions. I confirmed the issue wasn't with the VPN. Strangely, after a waiting a bit and restarting NTP, it did finally connect.
System two has a CyberPower UPS connected via USB and the NUT package is installed. After a while NUT loses the connection and can't reconnect to the UPS. I can disable an re-enable NUT and it will work for a little while before losing the connection again. Might try reinstalling NUT or a different USB port on the pack of the system to see if that fixes it.

Performance wise after the upgrade I did see some gains on system one: Running a single stream iperf3 test through the firewall between two 10Gbit hosts on different network segments used to give me around 6.5Gbit/s throughput. After upgrading to 23.01 I now see speeds around 7.5-8.0Gbit/s (and at times slightly higher).

As a final note, it's important to be patient with this upgrade. Even though both systems are fairly fast, system one did take around 5 minutes to come back up after the upgrade completed.

Gertjan

@chudak

Initially, I wanted to wait.

But I was already running the final 'last week' RC candidate.
With ZFS switching versions is a one click show (in case of). It's the perfect Plan B solution.
So I upgraded. No issues what so ever.

bcdouglas

@chudak I had good luck with the initial RCs, but am seeing a weird issue with my ipv4 gateway on the GA version.

After fiddling with my ipv6 config, my netgate 6100 ipv4 gateway disappears following a reboot. The family "did you break the Internet" factor is very high on 23.01 for me so far. I had to fail back to my previous Ubiquiti EdgeRouter house gateway which is 100% unsexy, but simply works.

All normal new release stuff I'm sure. I flashed my 6100 back to 22.05 all is good. I suggest holding off unless you are ready to fix something. Again, I had great luck with the early RCs. I hope to get back on 23.01 soon!

stephenw10

Is there an open bug that reflects what you're seeing? If you're seeing something unique we may not know about it to fix it.
https://redmine.pfsense.org/

pinballwiz

I upgraded an SG-5100 successful but there were some hurdles to overcome:

Ended up doing a fresh install of 23.01. It threw allot of PHP errors initially and crashed. Lost WebGUI and network connectivity shortly after boot (when logged into WebGUI). To resolve, I did the following:

Reinstalled Pfblocker-devel and updated to resolve most of the PHP errors.
Possible Captive Portal Bug causing the WebGUI and network connectivity loss? Had to exclude my Admin interface from Captive Portal with MAC pass through in order to restore both. Creating a new CP Zone for only the Admin interface enabled recreated the issue. No issues with CP enabled for other interfaces (VLANS).

AndyRH

Upgraded after 180 days of uptime.
I must be livin' right, memory and CPU usage are normal on first boot.
179 days left to tie.

chudak

@andyrh said in To 23.01 or not ? that is the question :):

Upgraded after 180 days of uptime.
I must be livin' right, memory and CPU usage are normal on first boot.
179 days left to tie.

Why magical 180 days uptime?

AndyRH

@chudak Nothing magical, just thought it was interesting that the last boot (power outage) worked out to be 180 days before I would have time to do the upgrade.

johnpoz

@chudak I just updated to 23.01 on my sg4860.. Went smooth no real issues that I have seen as of yet.. Still have a bit of testing to do.. but looks like my test vpn to my vps came up, my HE tunnel is up. I show all packages up.. Looking at the monitor graphs they all seem to be functioning. My haproxy stuff is working, clearly I have internet, etc. no issues dns (I use unbound in just normal resolver mode), my freerad is working because wifi client using eap-tls connected via cycle of its wifi, my external monitoring of services I provide via port forwards all show green again after warning me when down for the upgrade.

Only thing I see from quick check, is my tailscale test setup doesn't seem to have come up?
"# - not logged in, last login error=invalid key: API key does not exist"

@AndyRH I was at 188 days uptime before the update ;)

As always had a plan to recover if failed.

I had gotten 23.01 image from tac, ready at the standby.
I uninstalled all packages that I was not actively using - I normally have many packages installed that I don't use very often and just to help users that are using them. So reduced install to my core packages.. pfblocker, freerad, etc..
Took a manual backup of the config.
I rebooted pfsense pre update - not a bad idea since been up for some 188 days. And is recommended in the upgrade guide..
Connected to console so could watch the progress of the upgrade.

Upgrade took approx 15 minutes before I was logging into the gui again.

I hadn't put in the sg4860 setting in loader.conf.local - partly because I wanted to see the console error. Which I did see, it pretty much made use of console impossible with the flood of messages.. So I did the setting per the release notes, rebooted again and that is gone.

I kind of want to test rollback to 22.05 via boot loader stuff with zfs, but already have users connected to my plex it seems.. So that might have to wait for another day.. My change window is small normally - early morning is best time for me to do anything that takes services offline, like updating pfsense ;) Or I hear about - wife yelling from other room - is internet down heheheh

So from my perspective another great job from pfsense/netgate - there are always things that can go wrong in such a task. This wasn't some minor update.. This is a move to new freebsd base, update to php, etc. And users always have lots of crazy things different, different hardware, different settings.

So make sure you have your ducks in a row for recovery if the worse happens and should go fine - now off to see what is up with tailscale..

edit: well that didn't take long to figure out.. No wonder tailscale status not showing up - seems after logging into tailscale website that pfsense expired on "Expired Feb 5, 2023" - tells you how much I use it.. Easy enough to fix

edit2: upon my further looking validation that all green with everything.. I did see the states in monitoring were not working... Per this thread https://forum.netgate.com/topic/177931/system-states-graph-no-longer-working and all seems to be working now for states after apply of patches mentioned in there by jimp

Also validated openvpn client can connect to both my instances, udp and tcp. And tailscale client can connect as well.

mdp716

After seeing several posts from folks I view as experts (Lawrence Networks comes to mind right off) saying they're good with the UG to 23.01 I decided to go ahead and give it a go.

Started from 22.05 on a third party Intel platform with a J4125 CPU.

Installed packages were:

Backup
Cron
ntopng
openvpn-client-export
Service-Watchdog
Telegraf (disabled)

Since I have a functioning Boot Environments and backed up prior to UG I opted not to uninstall any of the packages on the first go around just to see what happens as prior upgrades (notably the CE --> Plus was particularly picky about installed packages) with them installed did not go well.

I'm happy to report that the Upgrade was successful at this point less than an hour in.

I did have to stop ntopng, reapply the backup of the configuration I had and then re-edit the ntopng.inc file to point ntopng to my SSD where I wanted the DB to write to before restarting ntopng. Note to netgate folks, sure wish you'd use the more mainstream ntopng version instead of the buggy/modified one you have as a package.

My OpenVPN set up is functioning as is all of the other packages as near as I can tell at this point.

The driving force for me was that I bought a Cisco AP and wanted to have it connect directly to another interface on the pfsense device instead of go through a switch connected to an interface on the pfsense device which is how I have an Orbi RBR50 set up and I intended to simply move the configurations and rules from the one interface to the other but then I read that there was a known issue with that and that it was fixed in 23.01 so I wanted to be on 23.01 before adding that interface and moving the configs/rules.

OffstageRoller

@johnpoz said in To 23.01 or not ? that is the question :):

I hadn't put in the sg4860 setting in loader.conf.local - partly because I wanted to see the console error. Which I did see, it pretty much made use of console impossible with the flood of messages.. So I did the setting per the release notes, rebooted again and that is gone.

Did you end up applying the ichsmb0 fix, ehci0 fix, or both?

johnpoz

@offstageroller I did this

hint.ichsmb.0.disabled=1

In the loader.conf.local - I don't recall seeing anything about echi0 ?? Guess I did a really shitty job this time of paying attention... I missed the pfblocker version thing in the release notes.. And now I seemed to have missed something else.. Even though I was aware of the ichsmb0 thing..

Yeah I was only seeing ichsmb0 in my console.. no echi0 issues that I saw. And once I rebooted after doing the above entry only my console was clean.

stephenw10

You only need ichsmb0 disabled.

Disabling EHCI also stops the error flood but it also disables the eMMC because that is USB connected in RCC-VE, including the 4860.

OffstageRoller

Thank you both @johnpoz and @stephenw10!

I'll apply this to my SG-4860, reboot, verify all is still well, and then upgrade to 23.01.

For anyone using Ansible with pfSense, here's what you can add to your yaml to accomplish this:

- name: workaround for cpu increase on sg-4860 with pfSense 23.01 and above
  lineinfile:
    path: /boot/loader.conf.local
    state: present
    create: true
    line: "hint.ichsmb.0.disabled=1"
    owner: root
    group: wheel
    mode: "0644"
  remote_user: root

Are the docs available for me to submit a PR/MR for so I can clean up the wording for others that have an SG-4860 (https://docs.netgate.com/pfsense/en/latest/releases/23-01.html)? Currently, the wording is vague to where it's not clear which one you should apply, if not both. But now that we know it's only ichsmb0, we can update the docs so users hopefully won't have to search through the forums and find posts like these to know for sure :).