Netgate 6100 crash report
-
I'm not 100% sure where to send this info so let me know and I'll endure it gets there.
I couple of days ago I noticed my phone started buffering on videos and shouldn't have. I restarted the phone not thinking much of it, but after it did it again, I logged into the router and noticed several services off and a crash report.
Crash report begins. Anonymous machine information:
amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05_1-n256108-459fc493a87: Wed Jun 28 04:26:04 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/obj/amd64/f2Em2w3l/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/
[PHP_errors (1).log](Invalid file type. Allowed types are: .png, .jpg, .bmp, .txt, .gif, .xls, .gz, .zip, .pcap, .pcapng, .7z, .xml, .jpeg, .diff, .patch, .tgz, .tar, .0, .cap)
Crash report details:PHP Errors:
[12-Nov-2023 00:18:22 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:18:23 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:18:23 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:18:24 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:18:32 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:18:33 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:19:26 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:19:40 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[11-Nov-2023 18:19:40 America/Chicago] PHP Fatal error: Uncaught Error: Undefined constant "CURLOPT_USERAGENT" in /usr/local/pkg/pfblockerng/pfblockerng.inc:167
Stack trace:
#0 /etc/inc/pkg-utils.inc(693): require_once()
#1 /etc/rc.start_packages(66): sync_package('pfBlockerNG')
#2 {main}
thrown in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 167
[12-Nov-2023 00:19:40 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:19:40 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:19:42 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[12-Nov-2023 00:19:44 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[11-Nov-2023 18:21:00 America/Chicago] PHP Fatal error: Uncaught Error: Undefined constant "CURLOPT_USERAGENT" in /usr/local/pkg/pfblockerng/pfblockerng.inc:167
Stack trace:
#0 /usr/local/www/widgets/widgets/pfblockerng.widget.php(36): require_once()
#1 /usr/local/www/index.php(430): include('/usr/local/www/...')
#2 {main}
thrown in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 167
[12-Nov-2023 00:21:00 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0
[11-Nov-2023 18:22:35 America/Chicago] PHP Fatal error: Uncaught Error: Undefined constant "CURLOPT_USERAGENT" in /usr/local/pkg/pfblockerng/pfblockerng.inc:167
Stack trace:
#0 /usr/local/www/widgets/widgets/pfblockerng.widget.php(36): require_once()
#1 /usr/local/www/index.php(430): include('/usr/local/www/...')
#2 {main}
thrown in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 167
[12-Nov-2023 00:22:35 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20220829/curl.so (Shared object "libssl.so.30" not found, required by "libcurl.so.4"), /usr/local/lib/php/20220829/curl.so.so (Cannot open "/usr/local/lib/php/20220829/curl.so.so")) in Unknown on line 0No FreeBSD crash data found.
-
@Flash-1 I've rebooted the router, I've tried shutting restarting the services, but the pfblocker related services, and well as the nut service remain off. Note: nut seems to have been a reoccurring problem since several versions ago when it was broken, reported, but I've not seen in a change long where it's been fixed (thought I may have overlooked it)
-
@Flash-1 said in Netgate 6100 crash report:
Note: nut seems to have been a reoccurring problem since several versions ago when it was broken, reported, but I've not seen in a change long where it's been fixed (thought I may have overlooked it)
NUT is an add-on package. Versions of pfSense and versions of add-on packages are not locked together, so it's kind of rare to find packages mentioned in the pfSense change log.
There is a new version of NUT in testing. You can find information regarding this in the NUT support thread.
-
@Flash-1 Did you by chance install or upgrade a package recently, after 23.09 was released?
-
@dennypage
Thank you sir, I’ll check that thread -
@SteveITS
Yes, pfBlocker, and WireGuard. I realize I could have been more specific on my 1st post. I’ve rebooted again, it seems to have hung on boot because no lights after 15 mins. I just power cycled it and I have the blue light in the left, but it doesn’t seem to be issuing IP’s… I can’t reach the login screen… working that now. I’m pretty certain I’m running 23.05 (the log above mentions that, but when I can get into the interface I’ll confirm). -
@Flash-1 said in Netgate 6100 crash report:
Yes, pfBlocker, and WireGuard.
Never ever never upgrade packages without upgrading pfSense itself first when any pfSense upgrade is offered! Updating packages in the scenario where a new pfSense version is released (but you haven't yet updated your firewall) results in the package update pulling in updated shared libraries that are then the wrong versions for pfSense itself. This leaves you with a broken install.
The Netgate team has looked at several ways of preventing this, but so far nothing seems to make it 100% foolproof to keep users from shooting both feet off by upgrading packages before updating pfSense itself when new pfSense versions are released. I can say that if you are a frequent visitor to these forums you can find many previous posts containing this very warning -- never update packages when a new pfSense version is released until after you first update pfSense itself to the latest version.
The solution now is going to probably be a full reinstall from a new 23.09 image and then import your most recent config backup.
-
@bmeeks
Thank you, for the record, I hit the button to check for it did not fine any pfsense updates. I then updated pfblocker and wireguard. Maybe 23.09 was available and it didn't find it? I could have gone out to the website and checked myself (instead of relying on the interface) but I didn't. When StevelTS mentioned 23.05 vs 23.09, I went to look on my phone and found a v23.09. At that point I had already pulled power to restart the router and can't get back into any sort of interface.Now it doesn't seem to respond to a factory reset either.... this is fun. sigh...
-
@Flash-1 I wish they’d make it harder to do that…
https://docs.netgate.com/pfsense/en/latest/solutions/netgate-6100/reinstall-pfsense.html
(Free ticket)
-
@Flash-1 said in Netgate 6100 crash report:
At that point I had already pulled power
That is also a very bad thing to do with pfSense appliances. They are full-fledged computers with spinning disks (even though in reality the disks are SSD, they are still written to very frequently for file system updates). When you just pull power without doing a proper shutdown sequence, you can wind up with a corrupted disk. That sounds like your problem now.
Connect the USB serial console cable that came with the firewall to your PC. You may need to download the USB serial console driver again on your PC. Then connect using PuTTY and try to see what is showing on the SG-6100's console.
-
@bmeeks said in Netgate 6100 crash report:
That is also a very bad thing to do with pfSense appliances
It's supposedly not as risky with ZFS. I don't recall when the 6100 came out relative to ZFS...want to say that was 22.01? So the 6100 I think was after that? If it wasn't using ZFS then reinstalling will use ZFS by default now.
@Flash-1 The square LED flashing orange is the upgrade available light (https://docs.netgate.com/pfsense/en/latest/solutions/netgate-6100/io-ports.html#led-patterns) though pfSense would have had to detect that, and it would be showing on the dashboard also then.
@bmeeks said in Netgate 6100 crash report:
find many previous posts containing this very warning
I put it in my sig a while back so it's in all of mine. :-/
ref: https://redmine.pfsense.org/issues/10464
-
@SteveITS said in Netgate 6100 crash report:
@Flash-1 I wish they’d make it harder to do that…
https://docs.netgate.com/pfsense/en/latest/solutions/netgate-6100/reinstall-pfsense.html
(Free ticket)
I have my home network on a backup router now. THANKYOU everyone for the quick responses, that made it easy for me to quickly get the back-up plan in place (Wife and I both work Remotely).I submitted a TAC ticket per the instructions you kindly linked to, THANK YOU for that.
Yeah, that would be nice but it's likely easier said than done (since they could omit those parts from the manual you so kindly provided the link for.... at least they were nice enough to include what to do in the documentation). Now, I'm not a programmer by any stretch of the imagination, but if was as simple as requiring an OS Version Flag for any Pfsense packages, they would have done it....or maybe more accurately, if they could do that in a way that wouldn't require extra work on these packages, they would have done it.... or maybe I'm the "fresh eyes" that provided a helpful suggestion (though I doubt it).
I grabbed the console cable jump down the console rabbit-hole.
-
While I waited on support to reply (which they did in literally 60 seconds) I connected to the console and started looking... something was definitely wrong. I save all of my previous configs prior to updating, and the last config I saved was from v23.05. So, I think when I checked for a pfsense update on the interface it didn't find it.. maybe a dropped packet somewhere along the way? I'm new to pfsense, but I normally check myself and, naturally, the time I didn't I got burned. I have the router reinstalled and the config file loaded.
I'll get some sleep and swap routers tomorrow. Thanks everyone, I'll post back when it's done.
-
If you have a config backup then re-installing to 23.09 clean and restoring the config is almost always the fastest way back up from a situation like that.
Steve
-
@stephenw10
It didn't take long for sure. Thank you to everyone here who assisted. I GREATLY appreciate it.I downloaded the image, followed the instructions (there were some subtle differences), but ultimately it worked. The Netgate has been swapped back over as of yesterday morning and working flawlessly.
I appreciate everyone's help.
