Crash report after upgrading to 2.6 yesterday
-
@lizznl I'm seeing the same. Refreshed the dashboard and there was an error present directly in the
Versionwidget:
I checked out the code in /etc/inc/pkg-utils.inc and it's definitely referring to a function that doesn't exist. At least temporarily to stop getting the messages, I created a patch to comment out the function which is presumably there to check to see if DNS is working.
--- /etc/inc/pkg-utils.inc.orig 2022-02-18 02:38:26.578257000 -0500 +++ /etc/inc/pkg-utils.inc 2022-02-18 02:40:58.082342000 -0500 @@ -1327,9 +1327,9 @@ function get_system_pkg_version($baseonly = false, $use_cache = true) { global $g; - if (!check_dnsavailable()) { - return false; - } +// if (!check_dnsavailable()) { +// return false; +// } $cache_file = $g['version_cache_file']; $rc_file = $cache_file . '.rc';I checked the code in 2.5.2 for
pkg-utils.incand that function wasn't mentioned there either so it must have been added in 2.6.0. I see the actual function is in system.inc so I am guessing that include file isn't being included as it should be. -
Actually, I think it is a problem with pfsense API patching the
system.incand removing the function. See pfsense API. I filed an issue. -
@mbentley Thanks a lot for looking into this. Found your issue reported aswell :) This happens also with the auto backup page. And update page;
Crash report begins. Anonymous machine information:
amd64
12.3-STABLE
FreeBSD 12.3-STABLE RELENG_2_6_0-n226742-1285d6d205f pfSenseCrash report details:
PHP Errors:
[18-Feb-2022 09:02:40 Europe/Amsterdam] PHP Fatal error: Uncaught Error: Call to undefined function check_dnsavailable() in /usr/local/www/services_acb.php:232
Stack trace:
#0 {main}
thrown in /usr/local/www/services_acb.php on line 232
[18-Feb-2022 09:07:00 Europe/Amsterdam] PHP Fatal error: Uncaught Error: Call to undefined function check_dnsavailable() in /usr/local/www/services_acb.php:232
Stack trace:
#0 {main}
thrown in /usr/local/www/services_acb.php on line 232
[18-Feb-2022 09:14:08 Europe/Amsterdam] PHP Fatal error: Uncaught Error: Call to undefined function check_dnsavailable() in /usr/local/www/services_acb.php:232
Stack trace:
#0 {main}
thrown in /usr/local/www/services_acb.php on line 232
[18-Feb-2022 09:14:55 Europe/Amsterdam] PHP Fatal error: Uncaught Error: Call to undefined function check_dnsavailable() in /etc/inc/pkg-utils.inc:1330
Stack trace:
#0 /usr/local/www/pkg_mgr_install.php(95): get_system_pkg_version(true, false)
#1 {main}
thrown in /etc/inc/pkg-utils.inc on line 1330No FreeBSD crash data found.
-
@lizznl Ah good catches. It's missing at least these two functions:
Not sure I would want to do a patch of a patched file but I've temporarily added those functions in on my
/etc/inc/system.incand that removes the need to apply that patch that removes the dns check function from being triggered and it also appears to fix the issue elsewhere. I updated the github issue to reflect that it looks like at least those two functions needing to be added to the patched version that gets applied for the pfsense API. -
If someone really does want a patch file that can be applied to an already patched
system.inc, here is the patch. I would only use this for pfsense 2.6.0 after the v1.3.3 pfsense api package has been installed:--- /etc/inc/system.inc.orig 2022-02-18 08:36:30.894924000 -0500 +++ /etc/inc/system.inc 2022-02-18 03:21:44.667401000 -0500 @@ -2704,4 +2704,52 @@ return $arp_table; } +function _getHostName($mac, $ip) { + global $dhcpmac, $dhcpip; + + if ($dhcpmac[$mac]) { + return $dhcpmac[$mac]; + } else if ($dhcpip[$ip]) { + return $dhcpip[$ip]; + } else { + $ipproto = (is_ipaddrv4($ip)) ? '-4 ' : '-6 '; + exec("/usr/bin/host -W 1 " . $ipproto . escapeshellarg($ip), $output); + if (preg_match('/.*pointer ([A-Za-z_0-9.-]+)\..*/', $output[0], $matches)) { + if ($matches[1] <> $ip) { + return $matches[1]; + } + } + } + return ""; +} + +function check_dnsavailable($proto='inet') { + + if ($proto == 'inet') { + $gdns = array('8.8.8.8', '8.8.4.4'); + } else { + $gdns = array('2001:4860:4860::8888', '2001:4860:4860::8844'); + } + $nameservers = array_merge($gdns, get_dns_nameservers()); + $test = 0; + + foreach ($gdns as $dns) { + if ($dns == '127.0.0.1') { + continue; + } else { + $dns_result = trim(_getHostName("", $dns)); + if (($test == '2') && ($dns_result == "")) { + return false; + } elseif ($dns_result == "") { + $test++; + continue; + } else { + return true; + } + } + } + + return false; +} + ?>There are no guarantees that this covers everything that is different and there may be other things different that are not covered by this patch that could cause problems. As always, make sure you have a backup!
-
@mbentley I resorted to restoring 2.5.2 from USB image and restoring that respective backup config. Is this something that you would suggest waiting until another release to fix?
-
@camelsaq - It's hard to say. I haven't done an extensive testing of the pfsense API package besides verifying I could create and delete static DHCP configs via API (that's my main use case). I will say that it does look like there are a number of differences from 2.5.2 and 2.6.0 in the
system.incfile so using the current pfsense api v1.3.3 might cause unexpected behaviors.I haven't looked at this diff between 2.6.0 and 2.5.2 to see if all of these might be able to be applied cleanly but just to show you the magnitude of the changes for the original
system.incfile (DO NOT USE THIS AS A PATCH!):diff --git a/src/etc/inc/system.inc b/src/etc/inc/system.inc index 39dab4b820..3423f43adb 100644 --- a/src/etc/inc/system.inc +++ b/src/etc/inc/system.inc @@ -5,7 +5,7 @@ * part of pfSense (https://www.pfsense.org) * Copyright (c) 2004-2013 BSD Perimeter * Copyright (c) 2013-2016 Electric Sheep Fencing - * Copyright (c) 2014-2021 Rubicon Communications, LLC (Netgate) + * Copyright (c) 2014-2022 Rubicon Communications, LLC (Netgate) * All rights reserved. * * originally part of m0n0wall (http://m0n0.ch/wall) @@ -672,7 +672,7 @@ function system_dhcpleases_configure() { } } -function system_get_dhcpleases() { +function system_get_dhcpleases($dnsavailable=null) { global $config, $g; $leases = array(); @@ -730,6 +730,7 @@ function system_get_dhcpleases() { $failover = false; $dedup_lease = false; $dedup_failover = false; + foreach ($leases_content as $line) { /* Skip comments */ if (preg_match('/^\s*(|#.*)$/', $line)) { @@ -739,9 +740,14 @@ function system_get_dhcpleases() { if (preg_match('/}$/', $line)) { if ($lease) { if (empty($item['hostname'])) { - $hostname = gethostbyaddr($item['ip']); - if (!empty($hostname)) { - $item['hostname'] = $hostname; + if (is_null($dnsavailable)) { + $dnsavailable = check_dnsavailable(); + } + if ($dnsavailable) { + $hostname = gethostbyaddr($item['ip']); + if (!empty($hostname)) { + $item['hostname'] = $hostname; + } } } $leases['lease'][] = $item; @@ -937,9 +943,10 @@ function system_staticroutes_configure($interface = "", $update_dns = false) { $gateways_arr = return_gateways_array(false, true); foreach ($static_routes as $rtent) { - /* Do not delete disabled routes on boot, - * see https://redmine.pfsense.org/issues/3709 */ - if (isset($rtent['disabled']) && platform_booting()) { + /* Do not delete disabled routes, + * see https://redmine.pfsense.org/issues/3709 + * and https://redmine.pfsense.org/issues/10706 */ + if (isset($rtent['disabled'])) { continue; } @@ -1034,6 +1041,32 @@ function system_staticroutes_configure($interface = "", $update_dns = false) { } } unset($gateways_arr); + + /* keep static routes cache, + * see https://redmine.pfsense.org/issues/11599 */ + $id = 0; + foreach ($config['staticroutes']['route'] as $sroute) { + $targets = array(); + if (is_subnet($sroute['network'])) { + $targets[] = $sroute['network']; + } elseif (is_alias($sroute['network'])) { + foreach (preg_split('/\s+/', $aliastable[$sroute['network']]) as $tgt) { + if (is_ipaddrv4($tgt)) { + $tgt .= "/32"; + } + if (is_ipaddrv6($tgt)) { + $tgt .= "/128"; + } + if (!is_subnet($tgt)) { + continue; + } + $targets[] = $tgt; + } + } + file_put_contents("{$g['tmp_path']}/staticroute_{$id}", serialize($targets)); + file_put_contents("{$g['tmp_path']}/staticroute_{$id}_gw", serialize($sroute['gateway'])); + $id++; + } } unset($static_routes); @@ -1063,6 +1096,52 @@ function system_staticroutes_configure($interface = "", $update_dns = false) { return 0; } +function delete_static_route($id, $delete = false) { + global $g, $config, $changedesc_prefix, $a_gateways; + + if (!isset($config['staticroutes']['route'][$id])) { + return; + } + + if (file_exists("{$g['tmp_path']}/.system_routes.apply")) { + $toapplylist = unserialize(file_get_contents("{$g['tmp_path']}/.system_routes.apply")); + } else { + $toapplylist = array(); + } + + if (file_exists("{$g['tmp_path']}/staticroute_{$id}") && + file_exists("{$g['tmp_path']}/staticroute_{$id}_gw")) { + $delete_targets = unserialize(file_get_contents("{$g['tmp_path']}/staticroute_{$id}")); + $delgw = lookup_gateway_ip_by_name(unserialize(file_get_contents("{$g['tmp_path']}/staticroute_{$id}_gw"))); + if (count($delete_targets)) { + foreach ($delete_targets as $dts) { + if (is_ipaddrv6($dts)) { + $family = "-inet6"; + } else { + $family = "-inet"; + } + $route = route_get($dts, '', true); + if (!count($route)) { + continue; + } + $toapplylist[] = "/sbin/route delete " . + $family . " " . $dts . " " . $delgw; + } + } + } + + if ($delete) { + unlink_if_exists("{$g['tmp_path']}/staticroute_{$id}"); + unlink_if_exists("{$g['tmp_path']}/staticroute_{$id}_gw"); + } + + if (!empty($toapplylist)) { + file_put_contents("{$g['tmp_path']}/.system_routes.apply", serialize($toapplylist)); + } + + unset($targets); +} + function system_routing_enable() { global $config, $g; if (isset($config['system']['developerspew'])) { @@ -1614,8 +1693,8 @@ global $ntp_poll_min_value, $ntp_poll_max_value; global $ntp_poll_min_default_gps, $ntp_poll_max_default_gps; global $ntp_poll_min_default_pps, $ntp_poll_max_default_pps; global $ntp_poll_min_default, $ntp_poll_max_default; -global $ntp_auth_halgos; -$ntp_poll_min_value = 4; +global $ntp_auth_halgos, $ntp_server_types; +$ntp_poll_min_value = 3; $ntp_poll_max_value = 17; $ntp_poll_min_default_gps = 4; $ntp_poll_max_default_gps = 4; @@ -1625,7 +1704,13 @@ $ntp_poll_min_default = 'omit'; $ntp_poll_max_default = 9; $ntp_auth_halgos = array( 'md5' => 'MD5', - 'sha1' => 'SHA1' + 'sha1' => 'SHA1', + 'sha256' => 'SHA256' +); +$ntp_server_types = array( + 'server' => 'Server', + 'pool' => 'Pool', + 'peer' => 'Peer' ); function system_ntp_poll_values() { @@ -1664,6 +1749,11 @@ function system_ntp_setup_gps($serialport) { } init_config_arr(array('ntpd', 'gps')); + $serialports = get_serial_ports(true); + + if (!array_key_exists($serialport, $serialports)) { + return false; + } $gps_device = '/dev/gps0'; $serialport = '/dev/'.basename($serialport); @@ -1751,6 +1841,12 @@ function system_ntp_setup_rawspeed($serialport, $baud) { function system_ntp_setup_pps($serialport) { global $config, $g; + $serialports = get_serial_ports(true); + + if (!array_key_exists($serialport, $serialports)) { + return false; + } + $pps_device = '/dev/pps0'; $serialport = '/dev/'.basename($serialport); @@ -1977,7 +2073,11 @@ function system_ntp_configure() { $ntpcfg .= 'pool '; $have_pools = true; } else { - $ntpcfg .= 'server '; + if (substr_count($config['ntpd']['ispeer'], $ts)) { + $ntpcfg .= 'peer '; + } else { + $ntpcfg .= 'server '; + } if ($config['ntpd']['dnsresolv'] == 'inet') { $ntpcfg .= '-4 '; } elseif ($config['ntpd']['dnsresolv'] == 'inet6') { @@ -1985,7 +2085,10 @@ function system_ntp_configure() { } } - $ntpcfg .= "{$ts} iburst"; + $ntpcfg .= "{$ts}"; + if (!substr_count($config['ntpd']['ispeer'], $ts)) { + $ntpcfg .= " iburst"; + } $ntpcfg .= system_ntp_fixup_poll_value('minpoll', $config['ntpd']['ntpminpoll'], $ntp_poll_min_default); $ntpcfg .= system_ntp_fixup_poll_value('maxpoll', $config['ntpd']['ntpmaxpoll'], $ntp_poll_max_default); @@ -2565,7 +2668,7 @@ function system_identify_specific_platform() { return (array('name' => 'RCC', 'descr' => 'Netgate XG-2758')); break; case 'SG-5100': - return (array('name' => 'SG-5100', 'descr' => 'Netgate SG-5100')); + return (array('name' => '5100', 'descr' => 'Netgate 5100')); break; case 'Minnowboard Turbot D0 PLATFORM': case 'Minnowboard Turbot D0/D1 PLATFORM': @@ -2600,9 +2703,9 @@ function system_identify_specific_platform() { break; case 'SYS-5018D-FN4T': if (strpos($hw_model, "D-1541") !== false) { - return (array('name' => 'XG-1541', 'descr' => 'Super Micro XG-1541')); + return (array('name' => '1541', 'descr' => 'Super Micro 1541')); } else { - return (array('name' => 'XG-1540', 'descr' => 'Super Micro XG-1540')); + return (array('name' => '1540', 'descr' => 'Super Micro XG-1540')); } break; case 'apu2': @@ -2632,7 +2735,7 @@ function system_identify_specific_platform() { $planar_product); if (isset($planar_product[0]) && $planar_product[0] == 'X10SDV-8C-TLN4F+') { - return array('name' => 'XG-1537', 'descr' => 'Super Micro XG-1537'); + return array('name' => '1537', 'descr' => 'Super Micro 1537'); } if (strpos($hw_model, "PC Engines WRAP") !== false) { @@ -2691,4 +2794,52 @@ function system_get_arp_table($resolve_hostnames = false) { return $arp_table; } +function _getHostName($mac, $ip) { + global $dhcpmac, $dhcpip; + + if ($dhcpmac[$mac]) { + return $dhcpmac[$mac]; + } else if ($dhcpip[$ip]) { + return $dhcpip[$ip]; + } else { + $ipproto = (is_ipaddrv4($ip)) ? '-4 ' : '-6 '; + exec("/usr/bin/host -W 1 " . $ipproto . escapeshellarg($ip), $output); + if (preg_match('/.*pointer ([A-Za-z_0-9.-]+)\..*/', $output[0], $matches)) { + if ($matches[1] <> $ip) { + return $matches[1]; + } + } + } + return ""; +} + +function check_dnsavailable($proto='inet') { + + if ($proto == 'inet') { + $gdns = array('8.8.8.8', '8.8.4.4'); + } else { + $gdns = array('2001:4860:4860::8888', '2001:4860:4860::8844'); + } + $nameservers = array_merge($gdns, get_dns_nameservers()); + $test = 0; + + foreach ($gdns as $dns) { + if ($dns == '127.0.0.1') { + continue; + } else { + $dns_result = trim(_getHostName("", $dns)); + if (($test == '2') && ($dns_result == "")) { + return false; + } elseif ($dns_result == "") { + $test++; + continue; + } else { + return true; + } + } + } + + return false; +} + ?>Honestly if you're already back up and running fine on 2.5.2, I would just wait until there is a new release of the pfsense api package for the 2.6.0 release if you depend on it's functionality but that is my opinion from just a user of the package.
-
pfSense API package, what is that ?
Some day there will be one ..... in a decade, maybe ;)During upgrades, most of not all "pfSense" core files are plain overwritten.
The pfSense GUI is mostly (very) 'old school' PHP (don't misunderstand : I love it, Perl is not my thing - PHP is soooooo easily readeable) . No university eduction needed to "copy this" ( and use "find here" so you can see in one glance where the function is used, and that it is defined in /etc/inc/system.inc - line 2816.
Compare your /etc/inc/system.inc with the github source repository. Remember, this is quasi open source : you have access to the original "should work" source code, as it was used to make pfSense version 2.6.0 - and the same code on your system.
So, compare, conclude, act and done.edit : in the future, you should select the branch used to build your pfSEnse version. The files in the main repository are live == altered as development continues. Right now, this isn't the case, as 2.6.0 came out just a couple of days ago.
You will probably find that your system.inc is different / broken / rubbish / whatever.
Conclusion : "something" went very wrong during an update.
Or, if your young : this is the first time a (hard) disk died on you : welcome to the club.Good news : (re) installing pfSense is a easy.
Don't try to repair /etc/inc/system.inc, it's not worth it, as any other files could have been altered to. -
@gertjan We are talking about https://github.com/jaredhendrickson13/pfsense-api as that is probably where the file is being modified from to specifically see this behavior, at least it was in my case - unsure about everyone else. I understand the update process and files being overwritten. The pfsense-api package also replaces the
system.incfile with a custom one to support the package. Replacing the file from source would break the pfsense-api package which is not what is wanted here.So it is likely that nothing went wrong during upgrade, the pfsense-api package hasn't included all of the changes since 2.6.0 was released as I am pretty sure the 2.6.0 package was just sort of a best effort, built for 2.6.0 package that wasn't really fully vetted.
-
@mbentley said in Crash report after upgrading to 2.6 yesterday:
https://github.com/jaredhendrickson13/pfsense-api
Dono who or what that is.
That is not a Netgate Official source. Just some guy who tries to implement an "Rest API" (good initiative btw, but it will take him years if he works 24/24h on it).Now I fully understand the why part of the issue.
https://github.com/jaredhendrickson13/pfsense-api/issues/199What I don't understand : if you install and use a native - non modified - pfSense, the issue doesn't exist.
When you install X, pfSense breaks.
My advise will be : repair X, or don't use it. -
@gertjan although I did have that api package installed, I received the error from outside of the API and inside pfsense. In fact I couldn't even access DHCP services under that menu with similar errors.
I also uninstalled the API before reverting and still received error above
-
@camelsaq So did you upgrade pfsense with the API installed the pfsense-api package and still received the error? If so, I am guessing that it would be because of how the install and removal functions - looks like it takes a backup of the original on install and on removal of the package, it copies it back which would align with your experience as it would be putting back the 2.5.2 file on top of 2.6.0 if I understand the process.
-
@mbentley yes when I upgraded to 2.6 the API was installed.
-
@mbentley when I reverted I did a factory reset back to the USB image. When restoring config backup from before I upgraded to 2.6 this did have the API and possibly had some API stuff in it. I did not receive error on 2.5.2 after restoring config, but did need to install the API back in order to get that working again.
Thanks so much for the insights here.
-
@camelsaq Sure thing - helpful to get your experience to see how it aligns with mine. Upon closer inspection, I see that there isn't an override file specific for 2.6.0 yet which I missed when installing the 2.6.0 pfsense-api package which explains why it uses what is the default "older" version of of the
system.incfile:pkg add https://github.com/jaredhendrickson13/pfsense-api/releases/download/v1.3.3/pfSense-2.6-pkg-API.txz && /etc/rc.restart_webgui Fetching pfSense-2.6-pkg-API.txz: 100% 230 KiB 235.2kB/s 00:01 Installing pfSense-pkg-API-1.3_3... Extracting pfSense-pkg-API-1.3_3: 100% Saving updated package information... done. Loading package configuration... done. Configuring package components... Custom commands... Menu items... done. Writing configuration... done. Creating backups of files to override... done. Checking pfSense version... done. WARNING: No overrides exist for 2.6.0-RELEASE, it may be unsupported. Using default overrides. Installing default file overrides... done. ...So yeah, that aligns a bit more with staying on 2.5.2 if you need a functional pfsense-api package without hacking it.
-
@camelsaq said in Crash report after upgrading to 2.6 yesterday:
back to the USB image. When restoring config backup
At that comment, you have the original /etc/inc/system.inc
If this file is broken, or or does't contain what it should contain, your pfSense wouldn't even boot up. Ir's pretty critical.Also, the function "check_dnsavailable" was added in 2.6.0 - and used since then in several other places. The API software changes core files on a pfSense ? ( ?? ! ???? ) I tend to say that you are not using pfSense any more.
So why even considering posting question here ;))Be careful, guys, when you use an external piece of software not written for the current pfSense software, you have a shoot in the foot situation.
This is valid for pfSense packages, and everything else.Edit : the subject of this thread is wrong - please edit it.
pfSense 2.6.0 didn't crash.
This is a "don't do this" issue that didn't end well.
Happens.
Tried X - X didn't work out. Ditched X for the time being. NeXt. -
@mbentley I removed the pfsense-api package for now and all works fine again after rebooting.
Hope there will come a patch on the pfsense-api side since I started to like pfsense even more when able to use API commands.. Else maybe consider something else which does support API natively.
-
@lizznl good to hear. For some reason I did not reboot after I removed API package and instead reverted pfsense version. May try to upgrade again and uninstall API first.
-
