What is: "pfr_update_stats: assertion failed." ?
-
I have been seeing the following messages on my serial console (scroll to see messages):
** Welcome to pfSense 2.2.2-RELEASE-pfSense (i386) on cox *** WAN (wan) -> fxp0 -> v4: {redacted}/1 LAN (lan) -> fxp1 -> v4: 192.168.0.1/23 OPT1 (opt1) -> fxp2 -> OPT2 (opt2) -> fxp3 -> 0) Logout (SSH only) 9) pfTop 1) Assign Interfaces 10) Filter Logs 2) Set interface(s) IP address 11) Restart webConfigurator 3) Reset webConfigurator password 12) pfSense Developer Shell 4) Reset to factory defaults 13) Upgrade from console 5) Reboot system 14) Disable Secure Shell (sshd) 6) Halt system 15) Restore recent configuration 7) Ping host 16) Restart PHP-FPM 8) Shell Enter an option: pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed. pfr_update_stats: assertion failed.
If left alone for some hours, I see one after another, as shown above.
Looking at the System Log –> General tab, I see things like this (note reverse chronological display order):
Jun 1 06:14:31 php: pfblockerng.php: [pfBlockerNG] Starting sync process. Jun 1 06:14:30 check_reload_status: Syncing firewall Jun 1 06:14:30 check_reload_status: Syncing firewall Jun 1 06:14:00 check_reload_status: Syncing firewall Jun 1 06:14:00 check_reload_status: Syncing firewall Jun 1 06:13:58 php: pfblockerng.php: [pfBlockerNG] Starting sync process. Jun 1 06:13:41 sshlockout[32901]: sshlockout/webConfigurator v3.0 starting up Jun 1 06:13:40 login: login on ttyu0 as root Jun 1 06:10:00 syslogd: kernel boot file is /boot/kernel/kernel Jun 1 06:10:00 syslogd: exiting on signal 15 Jun 1 06:09:53 syslogd: kernel boot file is /boot/kernel/kernel Jun 1 06:09:53 syslogd: exiting on signal 15 Jun 1 06:01:01 check_reload_status: Syncing firewall Jun 1 06:00:37 php: pfblockerng.php: [pfBlockerNG] Starting sync process. Jun 1 05:45:06 kernel: pfr_update_stats: assertion failed. Jun 1 05:36:41 kernel: pfr_update_stats: assertion failed. Jun 1 05:36:35 kernel: pfr_update_stats: assertion failed. Jun 1 05:36:32 kernel: pfr_update_stats: assertion failed. Jun 1 05:19:02 kernel: pfr_update_stats: assertion failed. Jun 1 05:18:31 check_reload_status: Reloading filter Jun 1 05:18:31 check_reload_status: Restarting OpenVPN tunnels/interfaces Jun 1 05:18:31 check_reload_status: Restarting ipsec tunnels Jun 1 05:18:31 check_reload_status: updating dyndns WANGW Jun 1 05:18:20 check_reload_status: Reloading filter Jun 1 05:18:20 check_reload_status: Restarting OpenVPN tunnels/interfaces Jun 1 05:18:20 check_reload_status: Restarting ipsec tunnels Jun 1 05:18:20 check_reload_status: updating dyndns WANGW Jun 1 05:01:12 check_reload_status: Syncing firewall Jun 1 05:00:45 php: pfblockerng.php: [pfBlockerNG] Starting sync process. Jun 1 04:53:10 kernel: pfr_update_stats: assertion failed. Jun 1 04:40:29 kernel: pfr_update_stats: assertion failed. Jun 1 04:31:43 kernel: pfr_update_stats: assertion failed. Jun 1 04:31:43 kernel: pfr_update_stats: assertion failed. Jun 1 04:31:42 kernel: pfr_update_stats: assertion failed. Jun 1 04:23:05 kernel: pfr_update_stats: assertion failed. Jun 1 04:01:03 check_reload_status: Syncing firewall Jun 1 04:00:37 php: pfblockerng.php: [pfBlockerNG] Starting sync process.
Just as I'm typing this, it popped up again.
I'm at a loss as to the problem. pfSense seems 100% functional. I've rebooted the firewall. I've looked at the pfBlockerNG log to see if the 'assertion failed' errors correlate with updates – they don't seem to.
Has anyone else seen these messages? Any idea as to the cause?
Thanks in advance.
-
Seems the developer who wrote the assertion doesnt know themselves.
http://pf.benzedrine.narkive.com/etdSLxlb/pfr-update-stats-assertion-failedIs your CPU a 32bit or 64bit processor and if 64bit could you update?
Likewise have you checked if your cpu needs a microcode update and same for your bios, any relevant updates which might resolve the problem?
-
Seems the developer who wrote the assertion doesnt know themselves.
http://pf.benzedrine.narkive.com/etdSLxlb/pfr-update-stats-assertion-failedThat was only 12 years ago, so I'm sure that a fix will be coming out any day now. :)
Is your CPU a 32bit or 64bit processor and if 64bit could you update?
Likewise have you checked if your cpu needs a microcode update and same for your bios, any relevant updates which might resolve the problem?
Thanks. It is a 32 bit Celeron M ULV and it never used to exhibit this behavior. It's a re-purposed Riverbed Steelhead WAN accelerator, which is, in turn, an Axiomtek NA-0043G.
See: http://www.axiomtek.net.cn/Download/Spec/na-0043.pdf
I've put an big, almost silent, fan on top, removing the screamer inside, and have swapped in a PATA SSD. It's quiet, cool, silent, and has four 10/100 Intel ports.
Sadly, there are no flash upgrades available – and I truly wish there were as it cannot boot from USB, making clean installations a painful procedure involving connecting optical drives via PATA cables to the inside.
-
Just because something hasnt happened before doesnt mean it cant happen in the future. Considering the differences between the amd and intel cpu's namely when the amd's get hot they hang where as the intel cpu's will adjust the clock speed down to a crawl to protect themselves, even things like dust building up in a system can cause problems after some years worth of use although I dont think dust is your problem here. I saw on the dragonflybsd board the same dev suggest it might have been a race condition with memory which could happen as these things are not built to last especially since lead was banned from solder by the EU some years back, for a while a lot of pcb's failed as manufacturers had to readjust machines to use more solder to compensate for the lack of weight on the solder joints.
Either way, lots of variables and not enough data to really say, but I'd go with what the dev who put the assertion in the bsd code with thats its likely a race condition and as you can see there's only been one reported incident online according to google until today that is.
-
Thanks for your reply.
If I were seeing lots of different failure modes, reboots, lock-ups, etc., I'd suspect hardware and cooling (I oversaw 3,000 PCs installed in industrial environments, so I've seen every kind of failure related to cooling – dust, bearing failures on fans, clogged filters, etc.). I've opened up this case within the last couple of weeks and have thoroughly blown it clean (there was not much dust to start with, either).
Temperature on the CPU is showing at 23.0C. 21% of memory is used. 3% of the disk is used. There's just no indication that the system is straining in the least.
I'm probably going to do a reinstall from ground up and see if I can correlate the problem to packages being added or settings being changed.
-
DO you have any packet captures to go back over when the problem occurred, that might be telling.