What is: "pfr_update_stats: assertion failed." ?

fmaxwell

I have been seeing the following messages on my serial console (scroll to see messages):

** Welcome to pfSense 2.2.2-RELEASE-pfSense (i386) on cox ***

 WAN (wan)       -> fxp0       -> v4: {redacted}/1
 LAN (lan)       -> fxp1       -> v4: 192.168.0.1/23
 OPT1 (opt1)     -> fxp2       ->
 OPT2 (opt2)     -> fxp3       ->
 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) pfSense Developer Shell
 4) Reset to factory defaults         13) Upgrade from console
 5) Reboot system                     14) Disable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.
pfr_update_stats: assertion failed.

If left alone for some hours, I see one after another, as shown above.

Looking at the System Log –> General tab, I see things like this (note reverse chronological display order):

Jun 1 06:14:31 	php: pfblockerng.php: [pfBlockerNG] Starting sync process.
Jun 1 06:14:30 	check_reload_status: Syncing firewall
Jun 1 06:14:30 	check_reload_status: Syncing firewall
Jun 1 06:14:00 	check_reload_status: Syncing firewall
Jun 1 06:14:00 	check_reload_status: Syncing firewall
Jun 1 06:13:58 	php: pfblockerng.php: [pfBlockerNG] Starting sync process.
Jun 1 06:13:41 	sshlockout[32901]: sshlockout/webConfigurator v3.0 starting up
Jun 1 06:13:40 	login: login on ttyu0 as root
Jun 1 06:10:00 	syslogd: kernel boot file is /boot/kernel/kernel
Jun 1 06:10:00 	syslogd: exiting on signal 15
Jun 1 06:09:53 	syslogd: kernel boot file is /boot/kernel/kernel
Jun 1 06:09:53 	syslogd: exiting on signal 15
Jun 1 06:01:01 	check_reload_status: Syncing firewall
Jun 1 06:00:37 	php: pfblockerng.php: [pfBlockerNG] Starting sync process.
Jun 1 05:45:06 	kernel: pfr_update_stats: assertion failed.
Jun 1 05:36:41 	kernel: pfr_update_stats: assertion failed.
Jun 1 05:36:35 	kernel: pfr_update_stats: assertion failed.
Jun 1 05:36:32 	kernel: pfr_update_stats: assertion failed.
Jun 1 05:19:02 	kernel: pfr_update_stats: assertion failed.
Jun 1 05:18:31 	check_reload_status: Reloading filter
Jun 1 05:18:31 	check_reload_status: Restarting OpenVPN tunnels/interfaces
Jun 1 05:18:31 	check_reload_status: Restarting ipsec tunnels
Jun 1 05:18:31 	check_reload_status: updating dyndns WANGW
Jun 1 05:18:20 	check_reload_status: Reloading filter
Jun 1 05:18:20 	check_reload_status: Restarting OpenVPN tunnels/interfaces
Jun 1 05:18:20 	check_reload_status: Restarting ipsec tunnels
Jun 1 05:18:20 	check_reload_status: updating dyndns WANGW
Jun 1 05:01:12 	check_reload_status: Syncing firewall
Jun 1 05:00:45 	php: pfblockerng.php: [pfBlockerNG] Starting sync process.
Jun 1 04:53:10 	kernel: pfr_update_stats: assertion failed.
Jun 1 04:40:29 	kernel: pfr_update_stats: assertion failed.
Jun 1 04:31:43 	kernel: pfr_update_stats: assertion failed.
Jun 1 04:31:43 	kernel: pfr_update_stats: assertion failed.
Jun 1 04:31:42 	kernel: pfr_update_stats: assertion failed.
Jun 1 04:23:05 	kernel: pfr_update_stats: assertion failed.
Jun 1 04:01:03 	check_reload_status: Syncing firewall
Jun 1 04:00:37 	php: pfblockerng.php: [pfBlockerNG] Starting sync process.

Just as I'm typing this, it popped up again.

I'm at a loss as to the problem. pfSense seems 100% functional. I've rebooted the firewall. I've looked at the pfBlockerNG log to see if the 'assertion failed' errors correlate with updates – they don't seem to.

Has anyone else seen these messages? Any idea as to the cause?

Thanks in advance.

firewalluser

Seems the developer who wrote the assertion doesnt know themselves.
http://pf.benzedrine.narkive.com/etdSLxlb/pfr-update-stats-assertion-failed

Is your CPU a 32bit or 64bit processor and if 64bit could you update?

Likewise have you checked if your cpu needs a microcode update and same for your bios, any relevant updates which might resolve the problem?

fmaxwell

@firewalluser:

Seems the developer who wrote the assertion doesnt know themselves.
http://pf.benzedrine.narkive.com/etdSLxlb/pfr-update-stats-assertion-failed

That was only 12 years ago, so I'm sure that a fix will be coming out any day now. :)

@firewalluser:

Is your CPU a 32bit or 64bit processor and if 64bit could you update?

Likewise have you checked if your cpu needs a microcode update and same for your bios, any relevant updates which might resolve the problem?

Thanks. It is a 32 bit Celeron M ULV and it never used to exhibit this behavior. It's a re-purposed Riverbed Steelhead WAN accelerator, which is, in turn, an Axiomtek NA-0043G.

See: http://www.axiomtek.net.cn/Download/Spec/na-0043.pdf

I've put an big, almost silent, fan on top, removing the screamer inside, and have swapped in a PATA SSD. It's quiet, cool, silent, and has four 10/100 Intel ports.

Sadly, there are no flash upgrades available – and I truly wish there were as it cannot boot from USB, making clean installations a painful procedure involving connecting optical drives via PATA cables to the inside.

firewalluser

Just because something hasnt happened before doesnt mean it cant happen in the future. Considering the differences between the amd and intel cpu's namely when the amd's get hot they hang where as the intel cpu's will adjust the clock speed down to a crawl to protect themselves, even things like dust building up in a system can cause problems after some years worth of use although I dont think dust is your problem here. I saw on the dragonflybsd board the same dev suggest it might have been a race condition with memory which could happen as these things are not built to last especially since lead was banned from solder by the EU some years back, for a while a lot of pcb's failed as manufacturers had to readjust machines to use more solder to compensate for the lack of weight on the solder joints.

Either way, lots of variables and not enough data to really say, but I'd go with what the dev who put the assertion in the bsd code with thats its likely a race condition and as you can see there's only been one reported incident online according to google until today that is.

fmaxwell

Thanks for your reply.

If I were seeing lots of different failure modes, reboots, lock-ups, etc., I'd suspect hardware and cooling (I oversaw 3,000 PCs installed in industrial environments, so I've seen every kind of failure related to cooling – dust, bearing failures on fans, clogged filters, etc.). I've opened up this case within the last couple of weeks and have thoroughly blown it clean (there was not much dust to start with, either).

Temperature on the CPU is showing at 23.0C. 21% of memory is used. 3% of the disk is used. There's just no indication that the system is straining in the least.

I'm probably going to do a reinstall from ground up and see if I can correlate the problem to packages being added or settings being changed.

firewalluser

DO you have any packet captures to go back over when the problem occurred, that might be telling.