Please Help: "Firewall Logs" Dashboard Widget Not Updating
-
Thanks to all for the assistance. I looked at the error and GUI logs and didn't see anything suggestive of a detected problem. I'm going to do some further research, possibly looking into a fresh install to an external USB drive attached to the Protectli.
-
@sissy said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
@nimrod said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
I also have Protectli FW4B running pfSense 2.5.2-RELEASE and i have tried everything you described and dashboards updates without any problems. Data gets updated if i change interval, number of entries, and even widget name. Data also gets updated if i open any menu in pfSense and close it down. It also gets updated if i refresh the page.
But does it auto-update if you just sit there and watch the widget without doing anything -- just start the dashboard and watch the firewall entries scroll? I assume that it does, but I want to be sure that we are on the same page.
Because my firewall is getting constantly spammed with IPv6 garbage, this was easy to check.
I have IPv6 disabled on mine. But I still get enough logged hits from botnets, script kiddies, and port scanners to keep things lively enough.
I would suggest clean install of pfSense since its obvious that your client machines are not the issue here.
I really hesitate to do a clean install on a firewall in front of multiple servers on a business Internet connection. I've got a massive number of rules, aliases (GeoIP, ASN, blacklist feeds, local blacklists and whitelists, etc.), and servers. I'll consider it, but I don't have a spare Protectli where I could do the install, bring it online, test it out, and then swap it back out if something went horribly wrong. If something does go wrong, I'm looking at websites, email for multiple domains, cloud storage, etc. going down for several domains.
Thank you so much for your very thorough testing and your detailed reply to me. I really appreciate it.
So i loaded widget again, set the refresh timer to 5 seconds, and just let it stand still. And yes, it updates every 5 seconds automatically. It scrolls down every time new entry shows up. No need to click anywhere or manually refresh the page.
Also. Protectli FW4B has classic SATA connector so you can take one of those spare SATA SSDs you have and do some testing with it.
Proprietary SATA power cable is provided with FW4B model as well as short SATA cable. So basically you have everything needed for testing. Also. FW4B has mounting holes for 3.5 inch SSD`s on the bottom cover. So if i were you, I would install both M2 and SATA SSD. Install pfSense on both of them. Configure the pfSense on primary M2 SDD, save the config.xml file, boot from secondary SATA SSD, restore config on it, and leave it like that as a spare drive in case something goes wrong. If Protectli box goes wrong, you can always take the internal SATA drive, attach it to another machine, reconfigure interfaces if needed, and you are good to go. As @johnpoz said, that is not a home network. You need to have backup option in case something goes wrong.
-
@nimrod said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
not a home network. You need to have backup option in case something goes wrong.
Even a home network should have backup plan ;) I use to have a USG3p router that was sitting on my shelf I could of leveraged if my sg4860 died (knock on wood).. But I set it up for my son when he bought his house.. I really should get him a sg1100 or something - but his network is so basic not really needed..
Current plan is just leverage pfsense VM on my nas.. I could have that fired up in a few minutes if need be. Worse thing that could fail would prob be my core switch.. But I have some entry level smart switches on the shelf that I could leverage to get most everything back up, etc. Wifi is 3 different AP, so even if one of those died, I could just move say the one in the back of the house in the guest room to kitchen or hall were more clients connect, etc.
Always have backup/contingency plans ;) I can leverage my phone tethered off my pfsense if my isp went out for any sort of extended time (again knock on wood).. That has been tested, and can have that online in a few minutes as well ;)
Think really gone off the rails from this threads topic - sorry ;)
-
Also. Protectli FW4B has classic SATA connector...
I built mine up from barebones, so I'm familiar with the internals and the options, but thank you for taking the time to so clearly document the SATA connections.
One reason why I went with mSATA was cooling. I'm not a bit fan (no pun intended) of adding heat sources to a fanless box.
I'd rather go with an external USB flash drive for a secondary install. That has the advantage of allowing me to configure it, test it, and then physically unplug it. I've seen too many cases of internal "backup" drives being damaged, electrically or through data corruption, to make me comfortable with having it 'on the bus' the whole time that the box is running.
Always have backup/contingency plans...
I've already got a configured backup firewall/router in case of failure. But it's not a pfSense box.
I have two RAID6 NASs, so I'm very familiar with the notion of backup/contingency plans. My primary system is Xeon-based with ECC RAM because I don't trust non-ECC RAM (too much time working on satellites to ignore the possibility of bit-flips from cosmic rays). It alternates backups between a locally attached USB hard drive and one of the RAID6 NASs. I've got current, bootable image backups of all three of my servers on external media, along with a spare, hardware-identical system into which I can plug the media should a server fail.
Backup firewalls are like backup parachutes -- really important to have, but not something you want to plan on deploying.
-
I discovered that the problem was caused by having an update interval that was too short. Any interval slower than 5 seconds on (on my particular hardware/pfSense configuration), causes the Firewall Logs widget to cease updating. My goal was to approach real-time reporting (it's on a network with very little traffic), so I was trying to use intervals of one and two seconds.
Were I back in my old assembly language days, I'd say that it looks like a timer interrupt service routine that doesn't complete before the time expires again, causing the interrupt to be reenabled after the timer has already expired. Hence, no edge to trigger the interrrupt. I realize that's almost certainly not what's happening here -- it's just an analogy.
It's one of those things that was simple once I found it, but I spent hours debugging before I happened upon the solution.
Thanks to all who helped on this. Perhaps a fix could be made to prevent this in the future. I tend to think that the acceptable interval time might vary by host hardware or even how busy the pfSense host is. But I could be wrong.
P.S. I do know that the problem still exists in 2.6.0 beta; as part of the troubleshooting, I installed that.
-- Sissy
-
@sissy - good find. But I just set mine to 5 seconds and not seeing any issues.. So its not something generic in pfsense.. Running on sg4860..
-
@johnpoz said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
@sissy - good find. But I just set mine to 5 seconds and not seeing any issues.. So its not something generic in pfsense.. Running on sg4860..
Mine works at 5 seconds or greater. Setting it to four or less causes it to not update.
This may be performance-based, so someone with faster hardware, or a less compute-intensive pfSense configuration, might be able to use a shorter update interval (maybe even 1 second). There are things I could do, such as turning off PowerD CPU throttling, simplifying my firewall configuration, experimenting with RAM disks. But I'm lazy, so I probably won't.
Perhaps it could be fixed in pfSense by changing the update interval to a delay between updates; the Firewall Log update finishes and then it waits however many seconds is specified before starting the next one. I'm looking at this as a black box rather than looking at source code, so I could be completely wrong.
-
Yeah, good catch, that's interesting. The widget should have a 5s minimum interval at least if it won't update faster.
https://redmine.pfsense.org/issues/12673
-
So I set it to 3, and from just quick test of sending some traffic to wan from outside.. Yeah its not updating.
I commented on the redmine with a link to this thread.
-
@johnpoz - TL;DR Mine stopped updating after some hours with a 5 second Update Interval. The problem may be load-dependent.
I just returned to my iMac Pro at 3:10AM, where I had left the pfSense dashboard displayed. The Firewall Logs widget, with a 5 second interval setting, had ceased updating about 7 hours ago.
Perhaps it stopped because my WAN port received a whole lot of unsavory, logged traffic at once -- more than the widget could process and display within one Update Interval timer cycle? Or maybe some other series of processes were depleting available CPU cycles, preventing the widget from completing the update fast enough?
I have now set the Update Interval to 10 seconds. I will see if updates cease at that setting.
I believe that the problem could be resolved by changing the Update Interval to a "rest" period after each update completes. That would free up CPU cycles (which is the raison d'être of the setting, after all) while not creating some sort of race condition between the Update Interval timer and the widget update display process.
Note: The Update Interval isn't really an interval. On my system, when it is set to five seconds, the updates are each several seconds "late" relative to five seconds (maybe coming every 7-9 seconds). That's not a problem, but it might be a clue.
-
Yeah, it 'feels' like something that takes too long to process which would be load dependent. Anything less that 5s always seems to fail though.
Add anything that might be relevant to the bug there.Steve
