Please Help: "Firewall Logs" Dashboard Widget Not Updating
-
You might also check the dynamic view works as expected in Status > Logs > Firewall.
-
When you say long standing, has it always done it? Started after an update?
I don't know if it has always done it or if it started after an update. I just noticed, more than a year ago, that the Firewall Logs widget was not updating. There wasn't some prior point in time where I consciously noticed that it was updating.
I assume it updates if you reload the dashboard it just doesn't auto-update?
That's correct
You might also check the dynamic view works as expected in Status > Logs > Firewall."
That one works fine. And that's how I normally would monitor firewall activity. So I'd glance on the Dashboard, see some entries there, and I just assumed that they were updating -- until I consciously noticed that they were not.
-
No errors in the System log or GUI log?
I've no idea what could cause that on the firewall really. If feels like a client side issues except it can't be if you've tried multiple clients and browsers. Or at least is very unlikely to be.
Steve
-
@nimrod said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
I also have Protectli FW4B running pfSense 2.5.2-RELEASE and i have tried everything you described and dashboards updates without any problems. Data gets updated if i change interval, number of entries, and even widget name. Data also gets updated if i open any menu in pfSense and close it down. It also gets updated if i refresh the page.
But does it auto-update if you just sit there and watch the widget without doing anything -- just start the dashboard and watch the firewall entries scroll? I assume that it does, but I want to be sure that we are on the same page.
Because my firewall is getting constantly spammed with IPv6 garbage, this was easy to check.
I have IPv6 disabled on mine. But I still get enough logged hits from botnets, script kiddies, and port scanners to keep things lively enough.
I would suggest clean install of pfSense since its obvious that your client machines are not the issue here.
I really hesitate to do a clean install on a firewall in front of multiple servers on a business Internet connection. I've got a massive number of rules, aliases (GeoIP, ASN, blacklist feeds, local blacklists and whitelists, etc.), and servers. I'll consider it, but I don't have a spare Protectli where I could do the install, bring it online, test it out, and then swap it back out if something went horribly wrong. If something does go wrong, I'm looking at websites, email for multiple domains, cloud storage, etc. going down for several domains.
Thank you so much for your very thorough testing and your detailed reply to me. I really appreciate it.
-
You could just get a spare drive and swap that in. Keep the old drive on hand so you can always fall back to it if the new install doesn't go smoothly.
Steve
-
@stephenw10 said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
No errors in the System log or GUI log?
I honestly haven't checked as I really have zero experience with them. I guess I have to learn something, dammit!
I've no idea what could cause that on the firewall really. If feels like a client side issues except it can't be if you've tried multiple clients and browsers. Or at least is very unlikely to be.
It seems pretty unlikely when I've tried it on two different systems, one being an iMac and the other being a Linux box, using a total of at least four different browsers between them.
-
@sissy said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
but I don't have a spare Protectli
Kind a of tangent here, and Steve's HDD suggest is prob the best option. But if you have that much that uses this firewall.. What would happen if it just failed - electronics tend to do that at the worse times..
There is one thing if just a home setup.. But for a business, there should always be failure plan for any hardware - either you can live with it being down to when you can get a replacement. Say you have a 4 hour sla or something.. If your running some DIY firewall, there should be "something" you can pull off a shelf and get everything back up..
Might be a good time to get that "spare" and then use it for your troubleshooting of this problem. And then you will have as backup as well.
-
@stephenw10 said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
You could just get a spare drive and swap that in. Keep the old drive on hand so you can always fall back to it if the new install doesn't go smoothly.
This is when I wish it was a "normal" drive. I've got more than a dozen SATA and USB drives, rotating and SSD, kicking around, including some spares for my NASs. But that box takes an mSATA SSD, of which I have no spares. Maybe I should order one or determine if I can configure it to boot from an external USB drive.
-
Maybe restore the config into a VM and see if the issue follows it. That would at least determine if it's a config problem, though I have no idea what that could be!
But John's point is good. If you have that much relying on it what's your recovery plan if/when it fails?
Steve
-
@johnpoz said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
@sissy said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
but I don't have a spare Protectli
Kind a of tangent here, and Steve's HDD suggest is prob the best option. But if you have that much that uses this firewall.. What would happen if it just failed - electronics tend to do that at the worse times.
I've got a stack of godawful-loud, 1U rack-mount, 8-gigabit port boxes lying around that I could press into service as a temporary firewall. Or I could configure a spare EdgeRouter X I have to fill in, albeit with less complex firewalling. Then I could get another Protectli overnight-shipped. One of the reasons for the Protectli is that there are no moving parts. No rotating hard drive. No fans. It's getting its AC power from a sine wave UPS.
-
Try importing your config into one of those then. If nothing else it proves your recovery plan.
-
Thanks to all for the assistance. I looked at the error and GUI logs and didn't see anything suggestive of a detected problem. I'm going to do some further research, possibly looking into a fresh install to an external USB drive attached to the Protectli.
-
@sissy said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
@nimrod said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
I also have Protectli FW4B running pfSense 2.5.2-RELEASE and i have tried everything you described and dashboards updates without any problems. Data gets updated if i change interval, number of entries, and even widget name. Data also gets updated if i open any menu in pfSense and close it down. It also gets updated if i refresh the page.
But does it auto-update if you just sit there and watch the widget without doing anything -- just start the dashboard and watch the firewall entries scroll? I assume that it does, but I want to be sure that we are on the same page.
Because my firewall is getting constantly spammed with IPv6 garbage, this was easy to check.
I have IPv6 disabled on mine. But I still get enough logged hits from botnets, script kiddies, and port scanners to keep things lively enough.
I would suggest clean install of pfSense since its obvious that your client machines are not the issue here.
I really hesitate to do a clean install on a firewall in front of multiple servers on a business Internet connection. I've got a massive number of rules, aliases (GeoIP, ASN, blacklist feeds, local blacklists and whitelists, etc.), and servers. I'll consider it, but I don't have a spare Protectli where I could do the install, bring it online, test it out, and then swap it back out if something went horribly wrong. If something does go wrong, I'm looking at websites, email for multiple domains, cloud storage, etc. going down for several domains.
Thank you so much for your very thorough testing and your detailed reply to me. I really appreciate it.
So i loaded widget again, set the refresh timer to 5 seconds, and just let it stand still. And yes, it updates every 5 seconds automatically. It scrolls down every time new entry shows up. No need to click anywhere or manually refresh the page.
Also. Protectli FW4B has classic SATA connector so you can take one of those spare SATA SSDs you have and do some testing with it.
Proprietary SATA power cable is provided with FW4B model as well as short SATA cable. So basically you have everything needed for testing. Also. FW4B has mounting holes for 3.5 inch SSD`s on the bottom cover. So if i were you, I would install both M2 and SATA SSD. Install pfSense on both of them. Configure the pfSense on primary M2 SDD, save the config.xml file, boot from secondary SATA SSD, restore config on it, and leave it like that as a spare drive in case something goes wrong. If Protectli box goes wrong, you can always take the internal SATA drive, attach it to another machine, reconfigure interfaces if needed, and you are good to go. As @johnpoz said, that is not a home network. You need to have backup option in case something goes wrong.
-
@nimrod said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
not a home network. You need to have backup option in case something goes wrong.
Even a home network should have backup plan ;) I use to have a USG3p router that was sitting on my shelf I could of leveraged if my sg4860 died (knock on wood).. But I set it up for my son when he bought his house.. I really should get him a sg1100 or something - but his network is so basic not really needed..
Current plan is just leverage pfsense VM on my nas.. I could have that fired up in a few minutes if need be. Worse thing that could fail would prob be my core switch.. But I have some entry level smart switches on the shelf that I could leverage to get most everything back up, etc. Wifi is 3 different AP, so even if one of those died, I could just move say the one in the back of the house in the guest room to kitchen or hall were more clients connect, etc.
Always have backup/contingency plans ;) I can leverage my phone tethered off my pfsense if my isp went out for any sort of extended time (again knock on wood).. That has been tested, and can have that online in a few minutes as well ;)
Think really gone off the rails from this threads topic - sorry ;)
-
Also. Protectli FW4B has classic SATA connector...
I built mine up from barebones, so I'm familiar with the internals and the options, but thank you for taking the time to so clearly document the SATA connections.
One reason why I went with mSATA was cooling. I'm not a bit fan (no pun intended) of adding heat sources to a fanless box.
I'd rather go with an external USB flash drive for a secondary install. That has the advantage of allowing me to configure it, test it, and then physically unplug it. I've seen too many cases of internal "backup" drives being damaged, electrically or through data corruption, to make me comfortable with having it 'on the bus' the whole time that the box is running.
Always have backup/contingency plans...
I've already got a configured backup firewall/router in case of failure. But it's not a pfSense box.
I have two RAID6 NASs, so I'm very familiar with the notion of backup/contingency plans. My primary system is Xeon-based with ECC RAM because I don't trust non-ECC RAM (too much time working on satellites to ignore the possibility of bit-flips from cosmic rays). It alternates backups between a locally attached USB hard drive and one of the RAID6 NASs. I've got current, bootable image backups of all three of my servers on external media, along with a spare, hardware-identical system into which I can plug the media should a server fail.
Backup firewalls are like backup parachutes -- really important to have, but not something you want to plan on deploying.
-
I discovered that the problem was caused by having an update interval that was too short. Any interval slower than 5 seconds on (on my particular hardware/pfSense configuration), causes the Firewall Logs widget to cease updating. My goal was to approach real-time reporting (it's on a network with very little traffic), so I was trying to use intervals of one and two seconds.
Were I back in my old assembly language days, I'd say that it looks like a timer interrupt service routine that doesn't complete before the time expires again, causing the interrupt to be reenabled after the timer has already expired. Hence, no edge to trigger the interrrupt. I realize that's almost certainly not what's happening here -- it's just an analogy.
It's one of those things that was simple once I found it, but I spent hours debugging before I happened upon the solution.
Thanks to all who helped on this. Perhaps a fix could be made to prevent this in the future. I tend to think that the acceptable interval time might vary by host hardware or even how busy the pfSense host is. But I could be wrong.
P.S. I do know that the problem still exists in 2.6.0 beta; as part of the troubleshooting, I installed that.
-- Sissy
-
@sissy - good find. But I just set mine to 5 seconds and not seeing any issues.. So its not something generic in pfsense.. Running on sg4860..
-
@johnpoz said in Please Help: "Firewall Logs" Dashboard Widget Not Updating:
@sissy - good find. But I just set mine to 5 seconds and not seeing any issues.. So its not something generic in pfsense.. Running on sg4860..
Mine works at 5 seconds or greater. Setting it to four or less causes it to not update.
This may be performance-based, so someone with faster hardware, or a less compute-intensive pfSense configuration, might be able to use a shorter update interval (maybe even 1 second). There are things I could do, such as turning off PowerD CPU throttling, simplifying my firewall configuration, experimenting with RAM disks. But I'm lazy, so I probably won't.
Perhaps it could be fixed in pfSense by changing the update interval to a delay between updates; the Firewall Log update finishes and then it waits however many seconds is specified before starting the next one. I'm looking at this as a black box rather than looking at source code, so I could be completely wrong.
-
Yeah, good catch, that's interesting. The widget should have a 5s minimum interval at least if it won't update faster.
https://redmine.pfsense.org/issues/12673
-
So I set it to 3, and from just quick test of sending some traffic to wan from outside.. Yeah its not updating.
I commented on the redmine with a link to this thread.
