Status: Monitoring is completely broken, pfSense 2.4.5

jimp · Mar 31, 2020, 5:55 PM

Might be your theme.

johnpoz · Mar 31, 2020, 6:06 PM

Yeah what theme is that, and what are you viewing on - phone/tablet? PC?

scurrier · Mar 31, 2020, 6:16 PM

@jimp

@jimp said in Status: Monitoring is completely broken, pfSense 2.4.5:

Might be your theme.

Changed the theme from pfSense-dark to pfSense. No dice.

@johnpoz said in Status: Monitoring is completely broken, pfSense 2.4.5:

Yeah what theme is that, and what are you viewing on - phone/tablet? PC?

It was pfSense-dark and I changed it now to pfSense. I'm viewing on a 15" 4k laptop. But last I checked the problem also existed on mobile, a Pixel 3 in Firefox.

jimp · Mar 31, 2020, 6:17 PM

If you open the browser console are there any errors there?

It's odd that it shows graph data but nothing in the box below... Almost has to be some kind of JavaScript issue, script blocking/ad blocking kind of thing if it's not the theme.

scurrier · Mar 31, 2020, 6:36 PM

@jimp Good idea, I need to remember to add this to my standard first line troubleshooting.

Console shows this.

Here's an expansion of the last one in red.

Some of those errors reminded me that I do have a security exception set up in Firefox. I have my own CA set up and pfSense has a cert from that CA. THe cert was made in 2015 and I never updated it when the Subject Alternate Name requirements came along, which I think is why I had made the exception. Regardless, none of the rest of pfSense has an issue with this and I'd think the exception should prevent the cert from causing problems on the page but I mention it in case someone knows more than me and it's relevant.

jimp · Mar 31, 2020, 6:37 PM

That isn't likely related. Though that last line with the TypeError seems more like a lead. I don't see that on mine which are working.

jimp · Mar 31, 2020, 6:41 PM

Are there any errors in the pfSense system log when you try to access that page?

Are you logging in as admin or another user?

scurrier · Mar 31, 2020, 6:43 PM

@jimp I forgot to include the console output upon trying to update the graph. Here it is, nothing interesting to my eye.

Literally nothing happens in the System > General logs when I initially open the page and then try to update the graph.

I am logged in as the base admin account.

scurrier · Apr 21, 2020, 5:13 PM

Anything else I could try?

bmeeks · Apr 21, 2020, 5:13 PM

@scurrier said in Status: Monitoring is completely broken, pfSense 2.4.5:

Anything else I could try?

I would suggest studying up on HSTS here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security, and then checking to see how this setting ties in with your setup. If Status Monitoring itself was broken in pfSense, there would be a ton of posts here about the issue. The fact there is not indicates your issue is very likely a local one. Perhaps it is being caused by the CA differences you mentioned? That's where I would be looking. The HSTS message will prevent certain things from loading in the browser, so that will break Status Monitoring for sure.

Gertjan · Apr 22, 2020, 8:57 AM

@scurrier : check System > Routing > Gateways
IS WAN really WAN (an old interface name ?) and now called WAN_DHCP ? (example) ?
What are the gateway names ?
Is monitoring activated for them ?

edit :

ls -al /var/db/rrd

old, abandoned rrd files (== abandoned interface names) might exist.

scurrier · Jul 22, 2020, 2:42 AM

Coming back to this issue since I am wanting to see just how unreliable my ISP has been since COVID-19 and I want to use the quality graphs to do it.

Regarding the security warning I mentioned earlier, that was just because I was accessing the web gui using a host name that I didn't put in the certificate. I tried using the IP I put in the certificate Common Name and now the browser fully accepts it:

...yet, the problem persists.

@Gertjan said in Status: Monitoring is completely broken, pfSense 2.4.5:

@scurrier : check System > Routing > Gateways
IS WAN really WAN (an old interface name ?) and now called WAN_DHCP ? (example) ?
What are the gateway names ?
Is monitoring activated for them ?

edit :
ls -al /var/db/rrd
old, abandoned rrd files (== abandoned interface names) might exist.

Thanks for your post. Sorry I somehow missed it earlier. To answer your questions:

Here's the WAN gateway:

When I have the Category of the Monitoring set to Packets or Traffic, I see WAN in the Graph dropdown. When I change the Category to Quality, I see WAN_DHCP. I guess that's because some Categories deal with Gateways and others deal with Interfaces. My WAN_DHCP gateway is on the WAN interface. Seems OK to me.

Here's my RRD files.

scurrier · Jul 22, 2020, 3:20 AM

Oh, snap. Looks like this bug #7656 is exactly what I'm encountering. The line numbers are different, but for me line 1142 of status_monitoring.php involving the same variables the bug discusses and they appear to be the culprit for me.

I don't know JavaScript or how to debug it in a browser, but I can fiddle with the best of 'em.

I set a breakpoint on 1141 of status_monitoring.php. This shows that timeFormat is undefined.

The reason timeFormat is undefined appears to be that data[0].step is 7200...

...yet timeLookup does not contain this value:

So... yada, yada, yada, this causes my problem?

Gertjan · Jul 22, 2020, 6:03 AM

@scurrier said in Status: Monitoring is completely broken, pfSense 2.4.5:

The line numbers are different

That code is very ancient.
What about hitting Ctrl-F5 to force-flush the browser cache - or even flush the cash, or use another browser to reproduce.

scurrier · Jul 22, 2020, 12:39 PM

@Gertjan I just confirmed that Ctrl + F5 does not solve the issue.

I've previously tried other browsers, no luck. It's a bug.

Gertjan · Jul 22, 2020, 1:44 PM

Were you using HTHS in the past (the browser you are using thinks so !) ? If so, the browser will recall that, and refuse new, non HSTS certificate. Which means that js files won't get loaded -> I'm reading out loud the messages from your logs - , which means functions and variables are not defined. That's what your issue is all about.

Ctrl-F5 will not wipe browser's SHTS memory. To wipe that one : go Google.

I known that IP's can be put into the SAN of a cert (some cert authorities are refusing that).
IMHO : bad practice. With upcoming IPv6 some what totally not manageable anyway.

What happens when you use a 'real' certificate, one that is signed by a known CA ?
Letsencrypt gives them for free this week !

Another thing to check : all pfSense files are actually updated, upgraded ?
A clean install could eliminate this question.

johnpoz · Jul 22, 2020, 2:35 PM

I would just set pfsense to not do https, hit via just plain jane http... Clear all the hsts stuff from your browser... Do you have issues now?

Crazy shit is going to happen if your js are not loading..

this takes all your certs and exceptions and non san in your certs out of the equation... I run my own ca on pfsense and create my own certs.. But It just seems problematic to do hsts on web gui that is only accessible by me, from my own secure network.. Currently its enabled, since I didn't check the box... But if your having issues loading stuff and your seeing errors about strict-transport-security... Then take that out of the equation for "testing".

Gertjan · Jul 22, 2020, 3:21 PM

@johnpoz said in Status: Monitoring is completely broken, pfSense 2.4.5:

But It just seems problematic to do hsts on web gui that is only accessible by me, from my own secure network.

Very true.
But it's is a button. So it takes hits, euh ... clicks ...

jimp · Jul 22, 2020, 3:28 PM

Being a security device, best practice is to enable security features like HSTS. Doesn't matter how (un)exposed the GUI is or where it can be reached from. Sure, you could use HTTP and telnet locally if you really wanted on an isolated management network, but it's better to do it as securely as possible.

scurrier · Jul 22, 2020, 4:19 PM

I can't check at the moment, but I am pretty sure the HSTS warning was there because I was accessing the webgui via a new host name that was not in the certificate. I made an exception in the browser to do this. Because of that, the browser rightly rejected the HSTS header because it didn't trust the site enough to honor such a permanent policy as HSTS.

I doubt HSTS has anything to do with this and I believe I've demonstrated the smoking gun above.