System states graph no longer working

[gisuck]

Hey all,

After upgrading to 23.01, I noticed that the System States graph in monitoring is no longer collecting data. Filter states, source and destination address is zeroed out. Only state changes are being logged.

login-to-view

A post on reddit asked me to try doing a reset data which did not resolve the issue. I'm not overly concerned since I'm just a home user but it's nice to look out for abnormalities on the network from time to time.

I'm open to suggestions on resolution.

Cheers.

[MoonKnight]

@gisuck

Same here.

login-to-view

[jimp]

Looks like something changed in egrep which is breaking the script lines that parse the state data.

I opened https://redmine.pfsense.org/issues/13966 to look into it.

[jimp]

So we found that not only was it newly broken, but the pattern wasn't properly matching NAT states for years. Reid crafted a new pattern that we've tested internally with good results, and it's in the repo now.

You can install the System Patches package and then create an entry for d9fa4584e3fb63d6051e9f1db7655f931cb1be19 to apply the fix.

After applying the patch you will need to either reboot or manually restart RRD graphing (Status > Monitoring, click the Wrench, then Display Advanced, then Disable Graphing and then repeat that but click Enable Graphing). Either way the RRD data collection script will get rewritten and should contain the new pattern.

[gisuck]

@jimp Seems to be working. Just applied the patch now and can see numbers being populated in the 1 hour time period.

Thanks!

[jimp]

You'll probably want this one, too. The graph wasn't even displaying the NAT data, but now that it's valid again, it should be turned back on:

diff --git a/blah/usr/local/www/rrd_fetch_json.php b/blah/usr/local/www/rrd_fetch_json.php
index df0401f96e89..4d7574819ac9 100644
--- a/blah/usr/local/www/rrd_fetch_json.php
+++ b/blah/usr/local/www/rrd_fetch_json.php
@@ -226,7 +226,8 @@ foreach ($side as $settings) {
                                $ds = "state changes";
                                break;
                        case "pfnat":
-                               $ignore = true;
+                               $unit_acronym = "";
+                               $ds = "NAT states";
                                break;
                        case "inpass":
                                $ninetyfifth = true;

[gisuck]

@jimp This is also working as well. Thanks for all of this! :)

[tman222]

Applied both patches and can also confirm that states (and NAT states) are now showing (again) on the RRD graphs. Thanks @jimp for the quick fix.

[michmoor]

@jimp I may not be looking at this correctly but what is the commit id for the NAT data?

[S762]

@jimp said in System states graph no longer working:

You'll probably want this one, too. The graph wasn't even displaying the NAT data, but now that it's valid again, it should be turned back on:

sorry but how do I/we run this one? I added the Patch package and seemed to have done that correctly but I'm missing something there. is that run from the command prompt?

[tman222]

Hi all -

See this Redmine issue for the NAT data patch:

https://redmine.pfsense.org/issues/13969

This fix was actually pushed to the dev snapshots. However, you can still apply it manually to 23.01.

Just go System > Patches > Add New Patch. Give the patch a description in the "Description" field, and then in the "Patch Contents" field copy the entire diff --git... code snippet posted above (this same code snippet is also listed in the Redmine issue linked to above). Click "Save" and then you can "Apply" the patch. Once the patch is applied, you'll have to disable and then re-enable graphing again to see the changes (as needed to be done with the other patch).

Hope this helps.

[jimp]

@michmoor said in System states graph no longer working:

@jimp I may not be looking at this correctly but what is the commit id for the NAT data?

It's in the ports repo:

https://github.com/pfsense/FreeBSD-ports/commit/6b39f88b22ace4d2b002ab3e7923db3d1d744bf3

But the Makefile change isn't relevant for patching a live system, so I posted a simplified version of the patch here in the thread and on Redmine.

@s762 said in System states graph no longer working:

@jimp said in System states graph no longer working:

You'll probably want this one, too. The graph wasn't even displaying the NAT data, but now that it's valid again, it should be turned back on:

sorry but how do I/we run this one? I added the Patch package and seemed to have done that correctly but I'm missing something there. is that run from the command prompt?

Create an entry in the system patches package, paste in the patch, then apply it.

https://docs.netgate.com/pfsense/en/latest/development/system-patches.html

[michmoor]

@jimp gotcha. Thanks

edit: Applied the patch and works without issues.

1. What is NAT state tracking exactly? How many translations are taking place?
2. The documentation needs updating to account for NAT along with a description.
   https://docs.netgate.com/pfsense/en/latest/monitoring/graphs/list.html#system-graphs

[jimp]

@michmoor said in System states graph no longer working:

@jimp gotcha. Thanks

edit: Applied the patch and works without issues.

1. What is NAT state tracking exactly? How many translations are taking place?

How many states in the state table involve a NAT translation

2. The documentation needs updating to account for NAT along with a description.
   https://docs.netgate.com/pfsense/en/latest/monitoring/graphs/list.html#system-graphs

Can you open an issue under https://redmine.pfsense.org/projects/pfsense-docs about that?

[johnpoz]

@jimp did it for them

https://redmine.pfsense.org/issues/13987

[michmoor]

@johnpoz Thank you

[johnpoz]

@michmoor NP - wasn't sure how familiar you were with creating redmines, if you have an account even.. figured just do it..