1100 upgrade, 22.05->23.01, high mem usage
-
@jrey said in 1100 upgrade, 22.05->23.01, high mem usage:
Yup I went a little further in testing, and pin pointed the exact step.
But I also then just disabled the 450.status-security in the config.
as it was configured out of box as (mail root) no one would ever see the output anyway.
following through I also sent the output to log files so in case I might like to look at it one day (NOT)this thread details all the steps I took and at what point it breaks, the settings quoted on the reference thread will do the same as disabling 450.status-security
there is nothing to see there anyway, especially as configured out of box.security_status_baseaudit_enable="NO"
security_status_pkg_checksum_enable="NO"
security_status_pkgaudit_enable="NO"I am active on that other thread too, but have not had a chance to perform those steps yet. Did it resolve the issue when you force the daily periodical post reboot?
-
Short answer = yes 100%
Long answer, why I disabled the whole thing
I used the disable the entire 450.status-security enabled "NO"
and yes all three cases with 100% after with no loss, one of the posts here shows/comments on that.
then I started to break it again to narrow down which specific step was causing it.at the end I just disable the entire security report, unless you make the other changes I documented earlier there is really nothing to see ;-)
have to change from "YES" to "NO"daily_status_security_enable="NO"
weekly_status_security_enable="NO"
monthly_status_security_enable="NO"but the changes in the other thread should also work as well, but they only disable specifically the base audit, pkg checksum and pkgaudit
ie the rest of the security report will still generate, and effectively go nowhere (you'll never see it) in the out of box config unless you also change the logging options as I documented earlier.honestly in the current state, the system security report is of little value.
For example, one of the checks it run is for "login failures:" and it hasn't picked up a single one. (and I've fat fingered my password more than once today.) My NAS picks up the log in failure from the syslog and notifies me almost instantly (within seconds that I can't spell my password) I've often got the email telling me about that before I've retyped it and actually logged in. LOLmaybe when FreeBSD 14 goes -RELEASE there may be value. But for now the value is keeping the memory footprint under 20% (it not a real goal, it's just where I flat lined before the upgrade) I support 40-50 devices LAN side and memory really never moves much (at 16% right now)
-
Personally, I'm reluctant to muck with crontab scripts at the moment, especially since Netgate has withdrawn release of 23.01 for smaller boxes. While they didn't actually say it, they seem to acknowledge that there is a real bug out there that they need to corner. So, I'm hoping that a 23.02 release might come out soon and solve this issue.
BTW, I just noticed that a new version of pfblockerng has appeared in System/Package Manager/Installed Packages. I have pfblockerng version 3.2.0_2, I will apply the update and see what happens.
-
Fair enough, I'm on a 2100 and no other issues
(well except it still does show as registered, but TAC told me that was a backend issue that would correct itself, when they fix the backend "shortly" that was last week, so I don't know how long "shortly" is. Not a panic at this point because they also told me it shows as registered on their side and is seeing the correct repo's and package availability etc. Sure they are swamped. The joy of releasing new stuff, been there, done that.
Cheers
-
@jrey said in 1100 upgrade, 22.05->23.01, high mem usage:
Short answer = yes 100%
Long answer, why I disabled the whole thing
I used the disable the entire 450.status-security enabled "NO"
and yes all three cases with 100% after with no loss, one of the posts here shows/comments on that.
then I started to break it again to narrow down which specific step was causing it.at the end I just disable the entire security report, unless you make the other changes I documented earlier there is really nothing to see ;-)
have to change from "YES" to "NO"daily_status_security_enable="NO"
weekly_status_security_enable="NO"
monthly_status_security_enable="NO"but the changes in the other thread should also work as well, but they only disable specifically the base audit, pkg checksum and pkgaudit
ie the rest of the security report will still generate, and effectively go nowhere (you'll never see it) in the out of box config unless you also change the logging options as I documented earlier.honestly in the current state, the system security report is of little value.
For example, one of the checks it run is for "login failures:" and it hasn't picked up a single one. (and I've fat fingered my password more than once today.) My NAS picks up the log in failure from the syslog and notifies me almost instantly (within seconds that I can't spell my password) I've often got the email telling me about that before I've retyped it and actually logged in. LOLmaybe when FreeBSD 14 goes -RELEASE there may be value. But for now the value is keeping the memory footprint under 20% (it not a real goal, it's just where I flat lined before the upgrade) I support 40-50 devices LAN side and memory really never moves much (at 16% right now)
Netgate suggested commenting out the 3 periodic lines in the /etc/crontab file. @jimp mentioned that those were not enabled in 22.05. I just made the edits and rebooted. We will know for sure tomorrow morning!
-
@beerguzzle said in 1100 upgrade, 22.05->23.01, high mem usage:
I have pfblockerng version 3.2.0_2, I will apply the update and see what happens.
it won't change the static memory loss caused by the security reports running, but it certainly has some great features.
I actually did the troubleshooting on the cron issue that is fixed in there. That was a fun weekend ;-). but all around that is a very good update they have done an excellent job pulling it all together so quickly. -
@defenderllc said in 1100 upgrade, 22.05->23.01, high mem usage:
Netgate suggested commenting out the 3 periodic lines
He made a patch already.
@beerguzzle said in 1100 upgrade, 22.05->23.01, high mem usage:
Netgate has withdrawn release of 23.01 for smaller boxes. While they didn't actually say it, they seem to acknowledge that there is a real bug
There were threads about it like this one. Early models of 1100/2100 had a small EFI partition, and the issue is an "out of space" copying to it. I'm not clear myself if that means "all sold with UFS" or just early models. I have a 2100 that had an 800K partition and had the problem. New installs and newer devices have ZFS and a 200 MB EFI partition so aren't affected. Per that thread Netgate was unable to duplicate the issue, at least as of this weekend, but stopped the updates anyway. A new install will use ZFS and the new file system layout so is unaffected. One can still request the 23.01 image file and reinstall fine.
I would normally have waited longer myself, knowing they skipped a FreeBSD version and jumped to PHP 8 with lots of coding changes, but was testing the 2100.
-
i saw that too, was sure, because I couldn't remember if it was enable in prior version or not.
@jimp suggests none of it was not enabled in prior versionThe change to crontab will for sure stop it and all the other reports it runs too.
again out of the box the way it was configured no one would have seen them anyway ..There are 3 fixes that will alleviate the problem caused by the security reports.
Dealers choice at this point.crontab is likely the best final solution since they say there is nothing else needed.
All good. Cheers
-
I just applied patch ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12 that was put out by the Netgate crew, see https://redmine.pfsense.org/issues/14016, and rebooted. Also see the discussion in the thread "23.1 using more RAM" about this patch.
After reboot, wired mem dropped from 55% to 33% on my 1100. I'll check it in the morning to see what happened at 3 AM.
-
@beerguzzle Hello, my first comment here. Same situation, SG-1100 with 85% memory in constant use. I applied that patch and Memory usage dropped to 35% after rebooting.
-
Checking my system this morning after applying patch ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12 and rebooting yesterday... Bliss! Nothing happened at 3 AM and my wired mem usage remains at about 35%. I consider this problem solved.
-
Patch "ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12" has completely resolved my SG-1100 memory problems!
-
@rpsmith said in 1100 upgrade, 22.05->23.01, high mem usage:
Patch "ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12" has completely resolved my SG-1100 memory problems!
+1
Applied patch and rebootet yesterday at 8:00pm
Regards
-
@fsc830 Applied this patch in my SG-3100, everything OK, memory usage didn't change at night.
Thanks -
S SteveITS referenced this topic on
-
@machasachaira How do we apply the patch?
JMV
-
@jmv43-0 Install the System Patches package and use the patch ID.
https://docs.netgate.com/pfsense/en/latest/development/system-patches.htmlPre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
Upvote 👍 helpful posts! -
@steveits I didn't know that way, I used the fetch command on the CLI to bring the file and replace the original.
Thanks.
-
@machasachaira :) System Patches is relatively new (1-2 years), and a wonderful idea. Netgate publishes a list of Recommended patches for the version you're on. Updating that package updates the list of patches. Any patch with a commit ID can also be pulled in.
-
@steveits Thanks
-
Applied this patch 2 days ago. Absolutely solved the 3am memory leak. However, dns broke on my sg 1100 the past two days. I had to restart the dns resolver service to restore dns. Anyone else experiencing this?
