1100 upgrade, 22.05->23.01, high mem usage

[DefenderLLC]

@jrey said in 1100 upgrade, 22.05->23.01, high mem usage:

@defenderllc

Short answer = yes 100%

Long answer, why I disabled the whole thing

I used the disable the entire 450.status-security enabled "NO"
and yes all three cases with 100% after with no loss, one of the posts here shows/comments on that.
then I started to break it again to narrow down which specific step was causing it.

at the end I just disable the entire security report, unless you make the other changes I documented earlier there is really nothing to see ;-)
have to change from "YES" to "NO"

daily_status_security_enable="NO"
weekly_status_security_enable="NO"
monthly_status_security_enable="NO"

but the changes in the other thread should also work as well, but they only disable specifically the base audit, pkg checksum and pkgaudit
ie the rest of the security report will still generate, and effectively go nowhere (you'll never see it) in the out of box config unless you also change the logging options as I documented earlier.

honestly in the current state, the system security report is of little value.
For example, one of the checks it run is for "login failures:" and it hasn't picked up a single one. (and I've fat fingered my password more than once today.) My NAS picks up the log in failure from the syslog and notifies me almost instantly (within seconds that I can't spell my password) I've often got the email telling me about that before I've retyped it and actually logged in. LOL

maybe when FreeBSD 14 goes -RELEASE there may be value. But for now the value is keeping the memory footprint under 20% (it not a real goal, it's just where I flat lined before the upgrade) I support 40-50 devices LAN side and memory really never moves much (at 16% right now)

Netgate suggested commenting out the 3 periodic lines in the /etc/crontab file. @jimp mentioned that those were not enabled in 22.05. I just made the edits and rebooted. We will know for sure tomorrow morning!

[jrey]

@beerguzzle said in 1100 upgrade, 22.05->23.01, high mem usage:

I have pfblockerng version 3.2.0_2, I will apply the update and see what happens.

it won't change the static memory loss caused by the security reports running, but it certainly has some great features.
I actually did the troubleshooting on the cron issue that is fixed in there. That was a fun weekend ;-). but all around that is a very good update they have done an excellent job pulling it all together so quickly.

[SteveITS]

@defenderllc said in 1100 upgrade, 22.05->23.01, high mem usage:

Netgate suggested commenting out the 3 periodic lines

He made a patch already.

@beerguzzle said in 1100 upgrade, 22.05->23.01, high mem usage:

Netgate has withdrawn release of 23.01 for smaller boxes. While they didn't actually say it, they seem to acknowledge that there is a real bug

There were threads about it like this one. Early models of 1100/2100 had a small EFI partition, and the issue is an "out of space" copying to it. I'm not clear myself if that means "all sold with UFS" or just early models. I have a 2100 that had an 800K partition and had the problem. New installs and newer devices have ZFS and a 200 MB EFI partition so aren't affected. Per that thread Netgate was unable to duplicate the issue, at least as of this weekend, but stopped the updates anyway. A new install will use ZFS and the new file system layout so is unaffected. One can still request the 23.01 image file and reinstall fine.

I would normally have waited longer myself, knowing they skipped a FreeBSD version and jumped to PHP 8 with lots of coding changes, but was testing the 2100.

[jrey]

@steveits

i saw that too, was sure, because I couldn't remember if it was enable in prior version or not.
@jimp suggests none of it was not enabled in prior version

The change to crontab will for sure stop it and all the other reports it runs too.
again out of the box the way it was configured no one would have seen them anyway ..

There are 3 fixes that will alleviate the problem caused by the security reports.
Dealers choice at this point.

crontab is likely the best final solution since they say there is nothing else needed.

All good. Cheers

[beerguzzle]

I just applied patch ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12 that was put out by the Netgate crew, see https://redmine.pfsense.org/issues/14016, and rebooted. Also see the discussion in the thread "23.1 using more RAM" about this patch.

After reboot, wired mem dropped from 55% to 33% on my 1100. I'll check it in the morning to see what happened at 3 AM.

[MachasaChaira]

@beerguzzle Hello, my first comment here. Same situation, SG-1100 with 85% memory in constant use. I applied that patch and Memory usage dropped to 35% after rebooting.

[beerguzzle]

Checking my system this morning after applying patch ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12 and rebooting yesterday... Bliss! Nothing happened at 3 AM and my wired mem usage remains at about 35%. I consider this problem solved.

[rpsmith]

Patch "ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12" has completely resolved my SG-1100 memory problems!

[FSC830]

@rpsmith said in 1100 upgrade, 22.05->23.01, high mem usage:

Patch "ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12" has completely resolved my SG-1100 memory problems!

+1

Applied patch and rebootet yesterday at 8:00pm

Regards

[mcury]

@fsc830 Applied this patch in my SG-3100, everything OK, memory usage didn't change at night.
Thanks

[JMV43 0]

@machasachaira How do we apply the patch?

JMV

[SteveITS]

@jmv43-0 Install the System Patches package and use the patch ID.
https://docs.netgate.com/pfsense/en/latest/development/system-patches.html

[MachasaChaira]

@steveits I didn't know that way, I used the fetch command on the CLI to bring the file and replace the original.

Thanks.

[SteveITS]

@machasachaira :) System Patches is relatively new (1-2 years), and a wonderful idea. Netgate publishes a list of Recommended patches for the version you're on. Updating that package updates the list of patches. Any patch with a commit ID can also be pulled in.

[JMV43 0]

@steveits Thanks

[mr.castoro]

Applied this patch 2 days ago. Absolutely solved the 3am memory leak. However, dns broke on my sg 1100 the past two days. I had to restart the dns resolver service to restore dns. Anyone else experiencing this?

[SteveITS]

@mr-castoro There are a bunch of DNS threads lately.
If you have Resolver set to forward, ensure DNSSEC is unchecked.

[MachasaChaira]

@mr-castoro Have you restarted your SG after the patch update? I have not experienced any problems after rebooting.

[mr.castoro]

@machasachaira yes, several times

[rpsmith]

@mr-castoro -- No problems with my SG-1100 and DNS but I use the "Forwarder" and not the "Resolver" and I point the Forwarder to my two Pi-hole IPs. Works great that way!

Also, instead of pointing it to a Pi-hole, you could just use: 9.9.9.9, 1.1.1.1 or 8.8.8.8 or some other external DNS.