sg-1100 22.01->23.05.1-high memory ussage after upgrade

lrossi

As the title says, i am experiencing unusually high memory usage after i upgraded from 22.01 to 23.05.1 and I'm wondering if anybody else is going through the same or has a solution.

I used to see this behavior on 22.01 until i applied the patch "ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12" which lowered my memory use to roughly 35% all the time.

Now, after the upgrade, my memory sits at 68%-71% which is twice as much as before. I rebooted the system several times after the upgrade but i end up with the same result.
Do any of you know if there is a patch or something else i could try to solve this?

stephenw10

@lrossi said in sg-1100 22.01->23.05.1-high memory ussage after upgrade:

ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12

That is in 23.05(.1).
Check the system activity, what do you see using the memory?

lrossi

@stephenw10 said in sg-1100 22.01->23.05.1-high memory ussage after upgrade:

ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12

My apologies. I seemed to have mixed the versions.
The version i had applied the patch to was 23.01 and it was included in the list of "recommended System Patches" so i just assumed that was the patch number i mentioned.

Should i try to apply patch ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12 to see if it helps?

To answer your question, unbound is using about 263M of memory but besides that i see nothing else. Well, telegraf seems to be using 4967M but that doesn't make any sense since this unit only has 1 GB of memory in total.

stephenw10

Do you have any large lists loaded in Unbound from pfBlocker?

What packages do you have installed?

lrossi

@stephenw10 not really. And even if i did, the lists did not change between updates so that wouldn't explain why I'm using more memory now than before.

The packages I have installed are

ACME
Avahi
HAProxy
nut
openvpn-client-export
pfBlockerNG
System_patches
Telegraf

lrossi

@stephenw10 now that i think about it, i did change from pfblockerNG-devel to regular pfblockerNG after i upgraded to 23.05.1

Do you think that change would produce the high memory usage?

stephenw10

No, in 23.05 they were merged, the two packages are identical.

If you restart Unbound or pfBlocker does the usage change?

lrossi

@stephenw10 I gave it a try but it didn't make a difference.

I also disabled the "telegraph" package for a bit, as i remember having memory issues with it in the past, but this didn't help either. I then restarted the "nut" package, as i read another thread where a user was having memory problems with that package, but this didn't help either.

For reference this is the memory graph for the last month. Can you guess where the update to 23.05.1 happened? Memory pfsense.png

It seems that the active memory increased from roughly 5% to 25%.

Do you happen to have any more ideas of things i could try?

Gertjan

@lrossi

I've seen this before on the forum :

but never on my own pfSense devices.

Not that "Laundry" is bad, but the presence == not zero says to me : something doesn't go as planned.
The laundry kernel process can't decide what to do with it. (last thread, last line).

These stand out for me :

HAProxy
Telegraf

because - don't laugh - I don't use them.
A way to debug is : stop using them - and memory issues go away ?

lrossi

@Gertjan Thanks for the input. I disabled the Telegraf service for a bit, just to see it makes a difference, but nothing changed.

About HAProxy, i will have to try tomorrow as the network users are starting to become active and they rely on that service for their normal operations.

However, i'll be surprised if this is the problem since HAProxy is widely used and i have not seen anybody talking about its high memory use.

lrossi

@Gertjan i hust stopped HAProxy for about 20 minutes and the memory usage didn't change.
Do you happen to have any other suggestions of things i could try?

My system is not crashing but i find it odd that it is using twice as much memory all of the sudden.

stephenw10

Mmm, if it's not actually climbing then it shouldn't be an issue. Interesting that it also shows far more variability.

lrossi

And just like that, the system started to behave as it did before the update (or similar).

I knew this forum was good, but i never imagined they could intimidate a system into fixing itself. Wow you guys are good.
pfsense 2023-08-01 1.png pfsense 2023-08-01 2.png

Even the variability that @stephenw10 noticed is gone. The situation with the "laundry" memory that @Gertjan saw is still there.

Now, what produced this behavior after the update? How it got fixed? I have learned to focus my time on problems and, right now, this is not one of them.

Thanks for trying to help.

stephenw10

Hmm, odd! Nothing logged at that time?

lrossi

@stephenw10 per my estimation, the change happened on 7/31 between 7:20 AM and 7:40 AM

The logs between that time are nothing out of the ordinary

Jul 31 08:00:01 	php 	78976 	[pfBlockerNG] Starting cron process.
Jul 31 07:00:49 	php 	1559 	[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Jul 31 07:00:01 	php 	1559 	[pfBlockerNG] Starting cron process.

Nothing else was logged during that period of time.

The only thing that concerns me is that I haven't seen any other reports of this on the SG-1100 which may implies there is something different and odd on my configuration or my hardware is starting to malfunction. Neither option is great.

stephenw10

I doubt it's a hardware issue causing memory use like that.