NG6100 MAX - pf+22.05 "wired" memory increasing over time
-
@heper said in NG6100 MAX - pf+22.05 "wired" memory increasing over time:
16: 16, 0,23018700, 510,1018355362, 0, 0
32: 32, 0, 15559, 5316,5358991388, 0, 0
64: 64, 0,11526267, 835,1410775321, 0, 0
128: 128, 0,34459501, 409,1895721235, 0, 0
256: 256, 0, 45512, 28573,6298090042, 0, 0
512: 512, 0, 5932, 1420,198321701, 0, 0Hmm, so ~4.4GB in the 128 zone, ~730MB in the 64 zone and ~360MB in 16.
If you run it again after some time are those all increasing at the same rate?
See this post for last time we looked at something similar: https://forum.netgate.com/post/1041601
Steve
-
@stephenw10
will add new timestamps as the day progresses2022/09/27-08:00
6404M Wired
zone16: 376MB
zone32: 0.5MB
zone64: 753MB
zone128: 4,504GB
attached dtrace_script_2022-09-27-0800.log.txt dtrace script output from post2022/09/27-09:00
wired 6453M
zone16: 380,465616mb
zone32: 3.554336mb
zone64: 755.222912mb
zone128: 4,537gb
dtrace attacheddtrace_script_2022-09-27_0911.log.txt2022/09/27-10:07
6475M wired
zone16: 381.96072MB
zone32: 3.560192MB
zone64: 758.20928MB
zone128: 4.55558976GB
dtrace: dtrace_script_2022-09-27-1007.log.txt2022/09/27-15:11
6625M Wired
zone16: 392.087136MB
zone32: 3.549216MB
zone64: 778.44768MB
zone128: 4.677006336GB
dtrace: dtrace_script_2022-09-27-1511.log.txtthat'll be all for me for today. gotta go do "stuff".
tonight i'll reboot the device remotely because i won't return to this school till friday & want to prevent it running out of memory (or start swapping) -
Ah, this is good data. We are reviewing it.
-
@stephenw10
do you need more data?
the dtrace script seems to target zone128. please let me know if you require other zones -
I believe we have found an issue in the code that pulls ethernet rules. It's particularly apparent where captive portal is running which is probably why you're seeing it more than most.
So we shouldn't need any more data but I'll reach out if we do.Yes, this:
https://github.com/freebsd/freebsd-src/commit/0044bd90f2397dfad5f4bbd12c64be86e0b7eb4a -
@stephenw10
Thanks for getting another cp related bug squashed.Will contact you again when I stumble upon the next rodent in the attic ....
-
@stephenw10 is there a redmine to follow up on the status of this bug?
(Didn't find one myself - but didn't look very hard) -
No, Kristof just went straight to the source! But there should be one to track it. I'll add one....
-
@stephenw10 when will a patch be released?
just got called in because school went offline:
tried to ssh remotely to reboot - then it got stuck somewhere
had to drive on site to powercycle the ng6100(yes i forgot to reboot before memory ran out ... )
-
It's patched in 23.01 if you're able to test that?
It's unlikely to be patched in 22.05 at this point because it would require a point release, the fix is in kernel.
Steve
-
@stephenw10 said in NG6100 MAX - pf+22.05 "wired" memory increasing over time:
It's patched in 23.01 if you're able to test that?
Is this Patch also in 2.7.0?
-
-
@stephenw10 test 23.01 on a production system with roughly 1k users ?
also: it's not mentioned in the 23.01 release notes posted alongside the beta announcement
-
Likely because it hadn't been marked as feedback yet. It is in 2.7 and 23.01 though. I've updated the ticket.
Steve
-
We need someone who was hitting this in 22.05 to test it in 23.01 to confirm it is fixed there.
Otherwise if it's not fixed in the current code it may not be before release.
Steve
-
@stephenw10 is netgate willing to verify/test the config of the school that hit this issue on 23.01 before i attempt the upgrade?
i can not afford to upgrade to a non/semi-working system that has bigger issue's then what i currently have.
so the verify / test would have to include the upgrade itself & basic connectivity appears to work in a lab environment. there are no addon packages installed.also: schools in belgium close down for the holidays. So between now & january 9th there will be no clients.
memory seems to increase only when there are "lots" of clients connected to CP. i can not observe memory-leakage during the weekends.also2: i read that the current builds still have some debugging enabled that cause decrease performance. how much % are we talking about on a ng6100?
conclusion: i'm willing to test this from january 9th if netgate can confirm there don't appear to be any issues upgrading from 22.05 to 23.01 with the config i have.
-
Well I can test a config in a 6100 running 23.01 but I obviously can't test it in your network environment.
-
@stephenw10 what's the preferred way to send you that config ?
-
You can upload it to me directly here:
https://nc.netgate.com/nextcloud/s/6ZZEPy4RsnWi6mb -
@stephenw10 you probably received the config at the link you provided.
there is a cron to restart dpinger to work around a different bug.
believe its https://redmine.pfsense.org/issues/11570 - anyhow in some scenarios dpinger keeps saying gateway is offline but in reality gw has been back up for a while - it has something todo with a PID getting stuck or something ?)could you let me know if the provided config seems to work on the alpha/beta/rc versions (whatever it is at this time).
