"cannot allocate memory" when there is enough memory
-
I have enabled second WAN interface and I believe this is when "cannot allocate memory" appeared in the logs, one is from node exported and another from arpresolve:
Oct 6 18:40:32 node_exporter 6258 ts=2023-10-06T15:40:32.570Z caller=collector.go:169 level=error msg="collector failed" name=uname duration_seconds=4.94e-06 err="cannot allocate memory" Oct 6 18:40:17 node_exporter 6258 ts=2023-10-06T15:40:17.569Z caller=collector.go:169 level=error msg="collector failed" name=uname duration_seconds=1.8051e-05 err="cannot allocate memory" Oct 6 18:40:12 node_exporter 6258 ts=2023-10-06T15:40:12.571Z caller=collector.go:169 level=error msg="collector failed" name=uname duration_seconds=2.1381e-05 err="cannot allocate memory" Oct 6 18:40:02 php-fpm 93831 /rc.filter_configure_sync: GW States: One or more gateways is down, flushing all states: WAN2_DHCP Oct 6 18:40:00 check_reload_status 400 Reloading filter Oct 6 18:40:00 php-fpm 70652 /rc.filter_configure_sync: GW States: One or more gateways is down, flushing all states: WAN2_DHCP Oct 6 18:40:00 php-fpm 42358 /rc.newwanip: rc.newwanip: on (IP address: 176.100.x.y) (interface: WAN2[opt1]) (real interface: vtnet1). Oct 6 18:40:00 php-fpm 42358 /rc.newwanip: rc.newwanip: Info: starting on vtnet1. Oct 6 18:40:00 php-fpm 93831 1.0.0.1|176.100.x.y|WAN2_DHCP|2.095ms|5.583ms|16%|down|highloss Oct 6 18:40:00 php-fpm 93831 /rc.openvpn: MONITOR: WAN2_DHCP has packet loss, omitting from routing group MultiWAN Oct 6 18:40:00 sshguard 80245 Now monitoring attacks. Oct 6 18:40:00 sshguard 22781 Exiting on signal. Oct 6 18:39:59 check_reload_status 400 rc.newwanip starting vtnet1 Oct 6 18:39:59 kernel arpresolve: can't allocate llinfo for 176.100.x.1 on vtnet1 Oct 6 18:39:59 kernel arpresolve: can't allocate llinfo for 176.100.x.1 on vtnet1 Oct 6 18:39:59 kernel arpresolve: can't allocate llinfo for 176.100.x.1 on vtnet1System memory usage is at ~23% out of 2G total according to dashboard and I have not seeing it spiking according to monitoring, but then it was node exporter that was crashing, so maybe it was running out of memory, just unable to send informtion about this. Looks like it happens when WAN2 connection goes down and states are flushed.
How do I debug this? I have seen several older existing topics about this, but they didn't contain applicable solutions to my situation.
-
Update, node exported just prints it all the time, really confusing.
-
@nazar-pc I don't have an answer for your memory error but the arpresolve error is not memory related:
https://forum.netgate.com/topic/54931/arpresolve-can-t-allocate-llinfo-for
https://forum.netgate.com/topic/149298/arpresolve-can-t-allocate-llinfo-for-192-168-100-1/ -
@SteveITS I just read them more carefully now, thanks!
That leaves node exporter then.
