Intermittently Unresponsive in Hyper-V
-
@stephenw10 errant GW disabled. System processed traffic overnight with the exception of nginx going down with a 502 Bad Gateway. Restart PHP-FPM got it back online.
System.log was full of this to the point it rolled over
Aug 18 09:45:05 firewall check_reload_status[66343]: Could not connect to /var/run/php-fpm.socketnignx had a few
Aug 18 09:47:10 firewall nginx: 2023/08/18 09:47:10 [error] 12916#100237: *2631 connect() to unix:/var/run/php-fpm.socket failed (61: Connection refused) while connecting to upstream, client: 192.168.1.195, server: , request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "192.168.1.1"kern.ipc.soacceptqueue = 128, so I bumped that up to 1024 via a system tunable and verified it applied.
-
Hmm, nothing logged to show why php stopped responding?
I suspect bumping that sysctl won't help if php never responds.
-
@stephenw10 what's the best way to increase logging for PHP?
-
There's no easy way to do that. You can check /tmp/php_errors.txt but I'd expect any php errors to be shown as an alert.
-
Logged in tonight to see SWAP was at 52%.
Ran top and see this:
-
Ah, so some stuck script using all the RAM. Interesting.
Try running top as
top -HaSPthat may show more.You might also try running
ps -auxwwdthat may show you hat script is running. -
SWAP use is down to 5% a few (15) hours later without my intervention:
Sorted and filtered for Swap:
ps -auxwwd results:
Stopped ntopng service and the stats seem similar:
top sorted by size:
top sorted by Swap:
and lastly:
-
Nothing shown in the process tree under php-fpm in the ps output?
-
Not nothing...
Sorry for truncating that output in the previous message.
-
Hmm, dissapointing, I was hoping it might show there.
I'm not really aware of anything that behaves like that but don't run Hyper-V. You might ask in the virtualisation sub forum.
-
Just posting here to let you know that I've also had similar issues with a system has been running stable for 5+ years on Hyper-V. Since upgrading to 23.05 (and now 23.05.1) this has begun happening to me. The only difference is that it occurs after about 14 days of uptime, so not easy to reproduce or troubleshoot.
I just had it happen about 45 minutes ago, but log rotation has already purged the relevant system logs. I also send syslog to another server, but of course it didn't receive anything during the outage. I doubled the log size for rotation and extended the retained logs to 21. I'll post another update to this thread next time it happens, and see if I can capture the relevant logs before they get purged.
-
Unless anyone is seeing this on another platform this thread would probably be better in the virtualization sub.
-
We've been stable for 4-5 days without touching anything. If it happens again, I'll hop to the other sub. Don't love that something is using a bit of swap and always at 2gb+ ram usage, but it's working well.
-
So it's been about 6 weeks since anything happened and we had a crash today. Just replying to add 2 pieces of information in case anyone comes across this post in the future. Today we rebooted the VM host for the first time in 8 weeks after pushing some Microsoft updates. Second, I tried to reboot from shell and it hung on stopping ntopng. I think the reboot is anecdotal/coincidental.
