pfSense 2.7.2 in Hyper-V freezing with no crash report after reboot
-
@maitops Running pfSense 2.7.2 (FBSD 14) with HAProxy as the only VM. Load wasn't high during the events, even occurring once at 3 AM with zero HAProxy load. The issue may not be HAProxy itself, but a kernel resource over time exhaustion?
I also have another router configured as a CARP secondary with the same configuration. If the primary goes down, the secondary experiences the same issue.
I found this sentence very interesting. Why is that? Maybe that's a starting point?
-
@Bismarck I will provide more context.
I made a cron script that detect if the hvevent issue is triggering and force the router to enter in a CARP maintenance mode. So the secondary is suppose to take the lead when the hvevent occurs. Once the cron worked at 3am, at 6am the second router triggered the hvevent issue too. So the 2nd router probably didn't had an exhaustion over time, it didn't took a lot of traffic during 3h in the night.
Btw the CARP maintenance mode can fail to release some VIP when the hvevent issue occurs. I trigger the CARP Maintenance mode with the web API of the OpnSense (probably work the same on Pfsense).
The VMs are not run on the same host, but all hosts are hyper-v windows server 2022 on AMD EPYC Genoa CPU.
-
@maitops Thanks for the detailed explanation.
No hvevent storm here for 6 days and 23.5 hours since my last update, but it probably needs at least 20 and more days to be significant.
Theory: Server 2022 Hyper-V power management, network driver changes may be incompatible with some FreeBSD kernel components, causing issues under certain conditions. Windows and Debian guests in Hyper-V Manager display more detailed information (e.g., RAM usage) than FreeBSD 14 guests. Interesting that the MS Hyper-V FreeBSD Guest compability list only goes to 13 and 2019, where pfSense runs just fine.
-
Hello, I'm struggling with simmilar case. pfSense with 6 interfaces, hosts suddenly lost connection from/to pfsense gateway, which means distruption of web services. It happens once a month, but last week it happened 3 times. Only restart can help. Today I swiched to UFS. If it not resolves the issue, I'll try with disabling pfblocker for achieve minimal resources consuming. I wonder whether pfsense 2.8.0 on FreeBSD 15 would be more stable or worse.
-
@Bismarck Hi,
The system is still running fine ?
-
Yes, no problems so far.
-
@maitops yes, since 10 days. I also disabled hn ALTQ support (no clue if it's necessary). Observing kernel hvevents, no issues. But I have to wait 2-3 months to say, that it's stable.
-
@maitops unfortunately, today morning we encountered network outages and the firewall needs to be restarted.
-
@Bismarck said in pfSense 2.7.2 in Hyper-V freezing with no crash report after reboot:
No hvevent storm here for 6 days and 23.5 hours since my last update,
So this was setting the power management to 'high power' in Hyper-V? Which presumably disables throttling down the VM in some way.
-
Yes, or just disabling the power management/green feature of the Nic should be enough, this is how it is right now on my Hyper-V host. There was a message (no error) in the Windows event logs about switching states or so, while the hvevent storm.
