pfSense 2.7.2 in Hyper-V freezing with no crash report after reboot

Bismarck

@maitops Thanks for the detailed explanation.

No hvevent storm here for 6 days and 23.5 hours since my last update, but it probably needs at least 20 and more days to be significant.

Theory: Server 2022 Hyper-V power management, network driver changes may be incompatible with some FreeBSD kernel components, causing issues under certain conditions. Windows and Debian guests in Hyper-V Manager display more detailed information (e.g., RAM usage) than FreeBSD 14 guests. Interesting that the MS Hyper-V FreeBSD Guest compability list only goes to 13 and 2019, where pfSense runs just fine.

jacolex

Hello, I'm struggling with simmilar case. pfSense with 6 interfaces, hosts suddenly lost connection from/to pfsense gateway, which means distruption of web services. It happens once a month, but last week it happened 3 times. Only restart can help. Today I swiched to UFS. If it not resolves the issue, I'll try with disabling pfblocker for achieve minimal resources consuming. I wonder whether pfsense 2.8.0 on FreeBSD 15 would be more stable or worse.

maitops

@Bismarck Hi,

The system is still running fine ?

Bismarck

@maitops

Yes, no problems so far.

jacolex

@maitops yes, since 10 days. I also disabled hn ALTQ support (no clue if it's necessary). Observing kernel hvevents, no issues. But I have to wait 2-3 months to say, that it's stable.

jacolex

@maitops unfortunately, today morning we encountered network outages and the firewall needs to be restarted.

stephenw10

@Bismarck said in pfSense 2.7.2 in Hyper-V freezing with no crash report after reboot:

No hvevent storm here for 6 days and 23.5 hours since my last update,

So this was setting the power management to 'high power' in Hyper-V? Which presumably disables throttling down the VM in some way.

Bismarck

@stephenw10

Yes, or just disabling the power management/green feature of the Nic should be enough, this is how it is right now on my Hyper-V host. There was a message (no error) in the Windows event logs about switching states or so, while the hvevent storm.

maitops

This post is deleted!

Bismarck

Quick update:

Had to reboot once because auf updates, but since than rock solid no incidence. Enabled all extra IP-Lists and Suricata and so again.

@maitops Just disable all energy saving features of the nic or select high performance power profile in windows for a test.

It must be the power state switching or the system tunables I did im my last update post.

maitops

@Bismarck I still have the issue, I have a mellanox connectx-6 Lx NIC, I just disabled the "Interupt Moderation" too

Bismarck

@maitops

I have a Intel 82599 and X550-AT2 in use.

Did you try the loader.conf.local and system tunables from the screenshot?

maitops

@Bismarck Yes, iI changed the loader.conf and rebooted.
the tunable, not everything, only:

hw.hvtimesync.sample_thresh
hw.hvtimesync.ignore_sync

i can try to set all the others too

stephenw10

You should set custom loader values in /boot/loader.conf.local (create that file) to prevent them being overwritten by the system.

shoulders

is this the same underlying issue as

https://forum.netgate.com/topic/197248/pfsense-2-7-2-ram-leak-wired-memory-pool/11

might be irrelevant it is on a hypervisor.

solution looks like it might to update to pfsense 2.8.0

maitops

@stephenw10 My bad, i'm trying with this.
I will tell you if it solved my issue.

jacolex

@maitops after upgrade to 2.8.0 - everything works fine for about 3 weeks. After that since yesterday I needed to reboot the pf 4 times.

Techniker_ctr

Just fyi, we recently upgraded one of our unused pfSense systems to 2.8, and it crashed about 4 hours after that. Looked the same as the crashes of 2.7, no systems behind that firewall so no traffic on it.

Our plan is to move about 50 firewalls to our new windows 2025 hypervisors and see if this fixes the problem, but as we still cannot really trigger it we need to wait at least half a year to be able to say its stable if no crashes occur.

Bismarck

@Techniker_ctr Today we had our first hvevent storm in a long time, only change was switching replica from one host to another 2 weeks ago.

To be honest I don't think its fixed in Server 2025 or pfSense 2.8.0.

stephenw10

Has anyone been able to replicate it with vanilla FreeBSD 15?

This is almost certainly something that would affect that and should be reported upstream. We don't do anything special for hyper-v in pfSense.

Reference FreeBSD thread (which looks like all same posters ) https://forums.freebsd.org/threads/hyperv-cpu-hvevent-goes-to-100.95981/