pfSense 2.7.2 in Hyper-V freezing with no crash report after reboot

Bismarck

Quick update:

Had to reboot once because auf updates, but since than rock solid no incidence. Enabled all extra IP-Lists and Suricata and so again.

@maitops Just disable all energy saving features of the nic or select high performance power profile in windows for a test.

It must be the power state switching or the system tunables I did im my last update post.

maitops

@Bismarck I still have the issue, I have a mellanox connectx-6 Lx NIC, I just disabled the "Interupt Moderation" too

Bismarck

@maitops

I have a Intel 82599 and X550-AT2 in use.

Did you try the loader.conf.local and system tunables from the screenshot?

maitops

@Bismarck Yes, iI changed the loader.conf and rebooted.
the tunable, not everything, only:

hw.hvtimesync.sample_thresh
hw.hvtimesync.ignore_sync

i can try to set all the others too

stephenw10

You should set custom loader values in /boot/loader.conf.local (create that file) to prevent them being overwritten by the system.

shoulders

is this the same underlying issue as

https://forum.netgate.com/topic/197248/pfsense-2-7-2-ram-leak-wired-memory-pool/11

might be irrelevant it is on a hypervisor.

solution looks like it might to update to pfsense 2.8.0

maitops

@stephenw10 My bad, i'm trying with this.
I will tell you if it solved my issue.

jacolex

@maitops after upgrade to 2.8.0 - everything works fine for about 3 weeks. After that since yesterday I needed to reboot the pf 4 times.

Techniker_ctr

Just fyi, we recently upgraded one of our unused pfSense systems to 2.8, and it crashed about 4 hours after that. Looked the same as the crashes of 2.7, no systems behind that firewall so no traffic on it.

Our plan is to move about 50 firewalls to our new windows 2025 hypervisors and see if this fixes the problem, but as we still cannot really trigger it we need to wait at least half a year to be able to say its stable if no crashes occur.

Bismarck

@Techniker_ctr Today we had our first hvevent storm in a long time, only change was switching replica from one host to another 2 weeks ago.

To be honest I don't think its fixed in Server 2025 or pfSense 2.8.0.

stephenw10

Has anyone been able to replicate it with vanilla FreeBSD 15?

This is almost certainly something that would affect that and should be reported upstream. We don't do anything special for hyper-v in pfSense.

Reference FreeBSD thread (which looks like all same posters ) https://forums.freebsd.org/threads/hyperv-cpu-hvevent-goes-to-100.95981/

shoulders

@Bismarck can you give me the full name of hvevent storm so I can understand what this means. Just a pointer will do

thanks

Bismarck

@shoulders The hvevent storm is causing excessive CPU usage, leading to system crashes or freezes.

root@proxy01:~ # top -aHSTn
last pid: 42090;  load averages:  3.50,  1.88,  0.94  up 0+14:51:36    12:33:11
252 threads:   13 running, 226 sleeping, 13 waiting
CPU:  1.5% user,  0.0% nice,  0.4% system,  0.0% interrupt, 98.1% idle
Mem: 135M Active, 1875M Inact, 1813M Wired, 1100M Buf, 7823M Free
Swap: 8192M Total, 8192M Free

   THR USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
100004 root        187 ki31     0B   128K RUN      1 872:01 100.00% [idle{idle: cpu1}]
100009 root        187 ki31     0B   128K CPU6     6 871:57 100.00% [idle{idle: cpu6}]
100003 root        187 ki31     0B   128K CPU0     0 871:36 100.00% [idle{idle: cpu0}]
100007 root        187 ki31     0B   128K CPU4     4 871:35 100.00% [idle{idle: cpu4}]
100005 root        187 ki31     0B   128K CPU2     2 871:33 100.00% [idle{idle: cpu2}]
hvevent storm ------>****100116 root        -64    -     0B  1472K CPU7     7   4:51 100.00% [kernel{hvevent7}]**** <------- hvevent storm  
100008 root        187 ki31     0B   128K RUN      5 871:22  98.97% [idle{idle: cpu5}]
100006 root        187 ki31     0B   128K CPU3     3 871:28  96.97% [idle{idle: cpu3}]
101332 www          21    0   537M    59M kqread   2  11:31   3.96% /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid{haproxy}
100902 www          21    0   537M    59M kqread   4  12:29   1.95% /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid{haproxy}
101333 www          23    0   537M    59M kqread   5  11:17   1.95% /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid{haproxy}
101334 www          21    0   537M    59M CPU1     1  10:57   1.95% /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid{haproxy}
100010 root        187 ki31     0B   128K RUN      7 867:24   0.00% [idle{idle: cpu7}]
100805 root         20    0    81M    52M nanslp   5   3:09   0.00% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_watcher.php interface routes alarm
100102 root        -64    -     0B  1472K -        0   1:14   0.00% [kernel{hvevent0}]
100104 root        -64    -     0B  1472K -        1   0:59   0.00% [kernel{hvevent1}]
100114 root        -64    -     0B  1472K -        6   0:56   0.00% [kernel{hvevent6}]
100106 root        -64    -     0B  1472K -        2   0:56   0.00% [kernel{hvevent2}]

Bismarck

For over 97 days nothing burger and now twice a week without touching that thing.

Today hvevent storm again, exactly 1 hour ago.

stephenw10

Is the phrase hvevent storm actually logged anywhere? In hyper-V maybe? I seem to recall it but I might be conflating several issues!

Bismarck

@stephenw10 No, that's just how I describe the incident, the hvevent (top -aHSTn -> kernel{hveventN]) is acting like a storm till the machine freeze and crash without logging or a dump.

Techniker_ctr

Yesterday we built a new pfSense 2.7.2 cluster, master firewall was running for over a week without problems, but about half an hour after setting up CARP and pfSync to the new slave it died with known hvevent problem. It then died several times, again and again.. Not sure but maybe it has something to do with either CARP/ConfigSync/pfSync or multicast traffic (because we know dying pfsense setups without carp configured, so might be multicast traffic in the network which triggers something).

We have had the same experience with our only OPNsense setup, of which the master is running smoothly since we removed the slave firewall.