pfSense CE 2.7.2 on Azure: boot hangs on Standard_D2s_v5 (MANA hardware), mlx5_core timeout / hn1 stuck DOWN

ronens

Running pfSense CE 2.7.2 as a custom NVA on Azure (WAN=hn0, LAN=hn1). Rock stable on Standard_D2s_v3. Resizing to Standard_D2s_v5 breaks boot completely - console loops forever on:

mlx5_core: WARN: wait_func:965:(pid 0): ACCESS_REG(0x805) timeout. Will cause a leak of a command resource

No SSH/HTTPS, Accelerated Networking already off on both NICs. This is Azure's new "MANA" network hardware, rolling out across the whole Dv5/Dsv5 family since 2026-05-26 - FreeBSD apparently isn't falling back to NetVSC cleanly like it's supposed to.

Tried Microsoft's official NVA opt-out (the "LegacyVMNVA" tag) on a disposable test VM at Standard_D2s_v5. That got rid of the mlx5_core loop, but caused a new hang instead: both hn0/hn1 and mce0/mce1 (mlx5en) come up, mce0/mce1 reach "link state UP", but hn1 (our LAN interface) goes to "link state DOWN" and boot freezes there for good.

Anyone else hit this on v5-generation Azure sizes? Any sysctl/loader.conf way to force pfSense to ignore the mlx5/mce interfaces and stick to hn0/hn1 only? Does a newer pfSense (newer FreeBSD base) handle MANA hardware better?

Staying on Standard_D2s_v3 for now, but there's pressure to move to v5. Happy to share more logs/screenshots.

Thanks!

ronens

UPDATE / SOLUTION FOUND - pfSense CE on Azure Standard_D2s_v5 (MANA) boot hang

Follow-up to my earlier post about pfSense CE 2.7.2 (FreeBSD 14) hanging on boot when resized
to Standard_D2s_v5 on Azure. Posting the resolution in case it helps anyone else hitting this.

ROOT CAUSE: Azure has been migrating the entire Intel-based Dv5/Dsv5/Ev5/Esv5 (etc.) VM family
to new "MANA" (Microsoft Azure Network Adapter) hardware. FreeBSD's hn(4)/mlx5_core driver
stack does not handle the VF pairing/failover cleanly on this hardware - depending on the
specific boot, we saw an infinite "mlx5_core: WARN: wait_func:965: ACCESS_REG(0x805) timeout"
loop, a lagg0 interface flapping between the NetVSC (hn0/hn1) and Mellanox VF (mce0/mce1)
interfaces, or an outright freeze right after early kernel init - all on the same VM/disk,
non-deterministically, depending on which physical host Azure placed the VM on at boot time.

WHAT WE TRIED (partial mitigations, none fully reliable):

Microsoft's official LegacyVMNVA=true VM tag (temporary NVA exemption, valid through
2027-05-31) - suppressed the original ACCESS_REG loop but didn't fully fix the hang.
FreeBSD loader tunable hw.hn.vf_transparent="1" (NOT hw.hn.enable_vf, which turned out not
to exist as a real sysctl on this kernel at all - a silent no-op). Combined with the tag,
this got a fully clean, working boot - but only reproduced about 1 out of 5 times across
repeated az vm deallocate + resize + az vm start cycles (i.e. fresh hardware placement).
Reliable enough to debug with, nowhere near reliable enough for production.

THE ACTUAL FIX: switch to Standard_D2as_v5 instead of Standard_D2s_v5 - same spec (2 vCPU,
8GB RAM), but the AMD-based "Da"/"Das" v5 family. Per Microsoft's own MANA documentation, this
series does not currently appear on the list of VM series being migrated to MANA hardware.

Tested 5 independent deallocate+resize+start cycles on Standard_D2as_v5 with ZERO mitigations
(no tag, no loader tunables) - 5 for 5 clean, immediate boots (SSH reachable in 30-40 seconds
every time). ifconfig shows only the standard hn0/hn1 NetVSC interfaces - no mce0/mce1 VF
device ever appears, confirming this hardware simply doesn't present the MANA-related NIC at
all (as opposed to being "protected" by a tag that could later expire).

We've since migrated our production pfSense from Standard_D2s_v3 straight to Standard_D2as_v5
and it's running cleanly with the same WAN/LAN config, OpenVPN, and FreeRADIUS setup as before.

TL;DR: if you need a v5-generation size for a custom-built pfSense/FreeBSD VM on Azure and hit
this MANA-related boot hang, try the AMD "Das_v5" family instead of the Intel "Dsv5" family
before going down the tag/tunable path - it may sidestep the whole problem.