[solved] pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V
-
This same scenario just played out in my environment as well. I spent hours on the line with my ISP trying to figure it out and left for the night with them escalating to a higher level technician.
It occurred to me after getting off the phone to search the internet for pfsense and hyper-v in the last week and it led me straight to this article. The hyper-v config changes for RSC were unnecessary for my setup (it was already $false).
I just spun up a new pfSense Hyper-V VM running 2.5.2 and restored from pfSense Auto Backup using the pre-upgrade auto config save, and everything is running perfectly. There were no Windows updates, no changes to Hyper-V / I reverted the settings I changed during diagnostics.
There's certainly something not right with 2.6.0. I re-learned that I need to make fewer major changes in one setting.
I took the opportunity to test pfSense 2.6.0 on various versions of Windows & Hyper-V. This same behavior occurred in Server 2016 (14393.4886), Windows 10 (19044.1526), and Windows 11 (22000.493).
-
@ttmcmurry said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:
I took the opportunity to test pfSense 2.6.0 on various versions of Windows & Hyper-V. This same behavior occurred in Server 2016 (14393.4886), Windows 10 (19044.1526), and Windows 11 (22000.493).
That must have been some immense work on your part. What have you tried other then changing the host os?
-
@bob-dig it didn't take that much time :) pfSense installs & configures fast! I grabbed two laptops, stuck some extra USB NICs on them, then probably spent 15 minutes on each installation to reproduce the issue. The ease of reproducing the issue across various hardware and Windows versions does speak to the consistency of the pfSense software; even though it has this undesirable problem, at least it's consistent and reproducible. Theoretically making finding the root cause easier.
Other things I tried
- Reboots
(Gateway, Switch, Host, VM) - On the same day I upgraded to 2.6, my ATT Gateway also got a firmware release. I jumped to conclusions & laid blame upon ATT and pursued them for a fix that was never to come.
- Examined Hyper-V vSwitch settings to ensure they were configured appropriately, bound to the correct physical uplinks (no changes made)
- Examined HV VM vNIC settings to ensure nothing has changed; set to pfSense recommendations (no changes made)
- Double checked my switching for loops/STP, logs, errors, unexpected BPDUs from someone adding a switch somewhere I didn't know about
- Interfaced with the ATT Gateway directly with laptop to test performance (this led to isolating Hyper-V as the problem)
- Upgraded Intel I350-T4 drivers & PROset to 27.0 (2022/02/09) which didn't fix or make anything worse.
- Reboots
-
And you are also seeing it specifically between VLANs on hn(4) NICs?
-
@stephenw10 Good day! In the pfSense VM, the interfaces are not associated with VLANs and there are no VLANs defined. From pfSense's perspective, it is working with native hn(x) interfaces.
Hyper-V's vSwitches are all untagged. All VM vNICs in HV are untagged.
VLANs exist past the physical uplinks in the Physical Switch.
-
Decision: do not use hyper-v as virtualization platform ))
Better try Proxmox VE (open source) -
@bob-dig said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:
It wouldn't been the first time I had to install pfSense fresh from the get-go after a new version. Whatever my usecase is, it might be special...
So I guess "This is the Way".Wasn't the way, creating a fresh pfSense-CE-2.6.0 sadly changed nothing.
If of interest:
-
@bob-dig yep. Thx for validating. 2.5.2 is fine, use that till they resolve the issue.
-
@ttmcmurry said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:
2.5.2 is fine, use that till they resolve the issue.
If netgate does, they seem to care only for azure and we all know since the exchange debacle, that the MS Cloud is not the same as MS's server products... So my hopes are rather low on this. Might have to try with DDA once again.
-
@bob-dig Or we switch to OPNsense which is working ok on the same system where pfSense is not working ok.
-
@dd I couldn't even install OPNsense on hyper-v not long ago, you had to do a trick, and those guys want to use the normal FreeBSD too.
I am stuck, if DDA is not going to work. I have hard time to change my firewall Distro but even more changing my Homserver OS...
-
@bob-dig They have new version 22.1 which is based on FreeBSD 13 and it's working ok. I have tried it. I think that problem with pfSense 2.6 is because it's based on FreeBSD 12.3. Same problem is with pfSense 2.7.0-DEVELOPMENT which is on FreeBSD 12.3 too. I think, fix will not be available for longer time but they must something to do because now is not pfSense 2.6 (and Plus) useable on Hyper-V.
-
@dd said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:
They have new version 22.1 which is based on FreeBSD 13 and it's working ok.
Huh? 22.01 is using 12.3 same as 2.6
-
@johnpoz We talked about OPNsense.
-
@dd said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:
because now is not pfSense 2.6 (and Plus) useable on Hyper-V.
Thanks for your heads-up regarding OPNsense and FreeBSD Version. But there are many people using the newest pfSense on hyper-v, they resolved their problems it seems, although I don't know what they are exactly doing with it. If you only use it as a firewall and not as a router, that would still work here too or lets say for me at least, your WAN-Speed was and is also affected.
So why is it so different, really making no sense to me.
@dd said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:
but they must something to do
I don't think so, we are not paying them and hyper-v isn't used much anyway, so... no.
-
Are you able to test FreeBSD 12.3 in a similar config?
-
@stephenw10 I can't. Is in the current release this already patched? I can't tell for sure, would be helpful to know.
-
No, that patch is not in 22.01 or 2.6.
https://github.com/pfsense/FreeBSD-src/tree/RELENG_2_6_0/sys/dev/hyperv/pcib
It's not yet in 22.05/2.7 either.
Steve
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
@dd said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:
@bob-dig They have new version 22.1 which is based on FreeBSD 13 and it's working ok. I have tried it. I think that problem with pfSense 2.6 is because it's based on FreeBSD 12.3. Same problem is with pfSense 2.7.0-DEVELOPMENT which is on FreeBSD 12.3 too. I think, fix will not be available for longer time but they must something to do because now is not pfSense 2.6 (and Plus) useable on Hyper-V.
I can 100% confirm this, tried it myself. I could also enable all offloading there.
This is from a VM with 4 Cores on another VLAN (and host).
What I had to do on both is disabling VMQ in the virtual NICs in hyper-V, otherwise there were some error messages in the console.
So no problem with FreeBSD 13 on Hyper-V (Server 2022) with the other thingy.
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
Can anyone seeing this test FreeBSD 12.3 directly?
This could be a simple fix if it's something we are setting in pfSense. Though I'm not sure what it could be.
Steve
