Thank you all for the help. I managed to solve the issue.
Please close the topic

solution: delete all the network's interfaces in the virt-manager, create everything and add static ip to the bridge of the LAN NIC :)

I would do it with Vlans.

I would use a direct cable from the modem to an esx host. use one nic on that host for a wswitch for internet. The rest i would build as vlans distributed over all esx servers.

Out of the box is 1 network is wan and has a gateway and other is lan then pfsense would nat.

And your running the test thru pfsense right.. vm1 is either client/server for iperf and vm2 is the opposite right. Did you switch directions. What speeds do you get then.

This thread while old shows link to old blog and also the 1 poster states he sees 8Gbps on esxi.. So you should be see way higher than that I would think.
https://forum.netgate.com/topic/111302/pfsense-tuning-for-10-gbit-throughput/6

also i have same problem. after 2.4.4 upgrade 100 mbps internet connection now 8mbsp - 10 mbps with speedtest.. big bug...

Thank you very much for your answers !

You guys @johnpoz and @stephenw10 are awesome !

I'll make some feedback on this topic later on

Kevin

If you wish to filter the Ubuntu host as well you will need to remove that NIC fro Ubuntu so only pfSense is getting a public IP.
Then add an internal adapter so Ubuntu has an interface in the pfSense LAN subnet. Unless it already does via the shared LAN NIC.

Though I've never actually tried that sort of setup in VB.

Steve

I think I just sorted it all out!

To let pfSense run on Hyper-V hosted on single network adapter Intel NUC with multiple VLANs, and together with virtual machines as well as other local Windows services and programs, I took following steps:

Run PowerShell as Administrator Run: Set-VMNetworkAdapterVlan -VMName pfSense -Trunk -AllowedVlanIdList 10-100 -NativeVlanId 1
note: pfSense is the name of Virtual Machine and AllowedVlanIdList covers range of VLANs configured for pfSense. NativeVlanId is left with default ID=1
more about PS command: https://docs.microsoft.com/en-us/powershell/module/hyper-v/set-vmnetworkadaptervlan?view=win10-ps Change Hyper-V Virtual Switch to enable VLAN = 10 (as configured on port 1 of the TP-Link managed switch - see my first post).
note: do not enable any VLAN identification on Virtual Network card configured for pfSense. Change for all other virtual machines VLAN to 10 (do not enable VLAN inside virtual machine - leave default VLAN ID = 0)

Final VLAN configuration as per component:

Intel NUC network card - no change, default config (no VLAN) Hyper-V Virtual Switch - enable VLAN = 10 pfSense virtual machine Network Adapter VLAN disabled every other Hyper-V virtual machine's Network Adapter VLAN = 10 no need to configure or change anything for Docker

I'm yet to test this configuration so any comments welcome.

Screenshot for reference.

0_1539058300648_50f60e35-81d1-4887-80d7-2eaccf3b4efa-image.png

You will have issues if you introduce login/passwords in your VM creations
https://azure.microsoft.com/fr-fr/blog/running-freebsd-in-azure/

Yon,

thank you so much... I was looking around for an answer everywhere for hours. Noone knew why FreeBSD based machines would always start from scratch. When you manually save the state of your vm hyper-v does it just fine, but when rebooting it sits there till $timeout and then nukes the vm. No idea why this actually happens. But disabling vss did the trick, reboot of host+guest in seconds to previous state!. (hyper-v-server 2016 testbench here)

I've spent hours on this and it seems I've solved my problem.

Now:

In pfSense web interface, it checks for updates without issues for real (previously it said it was running the latest version but now it says that update is available). In Ping menu it can ping all addresses without issues, e.g. 8.8.8.8, 192.168.2.1, google.com etc. LAN clients can access the Internet.

What exactly fixed my problems (from what I understand):

In [System - Routing - Gateways] there were 3 gateways configured (2 first ones remained from my testing environment). I had no idea that (a) pfSense would save them all even after I've changed the main gateway IP in vWAN interface, and that (b) it would also still use the very first gateway as by default.
I've removed first 2 gateways, leaving only current one. I think this fixed the issues in Ping menu and returned the ability of pfSense to actually access and share Internet access. In Hyper-V virtual switches settings, I've reverted back to vWAN with disabled "Allow management" and vLAN with enabled "Allow management". Network Connections section in Windows now looks like this: NIC1, set to something different from 1*.2*.3*.250, e.g. 1*.2*.3*.251 (also had to change the mask to allow that). NIC2, with all protocols/components except "Hyper-V Extensible Virtual Switch" disabled. vEthernet (LAN), set to use IP address 192.168.2.1, gateway is 192.168.2.2. Did not change addresses in pfSense, but just in case here they are: vWAN: 1*.2*.3*.250 (including all other parameters from our ISP) vLAN: 192.168.2.2.
Important thing to note about (3) and (4) is probably that the physical server should not use the same IP addresses as pfSense uses on any interface. Otherwise, from my observation, LAN clients could ping all Internet addresses (e.g. 8.8.8.8 and google.com) but were unable to load any website for browsing. Also, I'm not sure about DNS for now, but I've switched from DNS Resolver to DNS Forwarder. I was unable to fully understand how they work and why in most cases I'm unable to test DNS functionality right after I enable either service. Perhaps DNS services need some time to cache something, or I need to "Reset States" (have not tried this after changing DNS services yet). I'm testing DNS functionality like this in command line in Windows: nslookup google.com 192.168.2.2

I'm disappointed that pfSense made me believe it could connect to the Internet at first. I might have been able to solve the problem much faster if it displayed an error message on Dashboard about being unable to check for updates.

Yes the ability to choose a gateway group as the default is new in 2.4.4 as is the "Automatic" selection option. We have seen a few "quirks" with it, with disabled gateways especially. It's behaviour will be updated in 2.4.4p1.

Steve

Why would you get a mismatch error for a new interface?

The only time a mismatch happens is if it can't find an interface that must be there. For example if you had DMZ assigned and then removed that extra interface from the VM without removing it from pfSense first, that would be a mismatch.

All you did was add a new interface, the ones pfSense cared about were still there. That's not an error state.

just figured out the issue,
I had to set nativeVlanID to 0 in order not to lose my lan adapter connection:

but in the original command it has native vlan ID 1
Get-VMNetworkAdapter -VMName "vmname" | Where-Object {$_.MacAddress -eq "XXXXXXXXXXXX"} | Set-VMNetworkAdapterVlan -Trunk -AllowedVlanIdList "1-1000" -NativeVlanId 1

I found it in this instruction https://schimpie.net/2017/03/03/pfsense-vlan-with-hyper-v/

Even though setting native vlan ID to 0, it worked, but it does not make sense.

Got it fixed last night with a bit of fiddling around with open vswitch, it really did seem to be how openreach forward the service to plusnet.

Ok, So I've changed motherboards and CPU (one that allows PCI passthrough for intel NIC) and problem has been resolved.
UnRAID's VirtIO bridge seems to be unstable with pfsense (at least in my previous configuration) or a hardware related issue with previous config.