@Draven666:

Ok, I'll cut and paste (and slightly modify!) a message that I posted on the unRAID forum because it concerns virtualizing both products (unRAID and pfSense).

Let's start from the beginning. I built a pfSense server 3 or 4 years ago and I'm now in the process of upgrading it because it still run on version 2.1. I can really see your reaction, I know…I'm a bad guy but hey, if it ain't broke, don't fix it. So, I have a couple of questions for the community before diving head first in the upgrade process. First, since I'll probably upgrade the pfSense host machine, I would really like to virtualize it under ESXi 6.5. Is that possible and secure? Then, will I have to passthrough a couple of dedicated NIC to pfSense or virtual ones will do the trick? Since I'll run unRAID from the same box and probably a couple of Windows and/or Linux VM so, what kind of hardware can support this setup? pfSense doesn't need much so I don't think that I need a really powerful machine for them (unRAID and pfSense) but I would like to have some feedback from others. I have on hand an AMD Phenom II X4 945 or 965 Black Edition (can't remember exactly, but I can confirm upon request) on an Asus M4A89GTD Pro/USB3 or an Intel Q6600 on a P5K. From what I have read on the web, both of these boards don't support passthrough so, I'm looking at the Asus M5A99FX PRO R2.0. I would really like to find a board compatible with passthrough that I can use one of the processor I have on hand so, I can cut down the cost a little bit. I'll probably throw 16 or 32GB RAM, depending on the feedback I'll receive of this post. And, am I forced to use ECC ram for ESXi or non-ECC will do just fine?

For now, that's about it. I would like to thank everyone who took the time to read and answer my post.

Have a nice day.

pfSense is commonly virtualized, the security is good and the performance is good. It works on KVM, ESXI and Hyper-V, but is easiest to setup (GUI-wise) on the last two. (well… probably runs on Xen too, but it's not nearly as popular as the other 3 hypervisors mentioned)

You can choose to passthrough dedicated NICs, which would theoretically increase security, as the NICs are not shared with any other VMs, nor does the Hypervisor do any packet routing for you via vSwitches, but you lose some flexibility in configuration, as well as if you ever wanted to build a 2nd server and seamlessly vMotion/migrate the pfSense instance to the other host if the original host requires maintenance. That and you get simple backups, snapshot capability, etc. Still the option is yours.

My setup is as follows:

WAN connection VLAN2 on physical switch, trunked to ESXi.
ESXi host with 1 NIC (in reality there are more, but you only NEED one for this particular config)
vSwitch with portgroup WAN on VLAN 2 & regular LAN portgroup on native VLAN (0)/None
pfSense receives WAN signal on the VLAN 2 port, routes it through the LAN connection (OPT1, etc)

This is commonly known as a router-on-a-stick configuration, using a single NIC.

If you don't want to mess with VLANs or don't have a managed switch, then two NICs will be required on the host. Create 1 vSwitch with dedicated NIC for WAN, to be used exclusively for pfSense, and plug the WAN connection into that .
Create one or more vSwitches the LAN/OPT1/OPT2 connections, with the desired VMs also plugged into that switch for internet access. The LAN vSwitch NIC will provide internet access for the rest. You can create a vSwitch without a physical NIC attached to it if you only want to provide Internet access to to the VMs connected to it, and not the network at large.

Those CPUs are fine for pfSense, through running hot and power-hungryy for 24/7 use, but if you are going to run other VMs on it, it's probably OK :)

Now, the coolest thing you can do with this setup if you have another ESXI host with proper licensing (or VMUG learning license, $200 a year):

1. Have 2 hosts running in vCenter (the enterprise mgmt server for ESXI), identical vSwitch configurations, and be able to do a live migration of your router from one physical host to another without dropping a single packet.

2. Implement HA (high availability) monitoring so if one host or your pfSense VM goes down, it is restarted automatically on the other host.

Anyway, I'm a fan of virtualizing it, but be sure to know what you are doing, and understand the caveats of hosting your router on a VM sharing resources with other VMs, on a physical host that MAY need maintenance at times.

Turns out it was the virtIO causing issues and switched over to Intel Virtual NIC's

@kapara:

Since disabling have you had any issues?

Nope it has been up and running since I did the disable and I have had ZERO issues.

Yes if i remember right it's the same if you enable Hyper-V on your Windows installation -> Windows becomes a VM

Also Hyper-V is not reachable from the outside if you disable "Allow management operation system to share this network adapter" on
the virtual switch that is your WAN.

",two different network segments try communicate with each other must be used NAT"

No why does this seem to be a common thought.. Why would 2 different network segments connected to the same router need to be natted??  Do they overlap?  You do not need to nat between rfc1918 networks..

If your using KVM, have you read through the sticky
https://forum.pfsense.org/index.php?topic=88467.0

And where ae you placing these rules?  The default lan rules are any any… So if you bring up a vlan - lan should be able to talk to anything on the vlan out of the box.  If you can not - then you prob have a problem with the box on the vlan having a firewall.  Or maybe the vlan is not even correctly connected to pfsense.

Post up your rules on lan and vlans..

And how is your switch configured.  I have a gs108ev3 as well in my av cabinet that I run multiple vlans on..

Thank you for posting this, it's extremely helpful. I'm hopeful changes in 2.4 will benefit virtio performance? I'm not in a position where I can just pass through a nic dedicated to pfsense and so am at the mercy of virtio.

1. I need to setup the 3 physical NICS (LAN, WAN, MGT or maybe DMZ) with static IPs
You only need virtual switching.  Just add as many network adapters as youd like through hyper-v manager and your vm settings.

2. IP address(s) for virtual switch(s) and what types (ext, int or private?)
for outbound(wan) traffic, use an external switch and create at least one external network adapter for your pfsense vm.  You don't have to share this with management os, but take not that your management os wont have access to it.  in your pfsense configureation, this will be assigned an ip either by dhcp or staticely to match the external network.  create an internal switch for all other vms and even your host.  create adapters for all your vms and configure vms with pfsense internal ip address as default gateway.

3. IP addresses within pfsense (LAN, WAN etc).
pretty much answer to 2.
4. Endstate:  I have a WAN link with firewall rules applied and isolated from everything else, LAN link for filtered internet access and a LINK for management of pfsense (web interface and isolated to a workstation only).

I would suggest keeping it in an isolated environment until you are comfortable with it.  Then when you are sure of your abilities to manage it, put it into production.

heper - it's [iperf client] <-> [pfsense VM] <-> [iperf server] that all sit on the same switch;

"iperf client", "pfsense VM" and "iperf server" are each on their own hardware.

I don't this would be considered as multiple L3 setup right?

Been through all those suggestions but I appreciate the responses. I currently have 4 cores assigned, none seem to pin even under heavy load. The I350 nic settings I've left at their defaults. With the Broadcom's I had VMQ's disabled. The IPSec offload is enabled on the virtual NIC's. I'm starting to think that some how the Layer3 configuration is playing a role in the issue. I'm going to do a bit more research and follow up.

Hello heper.
I have looked on status/system log/gateway and there are messages like these:
Feb 9 13:24:48 dpinger WANGW 192.168.9.254: sendto error: 64

The 192.168.9.254 is the ISP's modem address.

On the other hand, this logs are real-time? the last message has date Feb 9 13:24:48, however, if this error message is related with my trouble, should there be messages from today's date? the pfSense firewall is currently operating and failing at every moment.

For the rest,  I have not found anything relevant in other logs options.

Thank you.
Luis

so my emX interfaces will not be changed in vmxX ?

No, of course not.

Does anyone know what the underlying issue of this was? Or if it is going to be resolved? I had 2.3.1 in Hyper-V on 2012R2 experiencing HEAVY packet loss when approaching 5mbps on our MPLS. Once I downgraded to 2.2.6 everything was fine again. I couldn't find any bug referencing this issue. I'm glad to find this problem is more widespread then just me.

Tenacity prevails!

If you assign the bridge to the LAN interface and add both your previous lan interface and the tap0 interface to it, then you can attain access to the internets for your vms.

1. Create a bridge in Interfaces:(assign):Bridges
2. Go to Interfaces:(assign) and determine which device is assigned to LAN
3. Go to Disgnostics:Command Prompt
4. Enter ifconfig bridge0 addm <device assigned="" to="" lan="">5. Go back to Interfaces:(assign) and set BRIDGE0 as the LAN
6. Add the previous LAN device
7. Enable that device
8. Go to Bridges:BRIDGE0 and assign your new device (OPT1?) to BRIDGE0

When you add the device to the bridge with the command prompt, you make certain that the bridge has access to the wire. 
When you add the device to the bridge in the GUI, you make that persist between reboots.

It works!  If anybody want to use Netflix with a Hurricane Electric IPv6 Tunnel you can use a VM to provide a barebones BIND install to filter out Netflix's IPv6 addresses.

If someone besides me shows interest in this thread, I could make the instructions more cohesive  ;D, but for now I'll leave this as is and hope it's useful to anyone else who wants to do this.</device>

Modern OSes like Windows Server boot in 5 seconds

VMware Workstation 12.5.2, running on an Intel i7-2600, takes about 15 seconds to boot a fully-patched Windows Server 2012R2 VM stored on an SSD.