We have three NICs. One for the WAN, one for the LAN and one is used to connect to the management interface of the modem. There are virtual switches on the WAN and LAN NICs plus an internal switch that's not connected to either NIC. The WAN switch is not shared with the OS. The LAN switch is shared with the OS. This allows there to be more than one instance of pfsense for testing. VMs can either connect to the LAN switch or with the internal switch (for testing).

Okay, i may have found the issue, though it should be solved in a driver update, but the issues seems to be the same i'm experiencing.
It relates to VMQ on networkcards.

Here is the article i found about it: http://www.aidanfinn.com/?p=16876

I will let you know if it solves my problem.

Thanks for pointing me in the right direction. The pfsense ISO's were corrupted on upload. It took me about 8 tries from more than a few different machines to get on to upload and have the same sha hash. I'm not sure what's causing it because other ISO's have uploaded without problems.

@kapara:

Jul 16 00:39:53 kernel calcru: runtime went backwards from 8791 usec to 4441 usec for pid 321 (devd)
Jul 16 00:39:53 kernel calcru: runtime went backwards from 1889 usec to 966 usec for pid 321 (devd)

Those are generally harmless, but there is a fix in 2.3.2 from Microsoft that makes it go away.

@kapara:

Jul 16 00:54:25 charon 08[CFG] <13> received proposals: IKE:BLOWFISH_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Jul 16 00:54:25 charon 08[CFG] <13> configured proposals: IKE:3DES_CBC/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Jul 16 00:54:25 charon 08[IKE] <13> received proposals inacceptable

Right there - your config doesn't match. Blowfish on one side, 3DES on the other.

Thank you for your help.

Currently all 3 ports are untagged trunk on vlan 1000 on my switch. pfSense1 on port1, pfSense2 on port2 and modem on port3.

I will change port 1 and 2 to tagged on switch and put both pfsense vm in the same port group in ESXi with vlan id 1000. Is this correct?

ok thanks, this is what I got

[root@localhost:~] esxcli storage core device vaai status get t10.ATA_____SanDisk_SDSSDXPS240G____________________162336401593________   VAAI Plugin Name:   ATS Status: unsupported   Clone Status: unsupported   Zero Status: unsupported   Delete Status: unsupported

it appears the SCSI UNMAP wont happen here.  thanks though

You will probably want to disable time sync. from there.
Don't want the host and NTP both trying to control the clock.

I thin provision my test VMs and give them ~8GB of space. Never had an issue with that size, even with some swap space configured, but none of them have much package data either.

@rudelerius:

@XanderVR:

Currently using pfSense with Hyper-V 2012 R2 without issues.
It nicely recognizes the virtual NIC's, and runs OK, even with VLAN
However if you want to work with VLAN tagging, the supported way of doing this is to add a virtual NIC for every VLAN you have in use, and set the VLAN tag on VM level settings

(Yes there is a workaround, however you might run into complications using this, as there is no official way to set a virtual machine NIC to trunk mode)

There is a small problem with the 1 NIC per VLAN solution that I ran into, in that there is a limitation in Hyper-V of 12 NICS per VM: 8 synthetic and 4 legacy NICS.  However, using Powershell, you can set a Hyper-V switch port to trunk mode: https://technet.microsoft.com/en-us/library/hh848475.aspx.

The following sets the port on the VM named Redmond to trunkmode and allows access to VLANs 1-100 and tags all untagged traffic to VLAN 10:

PS C:\> Set-VMNetworkAdapterVlan -VMName Redmond -Trunk -AllowedVlanIdList 1-100 -NativeVlanId 10

Thank you so much sir. After many many hours of search and research, your solution worked for me (Windows Server 2012 R2 + HyperV + pfSense 2.3.1)

Just one comment: on allowedVlanIdList, do not include VLAN 1, since on most switches, it´s the default untagged. On my environment (Dell Switches) it didn´t work at all until I used -AllowedVlanIdList 2-XXXX and -NativeVlanId 1 so I can access through my server

if not Hyper V then Vsphere , but technically it will work rite? Just put monitoring IP to ISP Router IP

@Crunk_Bass:

The CPU load of the pfSense VM was at 100% when running the transfer. So I decided to give it more vCPUs, which surprisingly resulted in a much lower transfer speed of 72.5MB/s.

Bad idea to give more vCpu than VM actually needs.
Also, I had VERY inconsistent speed measurements when running test directly from pfS box, like ~5 mB\s on it and whole 40 mB\s on machine behind pfS.

It seems to be a common misconception that multiple port nics are little switches..  Not sure how we kill off this misconception but it really needs to die..

The other misconception is that bridging these interfaces turns them into switches..  The closest it would come to would be a hub, and a shitty one at that.. Bridging has some specific use cases where it makes sense to do so.  Actual use case is when you change media type, say going to from a fiber connection to copper, or wifi to ethernet.

You can use multiple interfaces a lagg to loadbalance traffic through, again not optimal performance here.. If you need more than 1 gig for example you should use a 10ge interface ;)

Thank you for the replies.
Yes, running CARP. Not sure how other people report this as working. Yes, if you shut down one VM the other takes over. But, if an interface goes out, the backup never fully takes over, leaving a non-functioning Internet.
Yes, thought about scripting but not sure how to do it. And wouldn't want to bring down the whole vswitch.
I looked at the Gateway settings and monitoring is on.
Anyway, the easy solution is to pass through the NICs, which is what I've done. Works perfectly this way.
Thought maybe I was overlooking something.

@alfredo:

Interestingly, a 'top -SH' reveals

last pid:  9532;  load averages:  0.51,  0.20,  0.11                up 0+04:57:10  19:03:27
159 processes: 11 running, 118 sleeping, 30 waiting
CPU:  0.8% user,  0.0% nice,  5.6% system,  6.8% interrupt, 86.7% idle
Mem: 20M Active, 121M Inact, 225M Wired, 57M Buf, 7557M Free
Swap: 2047M Total, 2047M Free

PID USERNAME PRI NICE  SIZE    RES STATE  C  TIME    WCPU COMMAND
  11 root    155 ki31    0K  128K CPU4    4 296:40 100.00% idle{idle: cpu4}
  11 root    155 ki31    0K  128K CPU7    7 296:37 100.00% idle{idle: cpu7}
  11 root    155 ki31    0K  128K CPU1    1 295:31 100.00% idle{idle: cpu1}
  11 root    155 ki31    0K  128K RUN    3 296:35  96.97% idle{idle: cpu3}
  11 root    155 ki31    0K  128K CPU5    5 296:32  93.99% idle{idle: cpu5}
  11 root    155 ki31    0K  128K RUN    6 296:27  89.99% idle{idle: cpu6}
  11 root    155 ki31    0K  128K CPU2    2 296:37  86.96% idle{idle: cpu2}
  12 root    -92    -    0K  512K CPU0    0  3:55  55.96% intr{irq258: vmx0}
86328 root      52    0 56664K  6828K select  3  0:11  48.97% curl{curl}
  11 root    155 ki31    0K  128K RUN    0 292:34  48.00% idle{idle: cpu0}

Try with only 1 vCPU. Just try.

Does plaing with "Disabling checksum offload" change anything?

@sschaub:

@zehle:

Solved using Microsoft loopback adapter and then disabling tcp/ip v4 and v6 over the wan

I am trying to do a similar setup, and I don't see how the loopback adapter comes into play.

Ok, never mind. I think what is needed is to follow the instructions in the linked article to install the loopback adapter. Then, use internet connection sharing to share the loopback adapter with the Ad-hoc network.

FreeBSD 10 is NOT certified/supported on ESXi 5.1

Refer to:

https://forum.pfsense.org/index.php?topic=113220.0

Sounds like a shitty job to me, working with old shit day in and day out ;)

That your smb has hardware that is so old they can only run version that came out 5 years ago, again much suck working with such crap…  Sure they are big spendings and pay the big bugs out for support working with such old shit..

I can fire up windows 3.1 on a vm as well.. Doesn't mean something should be doing...

You need to add an additional nic to the pfSense vm.

http://itpro.outsidesys.com/2015/02/19/home-lab-with-pfsense-workstation/