also confirmed updating via "yum update" from a nonworking 5.7 to 5.9 resolved the horrific lag issues!

@cmb:

We do exactly as described frequently, nearly all our production firewalls in several colos, our office, homes, etc. run in ESX (ESXi technically, I and most use ESXi and ESX interchangeably these days). Never so much as a blip. So it's far from a general problem, tons of people do what you're doing with no issues.

Need some more troubleshooting, packet capture to see what gets where, check firewall states for what's getting passed, etc.

What version of ESXi are you running? The systems I have tried it on have all been ESXi 5.0 (I've not tried 5.1 yet). We have many of the same setup on ESXi 4.x without issue.

Nevermind. I did another install and it works fine now. The first install must have been messed up in some way, because I didn't really do anything different the second time around. I also didn't realize I had to add rules to the second LAN connection, and it works fine now too.

If virtualbox can pass the USB device directly to pfSense, it should work, provided that the modem works with pfSense at all.

The setup would be the same as any other multi-wan setup or 3G setup, plenty of docs/examples around on the wiki and forum for both.

The problem might be getting the USB device to pass into the VM. I believe the OSE version of virtualbox can't do that but the precompiled binary package from Oracle can, unless they fixed that since I last tried it.

Thanks to everyone for their hints and suggestions.

Here is what I ended up doing that seems to work.

Add a new firewall rule:
    proto: any, source: WAN, port: any, dest: any

then on a host in the Corp LAN:

route ADD 172.16.1.1 MASK 255.255.255.0 10.3.1.100

and voilà!

I am intrigued by the notion of doing partial NATing and am going to try playing around with that.

OK.  Is there a particular reason you want to pass through the one and only NIC to pfSense?

Indeed! In fact it looks like the disk emulation sometimes takes too long to read or write or it's not even able to read or write at all for some sort of disk emulation crash.
I have been using the latest vbox for a while and no errors at all, but sometimes, with the very same config, i install a newer pfsense snapshot (always working with the latest ones)
and the problem appears again.
I have been using a nice 2.1 snapshot that works perfectly but it's from November 2012. I'm afraid that installing a newer February 2013 snapshots the issue will appear again.

I noticed that disabling I/O host cache the issue is less frequent. Also with some snapshots virtual SATA controller seemed to work better than the default IDE controller.
In some snapshots SATA didn't work at all so not always i could test the SATA setup.

Hopefully this info will help someone.

I understand, I did not want it for free  :P
I could not find a source to learn more, I am stuck with this vnic issue :(

On site B you create two networks in Hyper-V of the external type
Lets call them Outside and Inside
You assign the wan nic to outside and the lan nic to inside

Make sure that the "allow management operating system …." checkbox is selected on inside and is NOT selected on outside

Now create a virtual machine for PFSense and give it one nic from Inside and one from Outside.

After the installation of PFSense is complete your wan traffic will com in on the Wan nic, pass through the Outside virtual switch and reach the Pfsense box outside interface.
Pfsense will then forward the trafic to its internal interface connected to the internal Virtual switch. Since the host machine is allowed to see that switch the trafic can reach it

In theory, you may have gotten past the hard part, and, again, in theory, it should work… till it doesn't, then you're really stuck.  But, hey, you've gotten this far and assuming it's not anything you (or clients) are financially dependent on, run with it.

Agreed proxmox is pretty good too… I like how proxmox uses a web interface and you dont have to have a windows box for esxi (correct me if im wrong) proxmox is fairly simple to learn and navigate,  it is based on debian too. but i do recommend proxmox or esxi.

@AlanMAC:

Do you happen to have another NIC laying around? I've just added a dual port ($45) to my proxmox setup and running two new LANs from pfSense now. I tried doing the virtual NIC, but kept running into problems, so I decided that instead of wasting my time, that I'd just add a dual port NIC…. I'll be adding another one soon as well.

thanks I did get it figured out and do have to say proxmox is pretty easy to learn dont really have any other issues but maybe some firewall/nat troubles

In the diagram the MGMT network is a separate NIC that's connected to the VMkernel Port Group (the management network) and nothing else.  That's just the way I did it because I had a spare NIC.  It's very common to leave the VMkernel Port Group and the VM Port Group (LAN) on the same vSwitch.

You shouldn't be thinking of pfSense as a switch, it's a router/firewall.  If you have multiple physical devices (wifi access point, PC, etc) to connect to the LAN you will need a physical switch, which I think is what Abdsalem referred to as a "pswitch".

Thanks for the feedback all.

I looked over my setup in pfsense and managed to resolve the issue.

Next problem…. how do I get PC's connected to different VLAN's to connect to the Internet ?

All VLAN's are on one switch

WAN interface is on another switch

Both switches have physical NICs

That VM must have been totally corrupted.  Didn't matter what I installed (2.0.1 or 2.0.2), I pretty much got the same results with VMware Tools.  Very odd.

I finished up deleting that VM and resurrecting a 2.1 VM from months ago.  Updated it to the 24 December build and, once I sorted out all the changes I've made since then, I'm back on line again.

I use vmware workstation 9 live cd install having no problems and it works fine

Thanks, I'll switch them over to e1000 and see if that makes any difference.

We are running RC 1 because in the final release the Captive Portal service has a bug where is doesn't accurately track mb usage via Radius.  That is what we use pfsense for to track user's internet usage for billing.

@Supermule:

Maybe because you had both WAN and LAN in the same physical network on the switch??

yup, with heavy network traffic, it would took few hours to freeze the pfsense box.

@quetzalcoatl:

Thanks matguy.

But when you said "pfSense won't really use much more than 2" did you mean 2 cores or 2 gigs of ram.
Since you then talk about ESX it sounds that you are talking about cores.

Besides all the PFSense stuff going on here i have a question for you.

If you meant cores that means that if i have a 6 core CPU and 2 VMs and i assign 6 cores to each one of them, those VMs will actually end up being slower than giving them only 3 cores each?

Because if one of the VMs is idle, the other one should be able to take advantage of all 6 cores, unless the idle VM is actually slowing down all 6 cores even if it's idle. Maybe it depends also on the OS you have inside the VMs.

TIA!

Yes, I was talking about cores.  Having multiple VMs with a couple vCPUs (assuming your VM host has, say, 4 or more cores) is fine as ESX(i) can schedule them easily.  When a single VM has as many (or close to) vCPus as cores in your host it can become difficult to schedule a busy VM as it may have to wait for enough cores to become available all at once.

Generally ESX(i) has to schedule all the cores of a multi-vCPU VM to run at the same time (I think the physical CPU may do some command re-shuffling, but as far as ESX(i) is concerned, they need to be fed to the CPUs at the same time.)  It needs to do that whether or not anything is actually happening on those vCPUs, so even an idle vCPU needs to be scheduled as though it was a busy one.

That causes 2 problems:  1, scheduling these large groups of vCPUs in an otherwise busy host, where that group of 6 vCPUs may have to wait a few, or many CPU cycles for enough cores to become free (think of it like a large family that all wants to ride the roller coaster together, they may have to wait for the next train or 2 to get enough open seats.)  2, filling an otherwise busy physical CPU with cycles that are forced idle by idle vCPUs that have to be scheduled when there may be only 1 or 2 that are actually processing anything.

Like I was saying, this may not be an issue for you if you have very few VMs running on that host, especially if the others are single vCPU VMs, or even 2 vCPU.  I share this more for others that may read this; it's probably not doing you any harm as long as you're not seeing contention or other instability.

I come from more dense environments, where a single host is probably hosting 10 to 30 VMs.  Even on hosts with 12 to 16 physical cores we generally put a limit on VMs to 4 vCPUs, and even then we generally require real justification for going over 2.