Looked around and found some solutions, and also decided a few things.

1.  Shutting down vm's nicely is not part of workstation.  There are a few hacks to get some scripts in windows to force the machines to shutdown, but it's not part of workstation.  So it's not quite a server replacement at this point.  I fail to see what's the point of having auto-startup, but no auto-shutdown as a feature of a shared vm.  I'm hoping by version 9 or so, they will actually get that working.

2.  The network adapter not getting marked as connected seems to be an issue, not just with workstation, but server 2.0 also.  The solution seems to be to mark some vmware servers as automatic (delayed).  The problem seems to happen when the server comes up too fast, before the network is up.  Delaying it, solves the problem.

My end solution ended up sticking with server 2.0 for the time being.  Workstation as a server seems to still be a bit immature (8 is 1st attempt to get the "shared" vm's working).  I'd go with ESX, but I am using the machine as a windows host for other reasons, so ESX is out (and I know I have a piece of hardware that has only windows drivers).

Just an FYI for anyone attempting the same thing…  PFsense on all these configurations works with no issues.

Thanks robi,

A nanobsd virtual machine is very usefull for tests.

att,
Marcello Coutinho

Found the issue and the resolution.

With Microsoft's VHDs if you create a drive with a size over about 31gb it sets the geometry to 255 sectors per track, fdisk does not like this at all!

Making a drive below 30gb will set the disk geometry to a happy 63 sectors per track which fdisk is fine with.

I was able to get tools installed on 2.1 (on esxi 5), and it runs the first time, but after a reboot it gives me an error saying tools aren't configured, and prompting to run vmware-config-tools (or whatever it the name is). Doing that tells me I don't have the proper compat6x libs installed (which they are). Uninstall the compat libs, reinstall them, config vmware tools and it runs. Till the next reboot. Kind of a pain.

Hello Tyger86,

I apologize if this is long winded, but I did something pretty similar to what you are doing, and you can find it here:
http://forum.pfsense.org/index.php/topic,42205.0.html

You did not specify your host OS, but I imagine if you are using VMWare Workstation that while the methods might change the solutions themselves won't be too much different.

My host was Windows, and I had VMWare Workstation 8 with PFSense on autoboot, and a Linux Web Server VM as well.

I used 2 NIC's, one for WAN, one for LAN.  I kept my Host inside the LAN, but separating it is pretty easy.

In any event, I will make some assumptions, feel free to correct me and I will do my best to revise my instructions:

You have a Windows Host
You plan to run PFSense in a VM, and 1 or more others
You want PFSense to act as the gateway for all VM's
WAN Port & Optionally LAN Port

First, where to put the host!

Inside the LAN:

On your WAN Adapter, disable all Windows services, this lets PFSense use this adapter without any interference.  Windows will not get an IP from the WAN and will not communicate through it.

Outside the WAN:

Do the opposite, keep all Windows services on in the WAN adapter, but turn them off on the LAN Adapter.

Separated from both WAN & LAN:

Simply turn off all windows services on both adapters.

Configuring the adapters, this is done with VMWare Workstation's Network Adapter Configuration utility.

By Default VMNet0 bridges any available connection, you have to change this or else you cannot create any other bridged adapters.  So simply specify what adapter VMNet0 is bridged to, and add a the second adapter to a new VMNet# as bridged also.

When setting up PFSense, be sure to make note of each adapters virtual mac address for the installation, and add both bridged adapters to the VM.

For all other VM's on your setup, add only the bridged LAN adapter.

If you plan to expand your Lab or add additional external servers in the future, I recommend connecting the LAN adapter to a Switch.

Potential Problems:

If the second adapter is not connected to a cable, Windows may turn that adapter off.  One solution is to simply connect it to another device, such as a Switch.

The alternative is to add a virtual loop back device, and bridge it to the LAN Adapter.

If you only have one adapter, you might be able to setup a Virtual Loop Back device as the second VMNet bridge.  Only one NIC required, but doesn't give you the ability to easily expand your Lab in the future.

I hope this helps, and look forward to hearing back.

Sorry, I missed yor earlier post.  If you run the Windows 7 Kiwi syslog server behind pfSense (both as VMs, with the Win 7 machine in a DMZ) you should be OK from a security point of view.

@luke240778:

Ok, so if i was to have another physical server as a squid server, can that still work with pfSense Lightsquid for reporting?  Or would i need to run lightsquid on the same box as squid?

I just like the lightsquid plugin on pfSense so was hoping that can be done.

You will have to run lightsquid on the same server as squid. If you wanted to run it on your pfsense vm, you would have to manually move the log file over. IMHO wouldn't be ideal and be more of PIA…

No, it's ok. There's definitely a big difference between not starting at all and just getting by.

Having the back door into the console would be useful, so perhaps this could be a good use of the spare nic.  You're right about the Realtek adapter too - best to focus on getting it up and running and then look at this later if needed.

Thanks for your help :)

@TooMeeK:

I gave it 1 CPU-core (Xeon E3 1220 3.1GHz) and 512MB RAM but it actually idled at around 50-60% CPU usage on the KVM host

This happend to me with HP Proliant ML110 G5 with Intel(R) Pentium(R) Dual  CPU  E2160  @ 1.80GHz that has no virtualization capabilites Intel® Virtualization Technology (VT-x)
http://ark.intel.com/products/29739/Intel-Pentium-Processor-E2160-(1M-Cache-1_80-GHz-800-MHz-FSB)
This also can happend with disabled VT-x in server BIOS. KVM is running then in basic mode, causing randomly pfSense hangs and 100% usage in idle.

Both VT-x and VT-d is enabled and working for sure which of course the performance of all the other linux and Windows based VMs with virtio confirms.

The high cpu usage is a result of about 5 mbit/s and a couple of hundred connections. When there is absolutely no traffic at all the idle usage is 1-5%.
As I said I get about 450 mbit/s max performance from pfsense (non routed, less than half that when routed) which is about as good as it gets with pfsense on KVM without better support in FreeBSD.

Until someone can show me hard iperf performance numbers that greatly surpasses 450mbit/s on KVM for pfSense I really don't believe that there is something wrong with my KVM-setup and not pfSense, at least as long as linux pfSense replacement vm to vm performance is 19,6 GBit/s, routed performance is 9,6gbit/s and physical host to linux vm is 36 gbit/s with iperf on the same KVM host.

Basically 1/40 of the network performance with pfSense vs linux on KVM.

Have You tried Proxmox 1.9 beta?
I'm also waiting for virtio drivers support.

Hi and thanks for info & discussion!

In fact, this configuration (2 interconnected ESXi hosts + 2 FreeBSDs with CARP) works even without VDS (ESXi 4.1). But there is one problem: packets sent to "internal" CARPed IP or other IPs on "internal" (VM's) interfaces of FreeBSD's are transmitted to both machines. I.e. we converted our switch into hub. It's problem if network bandwidth matters (I don't know about performance impact of this behavior in case of transmitting internally in one ESXi, but we use ESXi-ESXi HW network to send this packets). For external network it's not big problem as external HW switch are still works as switch.

Try enabling Spoofing of MAC addresses on the network adapter setting of the pfSense VM in Hyper-V.

Whenever I have problems with ARP this seems to sort it out.

I've also had this problem with Untangle so I guess that it's not a BSD problem.

Well I had solved this.
The first step is to create multiple-bridge script in /etc/xen/scripts

/etc/xen/scripts/network-bridge netdev=eth0 bridge=xenbr0 start /etc/xen/scripts/network-bridge netdev=eth1 bridge=xenbr1 start

now we need to edit xend-config.sxp
change
(network-script network-bridge)
to
(network-script multiple-bridge)
After xend restart or reboot

I get bridge name    bridge id              STP enabled    interfaces xenbr0          8000.0015c5f54501      no              eth0                                                         tap1.0                                                         tap2.0                                                         vif1.0                                                         vif2.0 xenbr1          8000.0015c5f54502      no              eth1                                                         tap1.1                                                         vif1.1

@tritron:

Esxi performance cost money. I would say that 0 dollars xenserver will be much faster.

Are we talking about money or performance?

I've tested both with redhat support and vmware support.
Both had same performance.

Vmware has good points and xen others.

Vmware is much more compatible then xen with those specific clients kernel versions requirements.

Btw. I do not like flames

Congratulations on you xen virtual server.

Does pfsense like one better than another? I'm using IDE right now. The choices are IDE, SATA. SAS and SCSI.