PfSense in VM, is there a good way to connect host machine to LAN?

johnpoz · Sep 25, 2014, 10:34 AM

Yes your host machine would have access to the internet.. Just don't connect the host to the physical wan or even virtual wan nic/switch and just use it as connection to physical world. The host machine, any other vms are only tied to the lan side physical nic.

Pfsense is the router between and the only thing that has a leg in both the wan and lan.

toysareforboys · Oct 28, 2014, 10:15 PM

I have the same/a similar question.

Right now I'm running pfsense on a dedicated PC, runs fine but now that I've got much faster internet and am doing traffic shaping for voip and IPTV I figure it'd benefit from a little more horsepower (plus I'd like to consolidate a bunch of my servers into one).

So, I got a monster server computer to use for pfsense. I plan on using virtualbox to host all my applications. I haven't been able to make this new server function as our main router when pfsense is running in virtual box :(

Here's my network diagram:

I've set up two bridged adapters in virtual box, one for the "wan" and one for the "lan", each one attached to the respective port on the NC360T. I set both adapters "promiscuous mode" to "all".

I had to clone the physical network card mac (#1, WAN) address into the virtualbox virtual network adapter (#1, WAN) in order for it to get an IP address from my ISP (my ISP only allows one IP address, and without cloning the mac address it was trying to get a second one!).

I can surf the internet from the server no problem (I'm assuming it's using the physical wan connection to do it, not through pfsense), but I can't surf from any computers on my network (they just get a "page can't be displayed"). From the machines on my network I can ping google.ca no problem, same with 192.168.1.1 and 8.8.8.8, and I can surf to 192.168.1.1 (pfsense gui) but not any other sites :(

I've tried:

Cloning the mac address from the physical lan card (#2, LAN) to the virtual network adapter (#2, LAN) but no change. I've tried manually specifying the IP address on the physical lan card (#2, LAN) to 192.168.1.1 but then all hell breaks loose. Machines on the network can't ping or surf to 192.168.1.1 and pinging google.ca results in 90% of the time "can't find host" and 10% of the time works.

What am I doing wrong?? :(

Thanks for any advice you can provide.

-Jamie M.

johnpoz · Oct 29, 2014, 12:02 PM

For why in the world would you be using virtual box running on windows? Run esxi and run pfsense and whatever say your windows 2k12 as VMs

And you don't need to clone anything.. In your current setup with windows being bound to that wan nic - unbind windows from it, so ti doesn't get an IP. Then only pfsense will ask for IP.

You windows host would only be bound to the lan nic.

No cloning of anything is required, and if you can ping google.ca but not browse it what are your lan rules on pfsense?

toysareforboys · Oct 29, 2014, 3:47 PM

@johnpoz:

For why in the world would you be using virtual box running on windows? Run esxi and run pfsense and whatever say your windows 2k12 as VMs

Ok, I'll check out esxi instead :)

@johnpoz:

And you don't need to clone anything.. In your current setup with windows being bound to that wan nic - unbind windows from it, so ti doesn't get an IP. Then only pfsense will ask for IP.

You windows host would only be bound to the lan nic.

So I just unbind TCP/IP v4 and v6 on the wan lan adapter?? That'll stop it from sending a DHCP request to my ISP?

@johnpoz:

No cloning of anything is required, and if you can ping google.ca but not browse it what are your lan rules on pfsense?

pfSense is a fresh brand new installation, no changes made, just assigned wan and lan cards.

Thanks again. I'll try esxi and unbind the wan card and see if I can make some magic happen :)

-Jamie M.

johnpoz · Oct 29, 2014, 4:50 PM

you won't have to unbind if you go with esxi its a different sort of setup. But yeah if your using virtual box, on the interface connected to wan just unbind all the protocols from windows on it and windows wont don't do anything with that interface other than bridge it to the virtual box virtual nic and pfsense will grab an IP from your isp.

toysareforboys · Oct 29, 2014, 4:56 PM

@johnpoz:

you won't have to unbind if you go with esxi its a different sort of setup. But yeah if your using virtual box, on the interface connected to wan just unbind all the protocols from windows on it and windows wont don't do anything with that interface other than bridge it to the virtual box virtual nic and pfsense will grab an IP from your isp.

Wow, you're not joking about "different sort of setup". esxi is making my brain hurt.

Before I go all in with esxi just answer me this one question:

1. Will my Windows 2012 R2 instance have direct access to all the different hard drives I have installed in this server, and will the i/o and network performance be as good as it was with just Windows 2012 R2 on it?

This "server" I have setup as a NAS with striped SSD drives and 10gbe peer to peer cards to handle 4k media content across three PC's. If the 10gbe network performance or the 2gb/sec read/write of the striped SSD drives suffers under esxi then I don't think it will be an option for this box :(

-Jamie M.

johnpoz · Oct 29, 2014, 8:37 PM

You didn't show any 10gbe cards in your setup that is for sure. Not that I saw

You can get direct access to the disk, what controller do you have in it? I just do a raw map to my disks so the vm can view the smart info, etc. I pull 100MBps from my VM nas without much issue. But these disks are nothing special, cheap storage drives 7200 rpm, etc.

But yes it is possible to do passthru to the Vm of the disks and network cards.

To be honest if you are using the box for that - I wouldn't be putting pfsense on it in a VM running in virtualbox. Just get a different box for pfsense would be a better option for sure!

toysareforboys · Oct 29, 2014, 8:44 PM

@johnpoz:

You didn't show any 10gbe cards in your setup that is for sure. Not that I saw

You can get direct access to the disk, what controller do you have in it? I just do a raw map to my disks so the vm can view the smart info, etc. I pull 100MBps from my VM nas without much issue. But these disks are nothing special, cheap storage drives 7200 rpm, etc.

But yes it is possible to do passthru to the Vm of the disks and network cards.

To be honest if you are using the box for that - I wouldn't be putting pfsense on it in a VM running in virtualbox. Just get a different box for pfsense would be a better option for sure!

I already have a different box for pfSense, I'm trying to consolidate :)

The diagram would have been too complicated to draw everything, I only drew what virtualbox and pfsense was going to be touching, there are four other network cards in there (three 10gbe and an onboard gigabit "management" one).

Alright, I'll give it a go and see how it works. Thanks for the detailed answers :)

-Jamie M.

johnpoz · Oct 30, 2014, 11:46 AM

Well use something else to consolidate too.. To me a box designed for HIGH IO both lan and disk doesn't seem like the ideal box to be running virtual software on top of, your virtual box idea.. Nor would it prob be a good candidate for visualization itself.

toysareforboys · Oct 31, 2014, 2:14 AM

Thanks so much!! Got everything up and working flawlessly. Man is esxi ever amazing, totally transparent performance wise :)

I downloaded esxi with an unlimited license (no time out, no ram resitrctions) from here: http://www.vmwarearena.com/2013/10/vsphere-55-download-free-esxi-55.html

I was able to re-install my Windows 2012 R2 and following this guide was able to directly map my SATA stripes into Windows (without losing any data on them), and then make them "online" with this guide. Performance is amazing.

I then installed pfSense using this guide: https://doc.pfsense.org/index.php/PfSense_2_on_VMware_ESXi_5

Everything is working great.

I forgot to copy down the mac address of my previous pfSense box so my cable modem is in provisioning mode so my speeds are terrible right now, not sure if I go and clone my mac now if it will kick out of provisioning mode or if it just has to expire the old mac.

Thanks again for all the great info.

One question: How do I expand the pfsense partition to fill the rest of the space on the disk I've assigned it? "Disk usage: 7% of 3.9G"???

-Jamie M.

johnpoz · Oct 31, 2014, 10:53 AM

why do think pfsense would need much space? I only gave it a 4GB disk as well.. My disk 22% of 2.9G

The Free lic from vmware is like 4TB host limit with 1TB vm limit, not sure what you think that website is giving away? Free has unlimited cpu cores as well with a limit of 8vcpu per guest.

Yeah its a great product – only stickler I have with the 5.5 is the client can not edit if you upgrade to version 10 on the hardware, only 9.. So you can upgrade to 10, and then ssh to the host and edit the vmx file to be 9, then you can edit hardware and such again with the vclient.

You are going to get way more performance out of esxi then you would running virtualbox on top of an OS. Don't get me wrong virtualbox is great and has lots of use cases. But if what you want to do is run VMs and get most power of the hardware to the VMs then no its not really the best use.

Been running pfsense on esxi for quite some time so if you have any questions just ask. So your getting your full IO on your disks then and network? Did you get your VM direct access to your 10G cards or are you just connecting that to a vswitch and using vmxnet3 virtual nics on your VM which are 10G, what kind of speeds are you getting? Any loss of throughput?

toysareforboys · Oct 31, 2014, 1:18 PM

@johnpoz:

why do think pfsense would need much space? I only gave it a 4GB disk as well.. My disk 22% of 2.9G

haha, ok. Is there a way to shrink my vmdk, I gave it 20gb on my precious boot SSD.

@johnpoz:

The Free lic from vmware is like 4TB host limit with 1TB vm limit, not sure what you think that website is giving away? Free has unlimited cpu cores as well with a limit of 8vcpu per guest.

When I downloaded ESXi from vmware directly, when I put in the license they gave me, it said it was going to expire in 60 days. When I googled that, peeps said just re-install every 60 days, but that website, when you click the link for "download vmware" it has a tag in it. It gave me a new license number which got rid of the expiry notice.

@johnpoz:

Yeah its a great product – only stickler I have with the 5.5 is the client can not edit if you upgrade to version 10 on the hardware, only 9.. So you can upgrade to 10, and then ssh to the host and edit the vmx file to be 9, then you can edit hardware and such again with the vclient.

At least it gives you a warning before you upgrade it to 10! I clicked that, and then it's like "you can only manage it with the web based/not free whatever" so I said no thanks :)

@johnpoz:

So your getting your full IO on your disks then and network? Did you get your VM direct access to your 10G cards

My drive/file/network permissions are completely messed up at the moment so haven't given it a good test. I was able to add the 10g cards as "pci device" directly to Win2012 R2 VM. I just dropped a file over the network at 500mb/sec and copied from stripe to stripe (internal) at 1.5gb/sec so it seems that everything is working at full speed or close enough to it :)

-Jamie M.

johnpoz · Oct 31, 2014, 1:37 PM

500mbps ?? You mean 500MBps ?? 500mb would be like watching paint dry on a 10Gb connection if you asked me.. I see high 800 to low 900's mbps on my cheap gig equipment, etc.. I pull 100MBps from my VM, etc.. b is bits, B is Bytes ;)

Yes the TRIAL expires every 60 days, just get a FREE license from VMware..

toysareforboys · Oct 31, 2014, 11:26 PM

@johnpoz:

500mbps ?? You mean 500MBps ?? 500mb would be like watching paint dry on a 10Gb connection if you asked me.. I see high 800 to low 900's mbps on my cheap gig equipment, etc.. I pull 100MBps from my VM, etc.. b is bits, B is Bytes ;)

500 (megabytes / second) = 4000 Mbps, not really sure what you didn't understand?

-Jamie M.

Derelict · Nov 1, 2014, 1:29 AM

XenServer. Free with all the goodies - iSCSI, Motion, HA. (All these VMs are on a FreeNAS iSCSI instance.)

![Screen Shot 2014-10-31 at 6.27.38 PM.png](/public/imported_attachments/1/Screen Shot 2014-10-31 at 6.27.38 PM.png)
![Screen Shot 2014-10-31 at 6.27.38 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2014-10-31 at 6.27.38 PM.png_thumb)

johnpoz · Nov 1, 2014, 11:00 AM

"500 (megabytes / second) = 4000 Mbps, not really sure what you didn't understand?"

There you say bytes – b is bites not Bytes, is what was confusing to me.. if you say "500mb/sec " pretty much everyone on the planet would read that is bits not Bytes ;)

And sure Xenserver is another type 1 option.. Much better than virtual box for something that is going to be a perm VM setup.

toysareforboys · Nov 1, 2014, 5:50 PM

@johnpoz:

"500 (megabytes / second) = 4000 Mbps, not really sure what you didn't understand?"

There you say bytes – b is bites not Bytes, is what was confusing to me.. if you say "500mb/sec " pretty much everyone on the planet would read that is bits not Bytes ;)

And sure Xenserver is another type 1 option.. Much better than virtual box for something that is going to be a perm VM setup.

Ahhhhh, lol. I figured if someone said mb/sec instead of mbps then you'd take it as mB, at least that's how I always do :)

-Jamie M.

johnpoz · Nov 1, 2014, 6:42 PM

No B is Bytes, b is bits - that is how it is everywhere!!

http://en.wikipedia.org/wiki/Bit

the lower-case letter b is widely used as well and was recommended by the IEEE 1541 Standard (2002). In contrast, the upper case letter B is the standard and customary symbol for byte.

doing /sec vs ps is 2 different ways to say the exact same thing per sec.

So your doing it wrong ;) And I can not believe you have ran into confusing before ;)

toysareforboys · Nov 1, 2014, 6:44 PM

@johnpoz:

No B is Bytes, b is bits - that is how it is everywhere!!

http://en.wikipedia.org/wiki/Bit

the lower-case letter b is widely used as well and was recommended by the IEEE 1541 Standard (2002). In contrast, the upper case letter B is the standard and customary symbol for byte.

doing /sec vs ps is 2 different ways to say the exact same thing per sec.

So your doing it wrong ;) And I can not believe you have ran into confusing before ;)

My bad. I'll be more clear in the future :D

-Jamie M.