Help on how to setup pfsense as a wireless AP/FW/router in virtual box

[hching]

Dear all,

I have been researching the topic on and off in the last two weeks and couldn't get it to work.

What I have is and old pc (Phenom II on 8gb of ram) with 2 nics and an USB wifi (requires zd1211rw driver) running virtual box in Debian Wheezy.

I want to bridge my wifi with all VMs running on my old PC. The bridge will also act as a DHCP server handing out IPs in 192.168.21.0/24 and connect to the internet via one of the nics.  Separately, there will be an independent wired network connected to the other physical nic under 192.168.22.0/24.

Here is the setup.

(internet) => pc (virtual box running pfsense 2.0.3-release AMD64) =>
WAN=em3 (dhcp)
LAN=bridge0 (dhcp server ip 192.168.21.1, 255.255.255.0 mask)
OPT1=em1 (dhcp server ip 192.168.22.1, 255.255.255.0 mask)
OPT2=em2 (no dhcp server)
OPT3=em0 (no dhcp server)

Where
em3=adapter 1=bridge adapter, eth0, deny promiscuous in virtual box network setting, currently connected to my router that connects to my isp for testing, eventually it will connect directly to my ISP.
em0=adapter 2=internal network, internet, deny promiscuous in virtual box network setting
em1=adapter 3=bridge adapter, eth1, deny promiscuous in virtual box network setting
em2=adapter 4=bridge adapter, wlan0*, deny promiscuous in virtual box network setting
bridge0= OPT2, OPT3

*wlan0 is setup by hostapd on Wheezy, wpa2 authenticated.  It is not bridged to anything.  Unfortunately, my zd1211rw is not detected by pfsense therefore I can't  set it up from within pfsense as an wifi adapter.

Advance:System Tunables:net.link.bridge.pfil_member =0;  net.link.bridge.pfil_bridge =1
firewall rules:
LAN: any LAN subnet can go to any, any OPT2 subnet can go to LAN subnet, any OPT3 subnet can go to LAN subnet
OPT1: any OPT1 subnet can go to any
OPT2: any OPT2 subnet can go to any, any LAN subnet can go to OPT2 subnet
OPT3: any OPT3 subnet can go to any, any LAN subnet can go to OPT3 subnet

Firewall:NAT:Outbound: set mode to "Automatic" all default rules were deleted.

My problem:
I can connect to internet, get ips from OPT2 (192.168.22.x) and it just works.
I can get ips (192.168.21.x) from bridge0 in my vms running in Wheezy and my laptop connected via wifi which is setup by hostapd on Wheezy.

The problem is none of the 192.168.21.x machines can connect to the internet.  I monitored the firewall log but nothing seemed out of the ordinary.

I have googled quite a bit but couldn't find anything definite and none of the solutions work with my setup.  I am still learning pfsense.  As such, I would be grateful if someone can give me pointers to the right direction as I can't even narrow things down to whether it is a problem with pfsense, wheezy, virtual box or hostapd on wheezy, or my wifi adapter.

As a side note, I have tried without any bridges and therefore 3 separate interfaces.  em0 and em1 always work but the em3 (wifi, setup on host and linked to pfsense vm as a regular network adapter) only gets an IP but no internet access.

Many thanks in advance.

[hching]

mild bump. =)

[kejianshi]

Do they have DNS?  Have you tried pinging the ip of a google server like:
ping 8.8.8.8

(I know its simple and you probably already did the simple checks)

[hching]

Hi kejianshi,

Many thanks for the reply.  No, I wouldn't assume I have done a good job at trouble shooting.  Routing and network has been one of my many weaknesses and I have not been exposed to pfsense until few weeks ago.  I am at most a pro-sumer.

Regarding DNS servers settings, using OpenDNS, I have tried the following:

1. setting it up at pfsense:System:General Setup:DNS Servers: 208.67.222.222, gateway=none
2. setting it up at the client (e.g. Windows:IPv4 Properties:General:Preferred DNS server addresses:208.67.222.222)

The results are the same.  The LAN:bridge0 interface offered a valid IP address but gave a time-out when I ping www.google.com.  No internet traffic gets through.  The OPT1:192.168.22.x interface just works.

As a side note, before I bridged, on OPT2:Wlan0 offered a valid IP through its DNS server.  I can't connect to the internet but when I ping www.google.com, I would get:

PING www.google.com (74.125.128.147) 56(84) bytes of data.
timeout…..

So it did a bit better as www.google.com was resolved to a valid external IP address, but then got stuck somewhere.

Unfortunately, every time I try to bring wlan0 to an interface, it seems to cause problems exchanging traffic to the outside world.

I thought about bridging the eth1 and wlan0 on Wheezy first before passing it as one interface (say br0) to pfsense through virtual box.  However, even if it works, it's less than ideal because it is one less interface pfsense can manage and further reduces the role pfsense can play in this set up, I think...

[kejianshi]

I'm pausing because I'm wondering if there isn't a better way to do what you are trying to do with the hardware you have in hand.  I have a habit of NOT virtualizing pfsense unless its unavoidable and going ahead and virtualizing everything else.  I've just had less headaches that way.
If every OS and every client I had was running in a VM, pfsense is the one thing I'd try to keep on hardware.
But I don't know how much hardware you have or how many client OS you must support.

That said, I do see people here talk of some success with virtual box.

My feeling is that if you want a pfsense VM to be able act as router for some other VMs on the same physical box, thats easy enough.  But when you start trying to get pfsense to ALSO act as router to some external physical machines or wifi APs, I really start thinking you need a different hypervisor than virtual box.  Something that can take exclusive control of physical NICs and hand it over to pfsense. (I'm no virtual box expert though).

I think v-sphere might do a much better job when you start talking about pfsense needing to manage a mix of virtual and physical clients.

[hching]

Hi kejianshi,

Thanks again for this.  This full size computer used to be my linux server for printing, mail,  ftp, ldap, simple routing (as a dual stack (IPv6/IPV4) router connected with a switch), web, cloud storage and media center but I re-purposed it in favor of a lower power mini ITX setup.  So I am really just experimenting and trying to push things a bit of how far things can go.

I wanted to pursue this route because I felt it is a flexible/easily scalable implementation.  I used to have routers running DD-WRT and hacked them to do all/some of the above but I ended up with many special purpose network devices and it became very cumbersome to manage.  It also became more expensive from a hardware and learning point of view as I found myself having to learn many different slightly different systems.  Performance also suffered because none of these boxes can give me real time data encryption quite like a current generation desktop cpu running say linux when moving large files through SATA or USB3.0 connected hard drives.

As such, I want to replace as many physical devices using VMs as possible.   I understand keeping pfsense (router) separate is superior because my network will not go down along with my computer but at this point I am willing to make the trade off. The VMs make it very easy to backup and restore.

Back on topic, to answer your question, I intend to connect less than 10 devices (including TVs, game consoles, etc) to this setup, if I get it to work without breaking things too much.

I feel I am getting quite close. pfsense is already talking to the two physical NICs and the virtual network.  The problem really is with wlan0 which is "translated" by virtualbox as just another wired NIC to pfsense, and pfsense has already managed to assign IPs to machines that are connected to this interface (bridged or unbridged).  I feel that I am just missing something obvious to get internet working due to inexperience… haha..  As I have time, I will continue to experiment with it and try to learn a bit more about virtual machines and pfsense.

Failing everything, one option is to buy a simple wireless switch to connect to my LAN facing NIC.  I need to get more ports anyway.   I will also look at v-sphere.  I know nothing about it at this point.

BTW, would you have any good suggestions if I want to learn how to analyze and trouble shoot network problems as a beginner? Should I learn how to analyze packets to trouble shoot things such as this?

[kejianshi]

Ohhhh.
Ummmmm….  Google?

[johnpoz]

"I will also look at v-sphere."

So this box id going to be your VM host?  Then yeah I would run esxi (vsphere) over virtualbox for sure!!

As to your wifi issue - just get a wireless router and use it as AP, any wireless router will do…  Don't you have one laying around?  What did you do for your router before?

I run my whole network off a VM pfsense on esxi box..  Router on vm for your physical network works great, there is little reason not to do it if you ask me..  And would be the 1st thing I would visualize not the last ;)

The LAN nic on my isxi host is connected to my physical switches, and both real and vms have access to the internet through pfsense vm.  I also have wlan on its own segment where pfsense is firewall between wlan segment and lan and internet and even a dmz segment that is vm access only through pfsense.

[kejianshi]

Minus the difference of opinion about virtual vs physical firewall, if you are already running pfsense in vsphere to manage both virtual and physical clients, you would probably be the perfect person to walk hching through it.

I MIGHT even consider doing it at 1 location because no one is there to fix anything if something should break so I do have an interest in running the least amount of hardware possible.  (Its the middle of nowhere)

[johnpoz]

Sure happy to walk you through it

Here is a basic diagram of how it would be setup in the most basic mode - 2 nics in your physical host. 1 to your internet (wan) other to physical lan (lan).

You could clearly get fancier with it - break out your vmkern port group to its own phy nic.  More nics in the host could allow you to breakout your wlan to its own physical segment, dmz or other firewalled segments.

Or you could also use vlans to run your different segments between vm and phy over just 1 physical nic.  Inside the host you could add as many vswitches or portgroups to breakout vlans, etc.

Other pic is my current vswitches in my esxi host.  See how pfsense is tied to wan, lan, wlan and dmz.

[kejianshi]

So much more sane than virtual box.

[johnpoz]

Virtualbox is more for running a VM on your PC to test something, etc.  I would not use it for setup like this.  Its great if you want to fire up another copy of windows or linux to test something or run some questionable code or visit a questionable site, etc.

You could set it up to work sure - but its much easier in something like esxi, which is FREE as well.

[kejianshi]

I do plan to set up pretty much exactly what you have laid out here in at least one place.  Eventually.  Perhaps after a stable release of 2.1 when I will be forced to visit that place again.

[johnpoz]

Oh so your waiting for 2.1 to release before you setup the VM environment.

Not sure what sort of access you have to this location.  But if make sure you can access the esxi host.. You could set it up now and then just update to 2.1 final.. If any issues you could correct remotely, etc.

[kejianshi]

No physical access except by long plane ride.