How to configure pfsense as vm in front of all vm server network?

vboxbsd

Hi, I'm trying to setup pfsense in a virtualbox vm to put in front of the existing vm servers, all running on mac os x:

–- webserver (192.168.160)
|
|
modem (192.168.1.254) --- mac os x (host 192.168.1.1) --- (192.168.1.2) pfsense (192.168.1.3) --- reverse proxy (192.168.140)--- mailserver (192.168.161)
|
|
--- dbserver (192.168.162)
Details

The above network currently works without pfsense, that is, the webserver is visible externally to the net via a single isp allocated IP address and DNS linked domain name.
It's the common setup of port 80 forwarded from modem to reverse proxy, which then directs queries by header/url to the webserver, and though insecure, it works great for testing.
Preferably I'd love to put pfsense between the modem and my mac, but that isn't a possibility due to my current real world constraints, so my goal is to secure the reverse proxy and backend servers on the mac as much as poss.

The reverse proxy and backend servers are all freebsd with pf running to limit access by address/port, and I'd like to use pfsense as their firewall, gateway (if I'm using term correctly) and DNS.
Local DNS is currently missing from the above setup, but will be important as I'd like to remove need for external mx record lookup for when emailing between backend servers (eg from webserver to mailserver).
If I can't use pfsense for DNS, I'll setup dnsmasq on the reverse proxy, but I'd rather pfsense handle DNS and it is my understanding that pfsense uses dnsmasq too?
But I'm getting ahead of my current issues… (tho' please chime in if you know how to do this with pfsense, as it will be integral to my final network setup).

Each virtualbox server, from reverse proxy to all backend serves are currently setup with with 2 network adapters - 1) em0 - bridged (wired), 2) em1 - host-only (vboxnet0, same IP as the mac host).
This has allowed external tcp and sftp to the webserver and ssh/sftp from host (mac os x) to each backend server.

END GOAL:
Once pfsense is working in front of the reverse proxy, I'd like to remove the bridged adapter from the reverse proxy and backend servers, and replace each with virtualbox's internal network adapter.
My goal is to remove external direct connection possibilities to the reverse proxy and backend servers by connecting them via their own private internal network fenced by their existing server based pf rules.
Only pfsense will have a bridged adapter in virtualbox to connect to the modem via the mac's cabled nic, while also having an internal network adapter to connect to the reverse proxy.
This will be a different internal network name to the one used by the backend servers to connect to the reverse proxy.

I've currently setup pfsense to have 2 bridged adapters; em0 for WAN, em1 for LAN, and an internal network adapter em2 for OPT1 (if needed - I'm unsure?)

Where I'm stumped

1) Why do I lose net connectivity from WAN (em0) of pfsense when I allocate it a static IP after setup?
I've tried the following to fix this:

setup a rule to allow all traffic through the WAN but pinging via the console doesn't work.
adding DNS records eg google's free public ones in Settings > General, but this hasn't helped.
DHCP is already disabled in virtualbox's host network adapter vboxnet0, but WAN doesn't use this anyway… it's using bridged mode.
DHCP server is disabled in pfsense on both WAN and LAN.

2) Why can't I access pfsense's LAN via a virtualbox host-only adapter and only via a bridged adapter? Host adapter would seem more secure?
Is this a DNS issue? If so, do I need add the LAN address/host of pfsense to my mac's /etc/hosts file to access the web configurator via host-adapter?
vboxnet0 is the same as the mac's static assigned IP 192.168.1.1

3) If I can get the web configurator accessible via LAN set as host-only adapter (not bridged mode), then do I need to create a rule to route traffic from it to the OPT1 internal network adapter (em2) for the reverse proxy to use?

4) Should I be bridging WAN to LAN on pfsense to make it a transparent firewall to pass external port 80 requests to the modem to go directly through pfsense to the reverse proxy?
5) Or would it be better (security?) if I don't bridge the WAN and LAN on pfsense but have it pass port 80 requests to LAN and NAT'd before going to the reverse proxy?
As you can see in the network layout all addresses are in the 192.168.1.0/24 net so it would seem a transparent firewall with WAN and LAN bridged and NAT disabled is the way to go?

6) If pfsense is setup as transparent firewall, then will LAN (em1) be configured as internal network adapter in pfsense's virtualbox settings, and will I have to use OPT1 (em2) with host-only adapter to access pfsense's web configurator and backend servers via ssh/sftp from the host (mac os x)?

7) Is my setup the most secure use of virtualbox network adapters - bridged for WAN on pfsense, internal network 1 for pfsense to reverse proxy and internal network 2 from reverse proxy to backend servers, with each backend server having a host-only adapter to allow for ssh/sftp?

I've google all of the above questions, read and tried howtos for setting up transparent firewalls with pfsense, and tried/tested many pfsense and virtualbox settings over the last 3 days.
But there appear to be too many gaps in my knowledge and variables for me to solve this one.
7 questions may seem a lot, but I'm sure I'm missing just a few vital bits of info, and if you can help me get the above layout working, I'll have a greater understanding of networking, pfsense, and even vms.
And no doubt helping me will help anyone else trying to setup this rather common (but not greatly documented) network setup using pfsense and virtualbox.

Many thanks in advance for reading the above wall of text!

I hope someone can help…?

SeventhSon

first of all… wow ;D

i myself have this setup, working perfectly fine (double NAT, but whatever ;)):

internet - pfsense hw (192.168.1.1) - (192.168.1.2) PC - VirtualBox -bridged- pfSense virtual WAN (192.168.1.3) -NAT- pfsense virtual LAN vboxnet0 (192.168.56.1)

I have another VM in vboxnet to configure the firewall with.

Is my setup the most secure use of virtualbox network adapters - bridged for WAN on pfsense, internal network 1 for pfsense to reverse proxy and internal network 2 from reverse proxy to backend servers, with each backend server having a host-only adapter to allow for ssh/sftp?

Seems good enough to me, although I'm not quite sure why you're doing this in VBox (I only use this setup for testing new things)

vboxbsd

Haha, thanks for reading my post and replying - I probably shoulda kept it shorter to get more replies - but I wanted to include the details.

Seems good enough to me, although I'm not quite sure why you're doing this in VBox (I only use this setup for testing new things)

That's one question down - only 6 to go!

To answer your question - I had the hardware equivalent of my vm network running in the past… but I'm 'resource constrained' now, so have to use vms. :(

If I can get pfsense working as described, I'll be crazy enough to put this the existing vm network into production.

The worst thing that can happen is that I'll have to restore vm snapshots or perhaps my whole mac drive from backup. (I may even try to run it from a bootable usb image as a precaution! =)

Those scenarios don't bother me, as once I get my servers live, I should start making enough $ to refresh my hardware and build the physical network out again.

The only thing holding me back now is getting pfsense integrated and working right!

If you could explain the best (most secure) way to use NAT in my layout, I'd really appreciate.

I don't mind changing addresses of any servers if it's going to increase security in anyway.

SeventhSon

I've never done filtering bridge tbh so I'm not really the person to talk to on that, but as far as I know you can use this:
http://doc.m0n0.ch/handbook/examples-filtered-bridge.html

the WAN would be your bridged network adapter to your real LAN and the OPT would be the vboxnet0.

vboxbsd

Thanks for the reply and link!

I've solved 1) and 2) by using the setup wizard and adjusting IP's - somehow settings stuck that didn't when I entered them bypassing the wizard.

I have the adapters right now for WAN and LAN, and after getting NAT working, will read that link thoroughly and look at making the firewall transparent by bridging WAN and LAN.

But for now I've decided NATing/port-forwarding will be more flexible in the short-term eg should I want pfsense to handle redundancy/load-balancing.

And it means I won't have to mess about with virtualbox adapters again for a little while! ;)

The problem I currently have appears to be concerned with nat-reflection…

As I wrote earlier, I can access the public IP from the mac host (and externally) without pfsense integrated.

This includes both the webserver over port 80 and my squirrelmail on email server over 443.

But with pfsense being port 80 forwarded to by the modem, I keep getting redirected to my modems web admin page over https (whereas normal access to it is over http).

The public ip isn't resolving externally, at least from my testing via a proxy, so I'm really confused/frustrated…bleh.

I've set up NAT and port-forwarding rules, tried the auto-generated ones from setting up NAT rules and auto-generated Easy Rules added from the firewall logs, as well as my own tweaks to each.

Before I used pfsense, I fixed the same issue with my modem to allow locally resolving the public IP, by telneting to the modem, enabling nat loopback and trying to delete the relevant wan http/https admin rule.

(For some reason I can't delete the https rule even as admin user as it does'nt recognise the wan group in the rule - though 'wan' is one of the actual group options for their ifdelete command! #)

None of the pfsense rules I've setup or are auto-generated redirect from http to https, and none of my reverse proxy rules could cause this redirection.

So… is the problem how nat-reflection is setup somewhere in pfsense?

I've tried 2 ways to fix this:

i) enabled Nat Reflection settings in my NAT rules (and tried disabling/system default)
ii) using split-dns by enabling dns-forwarding and adding host and domain entries for servers the reverse proxy listens for.

Perhaps I'm doing each wrong??

Once I have this solved, I should have pfsense doing everything needed including dns.

I hope someone has encountered this problem and has advice to fix it.

Thanks