ESXi 6, pfSense, and Plesk Panel 12.5 w/multiple public IP's

[TreeTopFlyer]

Has anybody tried this configuration?  Running the Plesk panel on Centos 6.7.  Using a /28 from Comcast.  I would like to be able to assign my public IP's directly to the Centos/Plesk box if possible and avoid 1:1 NAT which screws with the Plesk DNS.

I've been playing around for the last month (in my spare time) trying to make this work but I really need to get this functioning in the next couple of weeks to move some of my low traffic websites/email to my home office and get rid of 1 of my co-located servers.

I've actually gotten it to work with 5 domains each with a public IP a couple of times even though it shouldn't have been working (by my estimation).  If I wipe out the Centos/Plesk VM and start over, configure it identically, restore the backup from the previous functioning setup, it may or may not work.  It may just work for a couple of the domains.  This basic config consisted of IP .209 being my pfSense/LAN (which works fine) and assigning .210 thru .221 on the Plesk Box in a DMZ.  Transparent bridge. NO Virtual IP's.  If I make VIP's I lose all connectivity to that IP.  The first IP (.210) always seems to be available as long as there is no VIP created for that IP.

One other note:  I've added the Plesk firewall extension to help debug.  From inside my LAN I can reach all domain/IP's and test them with NMAP.  Everything is fine there.  But running NMAP from outside the box shows only the .209 & .210 ports as open.  The other IP's are recognized with tracert up to the WAN but no ports show available.

I have replaced my original pfSense Guide Book with the new pfSense Guide 2.  I have Googled until my eyes are crossed.  I have gone over the documentation on this website several times.  Seems like I can find individual pieces of the puzzle but I'm unable to put them all together . . .  ;D

I feel like I may be trying to over engineer this so if anyone has been able to make this work maybe you could point me in the right direction.  Just knowing someone else has been able to do this would help immensely . . .

If no one has any advice/pointers but are willing to help I will post complete configs and links to the examples I have tried and we can go from there.

Thanks

[johnpoz]

Question so comcast routes this /28 to you via some other transit network??  Or your trying to use some of the addresses behind pfsense while pfsense uses another of those address on its wan.. And your trying to bridge it??

If you want to play ISP - then have the networks actually ROUTED to you… Or go full blow transparent and put all your IPs behind pfsense on this /28, and then admin pfsense from a different interface..

[TreeTopFlyer]

Not sure what you mean by "want to play ISP".  I'm just trying to cut some costs by getting rid of a co-located server that's costing me $300 per month.

Comcast gives me a /28 which consists of the following:

x.x.x.222 is my gateway

x.x.x.209 thru x.x.x.221 are usable IP's

Modem is in bridge mode.

pfSense WAN is configured with x.x.x.209/255.255.255.240

LAN is 192.168.32.1/24  (MS 2008 R2, workstations, AP, etc)

VoIP is 192.168.33.1/24  (asterisk server with phones)

Plesk/Centos main IP is x.x.x.210

Within Plesk I add IP's x.x.x.211 thru x.x.x.221

I'm comfortable with ESXi & pfSense (have several systems like this at clients all with public IP).  Have a pfSense box at all my clients.

I've use Plesk for years on my dedicated servers simply because it's easy for the clients to take care of their own email and web stuff so I'd like to keep it on this system if possible.

I've never had need to set up a system with multiple public IP's in ESXi, pfSense, and Plesk.

I started here:  (this included creating the DMZ in the ESXi)
https://doc.pfsense.org/index.php/PfSense_2_on_VMware_ESXi_5#Creating_the_DMZ_network

Not sure why most of the screenshot images were included despite the disclaimer that "you're screen may be different".  Many images  different from what the author was talking about.  Not really whining, just pointing it out.  A tad bit hard to follow.

Then I went in to pfSense to configure the "System Tunable":
net.link.bridge.pfil_bridge:  1  (default 0)
net.link.bridge.pfil_member:  1  (default 1)
(I tried each variation of the above with no change other than which interface I set rules on)

I then created DMZ interface in pfSense using the 4th NIC, no IP config.

I then created a Bridge with WAN & DMZ, no IP config.

At this moment I only have 4 IP's added to Plesk: .211 thru .214 with .210 being the base IP for the Centos/Plesk VM.

I can RDP thru my VPN to another office and run NMAP to check for open ports against those IP's.  .210 & .214 are fine, .211, .212, & .213 show no ports open.

When I check the firewall logs I see no entries with the destination of .211, .212, or .213.  Entries show for .210 & .214.

I can manipulate the rules and see the results for .210 & .214 so I'm very comfortable with rules I have set up.

Just for reference this system is a Dell 710 server, dual hex core Xeons, 48 gb RAM, and 4 NIC's.

Again, I've found lot's of "pieces" to this puzzle but I just can't seem to put them together correctly.  That's why I was mainly asking if anyone had ever done this kind of configuration.


[johnpoz]

"Comcast gives me a /28 which consists of the following:
x.x.x.222 is my gateway
x.x.x.209 thru x.x.x.221 are usable IP's"

So that is is not a routed network, that is just a /28 they gave you..

"I then created a Bridge with WAN & DMZ, no IP config."

If you created a bridge you wouldn't put an IP on wan.. you would put it on the bridge interface..  Or none at all and just put your /28 Ip on your device behind pfsense with .222 being its gateway, not the pfsense IP..  In bridge mode pfsense is NOT a gateway for that bridge.

[TreeTopFlyer]

OK, thanks for the tip.  That kind of explains 1 of the images from the link to the pfSense docs showing the ESXi WAN with a red "X" on it even though he shows the pfSense console with a public IP for WAN and never mentions that you have to delete the WAN IP.

However when I remove the static public IP x.x.x.209/28 from the WAN interface and config the bridge with the static public IP x.x.x.209/28 I lose internet access on my LAN & VoIP interfaces.  I was unable to get out on the VPN to test from outside the network.

Again, thanks for the help.  I think I'm further along than I was . . . I'll try it again after some sleep and see if I can set up a rule to get out on the LAN.

[johnpoz]

Think you not grasping what a Transparent Firewall is..  It would not be the gateway.. its a BRIDGE..

It becomes a speedbump on the wire..  Since your /28 is not routed better options would prob be to do 1:1 natting of the stuff on that /28 you want behind pfsense.

[TreeTopFlyer]

Not sure where you come up with your assumptions . . .

I simply stated that when I did what you suggested my LAN interface & VoIP interface lost access to the internet.  As for the other half of the equation I couldn't test to see if that solved that half of the problem.

So you're saying that I can't use my 13 static IP's from Comcast with pfSense without doing NAT?  You're also saying that pfSense can't do a "true" DMZ & LAN?

If that's the case you should have just told me that and I'll move on to another option.

Trust me, I did my due diligence in trying to solve my problem.  I rarely call any form of tech support because it's usually a waste of my time.  You obviously seem to know what you're talking about.  Again, I've searched for a month trying to find an example of someone who's done this configuration before with no luck.  This was pretty much my last hope of getting a resolution so believe me when I say how much I dreaded having to post in the forum.

Thanks again for the info you provided.  I don't really think this is a pfSense problem as much as a Plesk issue.  Add to that I've never had to configure a bridge or DMZ in pfSense so in that area I'm a rookie and it's already frustrating enough to Google something and find 25 different fucking ways to do it and none of them work.

Sorry, I get a little bitchy when I haven't slept in 36 hours with the last 12 hours dealing with this problem.

Peace

[johnpoz]

What??

Dude if you want your firewall to be transparent - then make it TRANSPARENT… it is not the GATEWAY in that sort of setup..  Its a BRIDGE!!!

x.x.x.222 --- pfsense (bridge) ----- x.x.x.209, points to .222 as its GW, it doesn't point to pfsense!!

Or if your /28 was actually routed to you!!!

ISP x.x.x.1 - transit network x.x.x/30 - x.x.x.2 pfsense x.x.x.209 ---- x.x.x.208/28 devices

If your wanting to use pfsense as a gateway, then your going to have to NAT if the /28 is not routed too you!!!