Need some help installing PfSense in ESXi5.5 VM using 3 nics(two networks).

opjohnny

How come you made pfSense 192.168.1.3? Out of the box it takes 192.168.1.1 as its default LAN address.

~~Shouldn't specify a gateway on the LAN.~~ Sorry missed that.

Did you set up an allow rule on LAN2?

I actually made PfSense 192.168.1.4 after those screenshots were taken so I could access the webGUI on it from the second NIC in my RIG. My linksys is LAN-192.168.1.1/WAN-192.168.1.2 and my firebox is LAN-192.168.1.3.

I removed the gateway from the LAN interface/config page because I read that you don't do this in PfSense. No change.

I re-enabled DNSforwarder(originally disabled thinking the DNS 127.0.0.1 was the problem). No change, but I did learn in general settings you can disable the 127.0.0.1 loopback in your DNS list.

I enabled (Automatic) on the NAT > Outbound page thinking "oh wow, this has to be the problem!" No change. :\

I figured I'd have to set up firewall rules for LAN2 since it doesn't have the default ones but I was going to wait until I could figure out why I wasn't getting internet connectivity on either one(specifically LAN which does have the default rules) first. LAN2 won't even be used all that much, it's just for testing/playing with gear I bring home and the occasional server hosting when my gaming community servers are down for maintenance.

Could this be a weird issue with PfSense and the fact it's running inside a VM? I can't even seem to find anyone else that had this issue. Usually it was some painfully obvious setting for them that had to be changed. In my case all my settings look great, even Status > Interfaces looks normal. Everything looks like it's good to go but the net just won't work inside or outside of PfSense.

johnpoz

And did you reboot your SB6121 once you changed out your firebox for your pfsense VM? Or looks like you get a IP there on your pfsense wan.. But can you ping the gateway it gives you from pfsense even?

This is just a no brainer setup to be honest.. It should take no more than a few minutes to setup.. I run very sim setup using esxi on N40L box. I have never had issue one with it, and initial setup was a breeze.

You can get fancy with multiple lan segments after - be it they are connected to only other VMs or physical segment as well.

So in esxi - you should have 2 vswitches one connected to physical nic that is connected to your cable modem, second nic that is going to be your lan network. I would suggest you change your wrt54g to something other than .1 for starters on the default pfsense network.. You can always change it back to that if you want - but to get it up and running lets not have to deal with changing the pfsense default lan IPs, etc.

So lets say you changed your wrtg lan IP to 192.168.1.10/24

That is all working.. Then setup your pfsense VM with its WAN interface connected to your vswitch you that is connected to nic that will connect to your modem.. I change the mac so you for sure now which interface is which in your vm - see blow images. so 01 is WAN and 02 is LAN – so when you setup pfsense your sure which interfaces your connecting to which vswitches.

So once you have that up and running - and you can hit pfsense on its lan interface using the web gui from your physical network.. Then turn off your modem. Shutdown your pfsense vm.. Now connect your modem to the nic in your esxi host that is connected to your wan vswitch.. I have to assume your vmkern is connected to same nic as your lan vswitch will be.

Once your once your modem has rebooted and up and showing sync fire up your pfsense vm.. Bing bang zoom you should have internet and everything golden.

Now you can play with changing pfsense lan IP to whatever you want or whatever network you want. And or adding more network segments be them real or virtual. As you see from the screen shot I have 4 interface in pfsense - wan, lan, wlan and dmz - dmz is not tied to any physical network, etc.

Let me know happy to help - and have been running such a setup for quite some time, before esxi it was just vmware server, have used virtualbox, etc etc.. I would never go back to running pfsense or physical hardware to be honest - too many advantages to running your firewall/router in VM ;)

mac.png_thumb

opjohnny

Sorry for the delayed response. Just got home from work.

I always unplug my modem before disconnecting/reconnecting it to anything in order to ensure a new lease, or the correct lease, is applied.

I can ping the gateway I receive from Mediacom. Results below.

PING 173.17.240.1 (173.17.240.1): 56 data bytes
64 bytes from 173.17.240.1: icmp_seq=0 ttl=255 time=8.030 ms
64 bytes from 173.17.240.1: icmp_seq=1 ttl=255 time=8.627 ms
64 bytes from 173.17.240.1: icmp_seq=2 ttl=255 time=9.708 ms

--- 173.17.240.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 8.030/8.788/9.708/0.694 ms

Screenshot of ESXi Host > Configuration > Networking-

Screenshot of PfSense VM > Settings >

It is possible that I have the names of LAN1 and LAN2 mixed up in that last screenshot but I have them all figured out and working on PfSense so that shouldn't matter, I hope.

Should I go ahead and change the LAN/WAN IP of my wrt54gs and wipe/reinstall the PfSense VM or do you think it's possible to fix this issue without having to do all that?

johnpoz

Well your lan1 is disconnected from physical nic

And according to your vmkern port group that nic is connected to a 192.168.1 network (which you have labeled lan2? Thought lan2 was your 10 network?

opjohnny

Like I said I probably have the names reversed in ESXi. I lost track of which one was which when installing and doing the initial CLI configuration of PfSense(since it uses em0, em1, etc).

There are two vertically-stacked ports on the back of this server. 1 on the top and 2 on the bottom. According to PfSense Port 2 = LAN 1 which is 192.168.1.x and is also my vSphere management network, and Port 2 = LAN 2 which is 10.0.0.x. In ESXi I am pretty sure I have those names reversed but this shouldn't be a problem. Just confusing is all. The third port is a PCI NIC and I made that my WAN.

Using your MAC edit idea this would have been a lot less confusing, haha. Either way both LAN ports hand out the proper IP from the ranges I set in PfSense webAdmin, and according to webAdmin it's also successfully talking with my modem/mediacom, so I'm not sure why I'm not getting any internet connectivity.

johnpoz

Did you do anything other than set the IPs, lan2 or whatever your second lan is going to need a firewall rules to allow traffic.

But by default really it is click click on pfsense and you should be up and running on the internet. There really is nothing to do to have basic internet access - as long as your wan gets an IP and can talk, your normal lan by default has allow all rule and nat is automatic.

So unless you dicked with something it should be working.

If you say you can you get IPs from pfsense dhcp, and you can ping pfsense.. And pfsense can talk to its gateway and the internet then there should be no reason why it doesnt work.

opjohnny

I followed the install guide for PfSense in an ESXi VM exactly. I then got in to the webGUI and that's where I added the second LAN(10.0.0.x). I didn't mess with any settings other than DNSforwarder which I disabled and then reenabled.

I'm getting a WAN IP, WAN gateway, WAN DNS, I can ping and access PfSense from either LAN NIC directly or through my linksys on LAN 1, DHCP on both networks is working, I can still access vSphere Client on the 192.168.1.x LAN NIC, I can ping the WAN gateway within PfSense webGUI, etc. There just isn't any internet connectivity in PfSense or on either network, I can't ping or traceroute any outside hosts, etc.

I'm just as confused as you are, haha. From what I can see on my end I should have internet and I can see no reason why it wouldn't be working. I'm starting to wonder if it's my Dell PowerEdge 860 hardware, or the ESXi drivers for that hardware, that are the problem.

If there are any diagnostics I need to run or screenshots I need to take I'll be awake for about 2-3 more hours.

johnpoz

So are you not resolving dns? Do a traceroute to say 4.2.2.2 what do you get from a client

example

C:>tracert -d 4.2.2.2

Tracing route to 4.2.2.2 over a maximum of 30 hops

1 1 ms <1 ms <1 ms 192.168.1.253
2 25 ms 28 ms 13 ms 24.13.176.1
3 11 ms 11 ms 11 ms 68.85.131.149
4 17 ms 11 ms 11 ms 68.86.197.149

Juts need past your gateway - see that 24.13.178.1 that is my ISP gateway, I told it not to resolve hostnames with -d but works that way too.

C:>tracert 4.2.2.2

Tracing route to b.resolvers.Level3.net [4.2.2.2]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms pfsense.local.lan [192.168.1.253]
2 30 ms 29 ms 39 ms c-24-13-176-1.hsd1.il.comcast.net [24.13.176.1]
3 12 ms 11 ms 11 ms te-0-0-0-17-sur03.mtprospect.il.chicago.comcast.net [68.85.131.149]
4 12 ms 11 ms 11 ms 68.87.230.45
5 13 ms 15 ms 15 ms he-2-3-0-0-cr01.chicago.il.ibone.comcast.net [68.86.94.105]

Here is the thing on your 2nd lan that you put on OPT1, that would NEED to create a firewall rule. What does your traceroute look like from a client on your 192.168.1.0/24 network

I run mine on esxi 5.5 - there is nothing special you have to do..

Did you reboot your vm after you have changes its lan IP, etc. I have heard of people having issues when they change their lan network.. And you have nat on auto right?

Question are you running 32bit or 64 bit? I run

2.1-RELEASE (i386)
built on Wed Sep 11 18:16:50 EDT 2013
FreeBSD 8.3-RELEASE-p11

There is little point to running the BUGGY as shit if you watch the forums ;) 64bit unless your going to give it more than 4GB..

opjohnny

I'm running 2.1-RELEASE (i386), the latest one from the FTP. I read that 64bit was buggy and not worth running since PfSense doesn't require enough RAM to justify running 64bit.

I'll get those traceroute results from this rig on LAN1 of PfSense here in a few mins. Got a big update finishing up on something and I need that before I can swap the modem around. ;)

johnpoz

What adapter you using - I use just e1000 it works fine. What settings do you have on your vswitches? This is my wan vswitch and lan vswitch

wanvswitch.png_thumb

lanvswitch.png_thumb

opjohnny

I'm using whatever the defaults were. I know for sure the adapter type is E1000.

Here is a screencap of ipconfig /all and tracert -d 4.2.2.2 from this rig plugged directly in to LAN1-

opjohnny

I notice in the DNS field it's pulling the gateway/LAN IP of PfSense rather than the two DNS Mediacom usually gives me.

If I run an ipconfig /all on my rig on the firebox network right now I get…

DNS Servers . . . . . . . . . . . : 97.64.183.164
97.64.209.37

This seems like a step in the right direction unless I'm misunderstanding something here.

johnpoz

see hop 3.. Did you set a gateway on your lan or something?

You do not set gateways on lan interfaces!

opjohnny

There was one set and I removed it after finding out that I shouldn't.

johnpoz

Dude there is no possible way it should ever say that in a traceroute - unless it thought it needed to go out that interface to get somewhere. Can you post up your route table – here is mine.

See the default going out my ISP connection.

routepfsense.png_thumb

opjohnny

Sure, give me a few mins.

opjohnny

Ok so, I'm currently posting while online from PfSense.

Even though there were no gateways set on the LAN1 or LAN2 pages, on the Gateways page there were two different LAN gateways there for some reason. I deleted those, rebooted PfSense to continue towards getting you that screenshot and I hear Teamspeak say "Connected."

I'm going to connect my wrt54gs and see if all my stuff on my home network comes online. Brb.

opjohnny

My home network is now online and working. Full network and internet connectivity.

Still need to set those firewall rules for the second LAN which I'm not really sure how to do yet, but I'll cross that bridge when I get to it. I don't even have my switches here yet and currently have no hardware to play with on that "lab network" so it's no big deal right now. I just wanted my home network running and not being hindered by that firebox's 12 user limit.

Thank you so much for your help!

opjohnny

Meh, I was on a roll so I decided to do the LAN2 firewall rules before bed. Do these look ok? I want internet connectivity on LAN2 but I don't want it to see or access LAN1. However, I do want LAN1 to see and access LAN2 since I administrate most stuff on LAN2 from my rig on LAN1.

biggsy

You only need one rule on LAN2:

PASS
Proto: IPv4+IPv6
Source: ANY
S/Port: ANY
Dest: NOT LAN net
D/Port: ANY

This allows through any traffic coming in on pfSense's LAN2 interface as long as it does not have a destination address somewhere in the LAN subnet.