Need some help installing PfSense in ESXi5.5 VM using 3 nics(two networks).

opjohnny

Like I said I probably have the names reversed in ESXi. I lost track of which one was which when installing and doing the initial CLI configuration of PfSense(since it uses em0, em1, etc).

There are two vertically-stacked ports on the back of this server. 1 on the top and 2 on the bottom. According to PfSense Port 2 = LAN 1 which is 192.168.1.x and is also my vSphere management network, and Port 2 = LAN 2 which is 10.0.0.x. In ESXi I am pretty sure I have those names reversed but this shouldn't be a problem. Just confusing is all. The third port is a PCI NIC and I made that my WAN.

Using your MAC edit idea this would have been a lot less confusing, haha. Either way both LAN ports hand out the proper IP from the ranges I set in PfSense webAdmin, and according to webAdmin it's also successfully talking with my modem/mediacom, so I'm not sure why I'm not getting any internet connectivity.

johnpoz

Did you do anything other than set the IPs, lan2 or whatever your second lan is going to need a firewall rules to allow traffic.

But by default really it is click click on pfsense and you should be up and running on the internet. There really is nothing to do to have basic internet access - as long as your wan gets an IP and can talk, your normal lan by default has allow all rule and nat is automatic.

So unless you dicked with something it should be working.

If you say you can you get IPs from pfsense dhcp, and you can ping pfsense.. And pfsense can talk to its gateway and the internet then there should be no reason why it doesnt work.

opjohnny

I followed the install guide for PfSense in an ESXi VM exactly. I then got in to the webGUI and that's where I added the second LAN(10.0.0.x). I didn't mess with any settings other than DNSforwarder which I disabled and then reenabled.

I'm getting a WAN IP, WAN gateway, WAN DNS, I can ping and access PfSense from either LAN NIC directly or through my linksys on LAN 1, DHCP on both networks is working, I can still access vSphere Client on the 192.168.1.x LAN NIC, I can ping the WAN gateway within PfSense webGUI, etc. There just isn't any internet connectivity in PfSense or on either network, I can't ping or traceroute any outside hosts, etc.

I'm just as confused as you are, haha. From what I can see on my end I should have internet and I can see no reason why it wouldn't be working. I'm starting to wonder if it's my Dell PowerEdge 860 hardware, or the ESXi drivers for that hardware, that are the problem.

If there are any diagnostics I need to run or screenshots I need to take I'll be awake for about 2-3 more hours.

johnpoz

So are you not resolving dns? Do a traceroute to say 4.2.2.2 what do you get from a client

example

C:>tracert -d 4.2.2.2

Tracing route to 4.2.2.2 over a maximum of 30 hops

1 1 ms <1 ms <1 ms 192.168.1.253
2 25 ms 28 ms 13 ms 24.13.176.1
3 11 ms 11 ms 11 ms 68.85.131.149
4 17 ms 11 ms 11 ms 68.86.197.149

Juts need past your gateway - see that 24.13.178.1 that is my ISP gateway, I told it not to resolve hostnames with -d but works that way too.

C:>tracert 4.2.2.2

Tracing route to b.resolvers.Level3.net [4.2.2.2]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms pfsense.local.lan [192.168.1.253]
2 30 ms 29 ms 39 ms c-24-13-176-1.hsd1.il.comcast.net [24.13.176.1]
3 12 ms 11 ms 11 ms te-0-0-0-17-sur03.mtprospect.il.chicago.comcast.net [68.85.131.149]
4 12 ms 11 ms 11 ms 68.87.230.45
5 13 ms 15 ms 15 ms he-2-3-0-0-cr01.chicago.il.ibone.comcast.net [68.86.94.105]

Here is the thing on your 2nd lan that you put on OPT1, that would NEED to create a firewall rule. What does your traceroute look like from a client on your 192.168.1.0/24 network

I run mine on esxi 5.5 - there is nothing special you have to do..

Did you reboot your vm after you have changes its lan IP, etc. I have heard of people having issues when they change their lan network.. And you have nat on auto right?

Question are you running 32bit or 64 bit? I run

2.1-RELEASE (i386)
built on Wed Sep 11 18:16:50 EDT 2013
FreeBSD 8.3-RELEASE-p11

There is little point to running the BUGGY as shit if you watch the forums ;) 64bit unless your going to give it more than 4GB..

opjohnny

I'm running 2.1-RELEASE (i386), the latest one from the FTP. I read that 64bit was buggy and not worth running since PfSense doesn't require enough RAM to justify running 64bit.

I'll get those traceroute results from this rig on LAN1 of PfSense here in a few mins. Got a big update finishing up on something and I need that before I can swap the modem around. ;)

johnpoz

What adapter you using - I use just e1000 it works fine. What settings do you have on your vswitches? This is my wan vswitch and lan vswitch

wanvswitch.png_thumb

lanvswitch.png_thumb

opjohnny

I'm using whatever the defaults were. I know for sure the adapter type is E1000.

Here is a screencap of ipconfig /all and tracert -d 4.2.2.2 from this rig plugged directly in to LAN1-

opjohnny

I notice in the DNS field it's pulling the gateway/LAN IP of PfSense rather than the two DNS Mediacom usually gives me.

If I run an ipconfig /all on my rig on the firebox network right now I get…

DNS Servers . . . . . . . . . . . : 97.64.183.164
97.64.209.37

This seems like a step in the right direction unless I'm misunderstanding something here.

johnpoz

see hop 3.. Did you set a gateway on your lan or something?

You do not set gateways on lan interfaces!

opjohnny

There was one set and I removed it after finding out that I shouldn't.

johnpoz

Dude there is no possible way it should ever say that in a traceroute - unless it thought it needed to go out that interface to get somewhere. Can you post up your route table – here is mine.

See the default going out my ISP connection.

routepfsense.png_thumb

opjohnny

Sure, give me a few mins.

opjohnny

Ok so, I'm currently posting while online from PfSense.

Even though there were no gateways set on the LAN1 or LAN2 pages, on the Gateways page there were two different LAN gateways there for some reason. I deleted those, rebooted PfSense to continue towards getting you that screenshot and I hear Teamspeak say "Connected."

I'm going to connect my wrt54gs and see if all my stuff on my home network comes online. Brb.

opjohnny

My home network is now online and working. Full network and internet connectivity.

Still need to set those firewall rules for the second LAN which I'm not really sure how to do yet, but I'll cross that bridge when I get to it. I don't even have my switches here yet and currently have no hardware to play with on that "lab network" so it's no big deal right now. I just wanted my home network running and not being hindered by that firebox's 12 user limit.

Thank you so much for your help!

opjohnny

Meh, I was on a roll so I decided to do the LAN2 firewall rules before bed. Do these look ok? I want internet connectivity on LAN2 but I don't want it to see or access LAN1. However, I do want LAN1 to see and access LAN2 since I administrate most stuff on LAN2 from my rig on LAN1.

biggsy

You only need one rule on LAN2:

PASS
Proto: IPv4+IPv6
Source: ANY
S/Port: ANY
Dest: NOT LAN net
D/Port: ANY

This allows through any traffic coming in on pfSense's LAN2 interface as long as it does not have a destination address somewhere in the LAN subnet.

opjohnny

Are my rules set up properly for what I wanted on LAN2? Curious to know if I got anywhere close on those since I've never messed with firewall rules before, outside of Windows. :-[

Here is what I have now for LAN2…

Pass
ID: None
Proto: IPv4+6 TCP
Source: *
Port: *
Destination: ! LAN net
Port: *
Gateway: *
Queue: None
Schedule:
Description: Allow LAN2 to any except LAN.

johnpoz

So does your lan2 have ipv6 on it, if not prob want to just say ipv4, your source could be just lan2 net since what else would be coming into your lan2 interface?

But other than that looks right..

So example here are my wlan rules, this is is like a lan2 in your case - but I have another segment dmz besides my lan.

So the couple of pin holes I have - so ipad at the 2.230 address can go anywhere, lan, internet and dmz
so wlan can talk to my printer on lan at 1.50
so wlan can talk to my ntp server on lan at 1.40
so wlan can talk to internet and dmz, but not the lan network.

Hope that gives you some ideas how you would do rules that allow some traffic to start from your lan2 into your lan for exceptions, etc.

Now from my other segment the dmz, I have an alias called locals which has my lan and wlan network segments 192.168.1.0/24 and 192.168.2.0/24 in it

So this rules says dmz can go anywhere as long as its not either of those networks. Now if I want to allow dmz to talk to my printer or ntp I could put the same kind of rules I have in my wlan segment above that rule. So currently it can go to the internet, it could talk to my openvpn clients etc. But could not create traffic to either my lan or wlan segments.

Keep in mind both wlan and dmz could answer traffic that comes from lan.. Since my lan rules are allow any.

wlanrulesblocklan.png_thumb

dmzrulesnotlanorwlan.png_thumb

opjohnny

I'm curious about adding wifi(for home network) in to this thing and possibly doing away with my wrt54gs. My PE860 has one empty PCI slot. What are my options? Is there a compatible card list? The wrt54gs is B/G and WPA2/AES so I'd at least want those specs with the card. I'd lose my 'switch' on my home network but I've got two Netgear GS108's coming in that'll go on each network.

This weekend I'll be attempting a PfSense install and config on a Watchguard Firebox X-Core x700 for a friend. Currently trying to find a good mini-PCI wireless adapter to throw in it so I can do wireless for him as well. He was the original owner of this PE860 and ran PfSense on it(standalone, not VM) but his wife didn't like the noisy fans in it. This x700 should work nicely for him especially if I can get wireless going on it. ;)

johnpoz

I would never do wifi directly on pfsense - doing wireless in my mind is not the role of firewall/router - its is to firewall traffic and route.. Wireless is job of AP..