Routing between two LAN subnets - sounds simple but isn't working
-
Hi peoples,
I'm new to pfsense and am trying to set up a new subnet for testing in my VMware ESXi 5.1 environment.
My network is set up as follows:
Linux based firewall gateway to internet with modem > WAN / LAN 192.168.0.1 > 192.168.0.0/24 network with unmanaged gigabit switch.
My VMware environment is plugged into that switch and I have a distributed switch with two port groups - one with hosts on 192.168.0.0/24 and the other with hosts on 192.168.1.0/24.
I created a VM with three NICs and installed pfsense. The NICS are named WAN, LAN1 and LAN2 and WAN is configured with no IP and plugged into the 192.168.0.0/24 port group. LAN1 is also plugged into same port group and has IP 192.168.0.111 (given .1 and .11 were already in use). LAN2 is configured with 192.168.1.1.
I disabled all firewall rules in pfsense since I only need routing, but had rules in place on LAN1 and LAN2 allowing any protocol on LAN1 and LAN2 to any just in case.
I have a default route set up on LAN1 of 192.168.0.1 to allow access to the internet.
I can ping all IPs in 192.168.1.0 from hosts in 192.168.0.0/24, however from my PC on that network I cannot open remote desktop to my windows VMs on 192.168.1.0 network as it fails to connect. A look at netstat on the VM shows a connection being received with "SYN_RECEIVED" state. That, along with a tcpdump show the connection is received by the VM but the responses do not get back, all i saw was one way traffic.
So from the VM I can ping the pfsense router's interfaces on both LAN1 and LAN2, the default gateway AND the internet, for example google. Yet ping ANY other hosts on 192.168.0.0/24 network and the pings time out. A traceroute shows the first hop as 192.168.1.1 on the pfsense router but * * * beyond that.
I have tried with no success:
1) removing the default gateway 192.168.0.1 on LAN1 and on the router generally
2) adding gateways LAN1 192.168.0.111 and LAN2 192.1.1 on the respective interfaces - even though common sense and the routing table tell me they are not needed.I am tearing my hair out trying to work out why this does not work.
No firewalls are enabled on ANY systems on my LAN so that can be ruled out.
The default gateway of systems on 192.168.0.0/24 is 192.168.0.1 and on 192.168.1.2 it is 192.168.1.1 so that all looks fine.
This should be so simple so whats wrong????
-
For starters the current 2.2.x version of pfsense is not even supported until version 5.5u2
It is drop dead simple…
You say you have a "I have a distributed switch" so your running vcenter with enterprise licensing then? But you don't have smart/managed switches - really?
"192.168.0.0/24 network with unmanaged gigabit switch."
But then you have port groups on your vswitch that have both 0.0/24 and 1.0/24 networks on them? How exactly is that going to work connecting to a dumb physical switch??
How many physical networks do you have?? That vswitch connected to your physical network is where wan of pfsense should go. Doesn't seem like you have any other physical networks or vlans since all you have is a dumb switch.. So create new vswitches, no point in them being distributed unless your going to tie them to the physical world which doesn't really seem like you are..
Then create connect your pfsense lan and opt interfaces to these specific vswitches and connect whatever vms you want on those network segments to the same vswitch.. The gateway of these networks would be the IP of the pfsense interface you have on that vswitch.. I can show you a picture of my setup if helps..
So the phy int connected to wan vswitch is directly connected to my cable modem... Pfsense get as public IP on this interface.
vmkern and and lan phy nics are connected to same vlan ports on managed switch. wlan is connected to trunk port on phy switch and set for all vlans in the vswitch. On the pfsense interface on its virt interface I run multiple vlans as well as vlans on the AP connected to the phy switches in my network.
Then last you see the dmz, pfsense as a interface connect to this switch, and as you see its not connected to the physical world... But devices on this vswitch can route through pfsense interfaces to get to the phy network be it out the internet (wan) or out lan or wlan (and multiple vlans) etc..
Hope that helps... But if your running pfsense 2.2.x you need to be on 5.5u2 at min since this is when freebsd 10.1 support was added to esxi. Not saying it can not work, but there are some odd stuff that could/might happen, etc.. If you want to use pfsense 2.2.x then you really really should be on vsphere version that supports the OS fully..
I am currently running 6 build 3247720 without any problems at all.
-
Thanks for your reply johnpoz.
To answer your questions:
This is a test environment I ran at home for self education purposes and prototyping. You asked why run diatributed switches and the answer is right there "dustributed" - otherwise my VMs cannot be vmotioned - my cluster is set up as close to enterprise as possible since I set it up to study VCP and build experoence. At the moment I haven't got a managed switch. Also my cpus are not supported by 5.5, so 5.1u2 is aa far as I can go although having said that my hardware supposedly can't run 64bit guests yet lo and behold they run fine.
The only network existing outside vmware is the 192.168.0.0/24 network. 192.168.1.1 is purely behind the pfsense router.
I have two seperate port groups on the distributed switch - one has VMs on 192.168.0.0/24, the other has 192.168.1.0/24 VMs with pfsense having an interface on both.
-
you sure and the hell do not have to run distributed switches to vmotion. Not sure where you got that idea… Not sure where you got the idea your cpu is only support on 5.1u2, u3 isnt even support?? What cpu do you have??
But you stated you connected both port groups to the same unmanaged switch? So your running both address schemes over the same layer 2?
Can your vms connected to the pfsense lan side ping your lan side? You stated you turned of firewall, so you also disabled nat.. So does your internet router know how to get to the pfsense lan network? Most likely NOT.. So you would have to create a route on your internet
I have a default route set up on LAN1 of 192.168.0.1 to allow access to the internet.
Then it is NO Longer lan interface if you set a default gateway on it...
What exactly are you trying to do?
"LAN2 and WAN is configured with no IP and plugged into the 192.168.0.0/24 port group."
What are they going to do with no IPs? How is pfsense going to route anything if only has 1 interface with an IP?
Draw a picture please and state what exactly you want to happen.. Not sure how pfsense is going to do anything with an IP in only 1 network???
-
According to the compatibility matrix only 5.1 supports my CPUs which are opteron 285s with no virtualization instruction sets.
I was always under the belief that vswitches exist on the host only and don't talk across hosts and hence the distributed switch is used. I haven't used vswitches since esx 3.5 and vsphere 4 days.
My Internet router has a static route to 192.168.1.0/24 gateway 192.168.0.111
and I set up the default route on the LAN1 side of pfsense of 192.168.0.1 because I assumed it needed that to Know where to send any traffic for networks it does not know about.Internet–Gateway--Switch(192.168.0.0/24)---ESXiHosts---Dswitch---Portgroup(192.168.0.0/24)
(192.168.0.1) /
192.168.1.0 VMs---Portgroup(192.168.1.0) 192.168.0.0 VMs
\ /
(192.168.1.1) LAN2--pfsense--LAN1 (192.168.0.111)Does that make sense now? So everything 192.168.1.0 exists only inside Vsphere on the LAN2 side.
The whole thing with WAN interface being configured was another attempt to get this working. My first attempt was WAN and LAN but when it was not routing as expected I thought maybe this product is intended as an internet / LAN router needing NAT to route to WAN and I needed to use only LAN interfaces to allow routing between two LANs. So I set up WAN unconfigured and just used two LAN interfaces - however that didn't work either and here I am now. :)
-
The thing that I find bizarre about all this is that hosts on 192.168.0.0 network can ping, and receive replies from, hosts in 192.168.1.0.
Yet hosts on 192.168.1.0 cannot ping any hosts on 192.168.0.0 EXCEPT the pfsense routers IP 192.168.0.111 IP and the gateway 192.168.0.1 even when that is not set as the default route on the pfsense router.
Routing is working in one direction 192.168.0.0 > 192.168.1.0 and receiving the replies (at least for ICMP pings, not for actual connections such as RDP) yet not in the other.
No firewalls whatsoever in the way.
-
Dude what is the make and model of your hardware this cpu is in? And where are you looking?
If I look here an opteron 2xx
https://www.vmware.com/resources/compatibility/search.phpOnly supports 3.5 not 5 if your your running 5, then you can run 6..
Not sure how you configured, but why do you have 2 port groups tied to your physical network?? And not using vlans? Your just putting 2 different address schemes on the same physical wire than… They are all in the same layer 2..
Again lets ask this another way -- what exactly are you trying to accomplish here? Why not just put all your vms in 192.168.0.0/24 ??
If you want to use pfsense to route and or firewall.. Then do that.. Put pfsense wan in the 192.168.0.0/24 network.. Create 2 or 3 or ++++ new segments put them in their own vlan with their portgroup. If your wanting to connect your hosts and isolate vms into their own networks then you need a switch that supports vlans. Or you have to use isolated switches... You can not just plug everything into 1 dumb switch and use different address space on them..
You say your trying to simulate an enterprise setup, but you don't have a vlan capable switch? And your hosts only have 1 nic?
-
I can now rule out sharing two network ranges over one switch - I created a second distributed switch and moved the 192.168.1.0 network onto that distributed switch with no external uplinks. Now no traffic for 192.168.1.0/24 is on a switch used by 192.168.0.0/24.
Still doesn't work. Same identical issue.
As far as what I am trying to achieve - I want to create an isolated Windows test domain isolated from the rest of my network where it can be responsible for DNS and DHCP without affecting my existing setup.
As far as my setup is concerned cost is an issue, hence I have not bought managed switches. I had existing layer 2 switches so I made use of them. If I was to buy a switch it would have to be cisco so I could learn more about the industry standard.
For the record my specialization is UNIX/Linux. I haven't done networking since before VLANs and a lot has changed.
-
Dude lets do some basics here..
Lets do so basics.. this really should be clickity clickity.. I have been running pfsense on esxi since version 5 of esxi and has always just been clickity clickity to get it working..
So your physical network is 192.168.0.0/24 so connect pfsense wan to this network..
Then on your new vswitch be it d or just standard connect your pfsense lan interface.. Lets use say 192.168.2.0/24 since you have not mentioned this network. Put a vm in this same vswitch and it should get an IP address from pfsense dhcp server.. If it doesn't then you got something wrong.
Once you have wan/lan working.. You know have your isolated network.. Once you have it working with NAT, and your clients can get to your wan… Then you can turn worry about turning off nat if you want..
Again keep in mind pfsense 2.2 is NOT supported by your OLD as the hills version of esxi.. Your not installing the vmware tools are you??
