Unable to access Internet from virtual network
-
I'm currently configured with (to my knowledge) the firewall component turned off.
I have a virtual network configured, one of several, for my server devices. That network is configured for 10.1.10.0/24
The WAN port for PFSense is 192.168.1.175
Devices on the 10.1.10.0/24 network are configured with static IPs. 3 servers, each can ping the other two and can ping the 10.1.10.1 gateway that was configured for that virtual network.
The devices can't access internet at all, though I have a global WAN gateway configured with my internet router's address (192.168.1.1)
On the individual network interfaces configuration area the IPv4 Upstream gateway is "None". If I attempt to add a new gateway I get messages that my default gateway (192.168.1.1 as far as I know) is not in the right subnet.
What am I doing wrong and how do I add access to the internet to this virtual network (which I would then need to repeat on other virtual networks that I had created within the ESXi virtual network as well as within the pfsense configuration)
Thanks for any assistance!!!
-
Point of confusion - what should my LAN address be for the pfsense device? (Not the WAN address, but the LAN address)
-
Not clear what role pfSense plays in your network.
You have only mentioned the WAN address of pfSense.
Maybe a drawing could help for clarify.
-
Are you volunteering to make the drawing? (Not to give you a hard time, but drawings aren't necessarily that easy to achieve for all of us).
Here's a general picture of things:
Internet / World <---> ISP Router <---> Home network
That's the before picture, simple arrangement.
I'm now adding an ESXi host that is also on the Home network side of the ISP Router, so it would be something like this:
ISP Router <---> Home Network
^----> ESXi hostWithin ESXi host, I have multiple virtual networks carved up.
ESXi Host and Virtual networks:
VMX0 - WAN
VMX1 - LAN / Virtual only (though I want to allow access through to Internet)VMX1 includes multiple virtual networks:
Server1 - 10.1.10.0/24
Server2 - 10.1.15.0/24
Workstation1 - 10.1.20.0/24
Workstation2 - 10.1.30.0/24
DMZ1 - 172.xxxxxx/24
DMZ2 - 172.xxxxxx/24
External1 - 192.168.10.0/24
External2 - etc.Home network (or WAN for the pfsense firewall) is 192.168.1.xxx
PFSense is configured with two interfaces, one to VMX0, the other to VMX1
Trunk Port (4095) on the VMX1 virtual-only network
-
This should help make more sense. It only shows part of the network design, but enough to help make sense of things if above text wasn't clear enough.
-
To add here, the role for pfsense is to serve as a virtual router, for the devices within the "DOM" mentioned above. (The ESXi host actually). It is also supposed to serve as a firewall to keep things isolated between networks unless I turn off the firewall or add rules to allow specific traffic through.
In working through things as simply as possible, I have rules that should be allowing all traffic to flow between the virtual ports but that doesn't seem to be working, and even turning off the firewall entirely isn't allowing the traffic to flow outside of either virtual port on through to the internet.
-
I don't see the need of VLANs here, since alle those networks exists only in ESXi. Why don't you just add virtual network (vSwitches) in ESXi.
However, if you connect pfSense to a trunk port like that, you have to define each VLAN in pfSense. Interfaces > VLANs
Then go to Interfaces > Interface Assignments and add an interface for each VLAN and configure it.
The gateway 10.1.10.1 you mentioned above should also be one of these VLAN interfaces, I think.In the WAN interface settings add the 192.168.1.1 as upstream gateway.
-
@viragomann said in Unable to access Internet from virtual network:
I don't see the need of VLANs here, since alle those networks exists only in ESXi. Why don't you just add virtual network (vSwitches) in ESXi.
However, if you connect pfSense to a trunk port like that, you have to define each VLAN in pfSense. Interfaces > VLANs
Then go to Interfaces > Interface Assignments and add an interface for each VLAN and configure it.
The gateway 10.1.10.1 you mentioned above should also be one of these VLAN interfaces, I think.In the WAN interface settings add the 192.168.1.1 as upstream gateway.
There's more to it in terms of why the need, but the virtual networks are defined in ESXi and in pfsense, with each linked as expected.
The default gateway has been configured for each of the virtual lan interfaces.
For the WAN, the 192.168.1.1 address was added as the gateway. For the virtual networks, the configuration looks something the one that is shown here:
Wan Interface looks like this:
The virtual networks are mapped/identical between ESXi and pfsense.
-
More pictures:
-
so you what did you change the lan to on pfsense which defaults to 192.168.1.1 even..
You can not have 192.168.1 on your wan and your lan of of pfsense and expect anything to work.
And again WTF you doing vlans for in pfsense if all of those networks are only virtual and on your esxi host?
-
The LAN on the pfsense side should be 10.1.1.0/24
And again, the WTF is complicated, but expect it to work out to allowing for more firewall control over what passes between the virtual lans.
-
Its NOT going to work how you have it setup because your vswitches will not pass any tags to pfsense!!
You have zero use of tagging anything if all its going to be is virtual vswitches. Just create another vnic for pfsense and attached it to your vswitch you connect those vms too..
All of the networks in pfsense will be native with what pfsense thinks is a real interface - you have just as much control between your networks this way.
-
@terpfan1980 said in Unable to access Internet from virtual network:
The LAN on the pfsense side should be 10.1.1.0/24
And again, the WTF is complicated, but expect it to work out to allowing for more firewall control over what passes between the virtual lans.
@johnpoz said in Unable to access Internet from virtual network:
Its NOT going to work how you have it setup because your vswitches will not pass any tags to pfsense!!
You have zero use of tagging anything if all its going to be is virtual vswitches. Just create another vnic for pfsense and attached it to your vswitch you connect those vms too..
All of the networks in pfsense will be native with what pfsense thinks is a real interface - you have just as much control between your networks this way.
More info please? I'm following along with instructions that were written up by someone else and have been used by others. On the ESXi side, the same VLan ID's exist (is that the tagging you are referring to??) as are used on the pfsense side of things.
So I'm lost a bit on what you are saying about things not being tagged here.
Thanks for helping to explain it more clearly.
-
Another image from the instructions that I've been working from:
-
-
You would use the tags with those switches if going to physical network... I have no idea who wrote up instructions be makes zero sense like that - ZERO!! And they don't seem to understand basic networking in esxi..
When you set a vlan ID on a vswitch it does not pass the tag to the vnics connected to this vswitch... So if your not passing tags, you woudn't be setting up vlans in pfsense.. You only setup vlans in pfsense when pfsense has to deal with the tags...
If you want pfsense to deal with the tags, then it would just be 1 vnic and 1 vswitch and you would use 4095 on the vswitch so you would pass the tags to the vnic so that pfsense could deal with the tags.
Which you set a vlan id on a vswitch its like an access port on a cisco switch.. The vm device connected to it will not see any tags..
if your going to tag your vswithes like that it would be for them to connect to a physical network and to mark traffic towards the real network with the tags so that the real switch can handle the traffic correctly. But any vms connected to those port groups would not be seeing any tags.
His instructions say to create vlans on pfsense??? Or your thinking you need too? You would create another vnic for each vswitch/port group you create for whatever vlans.. But since nothing is going to to the real network just leave the vlan id as 0 on all of them and just name the vswitch/port group for whatever you want to use that network for.. Its like just using a different dumb switch for each network and real physical interface in pfsense connected to the different dumb switches.
-
@johnpoz said in Unable to access Internet from virtual network:
You would use the tags with those switches if going to physical network... I have no idea who wrote up instructions be makes zero sense like that - ZERO!! And they don't seem to understand basic networking in esxi..
When you set a vlan ID on a vswitch it does not pass the tag to the vnics connected to this vswitch... So if your not passing tags, you woudn't be setting up vlans in pfsense.. You only setup vlans in pfsense when pfsense has to deal with the tags...
If you want pfsense to deal with the tags, then it would just be 1 vnic and 1 vswitch and you would use 4095 on the vswitch so you would pass the tags to the vnic so that pfsense could deal with the tags.
Which you set a vlan id on a vswitch its like an access port on a cisco switch.. The vm device connected to it will not see any tags..
if your going to tag your vswithes like that it would be for them to connect to a physical network and to mark traffic towards the real network with the tags so that the real switch can handle the traffic correctly. But any vms connected to those port groups would not be seeing any tags.
His instructions say to create vlans on pfsense??? Or your thinking you need too? You would create another vnic for each vswitch/port group you create for whatever vlans.. But since nothing is going to to the real network just leave the vlan id as 0 on all of them and just name the vswitch/port group for whatever you want to use that network for.. Its like just using a different dumb switch for each network and real physical interface in pfsense connected to the different dumb switches.
The instructions say to create the same VLAN's on pfsense as were created on the ESXi side. The id's have to be the same in both places. On the ESXi side there is one additional VLAN ID (4095) for the Trunk port for pfSense LAN interface.
In my case (as in the case in the instructions), I've got one physical port on the Dell Optiplex Mini box. That physical port is used for the Management network for the ESXi host. (See vSwitch0 in the image sample in earlier reply)
pfsense has connection to that vSwitch as well as the Management Network for ESXi.
So, my understanding of things is that pfSense should be working as a router to allow connections through the various vlans back and forth and on to the outside world if desired. That's where I'm stuck. The communication between networks is happening, but I'm not getting communication out to the internet and beyond from any of the vlans that pfSense knows about (or that were created in the virtual-only switch on the ESXi side)
-
@terpfan1980 said in Unable to access Internet from virtual network:
The instructions say to create the same VLAN's on pfsense as were created on the ESXi side.
Then they are WRONG, or your not understanding them - what is the link to these instructions?.Or who wrote them doesn't understand how it works... Just plain freaking FACT!!! I have run pfsense on esxi since before there was esxi and ran it on vserver 1...
Yeah it can route just fine - has zero to do with creating vlans on pfsense if its never going to see the tags!!!
You came here for help, I am telling you how it works... You can listen or you can dick around with instructions that are just plain freaking WRONG!!! Please just read the esxi docs for EST or VST..
How is a vlan on pfsense going to work when it doesn't see any tags?
You only create vlans on pfsense with it will actually see a tag.. Be it physical or virtual.. That is not the case in such a setup. You need a different vnic for each vswitch you create..
-
@johnpoz said in Unable to access Internet from virtual network:
@terpfan1980 said in Unable to access Internet from virtual network:
The instructions say to create the same VLAN's on pfsense as were created on the ESXi side.
Then they are WRONG, or your not understanding them - what is the link to these instructions?.Or who wrote them doesn't understand how it works... Just plain freaking FACT!!! I have run pfsense on esxi since before there was esxi and ran it on vserver 1...
Yeah it can route just fine - has zero to do with creating vlans on pfsense if its never going to see the tags!!!
You came here for help, I am telling you how it works... You can listen or you can dick around with instructions that are just plain freaking WRONG!!! Please just read the esxi docs for EST or VST..
How is a vlan on pfsense going to work when it doesn't see any tags?
You only create vlans on pfsense with it will actually see a tag.. Be it physical or virtual.. That is not the case in such a setup. You need a different vnic for each vswitch you create..
Company proprietary Wiki site for the instructions, though I can clip some of material there (as included above)
Instructions for ProxMox have the same flow and also include pfSense in the same fashion.
I've not used pfSense that much (I've been a user without knowing it, first time working through the configuration myself). Similarly, my use of ESXi has been more limited, but I understand the basics of creating virtual switches, port groups, etc.
-
ESXi Networking
-
I would suggest you read up on vlans and tagging. A vlan only needs to created if there is going to be a TAG to interpret.. When you put the vlan ID on on a vswitch it will place that tag going to physical world for traffic coming from the vm side.. And traffic coming from the physical side it will hand off traffic tagged with that TAG to the devices connected to that port group/vswitch without the tag...
So no traffic going to pfsense vnic from vswitch with ID 10 for example will have any TAG on it - so how is the vlan interface of pfsense going to do anything with this traffic.
This is exactly how an access port works on any cisco switch with pvid set to your vlan ID..
Maybe proxmox or switches do not remove tags - but with esxi when you set a vlan id on a port group there are no tags sent towards the vm devices.. If you want tags to not be stripped you need to set 4095 on the port group/vswitch.
4095 makes sense when traffic will be coming from the physical world via a trunk port (cisco) with a tag on it - and you want pfsense to handle the traffic via what tag is on it.. When you create a port group with a ID, no tag will be sent towards the vm interface!! So how is vlan on pfsense going to have anything to work with?
Don't believe me if you don't want to - do your own research on how est and vst works in esxi.. But since your here in the first place it - I take their instructions are not working ;)
edit: the only way this could work is if you hairpin everything through your 4095 port group.. And all the other vlans are just portgroups on the same vswitch.. Or you actually go out to physical and come back.. Either way its all of useless hairpins..
Its not like vswitches or vnics cost you anything.. vs doing 1 vswitch with port groups and hairpins to vlans on pfsense just do native connections..
-
Back to update... a day or so after the discussion above, I was able to resolve the issues that I was having.
My best guess is that the issues that I was having related to my flipping the switch within pfSense to "Turn off the Firewall". Turning off the Firewall seemingly also turned off the NATting that I would have been relying upon.
Related, I had created "Any to Any" firewall rules, but then screwed up and had them only for TCP, and not for Any protocol.
With the rules changed to "Any" protocol, and not just TCP, things worked as documented (loosely documented, but with the pictures that were seen above, along with others) and the traffic flows as expected, from vlan to vlan, and from the vlans out to the internet as expected and desired in my case.
Another set of documentation (for Proxmox, which also uses pfSense in the same fashion) cover the use of multiple distinct firewall rules that would allow or restrict (default deny) traffic of various types. I had started with "Any / Any" allowed because I mostly am using pfSense as a router and didn't need to start with any blocking of traffic (I will get there eventually, but didn't need it to start).
This same configuration is used by several others (it numbers in the hundreds at this point) that are also running a home lab setup that normally would be using pfSense in the same way. Using it in this way allows for emulation of a larger corporate type network where traffic is isolated or at least the address ranges can more easily mimic what is typical in the corporate world.
Thanks for the assistance and discussion points along the way. It is valuable material to be aware of, and hopefully may help someone else in the future.
-
Not sure what guide you are following - but if its having users hairpin connections back and forth inside vswitches vs just crating native interfaces its not very efficient..
And is over complicating a simple setup..
-
@terpfan1980 said in Unable to access Internet from virtual network:
Back to update... a day or so after the discussion above, I was able to resolve the issues that I was having.
My best guess is that the issues that I was having related to my flipping the switch within pfSense to "Turn off the Firewall". Turning off the Firewall seemingly also turned off the NATting that I would have been relying upon.
Seemingly:
Disable Firewall
Disable all packet filtering. Note: This converts pfSense into a routing only platform!
Note: This will also turn off NAT! To only disable NAT, and not firewall rules, visit the Outbound NAT page.Related, I had created "Any to Any" firewall rules, but then screwed up and had them only for TCP, and not for Any protocol.
With the rules changed to "Any" protocol, and not just TCP, things worked as documented (loosely documented, but with the pictures that were seen above, along with others) and the traffic flows as expected, from vlan to vlan, and from the vlans out to the internet as expected and desired in my case.
Loosely documented:
