Can't connect to host in other network
-
Hi Forum
After many years of using pfsense i struggle with a problem that i can't fix on my own. Primary, i didn't know where to find the problem and don't know where the right forum to post it.
At a customer, i installed a new pfsense 2.7.2 on a baremetal server with different interfaces (IPv6 isn't in use at the moment and communication is blocked by pfsense):
WAN: DHCP from ISP, based on P2MP technology on VLAN11
mgmt (LAN): 10.10.0.254/24
client VLAN10: 10.10.10.254/24
dmz VLAN20: xx.xxx.xxx.17/28 (Public IP Range)
storage VLAN30: 10.10.30.254/24I configured the DMZ with this tutorial: https://docs.netgate.com/pfsense/en/latest/firewall/additional-ip-addresses.html#firewall-additional-address-routed
I prepared a webserver, based on nginx. this running on a vm (Debian 12 Bookworm) and have assignet the client-network 10.10.10.50/24. i can connect over ssh at using openVPN. then i add another network interface to the vm, assignet it to the dmz and configure the interface to xx.xxx.xxx.22/28. As gateway i change to the public gateway xx.xxx.xxx.17 and remove the local one. that works, the website is running and reachable from the internet. the outgoing traffic come from xx.xxx.xxx.22. but is not longer possible to connect to the ssh (port22) on the local client net over openvpn. also is not possible to connect from mgmt-lan. from pfsense itself, i can connect ssh on the debian server.
if i change back to the local gateway, it works and i can connect to the server per ssh.
Now, im not sure where the problem is located. i try different settings, any to any rules, set route on my pc and so on., checking network configs etc more then once. also i remove the interfaces and create again.
Can someone give me a hint where to start troubleshooting?
Greez Judge
-
@Judge_of_Death
Why do you want the vm to be multi-homed?Just remove the client VLAN and access it via the DMZ (public) IP.
Yes, you have to route the DMZ subnet over the VPN then, but should work. -
@viragomann said in Can't connect to host in other network:
Why do you want the vm to be multi-homed?
I would concur - multihoming can lead to problems (asymmetrical flow) , can lead to loss of security you have with a firewall, etc.
If you really want to do multihomed - for some reason, say bandwidth.. I use a multihomed on my pc and nas.. They are in the same network anyway - so no worries about loss of firewall controls. But only my pc and nas have 2.5ge interfaces.. And wanted to leverage that for data movement.
So these interfaces are on a different network, they have no gateways and they have no dns settings on them, etc. They are just leveraged when I access smb on the nas from my pc..
-
Why do you want the vm to be multi-homed?
because of security reason, the ssh should only accessable over his client ip. Also there are other tools like monitoring etc. that communicate explicit in the mgmt and client net.Thx 4 you reply.
Greez Judge
-
@johnpoz said in Can't connect to host in other network:
I would concur - multihoming can lead to problems (asymmetrical flow) , can lead to loss of security you have with a firewall, etc.
If you really want to do multihomed - for some reason, say bandwidth.. I use a multihomed on my pc and nas.. They are in the same network anyway - so no worries about loss of firewall controls. But only my pc and nas have 2.5ge interfaces.. And wanted to leverage that for data movement.
So these interfaces are on a different network, they have no gateways and they have no dns settings on them, etc. They are just leveraged when I access smb on the nas from my pc..
Same here. The mgmt-net is copper-based 1gbit for ilo, idrac, xcp-hosts and so on. All VLAN layed on one 10gbit fiber. The VLAN-networks are okay, they i tested on all direction. In the past, i got good experience with this setup.
The major problem is, when i change the gateway to the public gateway, it is not assesible from other networks. And i don't know why. And i don't know where to locate this problem. In the logs i cant find some information about blocked package, errors or something relevant to this.
Greez Judge
-
@Judge_of_Death said in Can't connect to host in other network:
when i change the gateway to the public gateway
What are you changing where.. Lan side network would never have a gateway set, unless that network is a transit/connector network - and if that is the case there should be zero devices on that network other than routers.. or yeah for sure your going to run into asymmetrical problems.
To put a device on a transit network, you would need to setup routing on this device to get to different networks, either the upstream router or the downstream router, etc.
-
What are you changing where.
The gateway of the interface on the vm:
As long i use the gaterway on local, i can connect from anywhere. When i change to public gateway, i can't connect
-
@Judge_of_Death said in Can't connect to host in other network:
When i change to public gateway, i can't connect
well duh.. What you be changing it to other than pfsense IP on that network.. What would be changing it too? If you have some other router on this network that it could use.. Then yeah your going to have a problem without specific host routing to say hey if traffic comes from this/these network use this router/gateway.. If not, ie default then use this gateway.
When you have multiple say internet connections to other networks via some other router. Its best to connect those to pfsense either directly or via a transit network.
And then do all your outer decisions on pfsense, on which connection to use to get where.
-
i didn't changing someone on pfsense, nor is there another router/ firewall in this network. Here a simple overview of the network:
i have 2 network's, one for internal shizzle like ssh connection, monitoring etc. one is the public ip net.
on the vm: as long the default gateway is set to local client network (10.10.10.254) i can ssh connect from openvpn (or mgmt, missing in the overview) to this vm. when i change the default gateway to the public-ip net (xx.xxx.xxx.17), i can't connect to ssh over openvpn (see picture in last post).
-
@Judge_of_Death
If the machine has no specific route to the source IP of the access it will response to its default gateway, and this might be the DMZ IP of pfSense. Hence you run into asymmetric routing.So you can solve it either by adding a static route for all subnets you want to access it from and point it to the client VLAN IP, or by masquerading the traffic on pfSense with an outbound NAT rule.
-
@viragomann said in Can't connect to host in other network:
So you can solve it either by adding a static route for all subnets you want to access it from and point it to the client VLAN IP, or by masquerading the traffic on pfSense with an outbound NAT rule.
yeah, add a route at the vm fix that problem.
I think is easier to config this in masquerading on the pfsense then on all clients that have the gw on public ip. I will follow up the solution when is running to help other with the same problem.
Thx a lot viragomann and johnpoz
Greez Judge
-
@Judge_of_Death yeah that would scream asymmetrical problem coming from a vpn client, that has some vpn tunnel IP..
Sure you can get to 10.10.10.50 pfsense would route your traffic there.. But that box then says oh some 10.10.200 (your tunnel network as example) hmmm, no route - send it to my default gateway (x.x.x.17).. pfsense says wtf is this, I have no state to allow this return traffic..
But if you would of sent the traffic back to 10.10.10.254, pfsense would have a state your good.
So you can work around this couple different ways, either source nat the traffic from your vpn going to 10.10.10.50 so it looks like it comes from 10.10.10.254 and not your tunnel network IP of your vpn client.
Or you can add a route on this .50 device that says hey if you want to talk to 10.10.200 (your tunnel network in my example) send it to 10.10.10.254 and not your default gateway. You wouldn't be able to use the x.x.x.22 address then to access your multihomed device.
