Routing from subnet does not belong to pfsense
-
@johnpoz I am sorry for reviving the old topic but I am facing the same issue behind pfsense.
I`ll try to explain in a bit more details:
-
Site A: A typical scenario where we have pfsense as router/firewall with Public IP managing a single LAN network 192.168.x.0/x
-
Site B: Cloud (let`s say AWS) has a VPS with Netmaker Server installed and accessible, ready to serve any authorized connection over his Public IP.
-
Site A - Client 1/Node 1 (192.168.1.100): A simple Laptop (Win, Linux. Mac - it doesn't really matter) is initiating Wireguard/Netmaker VPN connection to Site B Public IP (Netmaker Server), and... connection is established. The client gets internal IP from the Netmaker VPN let`s say 10.53.1.10. The Netmaker GW is 10.53.1.254. There is a Ping between the client 10.53.1.10 and the GW 10.53.1.254.
-
Site C - Client 2/Node 2: a different client in a different network behind some firewall, not pfSense with local IP 172.16.0.24 is initiating netmaker VPN connection to the public IP of Site B. connection is established and the IP assigned to the client C in the Netmaker network is 10.53.1.20. There is a ping between Site C 10.53.1.20 and Site A 10.53.1.254 no worries.
-
Site D - Client 3/Node 3: 3rd client in a row again in a totally different network behind some firewall, not pfsense with local IP 10.10.100.17 is initiating netmaker VPN connection to the public IP of Site B, the connection is established and the IP assigned to the client D in the Netmaker network is 10.53.1.30. There is a ping between Site D 10.53.1.30 and Site A 10.53.1.254 no worries.
Now comes the fun part:
Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)Site C and Site D can ping each other and can ping the GW/The Netmaker Server but they cannot ping the Site A which is behind pfSense.
Some references: Netmaker uses a UDP Hole Punching so he can be able to create p2p connection between each one of the clients/nodes which gives the ability to have a direct connection between different nodes/clients without having the route the traffic through the GW/Netmaker Server.
As far as I am aware the same issue is facing the ZeroTier which uses UDP Hole Punching as well.
I hope this explanation the give some light and understanding what the real issue is.
Still looking for a solution....
-
-
@josifbg-0 said in Routing from subnet does not belong to pfsense:
Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)Let me repeat this ZERO to do with pfsense..
If you tunnel "through" pfsense, doesn't matter what vpn software protocol.. Pfsense has zero to do with the routing inside that vpn, or what you can or can not do to other devices also connecting to that vpn..
If pfsense could do that - the the whole point of vpns would be pointless.. Pfsense has zero to do with the client making the connections routing through that tunnel.. If you can not get so some other site using your vpn your client is connected to, then its the vpn, the routes setup in the vpn, the firewalls on the end devices or the vpn.. But pfsense is oblivious to what is happening in the tunnel be it udp, icmp, ssh, http, it has no idea - all it knows if the outside of the tunnel.. Which it is allowing, or your client behind pfsense wouldn't be able to establish the connection..
A hole punch wouldn't be going through the tunnel.. Now if you block a client from going out the port that is trying to be punched you could have issues.. But traffic flowing through the tunnel no matter where its going to or coming from pfsense has nothing to do with that..
-
@johnpoz Thanks for your response.
So how would you explain that behind pfsense the client is not able to ping other nodes and to be pinged by any other nodes, but without pfsense there is no such issue at all?
-
@josifbg-0 they are not routing through the tunnel then.. If they are hole punching, then you would have to allow the actual connection outbound in the first place to allow for the whole punch. Maybe the port has to be static for what they are trying to do with the hole punch.
But traffic flowing though that tunnel pfsense has zero to do with any of that traffic.. Not unless pfsense was the actual endpoint in the vpn could it control traffic in or out of the tunnel.
-
here are some logs:
WAN udp 94.XX.XX.XX:13355 (192.168.20.125:51821) -> 18.192.50.92:45328 SINGLE:NO_TRAFFIC 901 / 0 155 KiB / 0 B
-
@johnpoz I totally agree with you and it was strange for me as well why this is happening.
"then you would have to allow the actual connection outbound " -> well this got to my mind as well but I am not sure which connection to open and where in the pfSense -
@josifbg-0 said in Routing from subnet does not belong to pfsense:
WAN udp 94.XX.XX.XX:13355 (192.168.20.125:51821) -> 18.192.50.92:45328 SINGLE:NO_TRAFFIC 901 / 0 155 KiB / 0 B
That is client 20.125 leaving psense wan - it gets no answer - how is that a pfsense thing? But notice your port on wan left at 13355 but the client source was 51821.. If in that udp packet it told this 18.192 box to come back on port 51821 - pfsense would not allow that traffic because it would be out of state an not able to hole punch.
You would need to set a static nat on such traffic - where pfsense when it nats it, if the source is 51821, when it leaves pfsense wan it would stay 51821 vs being port natted to that 13355 port.
https://docs.netgate.com/pfsense/en/latest/nat/outbound.html#static-port
-
@johnpoz This totally makes sense.
So I have to create static port 51821 right?
-
@josifbg-0 If that is the port your vpn client is going to use to tell the other site/client how to hole punch back directly to him yes.. Is there a setting for that port in the client?
Its a horrible design flaw in such clients... The other end that is going to hole punch back should just use the port it got the traffic from.. In your example that 13355 port..
-
@johnpoz From the Server side ... the Netmaker Server I am able to control that for the whole network or per each node/client
So, Yes I can make changes to the client, but again I am not really sure how to set it up in pfsense....
Here are some rulz I`ve created but still not working
the Outbound rule:
the port forward rule:
Here are the changes to the client/Node in the Netmaker server:
-
-
@josifbg-0 so working for you - did you change the port, your posting showing the 192 box using 51821, but what you posted shows port 51825?
-
@johnpoz Yes, the port is changed everywhere.
-
@josifbg-0 glad you got it sorted not a fan really of such methods. If they want to whole punch that is fine - but there is little reason to require a specific source port - just use the port the device talked you from. This allows for napt.. But if you know what the listen port is - you don't have to use hole punching, could just port forward those ports on the different locations.
-
@johnpoz Well the main reason for hole punching is that the port can be dynamic and the second reason is not to open any ports on your router.
This way you are able to securely create private tunnels between sites without opening any ports on your router.
-
@josifbg-0 said in Routing from subnet does not belong to pfsense:
hole punching is that the port can be dynamic
Not when the router does napt and changes the source port - that is my point.. There is no reason for the place your talking to can't just use the port it got traffic from to hole punch back.. As in your example the client behind pfsense used port 51821, but pfsense changed that port using napt to 13355 etc.. The whole punch would of worked if the answer would of came in on that port, etc.
-
@josifbg-0 Sorry for the late reply. I tried to replicate your setting but for some reason it did not work for me. Hope this may help others. Other user told me in the past to set the netmaker server as a relay for all of the other nodes. After doing so, you do not have to make any changes to pfsense but still able to ping and SSH to all nodes.
Firewall >> NAT >> Outbound >> automaticNo special rule in the firewall
At the Netmaker, the server node is a Relay
DoneNow I can access my home and work wherever I am at without opening any port and doing NAT'ing
-
@moussa854 said in Routing from subnet does not belong to pfsense:
netmaker server as a relay for all of the other nodes
In that mode all traffic would go through the tunnels the clients have setup.. No hole punching would be used.
-
Thank you, I tested it with no hole punching and I am still able to ping all nodes. I fixed my post. Thanks @johnpoz
-
@moussa854 I came to the same problem and after a lot of trial and error I found out that in order to UDP hole punch to work with pfsense you need to set Static Port on NAT outbound.
From the menu Firewall > NAT > Outbound select Mode = Manual and edit the auto created rule LAN to WAN and set static port.
Now everything should work ;)
