Routing from subnet does not belong to pfsense
-
@johnpoz said in Routing from subnet does not belong to pfsense:
once the client connects. What happens inside this vpn connection pfsense has no control or any clue to what you might be doing inside the vpn
This is my thought but not what I have experienced. While I am sitting at the same location, here is the ping while connected to teh main router before pfsense (no issue with
10.101.0.1nor10.101.255.254):aa@mms-MacBook ~ % ping 10.101.0.1 PING 10.101.0.1 (10.101.0.1): 56 data bytes 64 bytes from 10.101.0.1: icmp_seq=0 ttl=64 time=26.712 ms 64 bytes from 10.101.0.1: icmp_seq=1 ttl=64 time=25.334 ms 64 bytes from 10.101.0.1: icmp_seq=2 ttl=64 time=25.127 ms --- 10.101.0.1 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 25.127/25.724/26.712/0.703 ms aa@mms-MacBook ~ % ping 10.101.255.254 PING 10.101.255.254 (10.101.255.254): 56 data bytes 64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=26.939 ms 64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=24.541 ms 64 bytes from 10.101.255.254: icmp_seq=2 ttl=64 time=27.559 ms --- 10.101.255.254 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 24.541/26.346/27.559/1.301 msWhile connected to internet behind pfsense I can ping the Netmaker (WG) server but not the other node:
aa@mms-MacBook ~ % ping 10.101.255.254 PING 10.101.255.254 (10.101.255.254): 56 data bytes 64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=149.906 ms 64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=117.603 ms --- 10.101.255.254 ping statistics --- 3 packets transmitted, 2 packets received, 33.3% packet loss round-trip min/avg/max/stddev = 117.603/133.755/149.906/16.151 ms aa@mms-MacBook ~ % ping 10.101.0.1 PING 10.101.0.1 (10.101.0.1): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 --- 10.101.0.1 ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet lossI even made the MacBook at 10.101.0.2 leave the Netmaker network and join again and still can not ping 10.101.0.1 while I am behind the pfsense. The only clue that I see in the logs is
LAN Default deny rule IPv4 (1000000103)and theNO_TRAFFIC:SINGLEandSINGLE:NO_TRAFFICas mentioned above. -
@moussa854 Not sure what part your not getting.. Pfsense has Zero to do -- Let me repeat that again so its clear, ZERO to do with what happens inside a vpn tunnel.. There is no way pfsense can say oh you can't ping this..
If you can not talk to something on the wireguard network while you connected behind pfsense other than directly connected.. What are you connecting too?
Is your macbook actually connected to this wireguard behind pfsense? Or are you just pinging thie 10.101.255.254 address that is inside your isp network.
If pfsense could control what happens inside some tunnel you created from a client to some vpn server out on the internet - then the whole point of vpns would be completely utterly pointless.
Maybe your coming in from a different public IP then when you connect directly to the isp device? And this IP has permissions to talk to the other stuff on the vpn..
But what I am 200% positive of is pfsense has no way to control what happens inside that tunnel.. It could stop you from creating the tunnel.. But it sure can not say oh sorry you can not ping xyz inside the tunnel.. but you can ping this other IP..
-
@johnpoz I am sorry for reviving the old topic but I am facing the same issue behind pfsense.
I`ll try to explain in a bit more details:
-
Site A: A typical scenario where we have pfsense as router/firewall with Public IP managing a single LAN network 192.168.x.0/x
-
Site B: Cloud (let`s say AWS) has a VPS with Netmaker Server installed and accessible, ready to serve any authorized connection over his Public IP.
-
Site A - Client 1/Node 1 (192.168.1.100): A simple Laptop (Win, Linux. Mac - it doesn't really matter) is initiating Wireguard/Netmaker VPN connection to Site B Public IP (Netmaker Server), and... connection is established. The client gets internal IP from the Netmaker VPN let`s say 10.53.1.10. The Netmaker GW is 10.53.1.254. There is a Ping between the client 10.53.1.10 and the GW 10.53.1.254.
-
Site C - Client 2/Node 2: a different client in a different network behind some firewall, not pfSense with local IP 172.16.0.24 is initiating netmaker VPN connection to the public IP of Site B. connection is established and the IP assigned to the client C in the Netmaker network is 10.53.1.20. There is a ping between Site C 10.53.1.20 and Site A 10.53.1.254 no worries.
-
Site D - Client 3/Node 3: 3rd client in a row again in a totally different network behind some firewall, not pfsense with local IP 10.10.100.17 is initiating netmaker VPN connection to the public IP of Site B, the connection is established and the IP assigned to the client D in the Netmaker network is 10.53.1.30. There is a ping between Site D 10.53.1.30 and Site A 10.53.1.254 no worries.
Now comes the fun part:
Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)Site C and Site D can ping each other and can ping the GW/The Netmaker Server but they cannot ping the Site A which is behind pfSense.
Some references: Netmaker uses a UDP Hole Punching so he can be able to create p2p connection between each one of the clients/nodes which gives the ability to have a direct connection between different nodes/clients without having the route the traffic through the GW/Netmaker Server.
As far as I am aware the same issue is facing the ZeroTier which uses UDP Hole Punching as well.
I hope this explanation the give some light and understanding what the real issue is.
Still looking for a solution....
-
-
@josifbg-0 said in Routing from subnet does not belong to pfsense:
Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)Let me repeat this ZERO to do with pfsense..
If you tunnel "through" pfsense, doesn't matter what vpn software protocol.. Pfsense has zero to do with the routing inside that vpn, or what you can or can not do to other devices also connecting to that vpn..
If pfsense could do that - the the whole point of vpns would be pointless.. Pfsense has zero to do with the client making the connections routing through that tunnel.. If you can not get so some other site using your vpn your client is connected to, then its the vpn, the routes setup in the vpn, the firewalls on the end devices or the vpn.. But pfsense is oblivious to what is happening in the tunnel be it udp, icmp, ssh, http, it has no idea - all it knows if the outside of the tunnel.. Which it is allowing, or your client behind pfsense wouldn't be able to establish the connection..
A hole punch wouldn't be going through the tunnel.. Now if you block a client from going out the port that is trying to be punched you could have issues.. But traffic flowing through the tunnel no matter where its going to or coming from pfsense has nothing to do with that..
-
@johnpoz Thanks for your response.
So how would you explain that behind pfsense the client is not able to ping other nodes and to be pinged by any other nodes, but without pfsense there is no such issue at all?
-
@josifbg-0 they are not routing through the tunnel then.. If they are hole punching, then you would have to allow the actual connection outbound in the first place to allow for the whole punch. Maybe the port has to be static for what they are trying to do with the hole punch.
But traffic flowing though that tunnel pfsense has zero to do with any of that traffic.. Not unless pfsense was the actual endpoint in the vpn could it control traffic in or out of the tunnel.
-
here are some logs:
WAN udp 94.XX.XX.XX:13355 (192.168.20.125:51821) -> 18.192.50.92:45328 SINGLE:NO_TRAFFIC 901 / 0 155 KiB / 0 B
-
@johnpoz I totally agree with you and it was strange for me as well why this is happening.
"then you would have to allow the actual connection outbound " -> well this got to my mind as well but I am not sure which connection to open and where in the pfSense -
@josifbg-0 said in Routing from subnet does not belong to pfsense:
WAN udp 94.XX.XX.XX:13355 (192.168.20.125:51821) -> 18.192.50.92:45328 SINGLE:NO_TRAFFIC 901 / 0 155 KiB / 0 B
That is client 20.125 leaving psense wan - it gets no answer - how is that a pfsense thing? But notice your port on wan left at 13355 but the client source was 51821.. If in that udp packet it told this 18.192 box to come back on port 51821 - pfsense would not allow that traffic because it would be out of state an not able to hole punch.
You would need to set a static nat on such traffic - where pfsense when it nats it, if the source is 51821, when it leaves pfsense wan it would stay 51821 vs being port natted to that 13355 port.
https://docs.netgate.com/pfsense/en/latest/nat/outbound.html#static-port
-
@johnpoz This totally makes sense.
So I have to create static port 51821 right?
-
@josifbg-0 If that is the port your vpn client is going to use to tell the other site/client how to hole punch back directly to him yes.. Is there a setting for that port in the client?
Its a horrible design flaw in such clients... The other end that is going to hole punch back should just use the port it got the traffic from.. In your example that 13355 port..
-
@johnpoz From the Server side ... the Netmaker Server I am able to control that for the whole network or per each node/client
So, Yes I can make changes to the client, but again I am not really sure how to set it up in pfsense....
Here are some rulz I`ve created but still not working
the Outbound rule:
the port forward rule:
Here are the changes to the client/Node in the Netmaker server:
-
-
@josifbg-0 so working for you - did you change the port, your posting showing the 192 box using 51821, but what you posted shows port 51825?
-
@johnpoz Yes, the port is changed everywhere.
-
@josifbg-0 glad you got it sorted not a fan really of such methods. If they want to whole punch that is fine - but there is little reason to require a specific source port - just use the port the device talked you from. This allows for napt.. But if you know what the listen port is - you don't have to use hole punching, could just port forward those ports on the different locations.
-
@johnpoz Well the main reason for hole punching is that the port can be dynamic and the second reason is not to open any ports on your router.
This way you are able to securely create private tunnels between sites without opening any ports on your router.
-
@josifbg-0 said in Routing from subnet does not belong to pfsense:
hole punching is that the port can be dynamic
Not when the router does napt and changes the source port - that is my point.. There is no reason for the place your talking to can't just use the port it got traffic from to hole punch back.. As in your example the client behind pfsense used port 51821, but pfsense changed that port using napt to 13355 etc.. The whole punch would of worked if the answer would of came in on that port, etc.
-
@josifbg-0 Sorry for the late reply. I tried to replicate your setting but for some reason it did not work for me. Hope this may help others. Other user told me in the past to set the netmaker server as a relay for all of the other nodes. After doing so, you do not have to make any changes to pfsense but still able to ping and SSH to all nodes.
Firewall >> NAT >> Outbound >> automaticNo special rule in the firewall
At the Netmaker, the server node is a Relay
DoneNow I can access my home and work wherever I am at without opening any port and doing NAT'ing
-
@moussa854 said in Routing from subnet does not belong to pfsense:
netmaker server as a relay for all of the other nodes
In that mode all traffic would go through the tunnels the clients have setup.. No hole punching would be used.
