Routing from subnet does not belong to pfsense

[moussa854]

@johnpoz
If the Netmaker (WG) server showing that both 10.101.0.1 and 10.101.0.2 are connected to the same network 10.101.0.0/16 then there should be no issue pinging each other even if they are at different locations, correct?

[johnpoz]

@moussa854 if you have device behind pfsense connected to the wireguard network. Pfsense would have zero to do with what you do inside that connection.

Now if you could not connect to the wireguard, then maybe pfsense could be blocking that, but once the client connects. What happens inside this vpn connection pfsense has no control or any clue to what you might be doing inside the vpn.

[moussa854]

@johnpoz said in Routing from subnet does not belong to pfsense:

once the client connects. What happens inside this vpn connection pfsense has no control or any clue to what you might be doing inside the vpn

This is my thought but not what I have experienced. While I am sitting at the same location, here is the ping while connected to teh main router before pfsense (no issue with 10.101.0.1 nor 10.101.255.254):

aa@mms-MacBook ~ % ping 10.101.0.1    
PING 10.101.0.1 (10.101.0.1): 56 data bytes
64 bytes from 10.101.0.1: icmp_seq=0 ttl=64 time=26.712 ms
64 bytes from 10.101.0.1: icmp_seq=1 ttl=64 time=25.334 ms
64 bytes from 10.101.0.1: icmp_seq=2 ttl=64 time=25.127 ms

--- 10.101.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 25.127/25.724/26.712/0.703 ms
aa@mms-MacBook ~ % ping 10.101.255.254
PING 10.101.255.254 (10.101.255.254): 56 data bytes
64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=26.939 ms
64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=24.541 ms
64 bytes from 10.101.255.254: icmp_seq=2 ttl=64 time=27.559 ms

--- 10.101.255.254 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 24.541/26.346/27.559/1.301 ms

While connected to internet behind pfsense I can ping the Netmaker (WG) server but not the other node:

aa@mms-MacBook ~ % ping 10.101.255.254
PING 10.101.255.254 (10.101.255.254): 56 data bytes
64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=149.906 ms
64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=117.603 ms

--- 10.101.255.254 ping statistics ---
3 packets transmitted, 2 packets received, 33.3% packet loss
round-trip min/avg/max/stddev = 117.603/133.755/149.906/16.151 ms
aa@mms-MacBook ~ % ping 10.101.0.1    
PING 10.101.0.1 (10.101.0.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2

--- 10.101.0.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

I even made the MacBook at 10.101.0.2 leave the Netmaker network and join again and still can not ping 10.101.0.1 while I am behind the pfsense. The only clue that I see in the logs is LAN Default deny rule IPv4 (1000000103) and the NO_TRAFFIC:SINGLE and SINGLE:NO_TRAFFIC as mentioned above.

[johnpoz]

@moussa854 Not sure what part your not getting.. Pfsense has Zero to do -- Let me repeat that again so its clear, ZERO to do with what happens inside a vpn tunnel.. There is no way pfsense can say oh you can't ping this..

If you can not talk to something on the wireguard network while you connected behind pfsense other than directly connected.. What are you connecting too?

Is your macbook actually connected to this wireguard behind pfsense? Or are you just pinging thie 10.101.255.254 address that is inside your isp network.

If pfsense could control what happens inside some tunnel you created from a client to some vpn server out on the internet - then the whole point of vpns would be completely utterly pointless.

Maybe your coming in from a different public IP then when you connect directly to the isp device? And this IP has permissions to talk to the other stuff on the vpn..

But what I am 200% positive of is pfsense has no way to control what happens inside that tunnel.. It could stop you from creating the tunnel.. But it sure can not say oh sorry you can not ping xyz inside the tunnel.. but you can ping this other IP..

[Josifbg 0]

@johnpoz I am sorry for reviving the old topic but I am facing the same issue behind pfsense.

I`ll try to explain in a bit more details:

1. Site A: A typical scenario where we have pfsense as router/firewall with Public IP managing a single LAN network 192.168.x.0/x

2. Site B: Cloud (let`s say AWS) has a VPS with Netmaker Server installed and accessible, ready to serve any authorized connection over his Public IP.

3. Site A - Client 1/Node 1 (192.168.1.100): A simple Laptop (Win, Linux. Mac - it doesn't really matter) is initiating Wireguard/Netmaker VPN connection to Site B Public IP (Netmaker Server), and... connection is established. The client gets internal IP from the Netmaker VPN let`s say 10.53.1.10. The Netmaker GW is 10.53.1.254. There is a Ping between the client 10.53.1.10 and the GW 10.53.1.254.

4. Site C - Client 2/Node 2: a different client in a different network behind some firewall, not pfSense with local IP 172.16.0.24 is initiating netmaker VPN connection to the public IP of Site B. connection is established and the IP assigned to the client C in the Netmaker network is 10.53.1.20. There is a ping between Site C 10.53.1.20 and Site A 10.53.1.254 no worries.

5. Site D - Client 3/Node 3: 3rd client in a row again in a totally different network behind some firewall, not pfsense with local IP 10.10.100.17 is initiating netmaker VPN connection to the public IP of Site B, the connection is established and the IP assigned to the client D in the Netmaker network is 10.53.1.30. There is a ping between Site D 10.53.1.30 and Site A 10.53.1.254 no worries.

Now comes the fun part:

Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)

Site C and Site D can ping each other and can ping the GW/The Netmaker Server but they cannot ping the Site A which is behind pfSense.

Some references: Netmaker uses a UDP Hole Punching so he can be able to create p2p connection between each one of the clients/nodes which gives the ability to have a direct connection between different nodes/clients without having the route the traffic through the GW/Netmaker Server.

As far as I am aware the same issue is facing the ZeroTier which uses UDP Hole Punching as well.

I hope this explanation the give some light and understanding what the real issue is.

Still looking for a solution....

[johnpoz]

@josifbg-0 said in Routing from subnet does not belong to pfsense:

Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)

Let me repeat this ZERO to do with pfsense..

If you tunnel "through" pfsense, doesn't matter what vpn software protocol.. Pfsense has zero to do with the routing inside that vpn, or what you can or can not do to other devices also connecting to that vpn..

If pfsense could do that - the the whole point of vpns would be pointless.. Pfsense has zero to do with the client making the connections routing through that tunnel.. If you can not get so some other site using your vpn your client is connected to, then its the vpn, the routes setup in the vpn, the firewalls on the end devices or the vpn.. But pfsense is oblivious to what is happening in the tunnel be it udp, icmp, ssh, http, it has no idea - all it knows if the outside of the tunnel.. Which it is allowing, or your client behind pfsense wouldn't be able to establish the connection..

A hole punch wouldn't be going through the tunnel.. Now if you block a client from going out the port that is trying to be punched you could have issues.. But traffic flowing through the tunnel no matter where its going to or coming from pfsense has nothing to do with that..

[Josifbg 0]

@johnpoz Thanks for your response.

So how would you explain that behind pfsense the client is not able to ping other nodes and to be pinged by any other nodes, but without pfsense there is no such issue at all?

[johnpoz]

@josifbg-0 they are not routing through the tunnel then.. If they are hole punching, then you would have to allow the actual connection outbound in the first place to allow for the whole punch. Maybe the port has to be static for what they are trying to do with the hole punch.

But traffic flowing though that tunnel pfsense has zero to do with any of that traffic.. Not unless pfsense was the actual endpoint in the vpn could it control traffic in or out of the tunnel.

[Josifbg 0]

here are some logs:

WAN udp 94.XX.XX.XX:13355 (192.168.20.125:51821) -> 18.192.50.92:45328 SINGLE:NO_TRAFFIC 901 / 0 155 KiB / 0 B

[Josifbg 0]

@johnpoz I totally agree with you and it was strange for me as well why this is happening.
"then you would have to allow the actual connection outbound " -> well this got to my mind as well but I am not sure which connection to open and where in the pfSense

[johnpoz]

@josifbg-0 said in Routing from subnet does not belong to pfsense:

WAN udp 94.XX.XX.XX:13355 (192.168.20.125:51821) -> 18.192.50.92:45328 SINGLE:NO_TRAFFIC 901 / 0 155 KiB / 0 B

That is client 20.125 leaving psense wan - it gets no answer - how is that a pfsense thing? But notice your port on wan left at 13355 but the client source was 51821.. If in that udp packet it told this 18.192 box to come back on port 51821 - pfsense would not allow that traffic because it would be out of state an not able to hole punch.

You would need to set a static nat on such traffic - where pfsense when it nats it, if the source is 51821, when it leaves pfsense wan it would stay 51821 vs being port natted to that 13355 port.

https://docs.netgate.com/pfsense/en/latest/nat/outbound.html#static-port

[Josifbg 0]

@johnpoz This totally makes sense.

So I have to create static port 51821 right?

[johnpoz]

@josifbg-0 If that is the port your vpn client is going to use to tell the other site/client how to hole punch back directly to him yes.. Is there a setting for that port in the client?

Its a horrible design flaw in such clients... The other end that is going to hole punch back should just use the port it got the traffic from.. In your example that 13355 port..

[Josifbg 0]

@johnpoz From the Server side ... the Netmaker Server I am able to control that for the whole network or per each node/client

So, Yes I can make changes to the client, but again I am not really sure how to set it up in pfsense....

Here are some rulz I`ve created but still not working

the Outbound rule:

the port forward rule:

Here are the changes to the client/Node in the Netmaker server:

[Josifbg 0]

@johnpoz I did it! I guess I did it

Here are the final changes which has to be made in pfSense:

The Outbound Rule:

And some changes to the node which is behind pfSense in Netmaker Server:

That`s it!

Thank you @johnpoz! you`ve made my day :)

[johnpoz]

@josifbg-0 so working for you - did you change the port, your posting showing the 192 box using 51821, but what you posted shows port 51825?

[Josifbg 0]

@johnpoz Yes, the port is changed everywhere.

[johnpoz]

@josifbg-0 glad you got it sorted not a fan really of such methods. If they want to whole punch that is fine - but there is little reason to require a specific source port - just use the port the device talked you from. This allows for napt.. But if you know what the listen port is - you don't have to use hole punching, could just port forward those ports on the different locations.

[Josifbg 0]

@johnpoz Well the main reason for hole punching is that the port can be dynamic and the second reason is not to open any ports on your router.

This way you are able to securely create private tunnels between sites without opening any ports on your router.

[johnpoz]

@josifbg-0 said in Routing from subnet does not belong to pfsense:

hole punching is that the port can be dynamic

Not when the router does napt and changes the source port - that is my point.. There is no reason for the place your talking to can't just use the port it got traffic from to hole punch back.. As in your example the client behind pfsense used port 51821, but pfsense changed that port using napt to 13355 etc.. The whole punch would of worked if the answer would of came in on that port, etc.