Routing from subnet does not belong to pfsense
-
@johnpoz
You are right, I missed saying that the machine I use to ping the other node is connected to the WG vpn hosted on the Netmaker server in the cloud with an IP address of 10.101.0.2.
-
@moussa854 if you want clients behind pfsense to get to the network without having to use the wireguard client themselves - then you would need to join pfsense to it, and then route the traffic over the wireguard connection in pfsense.
-
What if I want to use the MacBook on 10.101.0.2 to ping the other nodes 10.101.255.254 (work) and 10.101.0.1 (not work) when behind pfsense, what do I need to do?
N.B. on the Netmaker side it looks like there is some kind of healthy connection for the MackBook (10.101.0.2) but I can not ping 10.101.0.1 (I am not sure why?)
If I connect the MackBook to the main router (before the pfsense) I can ping and SSH to 10.101.0.1
-
@moussa854
Not sure if this help but in LAN I have a ruleIPv4 * 10.0.30.0/16 * * * * noneand the in the States I gotLAN udp 192.168.30.51:1900 -> 10.101.0.2:57454 NO_TRAFFIC:SINGLE WAN udp WAN_IP:13300 (192.168.30.51:1900) -> 10.101.0.2:57454 SINGLE:NO_TRAFFICAnd the log show
LAN Default deny rule IPv4 (1000000103) -
Here is the ping results with the above setting
aa@MacBook ~ % ping 10.101.0.1 PING 10.101.0.1 (10.101.0.1): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 Request timeout for icmp_seq 3 --- 10.101.0.1 ping statistics --- 5 packets transmitted, 0 packets received, 100.0% packet loss aa@MacBook ~ % ping 10.101.255.254 PING 10.101.255.254 (10.101.255.254): 56 data bytes 64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=25.746 ms 64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=23.513 ms 64 bytes from 10.101.255.254: icmp_seq=2 ttl=64 time=23.235 ms 64 bytes from 10.101.255.254: icmp_seq=3 ttl=64 time=22.939 ms --- 10.101.255.254 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 22.939/23.858/25.746/1.109 ms -
@moussa854 I am at a loss to what you are not understanding about a 10.x address not being valid over the internet
That is a rfc1918 address. Your isp might use rfc1918 in their network that you can get to.. But sorry is it not possible for you to get to some 10.x address outside your or your isp network across the public internet.
You would have to connect to this vpn network over public IP space if you want to then route rfc1918 addresses via the vpn..
I can ping 10.x addresses inside my isp network as well - again they are RFC1918, they do not route over the public internet, you pinging some 10.x address inside your isp, is not you talking to some wireguard router with the same IP..
-
@johnpoz
If the Netmaker (WG) server showing that both10.101.0.1and10.101.0.2are connected to the same network10.101.0.0/16then there should be no issue pinging each other even if they are at different locations, correct?
-
@moussa854 if you have device behind pfsense connected to the wireguard network. Pfsense would have zero to do with what you do inside that connection.
Now if you could not connect to the wireguard, then maybe pfsense could be blocking that, but once the client connects. What happens inside this vpn connection pfsense has no control or any clue to what you might be doing inside the vpn.
-
@johnpoz said in Routing from subnet does not belong to pfsense:
once the client connects. What happens inside this vpn connection pfsense has no control or any clue to what you might be doing inside the vpn
This is my thought but not what I have experienced. While I am sitting at the same location, here is the ping while connected to teh main router before pfsense (no issue with
10.101.0.1nor10.101.255.254):aa@mms-MacBook ~ % ping 10.101.0.1 PING 10.101.0.1 (10.101.0.1): 56 data bytes 64 bytes from 10.101.0.1: icmp_seq=0 ttl=64 time=26.712 ms 64 bytes from 10.101.0.1: icmp_seq=1 ttl=64 time=25.334 ms 64 bytes from 10.101.0.1: icmp_seq=2 ttl=64 time=25.127 ms --- 10.101.0.1 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 25.127/25.724/26.712/0.703 ms aa@mms-MacBook ~ % ping 10.101.255.254 PING 10.101.255.254 (10.101.255.254): 56 data bytes 64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=26.939 ms 64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=24.541 ms 64 bytes from 10.101.255.254: icmp_seq=2 ttl=64 time=27.559 ms --- 10.101.255.254 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 24.541/26.346/27.559/1.301 msWhile connected to internet behind pfsense I can ping the Netmaker (WG) server but not the other node:
aa@mms-MacBook ~ % ping 10.101.255.254 PING 10.101.255.254 (10.101.255.254): 56 data bytes 64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=149.906 ms 64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=117.603 ms --- 10.101.255.254 ping statistics --- 3 packets transmitted, 2 packets received, 33.3% packet loss round-trip min/avg/max/stddev = 117.603/133.755/149.906/16.151 ms aa@mms-MacBook ~ % ping 10.101.0.1 PING 10.101.0.1 (10.101.0.1): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 --- 10.101.0.1 ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet lossI even made the MacBook at 10.101.0.2 leave the Netmaker network and join again and still can not ping 10.101.0.1 while I am behind the pfsense. The only clue that I see in the logs is
LAN Default deny rule IPv4 (1000000103)and theNO_TRAFFIC:SINGLEandSINGLE:NO_TRAFFICas mentioned above. -
@moussa854 Not sure what part your not getting.. Pfsense has Zero to do -- Let me repeat that again so its clear, ZERO to do with what happens inside a vpn tunnel.. There is no way pfsense can say oh you can't ping this..
If you can not talk to something on the wireguard network while you connected behind pfsense other than directly connected.. What are you connecting too?
Is your macbook actually connected to this wireguard behind pfsense? Or are you just pinging thie 10.101.255.254 address that is inside your isp network.
If pfsense could control what happens inside some tunnel you created from a client to some vpn server out on the internet - then the whole point of vpns would be completely utterly pointless.
Maybe your coming in from a different public IP then when you connect directly to the isp device? And this IP has permissions to talk to the other stuff on the vpn..
But what I am 200% positive of is pfsense has no way to control what happens inside that tunnel.. It could stop you from creating the tunnel.. But it sure can not say oh sorry you can not ping xyz inside the tunnel.. but you can ping this other IP..
-
@johnpoz I am sorry for reviving the old topic but I am facing the same issue behind pfsense.
I`ll try to explain in a bit more details:
-
Site A: A typical scenario where we have pfsense as router/firewall with Public IP managing a single LAN network 192.168.x.0/x
-
Site B: Cloud (let`s say AWS) has a VPS with Netmaker Server installed and accessible, ready to serve any authorized connection over his Public IP.
-
Site A - Client 1/Node 1 (192.168.1.100): A simple Laptop (Win, Linux. Mac - it doesn't really matter) is initiating Wireguard/Netmaker VPN connection to Site B Public IP (Netmaker Server), and... connection is established. The client gets internal IP from the Netmaker VPN let`s say 10.53.1.10. The Netmaker GW is 10.53.1.254. There is a Ping between the client 10.53.1.10 and the GW 10.53.1.254.
-
Site C - Client 2/Node 2: a different client in a different network behind some firewall, not pfSense with local IP 172.16.0.24 is initiating netmaker VPN connection to the public IP of Site B. connection is established and the IP assigned to the client C in the Netmaker network is 10.53.1.20. There is a ping between Site C 10.53.1.20 and Site A 10.53.1.254 no worries.
-
Site D - Client 3/Node 3: 3rd client in a row again in a totally different network behind some firewall, not pfsense with local IP 10.10.100.17 is initiating netmaker VPN connection to the public IP of Site B, the connection is established and the IP assigned to the client D in the Netmaker network is 10.53.1.30. There is a ping between Site D 10.53.1.30 and Site A 10.53.1.254 no worries.
Now comes the fun part:
Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)Site C and Site D can ping each other and can ping the GW/The Netmaker Server but they cannot ping the Site A which is behind pfSense.
Some references: Netmaker uses a UDP Hole Punching so he can be able to create p2p connection between each one of the clients/nodes which gives the ability to have a direct connection between different nodes/clients without having the route the traffic through the GW/Netmaker Server.
As far as I am aware the same issue is facing the ZeroTier which uses UDP Hole Punching as well.
I hope this explanation the give some light and understanding what the real issue is.
Still looking for a solution....
-
-
@josifbg-0 said in Routing from subnet does not belong to pfsense:
Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)Let me repeat this ZERO to do with pfsense..
If you tunnel "through" pfsense, doesn't matter what vpn software protocol.. Pfsense has zero to do with the routing inside that vpn, or what you can or can not do to other devices also connecting to that vpn..
If pfsense could do that - the the whole point of vpns would be pointless.. Pfsense has zero to do with the client making the connections routing through that tunnel.. If you can not get so some other site using your vpn your client is connected to, then its the vpn, the routes setup in the vpn, the firewalls on the end devices or the vpn.. But pfsense is oblivious to what is happening in the tunnel be it udp, icmp, ssh, http, it has no idea - all it knows if the outside of the tunnel.. Which it is allowing, or your client behind pfsense wouldn't be able to establish the connection..
A hole punch wouldn't be going through the tunnel.. Now if you block a client from going out the port that is trying to be punched you could have issues.. But traffic flowing through the tunnel no matter where its going to or coming from pfsense has nothing to do with that..
-
@johnpoz Thanks for your response.
So how would you explain that behind pfsense the client is not able to ping other nodes and to be pinged by any other nodes, but without pfsense there is no such issue at all?
-
@josifbg-0 they are not routing through the tunnel then.. If they are hole punching, then you would have to allow the actual connection outbound in the first place to allow for the whole punch. Maybe the port has to be static for what they are trying to do with the hole punch.
But traffic flowing though that tunnel pfsense has zero to do with any of that traffic.. Not unless pfsense was the actual endpoint in the vpn could it control traffic in or out of the tunnel.
-
here are some logs:
WAN udp 94.XX.XX.XX:13355 (192.168.20.125:51821) -> 18.192.50.92:45328 SINGLE:NO_TRAFFIC 901 / 0 155 KiB / 0 B
-
@johnpoz I totally agree with you and it was strange for me as well why this is happening.
"then you would have to allow the actual connection outbound " -> well this got to my mind as well but I am not sure which connection to open and where in the pfSense -
@josifbg-0 said in Routing from subnet does not belong to pfsense:
WAN udp 94.XX.XX.XX:13355 (192.168.20.125:51821) -> 18.192.50.92:45328 SINGLE:NO_TRAFFIC 901 / 0 155 KiB / 0 B
That is client 20.125 leaving psense wan - it gets no answer - how is that a pfsense thing? But notice your port on wan left at 13355 but the client source was 51821.. If in that udp packet it told this 18.192 box to come back on port 51821 - pfsense would not allow that traffic because it would be out of state an not able to hole punch.
You would need to set a static nat on such traffic - where pfsense when it nats it, if the source is 51821, when it leaves pfsense wan it would stay 51821 vs being port natted to that 13355 port.
https://docs.netgate.com/pfsense/en/latest/nat/outbound.html#static-port
-
@johnpoz This totally makes sense.
So I have to create static port 51821 right?
-
@josifbg-0 If that is the port your vpn client is going to use to tell the other site/client how to hole punch back directly to him yes.. Is there a setting for that port in the client?
Its a horrible design flaw in such clients... The other end that is going to hole punch back should just use the port it got the traffic from.. In your example that 13355 port..
-
@johnpoz From the Server side ... the Netmaker Server I am able to control that for the whole network or per each node/client
So, Yes I can make changes to the client, but again I am not really sure how to set it up in pfsense....
Here are some rulz I`ve created but still not working
the Outbound rule:
the port forward rule:
Here are the changes to the client/Node in the Netmaker server:
