Routing from subnet does not belong to pfsense

moussa854 · Oct 20, 2022, 1:48 PM

Hi,
I have a Netmaker installed in the cloud with subnet 10.101.0.0/16. I am able to ping all nodes on the Netmaker network without opening any port(s) when I am not behind pfsense. However, when I am behind pfsense (192.168.1.1/24) I am not able to ping other nodes on the Netmaker network. Pinging the Netmaker server works fine. I do not want to open any port.

I tried adding a rule on the LAN to allow 10.101.0.0/16 any port any where - did not work.

I tried Hybrid Outbound NAT rule to LAN and WAN 10.101.0.0/16 any port any where - did not work.

I appreciate any thoughts or suggestion.

johnpoz · Oct 20, 2022, 1:55 PM

@moussa854 said in Routing from subnet does not belong to pfsense:

cloud with subnet 10.101.0.0/16

That is not a public IP address - how exactly are you getting there, you have some vpn or something connection to this netmaker? Where exactly are you hosting it?

moussa854 · Oct 20, 2022, 1:59 PM

@johnpoz
The Netmaker is hosted in the cloud and it uses wireguard vpn with subnet of 0.101.0.0/16 to connect the nodes / computers togather

johnpoz · Oct 20, 2022, 2:02 PM

@moussa854 well if your routing traffic out wireguard.. to get to some 10.101.x address, then just nat your clients behinds psfense to the wireguard address you have.

moussa854 · Oct 20, 2022, 2:10 PM

@johnpoz
Just to be clear:

The wireguard vpn is not on pfsense, it is on the cloud.
pfsense does not have wireguard vpn installed
I can ping the Netmaker server @ 10.101.255.254 but not the connected node @ 10.101.0.1
I can ping both Netmaker server @ 10.101.255.254 and the other node @ 10.101.0.1 when I am not connected to pfsense

Is there anything wrong with these rules:

I tried adding firewall rule on LAN and Outbound NAT rule to LAN and WAN but that did not help. I always get Request timeout for icmp_seq.

moussa854 · Oct 20, 2022, 2:17 PM

@moussa854 Here are my NAT rules
login-to-view

johnpoz · Oct 20, 2022, 2:33 PM

@moussa854 said in Routing from subnet does not belong to pfsense:

I can ping the Netmaker server @ 10.101.255.254 but not the connected node @ 10.101.0.1

No your not... Your ISP might have that IP in their network... But rfc1918 space does not route over the public internet..

If you have no vpn connection to this cloud network, there is no way your talking to some 10.x address over the public internet.. Just doesn't work that way, you could talk to rfc1918 addresses that are on your ISP network..

If you want to connect to this stuff with device behind pfsense, then you would need to connect pfsense wireguard network..

moussa854 · Oct 20, 2022, 2:45 PM

@johnpoz
You are right, I missed saying that the machine I use to ping the other node is connected to the WG vpn hosted on the Netmaker server in the cloud with an IP address of 10.101.0.2.

login-to-view

johnpoz · Oct 20, 2022, 2:49 PM

@moussa854 if you want clients behind pfsense to get to the network without having to use the wireguard client themselves - then you would need to join pfsense to it, and then route the traffic over the wireguard connection in pfsense.

moussa854 · Oct 20, 2022, 3:07 PM

@johnpoz

What if I want to use the MacBook on 10.101.0.2 to ping the other nodes 10.101.255.254 (work) and 10.101.0.1 (not work) when behind pfsense, what do I need to do?

N.B. on the Netmaker side it looks like there is some kind of healthy connection for the MackBook (10.101.0.2) but I can not ping 10.101.0.1 (I am not sure why?)

If I connect the MackBook to the main router (before the pfsense) I can ping and SSH to 10.101.0.1

moussa854 · Oct 20, 2022, 4:15 PM

@moussa854
Not sure if this help but in LAN I have a rule IPv4 * 10.0.30.0/16 * * * * none and the in the States I got

LAN 	udp 	192.168.30.51:1900 -> 10.101.0.2:57454 	NO_TRAFFIC:SINGLE 	 	
WAN 	udp 	WAN_IP:13300 (192.168.30.51:1900) -> 10.101.0.2:57454 	SINGLE:NO_TRAFFIC

And the log show LAN Default deny rule IPv4 (1000000103)

moussa854 · Oct 20, 2022, 4:22 PM

Here is the ping results with the above setting

aa@MacBook ~ % ping 10.101.0.1 
PING 10.101.0.1 (10.101.0.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3

--- 10.101.0.1 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss
aa@MacBook ~ % ping 10.101.255.254
PING 10.101.255.254 (10.101.255.254): 56 data bytes
64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=25.746 ms
64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=23.513 ms
64 bytes from 10.101.255.254: icmp_seq=2 ttl=64 time=23.235 ms
64 bytes from 10.101.255.254: icmp_seq=3 ttl=64 time=22.939 ms

--- 10.101.255.254 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 22.939/23.858/25.746/1.109 ms

johnpoz · Oct 20, 2022, 4:31 PM

@moussa854 I am at a loss to what you are not understanding about a 10.x address not being valid over the internet

That is a rfc1918 address. Your isp might use rfc1918 in their network that you can get to.. But sorry is it not possible for you to get to some 10.x address outside your or your isp network across the public internet.

You would have to connect to this vpn network over public IP space if you want to then route rfc1918 addresses via the vpn..

I can ping 10.x addresses inside my isp network as well - again they are RFC1918, they do not route over the public internet, you pinging some 10.x address inside your isp, is not you talking to some wireguard router with the same IP..

moussa854 · Oct 20, 2022, 5:15 PM

@johnpoz
If the Netmaker (WG) server showing that both 10.101.0.1 and 10.101.0.2 are connected to the same network 10.101.0.0/16 then there should be no issue pinging each other even if they are at different locations, correct?

login-to-view

johnpoz · Oct 20, 2022, 5:35 PM

@moussa854 if you have device behind pfsense connected to the wireguard network. Pfsense would have zero to do with what you do inside that connection.

Now if you could not connect to the wireguard, then maybe pfsense could be blocking that, but once the client connects. What happens inside this vpn connection pfsense has no control or any clue to what you might be doing inside the vpn.

moussa854 · Oct 20, 2022, 6:20 PM

@johnpoz said in Routing from subnet does not belong to pfsense:

once the client connects. What happens inside this vpn connection pfsense has no control or any clue to what you might be doing inside the vpn

This is my thought but not what I have experienced. While I am sitting at the same location, here is the ping while connected to teh main router before pfsense (no issue with 10.101.0.1 nor 10.101.255.254):

aa@mms-MacBook ~ % ping 10.101.0.1    
PING 10.101.0.1 (10.101.0.1): 56 data bytes
64 bytes from 10.101.0.1: icmp_seq=0 ttl=64 time=26.712 ms
64 bytes from 10.101.0.1: icmp_seq=1 ttl=64 time=25.334 ms
64 bytes from 10.101.0.1: icmp_seq=2 ttl=64 time=25.127 ms

--- 10.101.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 25.127/25.724/26.712/0.703 ms
aa@mms-MacBook ~ % ping 10.101.255.254
PING 10.101.255.254 (10.101.255.254): 56 data bytes
64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=26.939 ms
64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=24.541 ms
64 bytes from 10.101.255.254: icmp_seq=2 ttl=64 time=27.559 ms

--- 10.101.255.254 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 24.541/26.346/27.559/1.301 ms

While connected to internet behind pfsense I can ping the Netmaker (WG) server but not the other node:

aa@mms-MacBook ~ % ping 10.101.255.254
PING 10.101.255.254 (10.101.255.254): 56 data bytes
64 bytes from 10.101.255.254: icmp_seq=0 ttl=64 time=149.906 ms
64 bytes from 10.101.255.254: icmp_seq=1 ttl=64 time=117.603 ms

--- 10.101.255.254 ping statistics ---
3 packets transmitted, 2 packets received, 33.3% packet loss
round-trip min/avg/max/stddev = 117.603/133.755/149.906/16.151 ms
aa@mms-MacBook ~ % ping 10.101.0.1    
PING 10.101.0.1 (10.101.0.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2

--- 10.101.0.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

I even made the MacBook at 10.101.0.2 leave the Netmaker network and join again and still can not ping 10.101.0.1 while I am behind the pfsense. The only clue that I see in the logs is LAN Default deny rule IPv4 (1000000103) and the NO_TRAFFIC:SINGLE and SINGLE:NO_TRAFFIC as mentioned above.

johnpoz · Oct 20, 2022, 6:50 PM

@moussa854 Not sure what part your not getting.. Pfsense has Zero to do -- Let me repeat that again so its clear, ZERO to do with what happens inside a vpn tunnel.. There is no way pfsense can say oh you can't ping this..

If you can not talk to something on the wireguard network while you connected behind pfsense other than directly connected.. What are you connecting too?

Is your macbook actually connected to this wireguard behind pfsense? Or are you just pinging thie 10.101.255.254 address that is inside your isp network.

If pfsense could control what happens inside some tunnel you created from a client to some vpn server out on the internet - then the whole point of vpns would be completely utterly pointless.

Maybe your coming in from a different public IP then when you connect directly to the isp device? And this IP has permissions to talk to the other stuff on the vpn..

But what I am 200% positive of is pfsense has no way to control what happens inside that tunnel.. It could stop you from creating the tunnel.. But it sure can not say oh sorry you can not ping xyz inside the tunnel.. but you can ping this other IP..

Josifbg 0 · Dec 30, 2022, 9:14 PM

@johnpoz I am sorry for reviving the old topic but I am facing the same issue behind pfsense.

I`ll try to explain in a bit more details:

Site A: A typical scenario where we have pfsense as router/firewall with Public IP managing a single LAN network 192.168.x.0/x
Site B: Cloud (let`s say AWS) has a VPS with Netmaker Server installed and accessible, ready to serve any authorized connection over his Public IP.
Site A - Client 1/Node 1 (192.168.1.100): A simple Laptop (Win, Linux. Mac - it doesn't really matter) is initiating Wireguard/Netmaker VPN connection to Site B Public IP (Netmaker Server), and... connection is established. The client gets internal IP from the Netmaker VPN let`s say 10.53.1.10. The Netmaker GW is 10.53.1.254. There is a Ping between the client 10.53.1.10 and the GW 10.53.1.254.
Site C - Client 2/Node 2: a different client in a different network behind some firewall, not pfSense with local IP 172.16.0.24 is initiating netmaker VPN connection to the public IP of Site B. connection is established and the IP assigned to the client C in the Netmaker network is 10.53.1.20. There is a ping between Site C 10.53.1.20 and Site A 10.53.1.254 no worries.
Site D - Client 3/Node 3: 3rd client in a row again in a totally different network behind some firewall, not pfsense with local IP 10.10.100.17 is initiating netmaker VPN connection to the public IP of Site B, the connection is established and the IP assigned to the client D in the Netmaker network is 10.53.1.30. There is a ping between Site D 10.53.1.30 and Site A 10.53.1.254 no worries.

Now comes the fun part:

Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)

Site C and Site D can ping each other and can ping the GW/The Netmaker Server but they cannot ping the Site A which is behind pfSense.

Some references: Netmaker uses a UDP Hole Punching so he can be able to create p2p connection between each one of the clients/nodes which gives the ability to have a direct connection between different nodes/clients without having the route the traffic through the GW/Netmaker Server.

As far as I am aware the same issue is facing the ZeroTier which uses UDP Hole Punching as well.

I hope this explanation the give some light and understanding what the real issue is.

Still looking for a solution....

johnpoz · Dec 30, 2022, 11:59 PM

@josifbg-0 said in Routing from subnet does not belong to pfsense:

Site A can ping Site B (the server/GW) no problem
Site A cannot ping any other Clients/Nodes (Site C and Site D)

Let me repeat this ZERO to do with pfsense..

If you tunnel "through" pfsense, doesn't matter what vpn software protocol.. Pfsense has zero to do with the routing inside that vpn, or what you can or can not do to other devices also connecting to that vpn..

If pfsense could do that - the the whole point of vpns would be pointless.. Pfsense has zero to do with the client making the connections routing through that tunnel.. If you can not get so some other site using your vpn your client is connected to, then its the vpn, the routes setup in the vpn, the firewalls on the end devices or the vpn.. But pfsense is oblivious to what is happening in the tunnel be it udp, icmp, ssh, http, it has no idea - all it knows if the outside of the tunnel.. Which it is allowing, or your client behind pfsense wouldn't be able to establish the connection..

A hole punch wouldn't be going through the tunnel.. Now if you block a client from going out the port that is trying to be punched you could have issues.. But traffic flowing through the tunnel no matter where its going to or coming from pfsense has nothing to do with that..

Josifbg 0 · Dec 30, 2022, 11:59 PM

@johnpoz Thanks for your response.

So how would you explain that behind pfsense the client is not able to ping other nodes and to be pinged by any other nodes, but without pfsense there is no such issue at all?