Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server

nazar-pc

I have reported https://forum.netgate.com/topic/177152/one-website-doesn-t-open-through-wireguard-vpn a while ago and didn't get any response. Since I have configured Wireguard server on pfSense and noticed that rules work properly there, but not on LAN, which is confusing and likely means it was not a Wireguard issue after all.

Basically I have the following:

LAN
Wireguard server on pfSense
Remote Wireguard server
URL alias that points to ~14k IP subnets

For both LAN and Wireguard server interface last rule is to send traffic through WAN (I have Multi-WAN configured to be specific) and rule before that is to send everything corresponding to that URL alias through remote Wireguard server:
Знімок екрана з 2023-07-30 19-02-45.png

This works great when connecting to pfSense through it's Wireguard server and doesn't work on LAN for most IPs most of the time (sometimes it does work, not sure why though, maybe rules are reloaded at the time or something, very strange).

I'm confused as to how this is possible with identical rules, any ideas how to debug this and where to look?

Bob.Dig

@nazar-pc Maybe you have to raise the Firewall Maximum Table Entries but that is just a wild guess. Are you sure that your WG-clients do use pfSense for all their traffic?

nazar-pc

@Bob-Dig said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

Maybe you have to raise the Firewall Maximum Table Entries but that is just a wild guess.

Default for "Firewall Maximum Table Entries" was 200k, increased to 1M and reloaded rules, no change.

@Bob-Dig said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

Are you sure that your WG-clients do use pfSense for all their traffic?

100%, both on the phone (Android) and laptop (Linux) when configured to block any traffic unless it is going through VPN I see the same behavior.

bingo600

@nazar-pc
To me it looks like the Lan rule (i suppose it's the TCP/UDP rule) , has been "hit" by 593.80MB data (See States collum)

Why it's not working (forwarding) is a totally different "issue".

But the rule seems to do (match) what it's supposed to.

Edit: Are you sure you are not "bitten" by some missing NAT if data origins from Lan

/Bingo

nazar-pc

@bingo600 said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

To me it looks like the Lan rule (i suppose it's the TCP/UDP rule) , has been "hit" by 593.80MB data (See States collum)

Why it's not working (forwarding) is a totally different "issue".

But the rule seems to do (match) what it's supposed to.

It works sometimes and matches some subnets, but not others, so I'm not surprised there is some traffic matching those rules. The challenge is to make it work predictably.

@bingo600 said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

Edit: Are you sure you are not "bitten" by some missing NAT if data origins from Lan

I have some port forwarding for NAT, but nothing that would affect this that I know of. I don't think I need NAT rules for VPN to work though. You suspect anything specific?

bingo600

@nazar-pc said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

Edit: Are you sure you are not "bitten" by some missing NAT if data origins from Lan
I have some port forwarding for NAT, but nothing that would affect this that I know of. I don't think I need NAT rules for VPN to work though. You suspect anything specific?

My guess would be that your External VPN provider would not be able to handle packages of they have an RFC1918 source ip.
Primarily because that would be a possible conflict with other users, using the same LAN/RFC1918 IP's as source ip.

Do you have some NAT in place for the WG_NET , where stuff is working ?

nazar-pc

@bingo600 said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

My guess would be that your External VPN provider would not be able to handle packages of they have an RFC1918 source ip.
Primarily because that would be a possible conflict with other users, using the same LAN/RFC1918 IP's as source ip.

I'm running remote VPN server myself using https://github.com/linuxserver/docker-wireguard.

@bingo600 said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

Do you have some NAT in place for the WG_NET , where stuff is working ?

No outbound NAT, zero firewall rules on WG interface that goes to remote server.

bingo600

@nazar-pc said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

I'm running remote VPN server myself using https://github.com/linuxserver/docker-wireguard.

Ah ... That makes things simpler i guess.

Now are you sure that you have a route back to pfSense LAN on your "remote vpn box" , and that you don't have any subnet overlaps on the boxes ?

nazar-pc

@bingo600 said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

Now are you sure that you have a route back to pfSense LAN on your "remote vpn box" , and that you don't have any subnet overlaps on the boxes ?

I certainly have strictly distinct subnets on two machines, no overlap.

About route back to pfSense I'm not sure what are asking specifically (my networking knowledge improves, but still basic). My understanding was that it is sufficient to have public IP on Wireguard server, while client doesn't have to have port exposed publicly, so I didn't do anything special there.

bingo600

@nazar-pc

In order for the remote box to send reply packets back to the pfSense LAN via WG interface.
You would have to add a route on the VPN box, that points your pfSense lan subnet "back to the pfSense box" , via the ip of the pfSense WG interface.

nazar-pc

@bingo600 said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

In order for the remote box to send reply packets back to the pfSense LAN via WG interface.
You would have to add a route on the VPN box, that points your pfSense lan subnet "back to the pfSense box" , via the ip of the pfSense WG interface.

I suspect that must be taken care of by default by linked container image. I don't quite understand why would it be any different for LAN than WG_SERVER interface, it still goes to the same VPN box and back to the same pfSense.

bingo600

@nazar-pc said in Rule doesn't apply to all IPs in the alias on LAN, but works fine for Wireguard server:

I suspect that must be taken care of by default by linked container image.

OK .. If you say so ... But i guess you'll never be able to use your lan.

I don't quite understand why would it be any different for LAN than WG_SERVER interface, it still goes to the same VPN box and back to the same pfSense.

My guess ... Because your WG interfaces belong to the WG_NET , and then that interface (subnet & route) is already known on the pfSense & WG Box.

Edit:
I don't use WG, so i don't know the "inners of how it works".
But i do know routing on pfSense & linux.

Edit2:
Maybe this can "inspire" on the linux side
https://www.laroberto.com/remote-lan-access-with-wireguard/
https://www.mickaelwalter.fr/extend-your-lan-with-wireguard/

nazar-pc

I looked at this again, just to test things out I have modified the rule to allow any protocol, not just 80/443 ports TCP/UDP and surely enough, traceroute shows that I am routed through WireGuard server:

nazar-pc@nazar-pc:~> traceroute a.b.c.d
traceroute to a.b.c.d (a.b.c.d), 64 hops max
  1   10.13.13.1  31,048ms  31,602ms  30,619ms 
  2   192.168.160.1  30,959ms  30,653ms  30,894ms 
...

Where 10.13.13.1 is a WireGuard server that pfSense is a client of. So apparently the packets are going the right way, the question then is why TCP/UDP packets are not working the way I expect them to.

I then checked if I can open it in the browser and... it does open
I have edited the rule back to what it was and it still works, killed the states associated with the rule and it still works.

I'm very confused right now
I'm happy that it started to work, but I do not understand why it does or rather why it didn't before with exactly the same settings

nazar-pc

@nazar-pc Hm, it is even more odd now...
So it stopped working a few more times. I think I figured out how to get it back though.

I generally use Firefox Nightly, it errors with PR_IO_TIMEOUT_ERROR even though state according to pfSense is created.

If I then open the same website in Chromium it will fail to load the first time without meaningful error, but Firefox starts working after this and Chromium after refresh does load website successfully as well. After this website works in both Firefox and Chromijm for some some, even if I kill the states in pfSense. If I don't open it for some time though, it will stop working again.

Any plausible explanations for this?

nazar-pc

Changed interface MTU to 1420 and it seems to work, might be related to https://bugzilla.mozilla.org/show_bug.cgi?id=1734110 or similar in Firefox

nazar-pc

Finally found long-term solution. WG interface (that is a WireGuard client of a remote WireGuard server) needed not only MTU set to 1420, but also MSS set to 1420, otherwise I suspect it didn't try to fragment packets and things didn't work properly.

The reason it did work properly with WG_SERVER instead of LAN I think is because WG_SERVER also has MTU set to 1420 on both ends, so client was already sending properly sized packets.