Port Forward within LAN via Wireguard VM

leonidas-o · Dec 12, 2022, 2:29 PM

@viragomann yes, wireguard is masquerading, I have the following in its "/etc/wireguard/tun0.conf"

PostUp = iptables -t nat -I POSTROUTING 1 -s 10.8.0.1/24 -o eth0 -j MASQUERADE; iptables -I INPUT 1 -i tun0 -j ACCEPT; iptables -I FORWARD 1 -i tun0 -o eth0 -j ACCEPT; iptables -I FORWARD 1 -i eth0 -o tun0 -j ACCEPT;
PostDown = iptables -t nat -D POSTROUTING 1 -s 10.8.0.1/24 -o eth0 -j MASQUERADE; iptables -D INPUT 1 -i tun0 -j ACCEPT; iptables -D FORWARD 1 -i tun0 -o eth0 -j ACCEPT; iptables -D FORWARD 1 -i eth0 -o tun0 -j ACCEPT;

If I ssh into wireguard or the mailserver vm, both can ping each other, and I'm redirecting all traffic through wireguard when connected (allowedIPs 0.0.0.0/0), making it a full VPN connection. Internet access is working from within wireguard.

The only anti lock out rule I found so far in pfsense was:

However opnsense has an anti lockout rule for nat port forwarding as well, this rule somehow is always on top and can't be even deleted. (Exclamation mark means: No redirect). So I thought maybe I need something like that in pfsense as well:

However, best practice is to use a different port for the Web configurator.

Mhh of course, makes totally sense. Have changed the port immediately.

Btw, sorry for switching between pfsense and opnsense, as a side note, I have two proxmox nodes, and on one node opnsense, on the other pfsense so I can compare both while setting up, to see which one fits my needs better.
And now I'm facing "gladly" the same issue/ same behaviour on both firewalls.

So coming back to the issue.
When connected to wireguard (yes, both proxmox nodes have a separate wireguard VM, traefik VM, etc.), I can't access the the services behind traefik proxy.
On pfsense I've added a port forwarding rule for 443:

And inside the rule, I've set "Nat reflection" to "Enabled (Nat+proxy)". When trying to access the service, the page starts to load, get stuck and after a longer period of time it fails.
I've also tried the global option in "System Advanced Firewall & NAT -> NAT Reflection mode for port forwards" to "NAT + proxy".
For both I tried even "Pure NAT", but same behaviour.

Do I have to add any other rules, nat port forwarding etc.?

viragomann · Dec 12, 2022, 4:25 PM

@leonidas-o said in Port Forward within LAN via Wireguard VM:

yes, wireguard is masquerading, I have the following in its "/etc/wireguard/tun0.conf"

PostUp = iptables -t nat -I POSTROUTING 1 -s 10.8.0.1/24 -o eth0 -j MASQUERADE; iptables -I INPUT 1 -i tun0 -j ACCEPT; iptables -I FORWARD 1 -i tun0 -o eth0 -j ACCEPT; iptables -I FORWARD 1 -i eth0 -o tun0 -j ACCEPT;
PostDown = iptables -t nat -D POSTROUTING 1 -s 10.8.0.1/24 -o eth0 -j MASQUERADE; iptables -D INPUT 1 -i tun0 -j ACCEPT; iptables -D FORWARD 1 -i tun0 -o

Are you sure that this rule works? 10.8.0.1/24 is not a proper network. I would change it to 10.8.0.024, assuming this is you VPN virtual IP pool.

However opnsense has an anti lockout rule for nat port forwarding as well, this rule somehow is always on top and can't be even deleted. (Exclamation mark means: No redirect).

What's the use of a nat port forwarding rule, which does forward nothing?
I don't think that you would need this in pfSense.

And inside the rule, I've set "Nat reflection" to "Enabled (Nat+proxy)". When trying to access the service, the page starts to load, get stuck and after a longer period of time it fails.

Any blocks regarding this in the firewall rules?
Ensure the "Log firewall default blocks" is enabled in the log settings.

leonidas-o · Dec 12, 2022, 8:20 PM

Are you sure that this rule works? 10.8.0.1/24 is not a proper network. I would change it to 10.8.0.024, assuming this is you VPN virtual IP pool.

@viragomann I've changed it to "10.8.0.0/24" but the behaviour is the same. I also think, if masquerading wouldn't work, I would have way more problems. With the old setting maybe it just masquerade all IPs from 10.8.0.1 - 10.8.0.254. Nevertheless, when connected to wireguard everything seems to work, as long as I don't try to access a service inside the LAN.
I even saw that behaviour for Gitlab and a Gitlab-runner. The Runner registeres itself on the gitlab instance via a url like "https://mygitlab.mydomain.com". And also this gitlab-runner appears to be offline, I think it has the same problem like me, when inside VPN.

What's the use of a nat port forwarding rule, which does forward nothing?

I think this is simply a safety net, as it is the first rule and that can't be changed, even if I create another port forward for 443, this rule should actually intercept the request and keep the GUI access intact. I only locked myself out, when I added a port forward for 443 and enabled the reflection option. I guess that's all it does. But yeah, actually I don't need it with the GUI on another port and knowing the behaviour.

Any blocks regarding this in the firewall rules?
Ensure the "Log firewall default blocks" is enabled in the log settings.

Yes the option is activated, just checked it. Unfortunately even with the adapted wireguard tun0.conf it is not working.

So you say, I only need a WAN based NAT port forward rule like in my last screenshot to traefik and inside it "enabled reflection". And that should work? I don't need any other LAN interface port forwarding rules or any other firewall rules?

I also went into the automatically generated firewall rule (generated out of the NAT port forward rule) and activated: "Log packets that are handled by this rule". But I don't see any blocks in "Status/ System Logs/ Firewall/ Normal View".

viragomann · Dec 13, 2022, 1:36 PM

@leonidas-o said in Port Forward within LAN via Wireguard VM:

I even saw that behaviour for Gitlab and a Gitlab-runner. The Runner registeres itself on the gitlab instance via a url like "https://mygitlab.mydomain.com".

I don't know, how Gitlab-runner works. So no idea, what's wrong with that.
Is it running on the clients site?

I think this is simply a safety net, as it is the first rule and that can't be changed, even if I create another port forward for 443, this rule should actually intercept the request and keep the GUI access intact. I only locked myself out, when I added a port forward for 443 and enabled the reflection option.

I got the idea, that the nat rule (with exclamation mark) prohibits the stated ports from being forwarded. That would make sense then.

Any blocks regarding this in the firewall rules?
Ensure the "Log firewall default blocks" is enabled in the log settings.

Yes the option is activated, just checked it. Unfortunately even with the adapted wireguard tun0.conf it is not working.

From the error you've described, my suspicion was an asymmetric routing issue. If that you should see out-of-state blocks on the LAN.

Generally the need of NAT reflection should be avoided if possible. The way to go is adding host overrides for the public host names to the internal DNS and point them to the servers, in your case to traefik.
However, this would require that you can provide the local DNS server to the VPN client. I don't know, if this can be done in wireguard. It is possible in OpenVPN at all. So the client uses the remote DNS, when connected.

Also I'm wondering, why you don't run WG on pfSense itself. Maybe this would make less troubles.

But to get a step beyond with your set up, you could sniff the traffic, while you try to access the web services. You can do it with Diagnostic > Packet capture.
Select the LAN interface and state the port 443, then start the capture and try to access the server from the VPN client.

If all works as expected you should see the packets coming from the wg server to the WAN address. Then from the LAN IP to the traefik IP, and responses in the other direction back.

leonidas-o · Dec 13, 2022, 1:36 PM

@viragomann I think the gitlab runner sends a simple post http request with a body to the specified URL. And that's the point actually. I have the "public" url in there (https://mygitlab.mydomain.com). So the runner registers itself with a token at that url. But now, the gitlab instance and the runner is inside the lan, and the runner can't reach the gitlab instance.
This is the same scenario like when I connect to wireguard, so I'm talking via the wireguard VM from within the lan, therefore all my requests, like trying to open services behind traefik. Also Gitlab is one of these services behind traefik which I can't reach.
So this isn't a Wireguard issue, this is a pure pfsense configuration issue. Within the LAN network, none of my services, including myself via Wireguard, can't reach other services when using public urls like https://myservice.mydomain.com.

Not using WG inside pfsense because, I had this all setup and running without pfsense, wireguard and everything was already working. I had some issues with the proxmox redirects and decided then, okay, let's put a real firewall in between instead messing around with iptables commands.

Btw do I maybe need other redirect rules here in my Proxmox/debian network interfaces file (see initial post at top), or is it enough only for the main "enp0s31f6" interface?

I do have a dns server, so I could add dns entries but this feels a bit to static (which isn't that bad actually, I could do that), but saying: "pfsense activate reflection" and then everything happens magically and simply works, feels way easier", even though it isn't the recommended way. But whatever I do right now, reflection simply isn't working, and I fell like, I will never know why.

I added dns Rewrites. An issue with this approach, the traefik VM is reachable at port 4430. So when coming from outside, I can simply enter the url and will be forwarded correctly to 4430:

WAN	TCP	*	*	WAN address	443 (HTTPS)	10.1.1.32	4430	traefik

However, this is not working from within LAN. So I added the following:

LAN	TCP	*	*	10.1.1.32	443 (HTTPS)	10.1.1.32	4430	traefik LAN 443 to 4430

I would now expect, from within LAN, entering the url (https://myservice.com) would forward me to the traefik VM but port 4430. Which isn't happening. Only when I explicitly say "https://myservice.com:4430" I reach traefik and the correct service behind it.

leonidas-o · Dec 13, 2022, 5:07 PM

I also wonder as the pfsense is already behind a NAT (see initial post on top), and is creating a NAT itself, so NAT in NAT, if this could be the cause for my issues. Do I need a nat outbound rule maybe?

viragomann · Dec 13, 2022, 5:48 PM

@leonidas-o said in Port Forward within LAN via Wireguard VM:

I also wonder as the pfsense is already behind a NAT

Yeah, so the WAN address is assigned to Proxmox.
The requested packet capture would show this.

So NAT reflection on pfSense is quite useless at all. It must be done on Proxmox.

The DNS overrides would only make sense if you can provide the DNS server to the wireguard client, as mentioned above.

If that is not possible you can add the NAT rules manually. But instead of the pfSense WAN address you have to use the real public IP as destination.
Apart from the port forwarding you also need an outbound NAT rule for masquerading.

Enable the hybrid mode and add a rule like this:
interface: LAN
source: the VPN tunnel pool
destination: traefik IP
translation: interface address

Try out.

leonidas-o · Dec 13, 2022, 6:05 PM

@viragomann I made it! lord have mercy, this killed me. Thank you very much for taking your time and describing everything, I appreciate that good sir.

In the end, what worked here for me. I added another port forward rule, basically the same with only two changes. Destination contains the public IP of the main enp0s31f6 interface and reflection is enabled on that rule only.

WAN	TCP/UDP	*	*	PUBLIC_IP 	443 (HTTPS)	10.1.1.32	4430	traefik reflection
WAN	TCP/UDP	*	*	WAN address	443 (HTTPS)	10.1.1.32	4430	traefik

Also in the global settings I checked: "Automatic Outbound NAT for Reflection". Now I can access the services from within LAN, I can access them from a regular Internet connection and gitlab-runner immediately connected to the server. Everything seems to work now.
Now I'm slowly going to step back, and never ever touch that beast again

viragomann · Dec 13, 2022, 6:19 PM

@leonidas-o
I didn't consider before, that you public host name resolves to the Proxmox interface IP, but not to pfSense WAN.
As mentioned, the NAT reflection should be done on Proxmox, then you would need nothing to configure on pfSense.

But anyway you don't need NAT reflection on pfSense for this now. It's useless, since nothing points to its WAN IP.
And the port forwarding rule with the WAN IP is useless as well.

The other rules with the public IP and the outbound NAT rule are now your manually added NAT reflection rules.

Now I'm slowly going to step back, and never ever touch that beast again

pfSense let you backup the settings easily in Diagnostics > Backup & Restore > Backup & Restore,
And on the Config History tab you it stores automatically all settings states.

leonidas-o · Dec 13, 2022, 6:38 PM

But anyway you don't need NAT reflection on pfSense for this now. It's useless, since nothing points to its WAN IP.
And the port forwarding rule with the WAN IP is useless as well.

@viragomann no I need both, I tested it. As soon as I remove the reflection from the port forward, the service is not accessible from within LAN. If I deactivate the WAN port forward Rule, I can't access it from the internet. Maybe because of the first main forward "everything" to pfsense rule in proxmox's network interfaces file. So I will leave as it is for now. I'm just happy that it finally works.
Yes, got a scheduled job doing VM backups every day.