WireGuard Asymetric NAT Issue when port forwarding from external server
-
Hello,
I've been fighting with this for over a week now and have now decided to reach out for some help.
I have a remote server running at VULTR with Wireguard setup on it. At home behind Starlink (CGNAT) I have a PFSense router running, with a peer to Wireguard outside setup and functional.
My goal is to forward some ports through Wireguard to the internal LAN so I can access things remotely. Specifically a helium miner for now. TCP port 44158 (forwarding to an internal server on 44158).
What's working:
- Tunnel / Peer configuration is working fine
- I can ping PFSense, and LAN from Vultr Server
- I can ping Vultr server from LAN and PFSense
- I can telnet to the open port on the LAN side from Vultr server just fine.
- I can ssh into the VULTR server just fine (aka: firewall is setup correctly)
- Iphone peer connected to vultr works just fine
What's not working:
- Port forwarding from external IP on Vultr to PFsense and through to a device on the LAN.
ISSUE:
The issue is any packets come in through the eth0 interface on VULTR, get set CLOSED:SYN_SENT and there is no reply.From my reading, this indicates there's a Asymetric NAT issue coming from PFSense. I've tried a plethora of rule combinations on PFSense on every interface to try and resolve it, but nothing has seemed to work yet.
I am pretty sure I am missing a "TCP TYPE: ALL, STATE: SLOPPY" rule somewhere per the documentation but I'm not sure where. And I possibly need a port forwarding rule, but unsure of this as telnet from vultr to LAN works just fine.
Versions
- PFSense 2.6.0-RELEASE
- WireGuard 0.1.6_1
Here are some pictures of my setup:
** Notes : Miner == 10.0.20.81
DEFAULT IPTABLES RULES ON VULTR
# Track VPN connection sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT # Enable VPN traffic on the listening port: 51820 sudo iptables -A INPUT -p udp -m udp --dport 51820 -m conntrack --ctstate NEW -j ACCEPT # TCP & UDP recursive DNS traffic sudo iptables -A INPUT -s 10.90.0.0/24 -p tcp -m tcp --dport 53 -m conntrack --ctstate NEW -j ACCEPT sudo iptables -A INPUT -s 10.90.0.0/24 -p udp -m udp --dport 53 -m conntrack --ctstate NEW -j ACCEPT # Allow forwarding of packets that stay in the VPN tunnel sudo iptables -A FORWARD -i wg0 -o wg0 -m conntrack --ctstate NEW -j ACCEPTWIREGUARD CONFIGURATION ON VULTR
[Interface] Address = 10.90.0.1/24 #MTU = 1420 MTU = 1280 ListenPort = 51820 PrivateKey = XXXXXXXXXXXXXX # ADDING IN HELIUM PostUp = iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu --set-mss 1280 PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE PostUp = iptables -A FORWARD -i enp1s0 -o wg0 -p tcp --syn --dport 44158 -m conntrack --ctstate NEW -j ACCEPT PostUp = iptables -A FORWARD -i enp1s0 -o wg0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT PostUp = iptables -A FORWARD -i wg0 -o enp1s0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT PostUp = iptables -t nat -A PREROUTING -i enp1s0 -p tcp --dport 44158 -j DNAT --to-destination 10.90.0.2 PostUp = iptables -t nat -A POSTROUTING -o wg0 -p tcp --dport 44158 -j SNAT --to-source 10.90.0.1 PostDown = iptables -D FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu --set-mss 1280 PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE PostDown = iptables -D FORWARD -i enp1s0 -o wg0 -p tcp --syn --dport 44158 -m conntrack --ctstate NEW -j ACCEPT PostDown = iptables -D FORWARD -i enp1s0 -o wg0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT PostDown = iptables -D FORWARD -i wg0 -o enp1s0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT PostDown = iptables -t nat -A PREROUTING -i enp1s0 -p tcp --dport 44158 -j DNAT --to-destination 10.90.0.2 PostDown = iptables -t nat -A POSTROUTING -o wg0 -p tcp --dport 44158 -j SNAT --to-source 10.90.0.1 #PFSense [Peer] PublicKey = XXXXXXXXXXXXXX AllowedIPs = 10.90.0.2/32,10.0.20.0/24 #AllowedIPs = 10.90.0.2/32 #Endpoint = 0.0.0.0:51820 #Iphone [Peer] PublicKey = XXXXXXXXXXXXXX AllowedIPs = 10.90.0.3/32WIREGUARD -> TUNNELS
WIREGUARD -> STATUS
WIREGUARD -> SETTINGS
WIREGUARD -> PEER
WIREGUARD -> TUNNEL SETTINGS
INTERFACES
INTERFACE -> SETTINGS
GATEWAYS
GATEWAY -> SETTINGS
NAT -> PORT FORWARDING
NAT -> OUTBOUND
FIREWALL -> FLOATING
FIREWALL -> LAN
FIREWALL -> WAN
FIREWALL -> WG0
FIREWALL -> WIREGUARD
Here is some debugging information:
LOG -> PFTOP
LOG -> STATES
TCPDUMP (TOP IS PFSENSE, BOTTOM IS VULTR)
TELNET FROM VULTR TO LAN HOST ON PORT 44158
root@wg:/etc/wireguard# telnet 10.0.20.81 44158 Trying 10.0.20.81... Connected to 10.0.20.81. Escape character is '^]'. /multistream/1.0.0 ^C Connection closed by foreign host.TCPDUMP - TELNET FROM EXTERNAL HOST, TO VULTR PORT 44158
19:39:27.076320 IP (tos 0x0, ttl 47, id 36001, offset 0, flags [DF], proto TCP (6), length 60) 10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0xa1ab (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105421334 ecr 0,nop,wscale 7], length 0 19:39:28.076284 IP (tos 0x0, ttl 47, id 36002, offset 0, flags [DF], proto TCP (6), length 60) 10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x9dc0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105422337 ecr 0,nop,wscale 7], length 0 19:39:30.100472 IP (tos 0x0, ttl 47, id 36003, offset 0, flags [DF], proto TCP (6), length 60) 10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x95e0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105424353 ecr 0,nop,wscale 7], length 0 19:39:34.220504 IP (tos 0x0, ttl 47, id 36004, offset 0, flags [DF], proto TCP (6), length 60) 10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x85c0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105428481 ecr 0,nop,wscale 7], length 0and
00:39:27.044678 IP (tos 0x0, ttl 47, id 36001, offset 0, flags [DF], proto TCP (6), length 60) 10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0xa1ab (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105421334 ecr 0,nop,wscale 7], length 0 00:39:28.047517 IP (tos 0x0, ttl 47, id 36002, offset 0, flags [DF], proto TCP (6), length 60) 10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x9dc0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105422337 ecr 0,nop,wscale 7], length 0 00:39:30.063467 IP (tos 0x0, ttl 47, id 36003, offset 0, flags [DF], proto TCP (6), length 60) 10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x95e0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105424353 ecr 0,nop,wscale 7], length 0 00:39:34.191522 IP (tos 0x0, ttl 47, id 36004, offset 0, flags [DF], proto TCP (6), length 60) 10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x85c0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105428481 ecr 0,nop,wscale 7], length 0ROUTE TABLE ON WIREGUARD SERVER
root@wg:/etc/wireguard# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default XXX.XXX.XXX.XXX 0.0.0.0 UG 100 0 0 enp1s0 10.0.20.0 0.0.0.0 255.255.255.0 U 0 0 0 wg0 10.90.0.0 0.0.0.0 255.255.255.0 U 0 0 0 wg0 XXX.XXX.XXX.254 XXX.XXX.XXX.XXX 255.255.255.255 UGH 100 0 0 enp1s0 XXX.XXX.XXX.0 0.0.0.0 255.255.254.0 U 0 0 0 enp1s0Thank you in advance for any help you can provide.
-
For anyone else finding this thread. I've found the solution.
Create a port forwarding rule
INTERFACE: WG0
PORT: 44158
DESTINATION: WG0
DEST PORT: 44158
REDIRECT TARGET IP: MINER IP
REDIRECT PORT: 44158Then everything works as expected.