Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    WireGuard Asymetric NAT Issue when port forwarding from external server

    Scheduled Pinned Locked Moved
    WireGuard
    pfsense 2.6.0 wireguard nat routing
    1
    2
    1.5k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      flewid
      last edited by

      Hello,

      I've been fighting with this for over a week now and have now decided to reach out for some help.

      I have a remote server running at VULTR with Wireguard setup on it. At home behind Starlink (CGNAT) I have a PFSense router running, with a peer to Wireguard outside setup and functional.

      My goal is to forward some ports through Wireguard to the internal LAN so I can access things remotely. Specifically a helium miner for now. TCP port 44158 (forwarding to an internal server on 44158).

      What's working:

      • Tunnel / Peer configuration is working fine
      • I can ping PFSense, and LAN from Vultr Server
      • I can ping Vultr server from LAN and PFSense
      • I can telnet to the open port on the LAN side from Vultr server just fine.
      • I can ssh into the VULTR server just fine (aka: firewall is setup correctly)
      • Iphone peer connected to vultr works just fine

      What's not working:

      • Port forwarding from external IP on Vultr to PFsense and through to a device on the LAN.

      ISSUE:
      The issue is any packets come in through the eth0 interface on VULTR, get set CLOSED:SYN_SENT and there is no reply.

      From my reading, this indicates there's a Asymetric NAT issue coming from PFSense. I've tried a plethora of rule combinations on PFSense on every interface to try and resolve it, but nothing has seemed to work yet.

      I am pretty sure I am missing a "TCP TYPE: ALL, STATE: SLOPPY" rule somewhere per the documentation but I'm not sure where. And I possibly need a port forwarding rule, but unsure of this as telnet from vultr to LAN works just fine.

      Versions

      • PFSense 2.6.0-RELEASE
      • WireGuard 0.1.6_1

      Here are some pictures of my setup:

      ** Notes : Miner == 10.0.20.81

      DEFAULT IPTABLES RULES ON VULTR

      # Track VPN connection
sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Enable VPN traffic on the listening port: 51820
sudo iptables -A INPUT -p udp -m udp --dport 51820 -m conntrack --ctstate NEW -j ACCEPT

# TCP & UDP recursive DNS traffic
sudo iptables -A INPUT -s 10.90.0.0/24 -p tcp -m tcp --dport 53 -m conntrack --ctstate NEW -j ACCEPT
sudo iptables -A INPUT -s 10.90.0.0/24 -p udp -m udp --dport 53 -m conntrack --ctstate NEW -j ACCEPT

# Allow forwarding of packets that stay in the VPN tunnel
sudo iptables -A FORWARD -i wg0 -o wg0 -m conntrack --ctstate NEW -j ACCEPT

      WIREGUARD CONFIGURATION ON VULTR

      [Interface]
Address = 10.90.0.1/24
#MTU = 1420
MTU = 1280
ListenPort = 51820
PrivateKey = XXXXXXXXXXXXXX

# ADDING IN HELIUM
PostUp = iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu --set-mss 1280
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
PostUp = iptables -A FORWARD -i enp1s0 -o wg0 -p tcp --syn --dport 44158 -m conntrack --ctstate NEW -j ACCEPT
PostUp = iptables -A FORWARD -i enp1s0 -o wg0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostUp = iptables -A FORWARD -i wg0 -o enp1s0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostUp = iptables -t nat -A PREROUTING -i enp1s0 -p tcp --dport 44158 -j DNAT --to-destination 10.90.0.2
PostUp = iptables -t nat -A POSTROUTING -o wg0 -p tcp --dport 44158 -j SNAT --to-source 10.90.0.1


PostDown = iptables -D FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu --set-mss 1280
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
PostDown = iptables -D FORWARD -i enp1s0 -o wg0 -p tcp --syn --dport 44158 -m conntrack --ctstate NEW -j ACCEPT
PostDown = iptables -D FORWARD -i enp1s0 -o wg0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostDown = iptables -D FORWARD -i wg0 -o enp1s0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostDown = iptables -t nat -A PREROUTING -i enp1s0 -p tcp --dport 44158 -j DNAT --to-destination 10.90.0.2
PostDown = iptables -t nat -A POSTROUTING -o wg0 -p tcp --dport 44158 -j SNAT --to-source 10.90.0.1

#PFSense
[Peer]
PublicKey = XXXXXXXXXXXXXX
AllowedIPs = 10.90.0.2/32,10.0.20.0/24
#AllowedIPs = 10.90.0.2/32
#Endpoint = 0.0.0.0:51820

#Iphone
[Peer]
PublicKey = XXXXXXXXXXXXXX
AllowedIPs = 10.90.0.3/32

      WIREGUARD -> TUNNELS
      wireguard tunnels

      WIREGUARD -> STATUS
      wireguard status

      WIREGUARD -> SETTINGS
      wireguard settings

      WIREGUARD -> PEER
      wireguard peer

      WIREGUARD -> TUNNEL SETTINGS
      alt text

      INTERFACES
      alt text

      INTERFACE -> SETTINGS
      alt text

      GATEWAYS
      alt text

      GATEWAY -> SETTINGS
      alt text

      NAT -> PORT FORWARDING
      alt text

      NAT -> OUTBOUND
      alt text

      FIREWALL -> FLOATING
      alt text

      FIREWALL -> LAN
      alt text

      FIREWALL -> WAN
      alt text

      FIREWALL -> WG0
      alt text

      FIREWALL -> WIREGUARD
      alt text

      Here is some debugging information:

      LOG -> PFTOP
      alt text

      LOG -> STATES
      alt text

      TCPDUMP (TOP IS PFSENSE, BOTTOM IS VULTR)
      alt text

      TELNET FROM VULTR TO LAN HOST ON PORT 44158

      root@wg:/etc/wireguard# telnet 10.0.20.81 44158
Trying 10.0.20.81...
Connected to 10.0.20.81.
Escape character is '^]'.
/multistream/1.0.0
^C
Connection closed by foreign host.

      TCPDUMP - TELNET FROM EXTERNAL HOST, TO VULTR PORT 44158

      19:39:27.076320 IP (tos 0x0, ttl 47, id 36001, offset 0, flags [DF], proto TCP (6), length 60)
    10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0xa1ab (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105421334 ecr 0,nop,wscale 7], length 0
19:39:28.076284 IP (tos 0x0, ttl 47, id 36002, offset 0, flags [DF], proto TCP (6), length 60)
    10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x9dc0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105422337 ecr 0,nop,wscale 7], length 0
19:39:30.100472 IP (tos 0x0, ttl 47, id 36003, offset 0, flags [DF], proto TCP (6), length 60)
    10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x95e0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105424353 ecr 0,nop,wscale 7], length 0
19:39:34.220504 IP (tos 0x0, ttl 47, id 36004, offset 0, flags [DF], proto TCP (6), length 60)
    10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x85c0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105428481 ecr 0,nop,wscale 7], length 0

      and

      
00:39:27.044678 IP (tos 0x0, ttl 47, id 36001, offset 0, flags [DF], proto TCP (6), length 60)
    10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0xa1ab (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105421334 ecr 0,nop,wscale 7], length 0
00:39:28.047517 IP (tos 0x0, ttl 47, id 36002, offset 0, flags [DF], proto TCP (6), length 60)
    10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x9dc0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105422337 ecr 0,nop,wscale 7], length 0
00:39:30.063467 IP (tos 0x0, ttl 47, id 36003, offset 0, flags [DF], proto TCP (6), length 60)
    10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x95e0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105424353 ecr 0,nop,wscale 7], length 0
00:39:34.191522 IP (tos 0x0, ttl 47, id 36004, offset 0, flags [DF], proto TCP (6), length 60)
    10.90.0.1.41496 > 10.90.0.2.44158: Flags [S], cksum 0x85c0 (correct), seq 2068783590, win 64240, options [mss 1280,sackOK,TS val 2105428481 ecr 0,nop,wscale 7], length 0

      ROUTE TABLE ON WIREGUARD SERVER

      root@wg:/etc/wireguard# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         XXX.XXX.XXX.XXX 0.0.0.0         UG    100    0        0 enp1s0
10.0.20.0       0.0.0.0         255.255.255.0   U     0      0        0 wg0
10.90.0.0       0.0.0.0         255.255.255.0   U     0      0        0 wg0
XXX.XXX.XXX.254 XXX.XXX.XXX.XXX 255.255.255.255 UGH   100    0        0 enp1s0
XXX.XXX.XXX.0   0.0.0.0         255.255.254.0   U     0      0        0 enp1s0

      Thank you in advance for any help you can provide.

      F 1 Reply Last reply Reply Quote 0
      • F
        flewid @flewid
        last edited by

        For anyone else finding this thread. I've found the solution.

        Create a port forwarding rule

        INTERFACE: WG0
        PORT: 44158
        DESTINATION: WG0
        DEST PORT: 44158
        REDIRECT TARGET IP: MINER IP
        REDIRECT PORT: 44158

        Then everything works as expected.

        1 Reply Last reply Reply Quote 0
        • First post
          Last post