Wireguard Routing Problems - Help wanted

gelcom · Aug 19, 2022, 2:02 PM

Hi all, any help is appreciated...

My local ISP blocks inbound connections on ports 80/443 and I need to host a few personnal websites locally.

To be able to do this I got a VPS account with ports 80/443 open at Oracle Cloud and I setup a WG tunnel between this VPS and local pfSense.

My VPS is very low on resources so running webservers or any other services in there is out of question.

Local pfSense routes incoming request to my webservers through HAProxy package.

pfSense is WG "server" (10.168.16.1) and VPS is WG "peer" (10.168.16.2)

pfSense WG server was setup and assigned to an interface as follows:

I have an "allow all" rule on pfSense WG interface:

At VPS side I have the following setup:

cat /etc/wireguard/wg0.conf

Client configuration
[Interface]
Address = 10.168.16.2/24 # private IP address of the VPN client.
DNS = 10.168.16.1
PrivateKey = hidden # The client_private.key value.

#Make sure packet forwarding is enabled
PreUp = sysctl -w net.ipv4.ip_forward=1

#Here we set the default rule for the FORWARD chain to DROP so that nobody can forward packets.
#Then we enable forwarding for packets coming from enp0s3 with a destination ports 80,443
PostUp = iptables -P FORWARD DROP
PostUp = iptables -I FORWARD -i enp0s3 -o %i -p tcp --syn -m multiport --dports 80,443 -m conntrack --ctstate NEW -j ACCEPT
PostUp = iptables -I FORWARD -i enp0s3 -o %i -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostUp = iptables -I FORWARD -i %i -o enp0s3 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
#Revert all when VPN is down
PreDown = iptables -D FORWARD -i enp0s3 -o %i -p tcp --syn -m multiport --dports 80,443 -m conntrack --ctstate NEW -j ACCEPT
PreDown = iptables -D FORWARD -i enp0s3 -o %i -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PreDown = iptables -D FORWARD -i %i -o enp0s3 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

#Next, we need to add a rule that changes the destination address in the TCP packet to 10.168.16.1 (the address of the other side of the tunnel)
#This rule only applies to packets coming from enp0s3 and with a destination ports 80,443
#The second rule changes the source address so that the remote VPN endpoint can send back the response to our server.
#When second rule is disabled it does not work anymore
PostUp = iptables -t nat -I PREROUTING -i enp0s3 -p tcp -m multiport --dports 80,443 -j DNAT --to-destination 10.168.16.1
#PostUp = iptables -t nat -I POSTROUTING -o %i -p tcp -m multiport --dports 80,443 -d 10.168.16.1 -j SNAT --to-source 10.168.16.2
#Revert all when VPN is shut down
PreDown = iptables -t nat -D PREROUTING -i enp0s3 -p tcp -m multiport --dports 80,443 -j DNAT --to-destination 10.168.16.1
#PreDown = iptables -t nat -D POSTROUTING -o %i -p tcp -m multiport --dports 80,443 -d 10.168.16.1 -j SNAT --to-source 10.168.16.2

[Peer]
PublicKey = hidden # The server_public.key value.
AllowedIPs = 10.168.16.1/32
Endpoint = vpn1.mypfsense.com:51821 # Public IP address of our VPN server and port number
PersistentKeepalive = 15

ip addr show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 02:00:17:02:29:4c brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.199/24 brd 10.0.0.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet6 fe80::17ff:fe02:294c/64 scope link 
       valid_lft forever preferred_lft forever
14: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8920 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.168.16.2/24 scope global wg0
       valid_lft forever preferred_lft forever

ip -4 route show table all

default via 10.0.0.1 dev enp0s3 proto dhcp src 10.0.0.199 metric 100 
10.0.0.0/24 dev enp0s3 proto kernel scope link src 10.0.0.199 
10.168.16.0/24 dev wg0 proto kernel scope link src 10.168.16.2 
169.254.0.0/16 dev enp0s3 proto dhcp scope link src 10.0.0.199 metric 100 
local 10.0.0.199 dev enp0s3 table local proto kernel scope host src 10.0.0.199 
broadcast 10.0.0.255 dev enp0s3 table local proto kernel scope link src 10.0.0.199 
local 10.168.16.2 dev wg0 table local proto kernel scope host src 10.168.16.2 
broadcast 10.168.16.255 dev wg0 table local proto kernel scope link src 10.168.16.2 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1

ip -4 rule show

0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

ip -6 route show table all
::1 dev lo proto kernel metric 256 pref medium

fe80::/64 dev enp0s3 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::17ff:fe02:294c dev enp0s3 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev enp0s3 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wg0 table local proto kernel metric 256 pref medium

ip -6 rule show

0:	from all lookup local
32766:	from all lookup main

wg

interface: wg0
  public key: (hidden)
  private key: (hidden)
  listening port: 60199

peer: (hidden)
  endpoint: (hidden):51821
  allowed ips: 10.168.16.1/32
  latest handshake: 22 seconds ago
  transfer: 62.17 KiB received, 69.86 KiB sent
  persistent keepalive: every 15 seconds

ip netconf

inet lo forwarding on rp_filter off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
inet enp0s3 forwarding on rp_filter off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
inet wg0 forwarding on rp_filter loose mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
inet all forwarding on rp_filter loose mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
inet default forwarding on rp_filter loose mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
inet6 lo forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
inet6 enp0s3 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
inet6 wg0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
inet6 all forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off 
inet6 default forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off

iptables-save -c

# Generated by iptables-save v1.8.4 on Fri Aug 19 13:31:42 2022
*filter
:INPUT ACCEPT [2953:228156]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [2926:244671]
[0:0] -A FORWARD -i wg0 -o enp0s3 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[0:0] -A FORWARD -i enp0s3 -o wg0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[80:5096] -A FORWARD -i enp0s3 -o wg0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m multiport --dports 80,443 -m conntrack --ctstate NEW -j ACCEPT
COMMIT
# Completed on Fri Aug 19 13:31:42 2022
# Generated by iptables-save v1.8.4 on Fri Aug 19 13:31:42 2022
*nat
:PREROUTING ACCEPT [12:716]
:INPUT ACCEPT [12:716]
:OUTPUT ACCEPT [5:284]
:POSTROUTING ACCEPT [20:1220]
[15:936] -A PREROUTING -i enp0s3 -p tcp -m multiport --dports 80,443 -j DNAT --to-destination 10.168.16.1
COMMIT
# Completed on Fri Aug 19 13:31:42 2022

iptables -t nat -nvL

Chain PREROUTING (policy ACCEPT 23 packets, 1300 bytes)
 pkts bytes target     prot opt in     out     source               destination
   20  1136 DNAT       tcp  --  enp0s3 *       0.0.0.0/0            0.0.0.0/0            multiport dports 80,443 to:10.168.16.1

Chain INPUT (policy ACCEPT 23 packets, 1300 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 77 packets, 5882 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 97 packets, 7018 bytes)
 pkts bytes target     prot opt in     out     source               destination

Using iptables rules on this VPS WG "peer" I forward all incoming connections on VPS's ports 80/443 to my local pfSense server running HAProxy so HAProxy can send the request to correct local webserver.

It's working: I can access both my websites from internet.

Problem is all inbound connections I see at pfSense shows VPS WG peer IP (10.168.16.2) as origin IP.

I need to implement GeoIP blocking on my webserver and so I need "source IP" to be shown as real IP and not NATed IP from WG peer.

If I comment out and disable "PostUp = iptables -t nat -I POSTROUTING -o %i -p tcp -m multiport --dports 80,443 -d 10.168.16.1 -j SNAT --to-source 10.168.16.2" from VPS WG iptables setup I start to see real IPs hitting my local pfSense (good) BUT when I do this my website stops showing at remote location (bad). It breaks. See above:

Attached I have 2 packet captures:
This one with the SNAT rule and all working but with NATed ip
This one without the SNAT rule, pfSense receiving real IPs but with broken setup. When I open this file with Wireshark I see a few lines marked in black but I don't understand how to "debug" that.

What am I doing wrong?

kind regards

stephenw10 · Aug 19, 2022, 6:41 PM

I assume the default route at the WG server end is via the WAN, not via the WG peer?

Do you have the WG gateway actually configured on the WG interface?
Without that you will not get the required reply-to tags on connections coming over the tunnel. And without those the replies are sent using the system routing table which probably means via the WAN.
Try running a pcap on the WAN. If you see the reply traffic from the server leaving there that's almost certainly the problem.

Steve

gelcom · Aug 19, 2022, 6:59 PM

@stephenw10 said in Wireguard Routing Problems - Help wanted:

I assume the default route at the WG server end is via the WAN, not via the WG peer?

Do you have the WG gateway actually configured on the WG interface?
Without that you will not get the required reply-to tags on connections coming over the tunnel. And without those the replies are sent using the system routing table which probably means via the WAN.

Default pfSense gateway is WAN but WG interface upstream gateway is VPS WG "server" IP:

Try running a pcap on the WAN. If you see the reply traffic from the server leaving there that's almost certainly the problem.
Steve

This is the pcap on WAN. I have no idea on how to debug that...

kind regards

stephenw10 · Aug 19, 2022, 8:28 PM

What is the external IP you were testing from? That's what you need to look for in that pcap. It should not be there.
Looks like it might have been 187.90.193.28? We see only SYN-ACKs in that pcap and from the WG tunnel IP which I assume HAProxy is listening on.

Check the ruleset in /tmp/rules.debug. Does the rule on the WG interface include the reply-to tags?

Steve

gelcom · Aug 19, 2022, 10:10 PM

@stephenw10 said in Wireguard Routing Problems - Help wanted:

What is the external IP you were testing from? That's what you need to look for in that pcap. It should not be there.

I didn't know I had to provide that. sorry...

I captured again. Now I know it´s 189.98.253.3:

I see some registers on wireshark with this ip address as destination. New pcap is here.

A few of these registers with this IP are marked in black on wireshark as "TCP retransmission".

@stephenw10 said in Wireguard Routing Problems - Help wanted:

Check the ruleset in /tmp/rules.debug. Does the rule on the WG interface include the reply-to tags?

No reply-tag in there. The only rule related to this WG interface is this one:

pass  in log  quick  on $VPN_SP_WG inet from any to any ridentifier 1660912179 keep state  label "USER_RULE: VPN_SP_WG - Allow All"

I see a few rules with "reply-to" but none of them related to Wireguard interface. A few related with WAN interface and another related to a OpenVPN interface I also have:

pass  in  quick  on $WAN reply-to ( vtnet0 192.168.15.1 ) inet proto udp  from any to 192.168.15.10 port 9700 ridentifier 1648051723 keep state  label "USER_RULE: Entrada VPN_IPTV"
pass  in  quick  on $WAN reply-to ( vtnet0 192.168.15.1 ) inet proto udp  from any to 192.168.15.10 port 9800 ridentifier 1648051756 keep state  label "USER_RULE: Entrada VPN_SP1_IN"
pass  in  quick  on $WAN reply-to ( vtnet0 192.168.15.1 ) inet proto udp  from any to 192.168.15.10 port 9900 ridentifier 1648051777 keep state  label "USER_RULE: Entrada VPN_LIVRE_IN"
pass  in  quick  on $WAN reply-to ( vtnet0 192.168.15.1 ) inet proto udp  from any to 192.168.15.10 port 51820 ridentifier 1651061012 keep state  label "USER_RULE: Entrada VPN_SCS_IN"
pass  in  quick  on $WAN reply-to ( vtnet0 192.168.15.1 ) inet proto udp  from any to 192.168.15.10 port 51821 ridentifier 1659534223 keep state  label "USER_RULE: Entrada VPN_SP1_BOTH"
pass  in log  quick  on $VPN_SP1_IN reply-to ( ovpns2 10.168.11.2 ) inet from any to any ridentifier 1659617774 keep state  label "USER_RULE: VPN_SP1_IN - Allow All ALL"

It's the missing reply-tag that causes the issue? If so how can I fix that?

kind regards

stephenw10 · Aug 19, 2022, 11:16 PM

Yes, you need the reply-to tags on the states created by the rule so the replies go back over the tunnel rather than by the default route.
Hmm, I expect it to add those on that rule since it has a gateway on the interface. I see that here on a test box:

pass  in  quick  on $WG0 reply-to ( tun_wg0 172.27.116.1 ) inet from any to any ridentifier 1660950022 keep state label "USER_RULE: Allow all" label "id:1660950022"

Does the gateway show as up?

Do you see the outbound route-to rule for the Wireguard interface?:

pass out  route-to ( tun_wg0 172.27.116.1 ) from 172.27.116.16 to !172.27.116.0/24 ridentifier 1000020513 keep state allow-opts label "let out anything from firewall host itself"

It's possible to disable reply-to on the pass rule but that would show as an advanced setting in your screenshot.

What pfSense version are you running? Which WG package version?

Steve

gelcom · Aug 19, 2022, 11:50 PM

@stephenw10 said in Wireguard Routing Problems - Help wanted:

Does the gateway show as up?

Yes...

Do you see the outbound route-to rule for the Wireguard interface?:

no... I have a few route-to rules but none of them related to WG interface

It's possible to disable reply-to on the pass rule but that would show as an advanced setting in your screenshot.

Indeed, my pass all rule is plain basic. No advanced options at all.

What pfSense version are you running? Which WG package version?

pfSense version 2.6.0-Release
Wireguard version 0.1.6_2

stephenw10 · Aug 20, 2022, 12:25 AM

Hmm, creates the tags for me just fine in 2.6 too.

You must have something else set/unset...

stephenw10 · Aug 20, 2022, 12:33 AM

Do you see the tun_wg1 gateway defined in the ruleset at all?

Did you maybe add it before the interface and it somehow ended up on the wrong interface? Hard to see how that would be possible...

What you are doing should work.

Steve

gelcom · Aug 20, 2022, 10:21 PM

@stephenw10 I deleted the WireGuard tunnel then I set it up all over again. Done the same thing at VPS. Rebooted remote VM and pfSense and it started working.

I have no idea what happened before but I thanks you for all the support you provided!!

Thanks a lot

:-)

kind regards