OpenVPN + WireGuard breaking DNS resolver. [SOLVED]

nimrod · Aug 2, 2022, 8:18 PM

So here is the new issue i discovered.

Im using MullvadVPN. I established WireGuard connection successfully by following these simple instructions. Im routing all the traffic through VPN. I have 3 subnets for various devices. They are:

Local subnet: 192.168.1.1/24
WiFi devices: 10.1.1.1/24
Cameras: 10.10.50.1/24

This setup has been working flawlessly for months now.

Few days ago, i created a OpenVPN connection while WireGuard tunel is still up and running. To setup OpenVPN i used MullvadVPN instructions. I know they are old, but they are working fine. In no time, OpenVPN client connected successfully, i created another interface, i updated NAT rules, and i created a new firewall rules to route all traffic on local subnet through OpenVPN tunel instead of WireGuard tunel.

So now it looks like this.

Local subnet: 192.168.1.1/24 - Going to OpenVPN
WiFi devices: 10.1.1.1/24 - Going to WireGuard
Cameras: 10.10.50.1/24 - Going to WireGuard

Amazingly, this was working perfectly fine for a few days, and then i had to reboot.

When i rebooted, i noticed that cameras are communicating, as well as my instant messaging software, however, DNS resolving stopped. Just to be clear, DNS resolver service is up and running. No issues there. However, not only that there was no DNS resolution on the subnet going through OpenVPN, there was no DNS resolution on subnets going through WireGuard either. To resolve this issue, i go to System > General Setup and i just click save button without changing anything. And DNS resolution starts working on both OpenVPN and WireGuard interfaces/subnets until next reboot.

Going to Services > DNS Resolver, changing nothing, and then clicking Save and Apply also temporary resolves this issue. I tried every DNS related setting and nothing helps. As soon as i reboot, DNS resolution is dead. If i remove OpenVPN interface, and disable OpenVPN client. Issue is completely gone. I can reboot pfSense, and all clients connect as soon as the VPN tunel is up. I also tried removing WireGuard completely, but it didnt help.

I was using ExpressVPN before, and i didnt had this issue. I assume its because ExpressVPN is providing DNS resolution through OpenVPN, and MullvadVPN does not. Mullvad provides two DNS IP`s that you must assign by putting them into the DHCP server DNS fields. Of course, this is not an option for me, because doing this gets around pfBlockerNG filtering. So i have to go to System > General and put those DNS IPs there and enable DNS Query Forwarding in Services > DNS Resolver.

So far i tried this with no success.

Restarting Unbound service
Putting a timer on Unbound service when pfSense boots up
Turning off DNS Query Forwarding
Port forwarding traffic on port 53 to all subnets.
Automatic, hybrid and manual NAT
Clean install of pfSense and starting from scratch
Uninstalling pfBlocker
Restaring WireGuard and OpenVPN services when system is up.
Applied all available patches by using System Patches package.

Only thing that works is:

Go to System > General and hit save button without changing anything.
Go to Services > DNS Resolver > hit Save and then Apply button without changing anything.
Completely removing OpenVPN interface and OpenVPN client.

These of course are temporary solutions that work until pfSense is rebooted.

Please help.

stephenw10 · Jul 22, 2022, 6:36 PM

So you are adding only 193.138.218.74 and 10.8.0.1 as DNS servers in System > General Setup and setting Unbound in Forwarding mode?
Have you set those as being on the OpenVPN gateway?

Unbound is probably trying to reach them before the VPN is established resulting in bad states on WAN or over Wireguard. You will need to see exactly what's happening in the failure situation and probably add rules to prevent that.

Steve

nimrod · Jul 22, 2022, 9:03 PM

@stephenw10 said in OpenVPN + WireGuard breaking DNS resolver.:

So you are adding only 193.138.218.74 and 10.8.0.1 as DNS servers in System > General Setup and setting Unbound in Forwarding mode?

Exactly.

And i also tried using just one of those IPs as well.

Have you set those as being on the OpenVPN gateway?

If we are talking about optional Gateway setting in System / General Setup, then yes. I tried using WireGuard as a gateway, rebooted, nothing. Then i set gateway to OpenVPN, rebooted, nothing. I did the same for both DNS IPs.

Unbound is probably trying to reach them before the VPN is established resulting in bad states on WAN or over Wireguard.

Just to be clear. I did clean install of pfSense without WireGuard and same thing happens. Then, i removed OpenVPN interface and disabled OpenVPN client completely. Then, i installed WireGuard package, set it up, rebooted pfSense box, and it all works flawlessly.

You will need to see exactly what's happening in the failure situation and probably add rules to prevent that.

Steve

Failure starts after reboot. Should i post logs from Status / System Logs / System / DNS Resolver section right after reboot ?

Thanks Steve.

stephenw10 · Jul 22, 2022, 9:14 PM

What I would do is look at the firewall states for outbound DNS connections in the failed condition and see which interfaces they are on. You can probably only connect to those DNS servers over the VPN so any connections that are leaving the WAN directly will fail.

nimrod · Jul 22, 2022, 11:25 PM

@stephenw10 said in OpenVPN + WireGuard breaking DNS resolver.:

What I would do is look at the firewall states for outbound DNS connections in the failed condition and see which interfaces they are on. You can probably only connect to those DNS servers over the VPN so any connections that are leaving the WAN directly will fail.

@stephenw10 you were absolutely right. Here is the part of the log.

WAN	icmp	192.168.5.2:23885 -> 192.168.5.1:23885	0:0
WAN	udp	192.168.5.2:123 -> 194.58.205.148:123	MULTIPLE:MULTIPLE
WAN	udp	192.168.5.2:123 -> 194.58.203.148:123	MULTIPLE:MULTIPLE
lo0	icmp	10.66.205.79:24517 -> 10.66.205.79:24517	0:0
lo0	icmp	10.16.0.2:24079 -> 193.138.218.74:24079	0:0
MULLVADVPN_WG	udp	10.66.205.79:20140 (10.10.50.3:42613) -> 188.120.127.79:443	MULTIPLE:MULTIPLE
MULLVADVPN_WG	udp	10.66.205.79:3124 (10.10.50.3:49582) -> 188.120.127.142:443	MULTIPLE:MULTIPLE
MULLVADVPN_WG	tcp	10.66.205.79:12740 (10.10.50.3:44841) -> 142.250.74.78:443	ESTABLISHED:ESTABLISHED
WAN	udp	192.168.5.2:31671 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	udp	192.168.5.2:29132 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	udp	192.168.5.2:58525 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	udp	192.168.5.2:33339 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	udp	192.168.5.2:42101 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	udp	192.168.5.2:64598 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	udp	192.168.5.2:33082 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	tcp	192.168.5.2:56200 -> 193.138.218.74:53	FIN_WAIT_2:FIN_WAIT_2
WAN	udp	192.168.5.2:29890 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	udp	192.168.5.2:13777 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	tcp	192.168.5.2:49574 -> 193.138.218.74:53	ESTABLISHED:ESTABLISHED
WAN	udp	192.168.5.2:28760 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	udp	192.168.5.2:8692 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	tcp	192.168.5.2:15643 -> 193.138.218.74:53	ESTABLISHED:ESTABLISHED
WAN	udp	192.168.5.2:8340 -> 193.138.218.74:53	MULTIPLE:SINGLE
WAN	tcp	192.168.5.2:16416 -> 193.138.218.74:53	ESTABLISHED:ESTABLISHED
WAN	udp	192.168.5.2:123 -> 193.182.111.141:123	MULTIPLE:SINGLE
MULLVADVPN_WG	tcp	10.66.205.79:55823 (10.10.50.3:46476) -> 188.120.127.78:443	ESTABLISHED:ESTABLISHED
MULLVADVPN_WG	tcp	10.66.205.79:39780 (10.10.50.3:44047) -> 142.250.74.67:80	ESTABLISHED:ESTABLISHED
MULLVADVPN_WG	tcp	10.66.205.79:23684 (10.10.50.3:42393) -> 142.250.74.164:443	ESTABLISHED:ESTABLISHED
MULLVADVPN_WG	udp	10.66.205.79:53224 (10.10.50.3:46297) -> 216.58.207.214:443	MULTIPLE:MULTIPLE
MULLVADVPN_WG	udp	10.66.205.79:24477 (10.10.50.3:49690) -> 188.120.127.76:443	MULTIPLE:MULTIPLE
MULLVADVPN_WG	udp	10.66.205.79:10898 (10.10.50.3:46000) -> 142.250.74.170:443	MULTIPLE:MULTIPLE
MULLVADVPN_WG	udp	10.66.205.79:34243 (10.10.50.3:44996) -> 142.250.74.164:443	MULTIPLE:MULTIPLE
MULLVADVPN_WG	udp	10.66.205.79:7771 (10.10.50.3:45347) -> 216.58.207.206:443	MULTIPLE:MULTIPLE
WAN	udp	192.168.5.2:58284 -> 193.138.218.74:53	MULTIPLE:SINGLE

I can clearly see that IP address 193.138.218.74, which is MullvadVPN DNS, is attempting to establish connection on 192.168.5.2 which is my WAN address. After i go to System > General and hit save button, those requests redirected to to MULLVADVPN_WG and MULLVADOVPN interfaces. And thats when it all starts working correctly.

Do i need to add

sleep xx

Where xx is the number of seconds in rc.bootup file, or theres more elegant way to solve this ?

Thanks.

stephenw10 · Jul 22, 2022, 11:33 PM

I would try adding a floating outbound block rule on WAN to match that and prevent any states opening on WAN.

nimrod · Jul 23, 2022, 12:30 AM

@stephenw10 said in OpenVPN + WireGuard breaking DNS resolver.:

I would try adding a floating outbound block rule on WAN to match that and prevent any states opening on WAN.

I tried but it didnt work. BlockWAN is the alias containing both MullvadVPN DNS IPs. I tried block and reject, and i enabled Quick option.

After reboot, state table still shows DNS requests made on port 53 on on my WAN IP 192.168.5.2. It seems like these DNS requests are taking place before firewall rules have the chance to block them.

Again, once the system is fully up, i just click save in System / General and everything starts working.

stephenw10 · Jul 23, 2022, 6:04 PM

The source port should be 'any' there not 53. If the look at the states created the source port is a random high numbered port.

Steve

nimrod · Jul 24, 2022, 12:44 AM

@stephenw10 said in OpenVPN + WireGuard breaking DNS resolver.:

The source port should be 'any' there not 53. If the look at the states created the source port is a random high numbered port.

Steve

I fixed that, but this rule blocks DNS resolution completely.

I tried setting port to "any" in destination, but same thing happens. Going to System / General and hitting save no longer works if this rule is enabled.

I experimented some more, and i made a tiny progress. If i set my DNS Server Settings like this

and then i select Mullvad OpenVPN interface as gateway in firewall rules, everything works fine after reboot. However, roles have been switched now. And all clients that use WireGuard gateway dont have DNS resolution or any traffic for that matter. WireGuard applet now shows 0 peers.

There is also no handshake

And going to System / General doesnt help like before.

stephenw10 · Jul 24, 2022, 3:49 PM

@nimrod said in OpenVPN + WireGuard breaking DNS resolver.:

and then i select Mullvad OpenVPN interface as gateway in firewall rules, everything works fine after reboot.

In what firewall rules? Traffic from the firewall itself cannot be policy routed. It will always use the system routing table which is why I was asking you about the static route to the DNS server added by setting a gateway against them.

Steve

nimrod · Jul 24, 2022, 3:49 PM

@stephenw10 said in OpenVPN + WireGuard breaking DNS resolver.:

@nimrod said in OpenVPN + WireGuard breaking DNS resolver.:

and then i select Mullvad OpenVPN interface as gateway in firewall rules, everything works fine after reboot.

In what firewall rules? Traffic from the firewall itself cannot be policy routed. It will always use the system routing table which is why I was asking you about the static route to the DNS server added by setting a gateway against them.

Steve

I think we didnt understand each other because im bad at explaining things. Sorry. Here is the entire configuration for the scenario that i described in the first post.

This is the status of the Mullvad OpenVPN client once its configured using their instructions.

As you can see, it connects with no issues.

Here is the interface configuration of the Mullvad OpenVPN client.

Here is the OpenVPN gateway that gets automatically created when i add and enable OpenVPN interface. I just added MullvadDNS IP as the gateway monitoring address.

This is the configuration under System / Advanced /
/ Miscellaneous

These are the manual NAT rules. Only local subnets are using OpenVPN and sometimes i switch to WG as well. I also have a WAN NAT rule so i can access DSL modem web configuration.

These are the firewall rules on the LAN interface aka Local Subnets.

These are the firewall rules on the WiFi interface aka WiFi Networks subnet. As you can see here all wifi devices are going through Mullvad WireGuard gateway. I just censored their descriptions and aliases containing their IP addresses.

And these are the firewall rules on the final subnet. A single PC running FreeBSD.

These are the settings under System / General Setup

These are the settings in Services / DNS Resolver / General Settings

This is the status of WireGuard VPN

This is how it all looks like in System / Routing / Gateways

This is how it all looks like after reboot, however, none of the 3 subnets have DNS resolution.

As i said before, i go to System / General Setup, i change nothing, hit the save button, and thats it.

Local Subnets start working without any issues going through OpenVPN gateway, and the other two subnets start working without any issues going through WireGuard gateway. Confirmend by using MullvadVPN connection check.

Its obvious that this setup works. Its just that it stops working after reboot. What am i doing wrong here ?

stephenw10 · Jul 25, 2022, 4:23 PM

When you add an alternative monitoring IP to a gateway a static route to that IP via the gateway is added to ensure it's actually monitoring the correct gateway. Since you are using the Mulvard DNS server there it means it can only ever connect over the WG VPN.

You have not set 'Skip rules when gateway is down' which means that means that if a gateway does go down the rules are just created without a gateway which here means traffic would just leave over the WAN. That's probably not what you want.

Steve

nimrod · Jul 25, 2022, 4:40 PM

@stephenw10 said in OpenVPN + WireGuard breaking DNS resolver.:

When you add an alternative monitoring IP to a gateway a static route to that IP via the gateway is added to ensure it's actually monitoring the correct gateway. Since you are using the Mulvard DNS server there it means it can only ever connect over the WG VPN.

Unbelivalble. I cant belive this was causing the issue.

You have not set 'Skip rules when gateway is down' which means that means that if a gateway does go down the rules are just created without a gateway which here means traffic would just leave over the WAN. That's probably not what you want.

Steve

Thank for very much for your help @stephenw10 !!!

Cheers.

stephenw10 · Jul 25, 2022, 4:50 PM

No worries. Let me know if that helps. There easily be more interactions happening there based on the connection timing.

Steve

nimrod · Jul 25, 2022, 4:56 PM

@stephenw10 said in OpenVPN + WireGuard breaking DNS resolver.:

No worries. Let me know if that helps. There easily be more interactions happening there based on the connection timing.

Steve

It works !!

I removed the monitoring IP`s on both gateways, and i enabled "Do not create rules when gateway is down" in System / Advanced / Miscellaneous.

After reboot, both WireGuard and OpenVPN clients connected as usual and all subnets are going through their designated gateways.

Once again, thank you @stephenw10 !!!