OpenVPN pfSense to pfSense (peer-to-peer) connected but not routing
-
@viragomann I'm making progress.
Part of the issue was a missing firewall rule at the client allowing in traffic over the OpenVPN interface.
Now the problem reduces to DNS, with two issues.
- The client DNS resolver doesn't seem to know about the server side DNS. It never tries to resolve queries for server-side FQDNs even though the override provides the server-side DN and the client config is set to "pull DNS"
- If I do a direct query to the server DNS with
dig @server.dns server-side-FQDNthe server-side DNS resolver refuses the query with "requested recursion not available".
Where should I look next?
-
@jhg
If you request the server side DNS directly either use only the host name or append a dot to the FQDN. Does it work then? -
@viragomann No I still get
WARNING: recursion requested but not availablewhen attempting to resolve server-side hostnames from a client-side host (both Windows and Linux) with or without a trailing dot. The unqualified hostname just returns NXDOMAIN as expected.The frustrating aspect of this is that I also have set up a "Remote Access" OpenVPN which works flawlessly with the Windows OpenVPN Connect client, and resolves DNS just fine. But I'd really like to get peer-to-peer working as well.
-
@jhg
Can you show the whole dig output? -
@viragomann Here is a complete example:
ON THE PFSENSE CLIENT BOX (ssh)
Query for a client-side host (success)
[2.7.2-RELEASE][admin@janus.jhmg.pvt]/root: dig home11.jhmg.pvt ; <<>> DiG 9.18.19 <<>> home11.jhmg.pvt ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59879 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1432 ;; QUESTION SECTION: ;home11.jhmg.pvt. IN A ;; ANSWER SECTION: home11.jhmg.pvt. 3600 IN A 192.168.10.11 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Wed Sep 18 11:09:34 PDT 2024 ;; MSG SIZE rcvd: 60Query for a server-side host (NXDOMAIN)
[2.7.2-RELEASE][admin@janus.jhmg.pvt]/root: dig nas.goow.lan ; <<>> DiG 9.18.19 <<>> nas.goow.lan ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 45776 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1432 ;; QUESTION SECTION: ;nas.goow.lan. IN A ;; AUTHORITY SECTION: . 2345 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2024091800 1800 900 604800 86400 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Wed Sep 18 11:09:43 PDT 2024 ;; MSG SIZE rcvd: 116Query for a server-side host forced to server-side DNS (also pfSense) host (success)
[2.7.2-RELEASE][admin@janus.jhmg.pvt]/root: dig @192.168.0.1 nas.goow.lan ; <<>> DiG 9.18.19 <<>> @192.168.0.1 nas.goow.lan ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33635 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1424 ;; QUESTION SECTION: ;nas.goow.lan. IN A ;; ANSWER SECTION: nas.goow.lan. 3600 IN A 192.168.0.105 ;; Query time: 13 msec ;; SERVER: 192.168.0.1#53(192.168.0.1) (UDP) ;; WHEN: Wed Sep 18 11:09:53 PDT 2024 ;; MSG SIZE rcvd: 57On a linux host in the client-side LAN
Query for a client-side host (success)
jhg@debian ~ $ dig home11.jhmg.pvt ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> home11.jhmg.pvt ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46356 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1432 ;; QUESTION SECTION: ;home11.jhmg.pvt. IN A ;; ANSWER SECTION: home11.jhmg.pvt. 3600 IN A 192.168.10.11 ;; Query time: 0 msec ;; SERVER: 2601:1c0:5601:6258:201:2eff:fe70:3bfe#53(2601:1c0:5601:6258:201:2eff:fe70:3bfe) (UDP) ;; WHEN: Wed Sep 18 11:15:42 PDT 2024 ;; MSG SIZE rcvd: 60Query for a server-side host (NXDOMAIN)
jhg@debian ~ $ dig nas.goow.lan ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> nas.goow.lan ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 21833 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1432 ;; QUESTION SECTION: ;nas.goow.lan. IN A ;; AUTHORITY SECTION: . 3563 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2024091800 1800 900 604800 86400 ;; Query time: 103 msec ;; SERVER: 2601:1c0:5601:6258:201:2eff:fe70:3bfe#53(2601:1c0:5601:6258:201:2eff:fe70:3bfe) (UDP) ;; WHEN: Wed Sep 18 11:15:50 PDT 2024 ;; MSG SIZE rcvd: 116Query for a server-side host forced to server-side DNS (also pfSense) host (recursion not available)
jhg@debian ~ $ dig @192.168.0.1 nas.goow.lan ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> @192.168.0.1 nas.goow.lan ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 24048 ;; flags: qr rd ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; Query time: 11 msec ;; SERVER: 192.168.0.1#53(192.168.0.1) (UDP) ;; WHEN: Wed Sep 18 11:16:07 PDT 2024 ;; MSG SIZE rcvd: 12Some additional info (client linux host)
ping to server pfSense (success)
jhg@debian ~ $ ping 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. 64 bytes from 192.168.0.1: icmp_seq=1 ttl=63 time=14.1 ms 64 bytes from 192.168.0.1: icmp_seq=2 ttl=63 time=10.4 ms ^C --- 192.168.0.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 10.417/12.269/14.122/1.852 mstraceroute with ICMP to server pfSense (success)
jhg@debian ~ $ sudo traceroute -In 192.168.0.1 [sudo] password for jhg: traceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 60 byte packets 1 192.168.10.254 0.324 ms 0.288 ms 0.287 ms 2 192.168.0.1 14.059 ms 14.058 ms 14.057 mstraceroute with UDP to server LAN host (success)
jhg@debian ~ $ sudo traceroute -n 192.168.0.69 traceroute to 192.168.0.69 (192.168.0.69), 30 hops max, 60 byte packets 1 192.168.10.254 0.281 ms 0.386 ms 0.267 ms 2 192.168.21.1 14.327 ms 14.323 ms 14.319 ms 3 192.168.0.69 14.334 ms * 14.325 ms -
@jhg said in OpenVPN pfSense to pfSense (peer-to-peer) connected but not routing:
Query for a server-side host (NXDOMAIN)
jhg@debian ~
$ dig nas.goow.lan; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> nas.goow.lan
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 21833
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1And you have configured a domain override for goow.lan and pointed it to the remote DNS server?
And also set the custom option as described above?Query for a server-side host forced to server-side DNS (also pfSense) host (recursion not available)
jhg@debian ~
$ dig @192.168.0.1 nas.goow.lan; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> @192.168.0.1 nas.goow.lan
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 24048
;; flags: qr rd ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not availableThis query was refused bei the DNS server. Maybe the client pfSense is refused as well.
You have to add ACLs to the server side DNS server for the client side LAN network and the OpenVPN tunnel network (used by the client pfSense).
Also ensure, that the OpenVPN is selected at "Network Interfaces" in the resolver settings if you have not selected "All".
This might presume, that you have assigned an interface to the OpenVPN server instance. -
@viragomann said in OpenVPN pfSense to pfSense (peer-to-peer) connected but not routing:
You have to add ACLs to the server side DNS server for the client side LAN network and the OpenVPN tunnel network (used by the client pfSense).
Then why does my "remote access" VPN to the same server, with Windows "OpenVPN Connect" client not require any of this? Everything, including DNS, just works, configured using the .ovpn file created with Client Export. It's rather surprising that peer-to-peer takes so many after-the-fact hacks to get working while "remote access" was trivial? Something here doesn't seem right.
I added the ACL on the server side and now directed DNS (i.e.
dig @[server-dns] hostname) now works, but it still doesn't work without specifying the server. Again, this all just works without any extra configuration changes on the "remote access" VPN.@viragomann said in OpenVPN pfSense to pfSense (peer-to-peer) connected but not routing:
Also ensure, that the OpenVPN is selected at "Network Interfaces" in the resolver settings if you have not selected "All".
"All" is selected... also the OpenVPN interface does not appear in the list.
The only missing piece now is getting the client pfSense resolver to recognize and resolve queries for the server-side LAN.
Monitoring DNS traffic with tcpdump, what I see is that when a client-side host issues a query for a server-side hostname (FQDN, with or without a trailing doc), the NXDOMAIN originates at the client side resolver. There is no attempt to forward the query via any other interface (VPN or WAN).
So next I tried adding a client-side Resolver Domain Override. Now instead of an immediate NXDOMAIN the request from a LAN host times out, but a directed query works. What I see in the packet trace is that the non-directed query appears to originate from
192.168.21.0(???) which is the tunnel interface's network (i.e. not a valid source IP address), while the directed query originates from the querying host's IP and resolves. Seeing a packet with an invalid source address indicates to me there's clearly a bug somewhere.Here's the tcpdump trace. The first line is the undirected (
dig server.lan.host) query showing the invalid source IP address. The second and third lines are from the directed (dig @192.168.0.1 ...) query.[2.7.2-RELEASE][admin@janus.jhmg.pvt]/root: tcpdump -ni ovpnc1 'not host 1.1.1.1 and ( port 53 or 853 )' tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ovpnc1, link-type NULL (BSD loopback), snapshot length 262144 bytes 18:06:50.429136 IP 192.168.21.0.25687 > 192.168.0.1.53: 31725+ [1au] A? portal.goow.lan. (44) 18:07:10.166701 IP 192.168.10.8.52665 > 192.168.0.1.53: 42081+ [1au] A? portal.goow.lan. (56) 18:07:10.180220 IP 192.168.0.1.53 > 192.168.10.8.52665: 42081* 1/0/1 A 192.168.0.69 (60)I feel like I'm playing whack-a-mole here for something that shouldn't be this hard.
-
@jhg said in OpenVPN pfSense to pfSense (peer-to-peer) connected but not routing:
Then why does my "remote access" VPN to the same server, with Windows "OpenVPN Connect" client not require any of this?
The remote access client gets a virtual IP out of the servers tunnel subnet. He uses this to access server side resources.
The tunnel network is considered as a local network in pfSense.However, if you access the remote DNS server across a site to site VPN, the clients IP is unknown on the server. Hence you have to create an ACL for the clients LAN.
@jhg said in OpenVPN pfSense to pfSense (peer-to-peer) connected but not routing:
What I see in the packet trace is that the non-directed query appears to originate from 192.168.21.0 (???) which is the tunnel interface's network (i.e. not a valid source IP address)
That's strange. I'd expect to see the clients VPN IP as source.
I just verified this on on of my installations, where I have exactly configured, what your need, an OpenVPN client connection to a server with DNS domain override to the remote DNS server.
I started a capture on the client interface and ran an nslookup for an (not-existing) host in the remote domain on a local Windows host.
Then in the capture output I saw packets originating from the virtual clients IP as expected.
So there might be something wrong in your setup apart from the DNS.What if you capture a ping from a client host to the server side?
-
@viragomann said in OpenVPN pfSense to pfSense (peer-to-peer) connected but not routing:
What if you capture a ping from a client host to the server side?
That looks completely normal. I set up tcpdump on all 4 interfaces (client LAN, client VPN, server VPN, server LAN) and everything has the correct source/destination addresses.
for completeness, here are the ping traces at all 4 points. Ping was performed from a Windows host on the client network (192.168.10.11) to a Linux host on the server LAN (192.168.0.69).
Client pfSense LAN interface (re0)
[2.7.2-RELEASE][admin@janus.jhmg.pvt]/root: tcpdump -ni re0 icmp and host 192.168.10.11 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on re0, link-type EN10MB (Ethernet), snapshot length 262144 bytes 22:12:58.041207 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 0, length 64 22:12:58.052766 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 0, length 64 22:12:59.046783 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 1, length 64 22:12:59.058258 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 1, length 64 22:13:00.058654 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 2, length 64 22:13:00.071337 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 2, length 64 22:13:01.068365 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 3, length 64 22:13:01.080533 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 3, length 64Client pfSense VPN tunnel (ovpnc1)
[2.7.2-RELEASE][admin@janus.jhmg.pvt]/root: tcpdump -ni ovpnc1 icmp and host 192.168.10.11 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ovpnc1, link-type NULL (BSD loopback), snapshot length 262144 bytes 22:12:58.041281 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 0, length 64 22:12:58.052737 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 0, length 64 22:12:59.046816 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 1, length 64 22:12:59.058241 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 1, length 64 22:13:00.058704 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 2, length 64 22:13:00.071292 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 2, length 64 22:13:01.068392 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 3, length 64 22:13:01.080514 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 3, length 64Server pfSense VPN tunnel (ovpns2)
[2.7.2-RELEASE][admin@fw.goow.lan]/root: tcpdump -ni ovpns2 icmp and host 192.168.10.11 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ovpns2, link-type NULL (BSD loopback), snapshot length 262144 bytes 22:12:58.045832 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 0, length 64 22:12:58.046197 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 0, length 64 22:12:59.051375 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 1, length 64 22:12:59.051793 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 1, length 64 22:13:00.064454 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 2, length 64 22:13:00.064898 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 2, length 64 22:13:01.074030 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 3, length 64 22:13:01.074303 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 3, length 64Server pfSense LAN interface (re1)
[2.7.2-RELEASE][admin@fw.goow.lan]/root: tcpdump -ni re1 icmp and host 192.168.10.11 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on re1, link-type EN10MB (Ethernet), snapshot length 262144 bytes 22:12:58.045860 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 0, length 64 22:12:58.046190 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 0, length 64 22:12:59.051392 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 1, length 64 22:12:59.051788 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 1, length 64 22:13:00.064480 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 2, length 64 22:13:00.064886 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 2, length 64 22:13:01.074044 IP 192.168.10.11 > 192.168.0.69: ICMP echo request, id 2336, seq 3, length 64 22:13:01.074295 IP 192.168.0.69 > 192.168.10.11: ICMP echo reply, id 2336, seq 3, length 64 -
@viragomann Found it:
System/General Setup/DNS Server Overrideseems to be a global setting that enables/disables the required functionality. For some reason it was disabled on my client pfSense.It seems you need all of the following non-default settings
Client
- System/General Setup/DNS Server Override ON
- DNS Resolver/Domain Override: [server domain]->[server DNS IP address]
- VPN Client/Tunnel Settings/"Pull DNS"
- Custom firewall rule on OpenVPN interface to allow incoming traffic
Server
- DNS Resolver: add an ACL permitting the remote LAN to query the server's DNS resolver
Some comments:
-
If you use the wizard to create multiple VPNs you'll get duplicate firewall rules for incoming VPN traffic
-
The "Pull DNS" setting (VPN/OpenVPN/Edit Client/Tunnel Settings) has the comment
If this option is set, pfSense will use DNS servers assigned by remote OpenVPN server for its own purposes (including the DNS Forwarder/DNS Resolver).
This needs to mention that it is overridden by the System/General Setup/DNS Server Override and has no effect if that setting is OFF.
-
@jhg said in OpenVPN pfSense to pfSense (peer-to-peer) connected but not routing:
It seems you need all of the following non-default settings
ClientSystem/General Setup/DNS Server Override ONAs mentioned multiple times, I think, this setting affects pfSense itself only, as long as you have not enabled DNS forwarding in the Resolver.
You still didn't mention if you have this.Anyway, it has no affect on a domains, which you have configured an override for.
VPN Client/Tunnel Settings/"Pull DNS"This also has no affect on a domains, which you have configured an override for. So you don't need to set this for your purposes and I never suggested to enable this option.
Custom firewall rule on OpenVPN interface to allow incoming trafficThat's pretty plausible. pfSense is a firewall, all intended traffic needs a rule.
Server
DNS Resolver: add an ACL permitting the remote LAN to query the server's DNS resolverThat's by design of Unbound (DNS Resolver). You need ACLs for all unknown source IPs.
Some comments:
If you use the wizard to create multiple VPNs you'll get duplicate firewall rules for incoming VPN trafficAlso note, that the rule tab "OpenVPN" is in fact an interface group including all OpenVPN instances your are running, can be servers or clients. Hence rules, you add there are applied to all.
For better separation you can assign interfaces to the OpenVPN instances. However, remember that rules on the interface group have priority over ones on a member interface.