openvpn causing resolver performance issue?

mikek · Oct 27, 2023, 9:58 PM

Really unclear as to the cause. hoping someone can help me.

using random made up domain name mix.com for test. but really doesn't matter what name.

10.35.53.1 is the dns server of my vpn provider. Accessed via a vpn client configuration. (not sure if that is relevant, but might be.
Disable DNS Rebinding Checks is not checked
resolver config:
python module enabled (pfblocker installed)
Enable Forwarding Mode not checked
custom options: server:include: /var/unbound/pfb_dnsbl.*conf

All from pfsense console

restart resolver.
run dns lookup for mix.com
enable openvpn server.
restart resolver
run dns lookup for mix.com

response time measured in seconds instead of ms. only thing different is i turned on the openvpn server service.
any ideas? can anyone else recreate this behavior?

Mike

johnpoz · Oct 27, 2023, 10:54 PM

@mikek said in openvpn causing resolver performance issue?:

Enable Forwarding Mode not checked

That what is the point of your vpn service dns? If your not forwarding.. You would just resolve - while sure pfsense would ask that server if 127.0.0.1 was down.. Any of your clients asking pfsense, that dns would never be used.

Just running a server instance of vpn should have zero to do with how unbound resolves..

So either something really wrong with your connection, I am taking your routing your traffic from pfsense out your vpn?

What if you disable your vpn client? What sort of cpu are you seeing on pfsense?

I would run +trace on pfsense to see where you could be having a delay.. 118 seems high to me, the ns for that domain are hosted by aws.. They should have something near you that answer fairly quickly.

;; QUESTION SECTION:                                                     
;mix.com.                       IN      NS                               
                                                                         
;; ANSWER SECTION:                                                       
mix.com.                86400   IN      NS      ns-1044.awsdns-02.org.   
mix.com.                86400   IN      NS      ns-1753.awsdns-27.co.uk. 
mix.com.                86400   IN      NS      ns-362.awsdns-45.com.    
mix.com.                86400   IN      NS      ns-999.awsdns-60.net.

; <<>> DiG 9.18.13 <<>> mix.com +trace
;; global options: +cmd
.                       70291   IN      NS      l.root-servers.net.
.                       70291   IN      NS      m.root-servers.net.
.                       70291   IN      NS      a.root-servers.net.
.                       70291   IN      NS      b.root-servers.net.
.                       70291   IN      NS      c.root-servers.net.
.                       70291   IN      NS      d.root-servers.net.
.                       70291   IN      NS      e.root-servers.net.
.                       70291   IN      NS      f.root-servers.net.
.                       70291   IN      NS      g.root-servers.net.
.                       70291   IN      NS      h.root-servers.net.
.                       70291   IN      NS      i.root-servers.net.
.                       70291   IN      NS      j.root-servers.net.
.                       70291   IN      NS      k.root-servers.net.
.                       70291   IN      RRSIG   NS 8 0 518400 20231109170000 20231027160000 46780 . rLPlXLbayZLknVgfPNQ1dKJ9vaPW/kj428WZkJscd7Mjo5ZBojLbPrn9 054sptxjr+6Qm1lA5HH6+H952D3f7P2mkf4ES+8Ot9N3fUXqwZXft+YV aZid+W5It63UEekzORMR2B3sFMU/zCfLSjGoDUQiR3rPYD8TbqhQS3L4 DAkjlSWApBdwId2NRJ+yRY9j1h7oBOB6eVkZEBRbiioYPvzyaaQSq+hI vQWkZEaahxecc7okt2UftofSSN54qABtn8mosvvTDQ0WT2AtGlqnByky LK81WKgHimZiDabV7oNtUOp2tGIUQlJyWY8oJsTF7qB7TD3ACUoD+zNd 8Mobaw==
;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 2 ms

com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20231109170000 20231027160000 46780 . xwYU+EK0KOcc/2wX0ZFzksEKRCtFCiAzGEy9Y65tNhboKdjQ7kLqIJAZ WJV3S4tcPUjcrYevmqP4wCTosBe5p1RZxZ6EfxgWKddhFkyJUl9H4tWw trU9B3oqhwEcR6gnKHIkY0jYXUzhDXT1AUhhnKxafqQVM2twSyRYClgp 13uvtCXb6RCDpPsr/XIl9J4DQeqm1GdrHbk9E2CJ1Ssmju6+AHrkkyBc kVgSqIDc1mGv0fL+JdqIOU+OXgmMgZioR9nUG6ZuyKwRUuz2CgucvVbi lX1KIVV4Ymf/PqksWzisedUHsdkQF/0pbHkXXdm/uxCejDD3vUfSzMQW N/hPcQ==
;; Received 1167 bytes from 198.97.190.53#53(h.root-servers.net) in 29 ms

mix.com.                172800  IN      NS      ns-362.awsdns-45.com.
mix.com.                172800  IN      NS      ns-999.awsdns-60.net.
mix.com.                172800  IN      NS      ns-1044.awsdns-02.org.
mix.com.                172800  IN      NS      ns-1753.awsdns-27.co.uk.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20231102042427 20231026031427 63246 com. JYZ3eaDCWcZRoClPmv/wR2prV2tdCly2XCk0xNbnvBIGUsZzdEu8pp/w ZJP0IA4eOj4IRyihXE87UHgnRVPwb9I2EQtMYH6ud+bwcG8ckJp0IpSd utt3c5BU1moKP9zcWO5o5pfY1Rsib2skoAaaNlaROcSRJLOSMDdhHv5n pbPpwC1W3O/j38TeR0i3hbld88REx38XDSRJEsuNG6Ikrg==
3JQSFDI8CCR36ME7R44PRGCIMTH23403.com. 86400 IN NSEC3 1 1 0 - 3JQT0JTPN8VVB5HFAC6V950MR4IJQ6CL NS DS RRSIG
3JQSFDI8CCR36ME7R44PRGCIMTH23403.com. 86400 IN RRSIG NSEC3 8 2 86400 20231103052943 20231027041943 63246 com. H445YmH4W46hyR6SpJw94UgNtVJ4WpJbgVOLSMv3wAz/Pp5YwdWSfdV5 Obxb52MgHitAehuwfoX/Nsu7ZivU51DDAdeN41KEzApilkCOCGNM16A4 7PpYfoVSsT/cQa6D6C4H9gfF5xm/td63NvkO99OIKWB6jNHINtyo2omG 6E4ISSHXCYmvEmPtlxoO+770cYCfRyS1KmaasM7SyfbLxg==
;; Received 738 bytes from 192.35.51.30#53(f.gtld-servers.net) in 37 ms

mix.com.                60      IN      A       34.192.56.160
mix.com.                60      IN      A       3.82.184.114
mix.com.                60      IN      A       52.21.23.66
mix.com.                60      IN      RRSIG   A 13 2 60 20231027235409 20231027215309 29285 mix.com. lcDcrggGIrQM3mVXxnTvpqUvNauwGNZF3R01GPxtM8ZZYzv+D0sGn6Mh kFgRHyHrdTmmz539ozKgywt+L2XX8A==
;; Received 187 bytes from 2600:9000:5306:d900::1#53(ns-1753.awsdns-27.co.uk) in 33 ms

Do you have IPv6? If you do, what does it look like when you only do ipv4?

[23.05.1-RELEASE][admin@sg4860.local.lan]/root: dig -4 mix.com +trace

; <<>> DiG 9.18.13 <<>> -4 mix.com +trace
;; global options: +cmd
.                       70271   IN      NS      i.root-servers.net.
.                       70271   IN      NS      j.root-servers.net.
.                       70271   IN      NS      k.root-servers.net.
.                       70271   IN      NS      l.root-servers.net.
.                       70271   IN      NS      m.root-servers.net.
.                       70271   IN      NS      a.root-servers.net.
.                       70271   IN      NS      b.root-servers.net.
.                       70271   IN      NS      c.root-servers.net.
.                       70271   IN      NS      d.root-servers.net.
.                       70271   IN      NS      e.root-servers.net.
.                       70271   IN      NS      f.root-servers.net.
.                       70271   IN      NS      g.root-servers.net.
.                       70271   IN      NS      h.root-servers.net.
.                       70271   IN      RRSIG   NS 8 0 518400 20231109170000 20231027160000 46780 . rLPlXLbayZLknVgfPNQ1dKJ9vaPW/kj428WZkJscd7Mjo5ZBojLbPrn9 054sptxjr+6Qm1lA5HH6+H952D3f7P2mkf4ES+8Ot9N3fUXqwZXft+YV aZid+W5It63UEekzORMR2B3sFMU/zCfLSjGoDUQiR3rPYD8TbqhQS3L4 DAkjlSWApBdwId2NRJ+yRY9j1h7oBOB6eVkZEBRbiioYPvzyaaQSq+hI vQWkZEaahxecc7okt2UftofSSN54qABtn8mosvvTDQ0WT2AtGlqnByky LK81WKgHimZiDabV7oNtUOp2tGIUQlJyWY8oJsTF7qB7TD3ACUoD+zNd 8Mobaw==
;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms

com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20231109170000 20231027160000 46780 . xwYU+EK0KOcc/2wX0ZFzksEKRCtFCiAzGEy9Y65tNhboKdjQ7kLqIJAZ WJV3S4tcPUjcrYevmqP4wCTosBe5p1RZxZ6EfxgWKddhFkyJUl9H4tWw trU9B3oqhwEcR6gnKHIkY0jYXUzhDXT1AUhhnKxafqQVM2twSyRYClgp 13uvtCXb6RCDpPsr/XIl9J4DQeqm1GdrHbk9E2CJ1Ssmju6+AHrkkyBc kVgSqIDc1mGv0fL+JdqIOU+OXgmMgZioR9nUG6ZuyKwRUuz2CgucvVbi lX1KIVV4Ymf/PqksWzisedUHsdkQF/0pbHkXXdm/uxCejDD3vUfSzMQW N/hPcQ==
;; Received 1167 bytes from 202.12.27.33#53(m.root-servers.net) in 54 ms

mix.com.                172800  IN      NS      ns-362.awsdns-45.com.
mix.com.                172800  IN      NS      ns-999.awsdns-60.net.
mix.com.                172800  IN      NS      ns-1044.awsdns-02.org.
mix.com.                172800  IN      NS      ns-1753.awsdns-27.co.uk.
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20231102042427 20231026031427 63246 com. JYZ3eaDCWcZRoClPmv/wR2prV2tdCly2XCk0xNbnvBIGUsZzdEu8pp/w ZJP0IA4eOj4IRyihXE87UHgnRVPwb9I2EQtMYH6ud+bwcG8ckJp0IpSd utt3c5BU1moKP9zcWO5o5pfY1Rsib2skoAaaNlaROcSRJLOSMDdhHv5n pbPpwC1W3O/j38TeR0i3hbld88REx38XDSRJEsuNG6Ikrg==
3JQSFDI8CCR36ME7R44PRGCIMTH23403.com. 86400 IN NSEC3 1 1 0 - 3JQT0JTPN8VVB5HFAC6V950MR4IJQ6CL NS DS RRSIG
3JQSFDI8CCR36ME7R44PRGCIMTH23403.com. 86400 IN RRSIG NSEC3 8 2 86400 20231103052943 20231027041943 63246 com. H445YmH4W46hyR6SpJw94UgNtVJ4WpJbgVOLSMv3wAz/Pp5YwdWSfdV5 Obxb52MgHitAehuwfoX/Nsu7ZivU51DDAdeN41KEzApilkCOCGNM16A4 7PpYfoVSsT/cQa6D6C4H9gfF5xm/td63NvkO99OIKWB6jNHINtyo2omG 6E4ISSHXCYmvEmPtlxoO+770cYCfRyS1KmaasM7SyfbLxg==
;; Received 738 bytes from 192.12.94.30#53(e.gtld-servers.net) in 40 ms

mix.com.                60      IN      A       34.192.56.160
mix.com.                60      IN      A       3.82.184.114
mix.com.                60      IN      A       52.21.23.66
mix.com.                60      IN      RRSIG   A 13 2 60 20231027235426 20231027215326 29285 mix.com. r2OeU/NLhlH/7ilHI9DRJebrx5YzKKUpOCvnpF8bWOd43BO+8Q4KBrsy yWF+kPKl72FculzVA02ZDVaXS/kP+w==
;; Received 187 bytes from 205.251.195.231#53(ns-999.awsdns-60.net) in 29 ms

mikek · Oct 28, 2023, 1:22 AM

@johnpoz
"That what is the point of your vpn service dns? If your not forwarding.. You would just resolve - while sure pfsense would ask that server if 127.0.0.1 was down.. Any of your clients asking pfsense, that dns would never be used."

Your correct. resolver should hit the root servers. 127.0.0.1 response would be from resolver response from root. but either configuration, forward or not forward doesn't change the behavior.

"Just running a server instance of vpn should have zero to do with how unbound resolves.." agree 100% which is why i am so confused!

"So either something really wrong with your connection, I am taking your routing your traffic from pfsense out your vpn?" everything works great until i start a server instance. i am in fact routing most traffic out the vpn with just a couple of host bypassing and going straight out. a routing issue would cause failure not slowdown, or maybe i am wrong about that. no clue.

cpu:

Never see it even go over 5%

shutdown vpn client to my provider:
restart resolver
run lookup:

start openvpn server
restart resolver
run lookup

start vpn
start openvpn server
restart resolver
run lookup

"I would run +trace on pfsense to see where you could be having a delay.. 118 seems high to me, the ns for that domain are hosted by aws.. They should have something near you that answer fairly "

i believe that the traffic is going out the vpn client when it is running (even the root server calls) which makes 118 perfectly fine, as the vpn endpoint is on the coast and i am in the center of the country. my issue is the 2 to 10 seconds response times when the openvpn server is running. something as you stated and by my understanding should have absolutely no effect on resolver performance.

Seems to be the combination of running the vpn client to my provider and having the openvpn server process running that causes this.

thanks
Mike

johnpoz · Oct 28, 2023, 1:22 AM

@mikek I run 2 instances of vpn server running.. different ports. And have a client connection to a vps of mine. I see no such issues. But I don't attempt to route pfsense traffic out the vpn client connection.

Listening on your wan IP on a port for a vpn connection should have nothing to do with unbound resolving something..

Only thing I could guess is you got something funked up.

Looks from your screenshots that its not when your running just the server, but once you connect to as a client.

On your unbound - set the outbound connection to what you want to use to resolve, be that just your wan, or your vpn client interface. don't set it to all.

mikek · Oct 28, 2023, 12:29 PM

@johnpoz

@johnpoz said in openvpn causing resolver performance issue?:

Only thing I could guess is you got something funked up.

only thing that makes sense. was hoping i could find it and fix it. but i have messed with all kinds of settings at this point with no effect.

going to document everything out, go to factory defaults and rebuild. testing along the way. only thing i can think to do at this point.

thanks
Mike

johnpoz · Oct 28, 2023, 7:20 PM

@mikek So to duplicate.. See I have 2 vpn server instances running, and a client connection to a vps I have out in Las Vegas.

So I told unbound to use the vpn client connection. I then did a dns leak test to validate that unbound was resolving from my vpn client connection.

Then did a query that would have to actually resolve and not be cached.. As you can see its a normal sort of resolve time.

here I put unbound back to just use my normal isp connection, and leak shows that using my isp IP to query.. And resolving is a bit faster, because not routing traffic out my vpn.

But as you can see resolving through my vpn client, while also having vpn server connections active does really play a part in time to resolve.

I could also make a connection to my vpn server from say my phone over cell if you feel that might be causing some issues? But here is the thing. Just listening on a port on your wan, should have zero to do with performance of your resolver - be it routing through your normal isp connection or some vpn client connection.

mikek · Oct 28, 2023, 7:20 PM

@johnpoz
appreciate the time and information. when i can afford some downtime. going to rebuild from scratch. maybe something is hung in the config or just flat messed up. your right. just starting an openvpn server and having it listen on the wan link "should" have absolutely no effect on performance of resolver. took me a few hours to initially isolate it because of that. but currently it is very reproduceable. even deleting the vpn server and rebuilding has no effect. issue remains.

Going to go through step by step testing all long the way from a default configuration. see if i can identify what i am doing exactly that presents this behavior.

johnpoz · Oct 28, 2023, 8:32 PM

@mikek The only thing that comes to mind if you had some sort of kill switch setup for your vpn, and you told unbound it could resolve using any of your interface.. If it tried to use your normal ISP and you blocked that traffic say outbound.. It would have to wait for timeouts, etc. until it tried your vpn connection.. So yeah that could take some time, ie seconds to finish up.

its possible your vpn is causing pain as well with trying to resolve, maybe they filter other dns??

I would let unbound either just use your normal isp connection to resolve, or if you set on using it through your vpn. Set unbound to only use that interface for its outbound, or just set it to forward to your vpn services dns server.

But the fact of just running a vpn service on your wan would/should/could not have any effect on unbound resolving.. That don't have anything to do with each other.

mikek · Oct 29, 2023, 12:12 PM

@johnpoz

@johnpoz said in openvpn causing resolver performance issue?:

its possible your vpn is causing pain as well with trying to resolve, maybe they filter other dns??

confirmed they do not filter anything I can find. pretty much just pass whatever traffic you send on through.

@johnpoz said in openvpn causing resolver performance issue?:

I would let unbound either just use your normal isp connection to resolve, or if you set on using it through your vpn. Set unbound to only use that interface for its outbound, or just set it to forward to your vpn services dns server.

ISP direct resolution would present a dns leak scenario on the vpn. not an optimal configuration. I tried changing the resolver interface binding, and it had no effect on the behavior.

@johnpoz said in openvpn causing resolver performance issue?:

But the fact of just running a vpn service on your wan would/should/could not have any effect on unbound resolving.. That don't have anything to do with each other.

I rebuilt everything from factory default last night. only difference is i setup the vpn server before i defined and configured the clients for my vpn service. everything functioning exactly as before with all dns traversing the vpn service. (no forwarding, so using root servers still)

The issue went away.

Don't really understand what was happening but would like to. I have a backup of the broken configuration. I might bring it up on a vm and investigate further. What you describe about a timeout scenario, seems to make a lot of sense. Just have no clue what would be timing out at the moment.