Problems with pfSense IPV6 DNS function (does it exist!?)

johnpoz · Jan 9, 2023, 7:22 PM

@louis2 said in Problems with pfSense IPV6 DNS function (does it exist!?):

used to validate DNS server certificates

this is big take away.. that fqdn is used on the cert to validate your talking to the right dot server..

So for example if I do a dot query to 1.1.1.1 and tell it to validate everything then using something other than the CN or what is in the SAN of the cert would fail.. A normal dot query should be validating this..

You can get all the info of CN and SANs that are in the cert with some use of openssl, notice the output below for the CN and SANs in that cert being used at 1.1.1.1 on port 853.

But from a kdig asking for validation you can see it fails if I use a fqdn that is not CN or SAN.

root@NewUC:/home/user# kdig -d @1.1.1.1 www.google.com +tls-ca
;; DEBUG: Querying for owner(www.google.com.), class(1), type(1), server(1.1.1.1), port(853), protocol(TCP)
;; DEBUG: TLS, imported 124 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG:  #1, C=US,ST=California,L=San Francisco,O=Cloudflare\, Inc.,CN=cloudflare-dns.com
;; DEBUG:      SHA-256 PIN: MnLdGiqUGYhtyinlrGTC4FZdDyDXv4NOWFGnXW3ur14=
;; DEBUG:  #2, C=US,O=DigiCert Inc,CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
;; DEBUG:      SHA-256 PIN: e0IRz5Tio3GA1Xs4fUVWmH1xHDiH2dMbVtCBSkOIdqM=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is trusted. 
;; TLS session (TLS1.3)-(ECDHE-X25519)-(ECDSA-SECP256R1-SHA256)-(AES-256-GCM)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 30091
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 1232 B; ext-rcode: NOERROR
;; PADDING: 405 B

;; QUESTION SECTION:
;; www.google.com.              IN      A

;; ANSWER SECTION:
www.google.com.         128     IN      A       142.250.191.132

;; Received 468 B
;; Time 2023-01-09 13:12:01 CST
;; From 1.1.1.1@853(TCP) in 61.0 ms
root@NewUC:/home/user#

But if I tell it wrong host name..

;; From 1.1.1.1@853(TCP) in 61.0 ms
root@NewUC:/home/user# kdig -d @1.1.1.1 www.google.com +tls-ca +tls-host=badname
;; DEBUG: Querying for owner(www.google.com.), class(1), type(1), server(1.1.1.1), port(853), protocol(TCP)
;; DEBUG: TLS, imported 124 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG:  #1, C=US,ST=California,L=San Francisco,O=Cloudflare\, Inc.,CN=cloudflare-dns.com
;; DEBUG:      SHA-256 PIN: MnLdGiqUGYhtyinlrGTC4FZdDyDXv4NOWFGnXW3ur14=
;; DEBUG:  #2, C=US,O=DigiCert Inc,CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
;; DEBUG:      SHA-256 PIN: e0IRz5Tio3GA1Xs4fUVWmH1xHDiH2dMbVtCBSkOIdqM=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is NOT trusted. The name in the certificate does not match the expected. 
;; WARNING: TLS, handshake failed (Error in the certificate.)
;; ERROR: failed to query server 1.1.1.1@853(TCP)
root@NewUC:/home/user#

if I use correct hostname, see the below openssl output to what all the valid SANs are in that cert

root@NewUC:/home/user# kdig -d @1.1.1.1 www.google.com +tls-ca +tls-host=one.one.one.one
;; DEBUG: Querying for owner(www.google.com.), class(1), type(1), server(1.1.1.1), port(853), protocol(TCP)
;; DEBUG: TLS, imported 124 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG:  #1, C=US,ST=California,L=San Francisco,O=Cloudflare\, Inc.,CN=cloudflare-dns.com
;; DEBUG:      SHA-256 PIN: MnLdGiqUGYhtyinlrGTC4FZdDyDXv4NOWFGnXW3ur14=
;; DEBUG:  #2, C=US,O=DigiCert Inc,CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
;; DEBUG:      SHA-256 PIN: e0IRz5Tio3GA1Xs4fUVWmH1xHDiH2dMbVtCBSkOIdqM=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is trusted. 
;; TLS session (TLS1.3)-(ECDHE-X25519)-(ECDSA-SECP256R1-SHA256)-(AES-256-GCM)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 62622
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 1232 B; ext-rcode: NOERROR
;; PADDING: 405 B

;; QUESTION SECTION:
;; www.google.com.              IN      A

;; ANSWER SECTION:
www.google.com.         289     IN      A       142.250.190.4

;; Received 468 B
;; Time 2023-01-09 13:14:10 CST
;; From 1.1.1.1@853(TCP) in 63.2 ms
root@NewUC:/home/user#

As to your log - seems it pretty clear to why its refusing you, that "::/128 refused" where that is coming from not sure.. But that would explain why its refusing you.

root@NewUC:/home/user# openssl s_client -connect 1.1.1.1:853 </dev/null 2>/dev/null | openssl x509 -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0d:1c:7a:f2:8e:5f:27:17:db:b2:7f:41:08:20:bd:f7
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = US, O = DigiCert Inc, CN = DigiCert TLS Hybrid ECC SHA384 2020 CA1
        Validity
            Not Before: Sep 13 00:00:00 2022 GMT
            Not After : Sep 13 23:59:59 2023 GMT
        Subject: C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = cloudflare-dns.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:fc:3e:51:ef:74:1d:c6:da:78:ba:ae:a5:8a:4a:
                    dd:d9:0b:e6:e2:5b:57:31:57:de:d3:bf:b6:d9:8a:
                    3b:4f:d2:54:54:88:cf:bd:2e:65:e7:66:eb:c5:df:
                    d0:31:54:52:a7:2c:ee:12:86:a3:9a:66:c1:ea:06:
                    79:03:ba:1b:f0
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                0A:BC:08:29:17:8C:A5:39:6D:7A:0E:CE:33:C7:2E:B3:ED:FB:C3:7A
            X509v3 Subject Key Identifier: 
                D2:63:BA:94:D6:54:7F:4C:85:14:08:3A:1C:85:56:29:EF:59:8F:CC
            X509v3 Subject Alternative Name: 
                DNS:cloudflare-dns.com, DNS:*.cloudflare-dns.com, DNS:one.one.one.one, IP Address:1.0.0.1, IP Address:1.1.1.1, IP Address:162.159.36.1, IP Address:162.159.46.1, IP Address:2606:4700:4700:0:0:0:0:1001, IP Address:2606:4700:4700:0:0:0:0:1111, IP Address:2606:4700:4700:0:0:0:0:64, IP Address:2606:4700:4700:0:0:0:0:6400
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points: 
                Full Name:
                  URI:http://crl3.digicert.com/DigiCertTLSHybridECCSHA3842020CA1-1.crl
                Full Name:
                  URI:http://crl4.digicert.com/DigiCertTLSHybridECCSHA3842020CA1-1.crl
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.2
                  CPS: http://www.digicert.com/CPS
            Authority Information Access: 
                OCSP - URI:http://ocsp.digicert.com
                CA Issuers - URI:http://cacerts.digicert.com/DigiCertTLSHybridECCSHA3842020CA1-1.crt
            X509v3 Basic Constraints: 
                CA:FALSE
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : E8:3E:D0:DA:3E:F5:06:35:32:E7:57:28:BC:89:6B:C9:
                                03:D3:CB:D1:11:6B:EC:EB:69:E1:77:7D:6D:06:BD:6E
                    Timestamp : Sep 13 22:48:42.079 2022 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:13:E1:7D:A3:DD:99:5A:FF:0B:9F:41:A7:
                                37:EA:F3:4D:82:23:03:09:43:DD:AC:2D:C8:9C:C8:4E:
                                3C:34:32:F7:02:20:35:1A:CE:22:F9:06:48:3D:8D:0D:
                                C9:EB:9D:6C:92:C2:03:BA:59:D6:94:EE:8F:71:38:E8:
                                26:00:12:76:48:0F
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 35:CF:19:1B:BF:B1:6C:57:BF:0F:AD:4C:6D:42:CB:BB:
                                B6:27:20:26:51:EA:3F:E1:2A:EF:A8:03:C3:3B:D6:4C
                    Timestamp : Sep 13 22:48:42.151 2022 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:A2:B6:45:5D:DE:32:F8:03:3C:9C:BD:
                                5F:49:1B:8B:79:1B:FB:39:68:73:53:43:83:4E:77:8A:
                                37:D2:56:76:E2:02:21:00:84:92:F2:CA:5E:AB:16:F4:
                                50:EC:8D:95:F0:84:36:B0:A8:99:B8:75:05:9E:A0:4A:
                                F7:C2:16:F6:8E:23:1C:4F
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : B3:73:77:07:E1:84:50:F8:63:86:D6:05:A9:DC:11:09:
                                4A:79:2D:B1:67:0C:0B:87:DC:F0:03:0E:79:36:A5:9A
                    Timestamp : Sep 13 22:48:42.231 2022 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:9A:01:8C:AF:2F:41:72:8B:57:0B:1C:
                                1C:07:90:22:B1:C3:D7:FE:80:BB:FB:D2:B5:AA:CE:60:
                                5F:0F:62:0E:DF:02:20:0A:8C:4B:CE:DB:26:DF:B8:5E:
                                67:D6:9E:68:7C:CE:C4:D4:15:9A:53:3B:80:1F:07:F6:
                                6B:80:6B:07:18:86:A0
    Signature Algorithm: ecdsa-with-SHA384
    Signature Value:
        30:65:02:30:19:d7:9a:79:6e:69:cc:19:6f:dd:c3:d0:36:c8:
        54:d0:4e:f4:6f:1d:46:c0:ca:2f:fb:22:19:95:f8:70:75:0b:
        59:d9:ad:16:31:6e:1a:26:1b:16:df:71:0d:55:cd:4a:02:31:
        00:ea:dc:2e:09:98:fe:b1:fb:07:f5:31:5f:b6:7c:1a:be:38:
        d9:c5:39:9a:6a:0b:9d:8f:54:67:13:4b:c1:12:86:26:2a:11:
        6d:98:d3:91:1d:02:eb:e1:77:76:71:37:8e
root@NewUC:/home/user#

If you don't want to look through all that, the important info from the cert is this

X509v3 Subject Alternative Name:
DNS:cloudflare-dns.com, DNS:*.cloudflare-dns.com, DNS:one.one.one.one, IP Address:1.0.0.1, IP Address:1.1.1.1, IP Address:162.159.36.1, IP Address:162.159.46.1, IP Address:2606:4700:4700:0:0:0:0:1001, IP Address:2606:4700:4700:0:0:0:0:1111, IP Address:2606:4700:4700:0:0:0:0:64, IP Address:2606:4700:4700:0:0:0:0:6400

big fan of dig, and use it pretty much everyday - but kdig is much better with working with dot and or doh. And with anything cert related use of openssl is your go to tool.

cursixx · Jan 10, 2023, 6:53 AM

I did a factory reset of my config and set my WAN and LAN interfaces (dev 2.7 1/9). Never even logged in and I'm still getting query refused with ipv6. I just wanted to rule out any of my config changes as the issue.

Then I loaded a clean install of stable 2.6 and it seems to be working as expected

Bob.Dig · Feb 6, 2023, 7:29 AM

@marcosm I can confirm this bug for 23.01.r.20230202.1645 too. At least for ULAs.

jimp · Feb 6, 2023, 3:01 PM

I still can't reproduce this here. Fresh install, DHCPv6 WAN / Track6 LAN, it boots up and has the prefix for LAN allowed in Unbound, and a client on LAN gets an address and can query it properly.

If that doesn't work I suggest first checking your WAN to make sure you have the correct settings there for DHCPv6 (especially the delegation size), and check to see what is present on the LAN interface(s) and in /var/unbound/access_lists.conf.

There could be some kind of timing issue where the address gets added to LAN but unbound doesn't reload after that, but I'm not aware of anything that would have changed in that scenario specifically.

watkins13 · Feb 13, 2023, 10:04 PM

I seem to be running into this REFUSED query response issue as well—or at the least a very similar issue—on the current pfSense Plus 23.01-RC build running on my Netgate 6100.

Refused IPv6 query output:

Arashi ~ % dig @2600:1700:2320:5fcf:92ec:77ff:fe21:3f06 dns.google. AAAA

; <<>> DiG 9.10.6 <<>> @2600:1700:2320:5fcf:92ec:77ff:fe21:3f06 dns.google. AAAA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 46011
;; flags: qr rd ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; Query time: 57 msec
;; SERVER: 2600:1700:2320:5fcf:92ec:77ff:fe21:3f06#53(2600:1700:2320:5fcf:92ec:77ff:fe21:3f06)
;; WHEN: Mon Feb 13 13:26:40 PST 2023
;; MSG SIZE  rcvd: 12

Working IPv4 query output:

Arashi ~ % dig @10.13.1.1 dns.google. AAAA

; <<>> DiG 9.10.6 <<>> @10.13.1.1 dns.google. AAAA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46994
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1432
;; QUESTION SECTION:
;dns.google.			IN	AAAA

;; ANSWER SECTION:
dns.google.		900	IN	AAAA	2001:4860:4860::8844
dns.google.		900	IN	AAAA	2001:4860:4860::8888

;; Query time: 484 msec
;; SERVER: 10.13.1.1#53(10.13.1.1)
;; WHEN: Mon Feb 13 13:18:02 PST 2023
;; MSG SIZE  rcvd: 95

From what I can tell so far, the issue seems to be that expected access-control lines for the LAN interface are not auto-added the lines for the IPv6 parts of the LAN interface and only generating access-control entries for the IPv4 LAN network.

IPv4 queries function as expected on the LAN, and IPv6 DNS queries on the LAN interface are answered with a REFUSED status. I checked to see confirm that IPv6 DNS packets were reaching the router past the firewall rules. I further confirmed that by adding in an explicit firewall rule to allow it at the top of the LAN list to confirm that again.

uname -v output:

FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256016-ee43ebe4124: Wed Feb  8 14:43:16 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/B5BT22YU/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/amd64.amd64/sys/pfSense

Current pfSense settings:

My WAN interface is using DHCP6 for IPv6, and my LAN is in the Track Interface type with WAN set as the IPv6 Interface to track.
The Disable Auto-added Access Control for the DNS Resolver is not checked.
The DNS Resolver (unbound) daemon is configured to listen on ALL interfaces.
DHCPv6 Server is enabled on the LAN with a delegated prefix length of 64, and the Router Advertisements mode is set to Assisted and Use same settings as DHCPv6 server is enabled/checked.
Looking at the generated /var/unbound/access_lists.conf file, it appears to have only generated IPv4 access-control lines. I do not see any lines for the LAN IPv6 delegated prefix.

[23.01-RC][root@pfSense-ng6100.home.arpa]/root: netstat -an | grep -E '\.53\b'
tcp4       0      0 *.53                   *.*                    LISTEN     
tcp6       0      0 *.53                   *.*                    LISTEN     
udp4       0      0 *.53                   *.*                    
udp6       0      0 *.53                   *.*

[23.01-RC][root@pfSense-ng6100.home.arpa]/root: cat /var/unbound/access_lists.conf
access-control: 127.0.0.1/32 allow_snoop
access-control: ::1 allow_snoop
access-control: 10.13.1.0/24 allow
access-control: 127.0.0.0/8 allow
access-control: ::1/128 allow

watkins13 · Feb 13, 2023, 10:13 PM

@jimp

Is there a way I can enable/view logs for the generated auto-added access control items?

For now, I can workaround the missing IPv6 access-control lines for the DNS access lists by adding a custom Allow access list entry in the DNS Resolver settings to cover my current /64 LAN delegated prefix, but that workaround is brittle and would break if my delegated prefix changes at any point.

[23.01-RC][root@pfSense-ng6100.home.arpa]/root: cat /var/unbound/access_lists.conf
access-control: 127.0.0.1/32 allow_snoop
access-control: ::1 allow_snoop
access-control: 10.13.1.0/24 allow 
access-control: 127.0.0.0/8 allow 
access-control: ::1/128 allow 
#Custom IPv6 Allow Workaround
access-control: 2600:1700:2320:5fcf::/64 allow

Working query with workaround:

Arashi ~ % dig @2600:1700:2320:5fcf:92ec:77ff:fe21:3f06 dns.google. AAAA

; <<>> DiG 9.10.6 <<>> @2600:1700:2320:5fcf:92ec:77ff:fe21:3f06 dns.google. AAAA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6924
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1432
;; QUESTION SECTION:
;dns.google.			IN	AAAA

;; ANSWER SECTION:
dns.google.		900	IN	AAAA	2001:4860:4860::8844
dns.google.		900	IN	AAAA	2001:4860:4860::8888

;; Query time: 378 msec
;; SERVER: 2600:1700:2320:5fcf:92ec:77ff:fe21:3f06#53(2600:1700:2320:5fcf:92ec:77ff:fe21:3f06)
;; WHEN: Mon Feb 13 13:31:02 PST 2023
;; MSG SIZE  rcvd: 95

thebear · Feb 16, 2023, 6:14 PM

@jimp I was triggered by your DHCP-PD size. My ISP provides a full /48 within PPPoE.

Then it should be configured as SIZE: 48?

More screenshots here https://forum.netgate.com/topic/177863/23-01-unable-to-answer-dns-queries-after-upgrade/26?_=1676570089641

@jimp said in [Problems with pfSense IPV6 DNS function (does it

If that doesn't work I suggest first checking your WAN to make sure you have the correct settings there for DHCPv6 (especially the delegation size)

jimp · Feb 23, 2023, 2:38 PM

I was finally able to find a box in my lab that hit this, so I was able to find out what was happening there. I reopened https://redmine.pfsense.org/issues/13851 for it.

I don't know how/why I wasn't hitting it on more things, but apparently most of the boxes I was checking for IPv6 had specific local bindings and were not set to "all" for the local network interface.

You can install the System Patches package and then create an entry for 46b159032fef8c78783aa1a749d2238cfed7ac0d to apply the fix.

thebear · Feb 23, 2023, 3:25 PM

@jimp said in Problems with pfSense IPV6 DNS function (does it exist!?):

I was finally able to find a box in my lab that hit this, so I was able to find out what was happening there. I reopened https://redmine.pfsense.org/issues/13851 for it.

I don't know how/why I wasn't hitting it on more things, but apparently most of the boxes I was checking for IPv6 had specific local bindings and were not set to "all" for the local network interface.

You can install the System Patches package and then create an entry for 46b159032fef8c78783aa1a749d2238cfed7ac0d to apply the fix.

Thanks for the quick fix that looks way better!

Bob.Dig · Feb 23, 2023, 3:31 PM

@jimp said in Problems with pfSense IPV6 DNS function (does it exist!?):

46b159032fef8c78783aa1a749d2238cfed7ac0d

That fix worked for me too, took some seconds after the reboot that I did.

PS C:\Users\Bobby> nslookup reddit.com fd7b:97bf:48b9:100:192:168:100:1
Server:  pfSense.home.arpa
Address:  fd7b:97bf:48b9:100:192:168:100:1

Non-authoritative answer:
Name:    reddit.com
Addresses:  2a04:4e42:200::396
          2a04:4e42::396
          2a04:4e42:600::396
          2a04:4e42:400::396
          151.101.193.140
          151.101.129.140
          151.101.65.140
          151.101.1.140

JimBob Indiana · Feb 24, 2023, 4:28 PM

@jimp said in Problems with pfSense IPV6 DNS function (does it exist!?):

46b159032fef8c78783aa1a749d2238cfed7ac0d

If we're using 2.7 should we apply this patch?

jimp · Feb 24, 2023, 4:31 PM

@jimbob-indiana said in Problems with pfSense IPV6 DNS function (does it exist!?):

@jimp said in Problems with pfSense IPV6 DNS function (does it exist!?):

46b159032fef8c78783aa1a749d2238cfed7ac0d

If we're using 2.7 should we apply this patch?

You can for now, once we have dev snapshots building again you can just upgrade to the latest snapshot and get the fix. Or you could gitsync, but that's more risky.

watkins13 · Mar 19, 2023, 9:15 PM

I applied the 46b159032fef8c78783aa1a749d2238cfed7ac0d commit patch and can add my own confirmation to the group that the generated DNS access lists are working as expected.

I want to personally thank you @jimp for the work you put into this. :D I'm sure you've been busy with more higher-priority work related to the 23.01 release with how monumental of a change it really is. Changing both the PHP major version platform and language differences at the same time as re-basing work with skipping the entire FreeBSD 13.x series and going straight to the FreeBSD 14 code branch. That's no small feat to change both of those in a single release target, so bravo!