No DNS after upgrading from 23.01 to 23.05 - unbound issue?

haraldinho

@haraldinho I did some further digging.

I managed to get it working when disabling "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers". However, the system was very sluggish.

I then changed the DNS servers to Google's and enabled "Use SSL/TLS for outgoing DNS Queries to Forwarding Servers" again. That worked too, however, the system was still sluggish.

I then observed something weird checking top: check_reload_status is eating my CPU...

last pid: 78728;  load averages:  1.53,  1.40,  1.19                                                                                                                         up 0+00:28:07  21:54:34
96 processes:  2 running, 94 sleeping
CPU:  5.8% user, 11.3% nice, 15.3% system,  0.0% interrupt, 67.6% idle
Mem: 611M Active, 260M Inact, 683M Wired, 56K Buf, 6232M Free
ARC: 281M Total, 26M MFU, 245M MRU, 548K Anon, 1369K Header, 7529K Other
     237M Compressed, 574M Uncompressed, 2.42:1 Ratio
Swap: 1024M Total, 1024M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
  431 root          1 154   20    13M  2976K CPU3     3  26:26  98.81% check_reload_status
84529 root          8  20    0   420M   339M nanslp   2   8:31  26.26% suricata
51433 myuser        1  20    0    14M  3940K CPU2     2   0:00   0.18% top
82461 unbound       4  20    0   104M    70M kqread   2   0:05   0.07% unbound
48288 myuser        1  20    0    21M  9960K select   2   0:00   0.03% sshd
46701 root          1  20    0    24M  9120K select   1   0:00   0.02% mpd5
37389 zabbix        1  20    0    24M    10M nanslp   3   0:00   0.01% zabbix_agentd
18239 root          5  68    0    13M  2848K uwait    0   0:00   0.01% dpinger
96372 root          1  20    0    12M  2308K select   3   0:00   0.01% powerd
85553 root          1  20    0    21M  7508K select   1   0:00   0.01% ntpd
95822 dhcpd         1  20    0    25M    13M select   0   0:00   0.01% dhcpd
37512 root          1  20    0    45M    35M bpf      2   0:00   0.01% arpwatch
37309 root          1  20    0    45M    35M bpf      3   0:00   0.01% arpwatch
  392 root          1  20    0   112M    30M kqread   0   0:00   0.00% php-fpm

Output of ps uxawww

[23.05-RELEASE][myuser@myrouter]/home/myuser: ps uxawww
USER         PID  %CPU %MEM    VSZ    RSS TT  STAT STARTED     TIME COMMAND
root          11 277.0  0.0      0     64  -  RNL  21:26   50:22.51 [idle]
root         431 100.0  0.0  13244   2976  -  RNs  21:26   17:44.27 /usr/local/sbin/check_reload_status
root       84529  26.9  4.2 424212 346632  -  Ss   21:27    5:45.64 /usr/local/bin/suricata -i ix1 -D -c /usr/local/etc/suricata/suricata_281_ix1/suricata.yaml --pidfile /var/run/suricata_ix1281.pid
root           0   4.1  0.0      0   1520  -  DLs  21:26    1:15.90 [kernel]
root           1   0.0  0.0  11352   1216  -  SLs  21:26    0:00.09 /sbin/init
root           2   0.0  0.0      0     64  -  WL   21:26    0:00.44 [clock]
root           3   0.0  0.0      0     80  -  DL   21:26    0:00.00 [crypto]
root           4   0.0  0.0      0     48  -  DL   21:26    0:00.00 [cam]
root           5   0.0  0.0      0     16  -  DL   21:26    0:00.00 [busdma]
root           6   0.0  0.0      0    928  -  DL   21:26    0:00.71 [zfskern]
root           7   0.0  0.0      0     16  -  DL   21:26    0:00.30 [pf purge]
root           8   0.0  0.0      0     16  -  DL   21:26    0:00.13 [rand_harvestq]
root           9   0.0  0.0      0     16  -  DL   21:26    0:00.00 [mmcsd0: mmc/sd card]
root          10   0.0  0.0      0     16  -  DL   21:26    0:00.00 [audit]
root          12   0.0  0.0      0    288  -  WL   21:26    0:01.54 [intr]
root          13   0.0  0.0      0     64  -  DL   21:26    0:00.02 [ng_queue]
root          14   0.0  0.0      0     48  -  DL   21:26    0:00.00 [geom]
root          15   0.0  0.0      0     16  -  DL   21:26    0:00.00 [sequencer 00]
root          16   0.0  0.0      0     80  -  DL   21:26    0:00.02 [usb]
root          17   0.0  0.0      0     16  -  DL   21:26    0:00.01 [acpi_thermal]
root          18   0.0  0.0      0     16  -  DL   21:26    0:00.00 [acpi_cooling0]
root          19   0.0  0.0      0     16  -  DL   21:26    0:00.00 [mmcsd0boot0: mmc/sd]
root          20   0.0  0.0      0     16  -  DL   21:26    0:00.00 [mmcsd0boot1: mmc/sd]
root          21   0.0  0.0      0     48  -  DL   21:26    0:00.20 [pagedaemon]
root          22   0.0  0.0      0     16  -  DL   21:26    0:00.00 [vmdaemon]
root          23   0.0  0.0      0     80  -  DL   21:26    0:00.03 [bufdaemon]
root          24   0.0  0.0      0     16  -  DL   21:26    0:00.01 [vnlru]
root          25   0.0  0.0      0     16  -  DL   21:26    0:00.01 [syncer]
root          26   0.0  0.0      0     16  -  DL   21:26    0:00.00 [ALQ Daemon]
root          28   0.0  0.0  12672   2248  -  Is   21:27    0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
root         392   0.0  0.4 114208  30480  -  Ss   21:26    0:00.04 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root         433   0.0  0.0  13244   2692  -  IN   21:26    0:00.00 check_reload_status: Monitoring daemon of check_reload_status (check_reload_status)
root         450   0.0  0.0  12672   2272  -  I    21:27    0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts  (minicron)
root         717   0.0  0.0  12672   2244  -  Is   21:27    0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data
root         859   0.0  0.1  14364   4780  -  Ss   21:26    0:00.22 /sbin/devd -q -f /etc/pfSense-devd.conf
root        1065   0.0  0.0  12672   2268  -  I    21:27    0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data  (minicron)
root        6755   0.0  0.1  21020   9632  -  Ss   21:45    0:00.03 sshd: myuser [priv] (sshd)
root        8712   0.0  0.0  13416   3180  -  I    21:29    0:00.01 /bin/sh /usr/local/sbin/pfSense-repo-setup
root       11482   0.0  0.1  17064   5080  -  I    21:29    0:00.03 /usr/local/sbin/pfSense-repoc-static (pfSense-repoc-stati)
root       16125   0.0  0.9 150060  74852  -  I    21:27    0:00.31 php-fpm: pool nginx (php-fpm)
root       17459   0.0  0.9 150060  74528  -  I    21:27    0:00.64 php-fpm: pool nginx (php-fpm)
root       18239   0.0  0.0  13520   2848  -  Is   21:27    0:00.17 /usr/local/bin/dpinger -S -r 0 -i WAN_PPPOE -B xxx.xxx.xxx.xxx -p /var/run/dpinger_WAN_PPPOE~xxx.xxx.xxx.xxx~195.190.228.37.pid -u /var/run/dpinger_WAN_PPPOE~xxx.xxx.xxx.xxx~195.190.228.37.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 195.190.228.37
root       22644   0.0  0.9 150060  74992  -  S    21:27    0:01.46 php-fpm: pool nginx (php-fpm)
root       23252   0.0  0.1  20816   8868  -  Ss   21:27    0:00.00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
root       24878   0.0  0.0  12680   2396  -  Is   21:27    0:00.01 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d myown.lan -p /var/run/unbound.pid -u /var/unbound/dhcpleases_entries.conf -h /etc/hosts
root       28490   0.0  0.0  12872   2676  -  Is   21:27    0:00.02 /usr/sbin/cron -s
root       28747   0.0  0.9 148012  74600  -  I    21:27    0:00.52 php-fpm: pool nginx (php-fpm)
root       29386   0.0  0.0  13416   3180  -  I    21:29    0:00.01 /bin/sh /usr/local/sbin/pfSense-repo-setup
root       35112   0.0  0.0  13336   3028  -  Is   21:34    0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid
root       35521   0.0  0.0  12676   2364  -  S    21:34    0:00.00 /bin/cat
root       35525   0.0  0.1  19336   5496  -  SC   21:34    0:00.00 /usr/local/libexec/sshg-parser
root       35703   0.0  0.0  13280   2904  -  IC   21:34    0:00.00 /usr/local/libexec/sshg-blocker
root       35876   0.0  0.0  13336   3036  -  I    21:34    0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid
root       36064   0.0  0.0  13336   3024  -  I    21:34    0:00.00 /bin/sh /usr/local/libexec/sshg-fw-pf
root       36552   0.0  0.4  45884  36304  -  Ss   21:28    0:00.05 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_ix1.dat -i ix1 -w myemail@mydomain.com
root       37023   0.0  0.4  45884  36304  -  Ss   21:28    0:00.05 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_ix1.70.dat -i ix1.70 -w myemail@mydomain.com
zabbix     37226   0.0  0.1  24516  10100  -  I    21:27    0:00.00 /usr/local/sbin/zabbix_agentd -c /usr/local/etc/zabbix62/zabbix_agentd.conf
root       37309   0.0  0.4  45884  36304  -  Ss   21:28    0:00.05 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_ix1.200.dat -i ix1.200 -w myemail@mydomain.com
zabbix     37389   0.0  0.1  24516  10472  -  S    21:27    0:00.12 zabbix_agentd: collector [idle 1 sec] (zabbix_agentd)
zabbix     37392   0.0  0.1  24652  10688  -  S    21:27    0:00.50 zabbix_agentd: listener #1 [waiting for connection] (zabbix_agentd)
root       37512   0.0  0.4  45884  36304  -  Ss   21:28    0:00.05 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_ix1.30.dat -i ix1.30 -w myemail@mydomain.com
zabbix     37524   0.0  0.1  24652  10756  -  S    21:27    0:00.52 zabbix_agentd: listener #2 [waiting for connection] (zabbix_agentd)
root       37607   0.0  0.4  45884  36308  -  Ss   21:28    0:00.05 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_ix1.40.dat -i ix1.40 -w myemail@mydomain.com
zabbix     37842   0.0  0.1  24652  10588  -  S    21:27    0:00.50 zabbix_agentd: listener #3 [waiting for connection] (zabbix_agentd)
zabbix     38167   0.0  0.1  24652  10392  -  S    21:27    0:00.08 zabbix_agentd: active checks #1 [idle 1 sec] (zabbix_agentd)
root       38276   0.0  0.4  45884  36304  -  Ss   21:28    0:00.05 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_ix1.50.dat -i ix1.50 -w myemail@mydomain.com
root       38667   0.0  0.1  17064   5076  -  I    21:30    0:00.03 /usr/local/sbin/pfSense-repoc-static (pfSense-repoc-stati)
root       38862   0.0  0.4  45884  36308  -  Ss   21:28    0:00.05 /usr/local/sbin/arpwatch -Z -f /usr/local/arpwatch/arp_ix1.60.dat -i ix1.60 -w myemail@mydomain.com
root       40784   0.0  0.9 150060  74472  -  I    21:27    0:00.28 php-fpm: pool nginx (php-fpm)
root       46701   0.0  0.1  24224   9120  -  Ss   21:27    0:00.16 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient
root       46998   0.0  0.0  12672   2164  -  SNC  21:45    0:00.00 sleep 60
myuser     48288   0.0  0.1  21020   9960  -  S    21:45    0:00.00 sshd: myuser@pts/0 (sshd)
root       51143   0.0  0.0  12832   3064  -  Ss   21:27    0:00.40 /usr/sbin/syslogd -O rfc5424 -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
root       55651   0.0  0.0  12712   2344  -  S    21:27    0:00.03 /usr/local/sbin/igmpproxy /var/etc/igmpproxy.conf
root       57587   0.0  0.0  13416   3184  -  I    21:37    0:00.01 /bin/sh /usr/local/sbin/pfSense-repo-setup
root       59919   0.0  0.1  17064   5076  -  I    21:37    0:00.03 /usr/local/sbin/pfSense-repoc-static (pfSense-repoc-stati)
root       60355   0.0  0.0  12956   2716  -  Is   21:27    0:00.00 dhclient: ix3.4 [priv] (dhclient)
root       62452   0.0  0.1  16792   6084  -  Is   21:28    0:00.00 /usr/local/sbin/upsmon
_dhcp      62504   0.0  0.0  12960   2912  -  Is   21:27    0:00.05 dhclient: ix3.4 (dhclient)
nut        62625   0.0  0.1  16928   6412  -  S    21:28    0:00.03 /usr/local/sbin/upsmon
root       67682   0.0  0.0  12820   2592  -  Is   21:27    0:00.00 /usr/local/sbin/dhcp6c -d -c /var/etc/dhcp6c.conf -p /var/run/dhcp6c.pid pppoe0
root       77058   0.0  0.0  13336   3004  -  SN   21:27    0:00.32 /bin/sh /var/db/rrd/updaterrd.sh
unbound    82461   0.0  0.9 106492  71632  -  Ss   21:27    0:03.90 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
root       82671   0.0  0.1  15032   5428  -  Ss   21:28    0:00.30 /usr/local/sbin/vnstatd -d -p /var/run/vnstat/vnstat.pid --config /usr/local/etc/vnstat.conf
root       84525   0.0  0.0  13416   3172  -  I    21:34    0:00.01 /bin/sh /usr/local/sbin/pfSense-repo-setup
root       85553   0.0  0.1  22000   7508  -  Ss   21:27    0:00.16 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
root       86095   0.0  0.1  29060   8400  -  Is   21:27    0:00.00 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
root       86321   0.0  0.1  31620   9744  -  I    21:27    0:00.10 nginx: worker process (nginx)
root       86345   0.0  0.9 152980  77468  -  I    21:27    0:01.27 php-fpm: pool nginx (php-fpm)
root       86520   0.0  0.1  31620  10776  -  S    21:27    0:00.34 nginx: worker process (nginx)
root       86958   0.0  0.1  17064   5076  -  I    21:34    0:00.03 /usr/local/sbin/pfSense-repoc-static (pfSense-repoc-stati)
root       87524   0.0  0.0  13428   3604  -  Ss   21:27    0:00.15 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root       90583   0.0  0.1  18484   7856  -  Ss   21:27    0:00.29 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1/config.ovpn
root       93835   0.0  0.0  38632   3748  -  Is   21:27    0:00.02 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
dhcpd      95822   0.0  0.2  25744  13336  -  Ss   21:27    0:00.13 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid ix1 ix1.30 ix1.40 ix1.50 ix1.60 ix1.70 ix1.80
root       95853   0.0  0.1  17008   6276  -  Ss   21:27    0:00.11 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid
root       96372   0.0  0.0  12776   2308  -  Ss   21:27    0:00.09 /usr/sbin/powerd -b hadp -a hadp -n hadp
root       98545   0.0  0.0  12672   2248  -  Is   21:27    0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
root       98937   0.0  0.0  12672   2268  -  I    21:27    0:00.00 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
root       99282   0.0  0.0  12672   2244  -  Is   21:27    0:00.00 /usr/local/bin/minicron 300 /var/run/ipsec_keepalive.pid /usr/local/bin/ipsec_keepalive.php
root       99296   0.0  0.9 150060  74544  -  I    21:27    0:00.40 php-fpm: pool nginx (php-fpm)
root       99895   0.0  0.0  12672   2264  -  I    21:27    0:00.00 minicron: helper /usr/local/bin/ipsec_keepalive.php  (minicron)
root       19698   0.0  0.0  13336   2808 u0- I    21:27    0:00.00 /bin/sh /etc/rc.update_pkg_metadata now
root       20522   0.0  0.0  13336   2804 u0- I    21:27    0:00.00 /bin/sh /usr/local/sbin/pfSense-upgrade -uf
root       22488   0.0  0.0  12672   2156 u0- I    21:27    0:00.00 /usr/bin/lockf -s -t 5 /tmp/pfSense-upgrade.lock /usr/local/libexec/pfSense-upgrade -uf
root       22828   0.0  0.0  13336   3176 u0- I    21:27    0:00.01 /bin/sh /usr/local/libexec/pfSense-upgrade -uf
root       31184   0.0  0.0  13336   2772 u0- I    21:27    0:00.00 /bin/sh /usr/local/sbin/pfSense-repo-setup
root       34658   0.0  0.1  17064   5064 u0- I    21:27    0:00.04 /usr/local/sbin/pfSense-repoc-static (pfSense-repoc-stati)
root       39273   0.0  0.0  13216   2896 u0  Is   21:27    0:00.01 login [pam] (login)
root       39506   0.0  0.0  13336   3300 u0  I    21:27    0:00.01 -sh (sh)
root       54651   0.0  0.0  13336   3040 u0  I+   21:27    0:00.00 /bin/sh /etc/rc.initial
root       37956   0.0  0.0  13216   2892 v0  Is   21:27    0:00.01 login [pam] (login)
root       39754   0.0  0.0  13336   3296 v0  I    21:27    0:00.01 -sh (sh)
root       41655   0.0  0.0  13336   3040 v0  I+   21:27    0:00.00 /bin/sh /etc/rc.initial
root       38116   0.0  0.0  12800   2344 v1  Is+  21:27    0:00.00 /usr/libexec/getty Pc ttyv1
root       38214   0.0  0.0  12800   2348 v2  Is+  21:27    0:00.00 /usr/libexec/getty Pc ttyv2
root       38217   0.0  0.0  12800   2344 v3  Is+  21:27    0:00.00 /usr/libexec/getty Pc ttyv3
root       38564   0.0  0.0  12800   2344 v4  Is+  21:27    0:00.00 /usr/libexec/getty Pc ttyv4
root       38875   0.0  0.0  12800   2348 v5  Is+  21:27    0:00.00 /usr/libexec/getty Pc ttyv5
root       38890   0.0  0.0  12800   2352 v6  Is+  21:27    0:00.00 /usr/libexec/getty Pc ttyv6
root       39224   0.0  0.0  12800   2348 v7  Is+  21:27    0:00.00 /usr/libexec/getty Pc ttyv7
myuser     48413   0.0  0.1  13804   4268  0  Ss   21:45    0:00.02 -tcsh (tcsh)
myuser     55806   0.0  0.0  13400   3256  0  R+   21:45    0:00.00 ps uxawww

Who sees what's going wrong in my setup?

SteveITS

@haraldinho See Jim-p’s comment here: https://www.reddit.com/r/PFSENSE/comments/5cm6sg/usrlocalsbincheck_reload_status_using_100_of_my/

Is your WAN going down/up?

haraldinho

@SteveITS I'm not sure why it happened, but the high load seems to have resolved itself after one of the many reboots I have done. The original problem remains however, DNS remains behaving erratic.

I changed the DNS from Google to Cloudflare and re-enabled DNS over TLS to see if that would help, but it doesn't.

Things I notice:

Package manager is not able to retrieve list of available packages. Dashboard is not able to load 'Latest version' to determine if there is an update.
Time it takes for 127.0.0.1 to resolve is long (more than 10 seconds is not unusual, sometimes it does not respond at all see below screenshot).
The forwarding part seems to work properly and fast, see below log

Schermafbeelding 2023-06-11 om 14.24.15.png

2023-06-11 14:21:49.471930+02:00	unbound	36056	[36056:0] debug: cache memory msg=80110 rrset=78875 infra=8306 val=0
2023-06-11 14:21:49.471762+02:00	unbound	36056	[36056:0] info: finishing processing for gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.471659+02:00	unbound	36056	[36056:0] info: query response was ANSWER
2023-06-11 14:21:49.471618+02:00	unbound	36056	[36056:0] info: reply from <.> 1.1.1.2#853
2023-06-11 14:21:49.471566+02:00	unbound	36056	[36056:0] info: response for gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.471512+02:00	unbound	36056	[36056:0] info: iterator operate: chased to get-bx.g.aaplimg.com. A IN
2023-06-11 14:21:49.471481+02:00	unbound	36056	[36056:0] info: iterator operate: query gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.471439+02:00	unbound	36056	[36056:0] debug: iterator[module 0] operate: extstate:module_wait_reply event:module_event_reply
2023-06-11 14:21:49.470854+02:00	unbound	36056	[36056:3] debug: cache memory msg=80110 rrset=78875 infra=8306 val=0
2023-06-11 14:21:49.470708+02:00	unbound	36056	[36056:3] info: finishing processing for gsp-ssl.ls-apple.com.akadns.net. HTTPS IN
2023-06-11 14:21:49.470666+02:00	unbound	36056	[36056:3] info: resolving gsp-ssl.ls-apple.com.akadns.net. HTTPS IN
2023-06-11 14:21:49.470621+02:00	unbound	36056	[36056:3] info: resolving gsp-ssl.ls-apple.com.akadns.net. HTTPS IN
2023-06-11 14:21:49.470572+02:00	unbound	36056	[36056:3] info: resolving gsp-ssl.ls-apple.com.akadns.net. HTTPS IN
2023-06-11 14:21:49.470519+02:00	unbound	36056	[36056:3] info: query response was CNAME
2023-06-11 14:21:49.470489+02:00	unbound	36056	[36056:3] info: reply from <.> 1.1.1.2#853
2023-06-11 14:21:49.470456+02:00	unbound	36056	[36056:3] info: response for gsp-ssl.ls-apple.com.akadns.net. HTTPS IN
2023-06-11 14:21:49.470419+02:00	unbound	36056	[36056:3] info: sanitize: removing extraneous answer RRset: gspx-ssl.ls.apple.com. CNAME IN
2023-06-11 14:21:49.470386+02:00	unbound	36056	[36056:3] info: sanitize: removing extraneous answer RRset: gsp-ssl-geomap.ls-apple.com.akadns.net. CNAME IN
2023-06-11 14:21:49.470337+02:00	unbound	36056	[36056:3] info: iterator operate: query gsp-ssl.ls-apple.com.akadns.net. HTTPS IN
2023-06-11 14:21:49.470291+02:00	unbound	36056	[36056:3] debug: iterator[module 0] operate: extstate:module_wait_reply event:module_event_reply
2023-06-11 14:21:49.465750+02:00	unbound	36056	[36056:0] debug: cache memory msg=80110 rrset=78875 infra=8306 val=0
2023-06-11 14:21:49.465700+02:00	unbound	36056	[36056:0] debug: sending to target: <.> 1.1.1.2#853
2023-06-11 14:21:49.465667+02:00	unbound	36056	[36056:0] info: sending query: get-bx.g.aaplimg.com. A IN
2023-06-11 14:21:49.465624+02:00	unbound	36056	[36056:0] info: processQueryTargets: gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.465575+02:00	unbound	36056	[36056:0] info: resolving gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.465529+02:00	unbound	36056	[36056:0] info: resolving gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.465479+02:00	unbound	36056	[36056:0] info: resolving gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.465427+02:00	unbound	36056	[36056:0] info: query response was CNAME
2023-06-11 14:21:49.465395+02:00	unbound	36056	[36056:0] info: reply from <.> 1.1.1.2#853
2023-06-11 14:21:49.465363+02:00	unbound	36056	[36056:0] info: response for gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.465327+02:00	unbound	36056	[36056:0] info: sanitize: removing extraneous answer RRset: get-bx.g.aaplimg.com. A IN
2023-06-11 14:21:49.465296+02:00	unbound	36056	[36056:0] info: sanitize: removing extraneous answer RRset: gspx-ssl.ls.apple.com. CNAME IN
2023-06-11 14:21:49.465262+02:00	unbound	36056	[36056:0] info: sanitize: removing extraneous answer RRset: gsp-ssl-geomap.ls-apple.com.akadns.net. CNAME IN
2023-06-11 14:21:49.465206+02:00	unbound	36056	[36056:0] info: iterator operate: query gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.465160+02:00	unbound	36056	[36056:0] debug: iterator[module 0] operate: extstate:module_wait_reply event:module_event_reply
2023-06-11 14:21:49.457484+02:00	unbound	36056	[36056:0] debug: cache memory msg=80110 rrset=78875 infra=8306 val=0
2023-06-11 14:21:49.457433+02:00	unbound	36056	[36056:3] debug: cache memory msg=80110 rrset=78875 infra=8306 val=0
2023-06-11 14:21:49.457381+02:00	unbound	36056	[36056:0] debug: sending to target: <.> 1.1.1.2#853
2023-06-11 14:21:49.457331+02:00	unbound	36056	[36056:3] debug: sending to target: <.> 1.1.1.2#853
2023-06-11 14:21:49.457286+02:00	unbound	36056	[36056:0] info: sending query: gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.457241+02:00	unbound	36056	[36056:3] info: sending query: gsp-ssl.ls-apple.com.akadns.net. HTTPS IN
2023-06-11 14:21:49.457196+02:00	unbound	36056	[36056:0] info: processQueryTargets: gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.457160+02:00	unbound	36056	[36056:3] info: processQueryTargets: gsp-ssl.ls-apple.com.akadns.net. HTTPS IN
2023-06-11 14:21:49.457042+02:00	unbound	36056	[36056:0] info: resolving gsp-ssl.ls-apple.com.akadns.net. A IN
2023-06-11 14:21:49.456976+02:00	unbound	36056	[36056:3] info: resolving gsp-ssl.ls-apple.com.akadns.net. HTTPS IN
2023-06-11 14:21:49.456906+02:00	unbound	36056	[36056:0] debug: iterator[module 0] operate: extstate:module_state_initial event:module_event_new
2023-06-11 14:21:49.456824+02:00	unbound	36056	[36056:3] debug: iterator[module 0] operate: extstate:module_state_initial event:module_event_new

I still feel that unbound itself is the culprit, but it does not spit out any useful log entries.

SteveITS

@haraldinho is IPv6 working on the router itself? If it isn’t connections will fail if it tries IPv6 first. There is a checkbox in the pfSense settings to prefer IPv4 for (only) the router.

Does it work if you disable forwarding temporarily?

haraldinho

@SteveITS Hey Steve, I appreciate your help. I just tried both settings that you indicated, but they do not resolve the issue. DNS is still erratic and I cannot see available packages in Package Manager and also it does not show the update status on the dashboard.

I had disabled all IPv6 on all interfaces previously.

I was going through the output of ps -auxwwd and my eye caught this section, I am not sure if it is related:

root       46599   0.0  0.0  13336  2820 u0- I    15:02    0:00.00 |-- /bin/sh /etc/rc.update_pkg_metadata now
root       47042   0.0  0.0  13336  2812 u0- I    15:02    0:00.00 | `-- /bin/sh /usr/local/sbin/pfSense-upgrade -uf
root       49188   0.0  0.0  12672  2160 u0- I    15:02    0:00.00 |   `-- /usr/bin/lockf -s -t 5 /tmp/pfSense-upgrade.lock /usr/local/libexec/pfSense-upgrade -uf
root       49640   0.0  0.0  13336  3180 u0- I    15:02    0:00.01 |     `-- /bin/sh /usr/local/libexec/pfSense-upgrade -uf
root       56944   0.0  0.0  13336  2780 u0- I    15:02    0:00.00 |       `-- /bin/sh /usr/local/sbin/pfSense-repo-setup
root       59363   0.0  0.1  17064  5068 u0- I    15:02    0:00.04 |         `-- /usr/local/sbin/pfSense-repoc-static (pfSense-repoc-stati)

It is still there after a reboot. Not sure what it exactly indicates, but it suggests the upgrade did not completely finish? Any thoughts?

Is there any other logging that I could inspect that would be able to shed more light on the DNS matter?

haraldinho

@SteveITS @stephenw10 The issue in this post looks really very much similar to my problem.

TAC57

@haraldinho Our problems look very similar!

My Package manager is not able to retrieve a list of packages. I've tried to delete packages I don't need and I can't.
I can't load Latest version.
System is very sluggish going from tab to tab.

Are you also getting this Notice: 'An error occurred while uploading the encrypted Netgate pfSense Plus configuration to https://acb.netgate.com/save ( Unable to resolve acb.netgate.com ) @ 2023-06-09 18:26:04'

If you come up with something I'd really appreciate hearing from you.

Can you recommend absolute basic DNS Resolver settings just to see if I can get this working.

Gertjan

@TAC57 said in No DNS after upgrading from 23.01 to 23.05 - unbound issue?:

Can you recommend absolute basic DNS Resolver settings

The default resolver settings.
The one Netgate has put in place when you've installed pfSense.

Note : The certificate selected isn't important, just pick 'one'.
You can leave the Custom option box empty.

and :

If your uplink isn't discriminating your "Internet" access (read : your ISP isn't messing around with 'what you are allowed to visit - and what not) these settings work perfectly well.

haraldinho

@Gertjan Are the settings in the images the default 23.05 resolver settings or are they representing your working 23.05 configuration?
Are you not forwarding?

Gertjan

@haraldinho said in No DNS after upgrading from 23.01 to 23.05 - unbound issue?:

or are they representing your working 23.05 configuration?

Good question.
The first image : 99 % default, as I'm pretty sure about all these settings.

The second image : I enforce DNSSEC, and keep the local cache updated when resolved host names TTL reaches zero.

Of course I'm not forwarding, as I don't feel the need to hand over all DNS request to some other company. I never understood why I have to do this. I chose pfSense because it can resolve for itself.

Edit : the settings I'm using are pretty much the same for the last .... decade.
I've tested forwarding to 1.1.1.1, 8.8.8.8 etc to port 53 and 853 (TLS) - mixing IPv4 and Ipv6.
Worked perfectly fine for me.
I'm using a Netgate 4100 - and my ISP doesn't play tricks on me ( I guess ).

TAC57

@Gertjan I made all the changes above, rebooted, and thought I was in business. I could get my package list and could (did) delete a number of packages I don't need that I couldn't delete previously. DNS Lookup responses looked good. I didn't have a notice that an error occurred while upload pfSense config.

After logging off and back on I'm back to the way it was. One difference though, DNS Lookup no reports back immediately instead of taking awhile. I'm sure this is because Name server is 127.0.0.1.

Gertjan

@TAC57

"1 ms" because "google.com was already in the local DNS (resolver) cache.

More typical :

You should see hundreds if not thousands if entries here :

This line (arrow) :

should occur as less frequent as possible.

haraldinho

@Gertjan @TAC57 Have a look at this topic. This is exactly what we are experiencing in my opinion. There seems also to be a solution by turning off ASLR, but I need to dig into it more. It's a long thread, I need to read it carefully.

TAC57

@haraldinho Jeez, that is a long thread, thanks for pointing it out! Per my other topic I jumped back to 2.6.0 on my backup box and everything is working just fine now.

How do I turn off 'ASLR'? I'm be glad to jump over to my 23.05 install and give it a try.

Thanks again.

haraldinho

@TAC57 Search the thread, it is in there somewhere

Gertjan

@haraldinho said in No DNS after upgrading from 23.01 to 23.05 - unbound issue?:

@Gertjan @TAC57 Have a look at this topic. This is exactly what we are experiencing in my opinion. There seems also to be a solution by turning off ASLR, but I need to dig into it more. It's a long thread, I need to read it carefully

I don't need to re read that thread - I was posting there.

Btw : Before, using 23.01 the ALSR bit of my /usr/local/sbin/unbound as unset.
Now, using 23.05, it is set.

[23.05-RELEASE][root@pfSense.going.down]/root: elfctl /usr/local/sbin/unbound
File '/usr/local/sbin/unbound' features:
noaslr          'Disable ASLR' is set.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user VA to 48bit' is unset.

But, as shown in the other tread, I could not find any issues while using unbound in forwarding mode to 8.8.8.8 or 1.1.1.1. I've been forwarding for two weeks, never had any issues.

I'm back to resolving mode as it is 'less hassle' and 'works out of the box'. That is, if your uplink (ISP) isn't playing tricks on you.

haraldinho

@Gertjan @TAC57 @SteveITS There seems to be some good news: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270912

"Jaap Akkerhuis 2023-06-01 12:41:18 UTC
A fix is developed by upstairs. There will be a new release within weeks with this fix. For the inpatients among us, a prerelease is made available https://github.com/NLnetLabs/unbound/issues/887#issuecomment-1570136710."