"Starting DNS resolver" startup delay

baah

Hello,

I'm testing pfSense in Hyper-V and it appears that I'm stuck with a problem. It takes too long for pfSense 2.6.0 to start. The delay occurs on the "Starting DNS resolver" step.

I would greatly appreciate any suggestion on how to troubleshoot this problem further.

First I noticed the problem on my two pfSense VMs and it takes ~50 seconds to start DNS resolver. With a clean pfSense VM (right after initial installation and configuration) it takes 10 seconds to start DNS resolver. Shouldn't DNS resolver start instantly?

I guess that I have misconfigured something in pfSense so that it takes 50 seconds for DNS resolver to start. But what about that 10-second delay with a clean pfSense installation?

I've tried disabling DNS forwarder and DNSSec and it doesn't help. Disabling DNS Resolver itself helps eliminate the delay, but this obviously is not a solution.

BTW, I have another problem when pfSense sporadically hangs and becomes unresponsive (absolutely non functional, terminal doesn't work, web console doesn't work, routing etc doesn't work), but I first want to tackle the DNS resolver problem. Sporadically hangs means that I've been testing pfSense for the last 3 days and my both VMs hang several times a day. I have to reboot the VMs to make them work again. I was first assuming that the problem is somehow related to time synchronization in Hyper-V, but the problem persists after disabling "Time synchronization" in VM's properties.

And just FYI, here is a screenshot of pfSense in non-functioning state:

Thank you!

SteveITS

@baah said in "Starting DNS resolver" startup delay:

Shouldn't DNS resolver start instantly

Pretty much. Are you using pfBlocker with large lists maybe?

baah

@steveits said in "Starting DNS resolver" startup delay:

@baah said in "Starting DNS resolver" startup delay:

Shouldn't DNS resolver start instantly

Pretty much. Are you using pfBlocker with large lists maybe?

Nope, pfBlocker isn't enabled in my case unless this package is pre-installed or pre-enabled (I assume it's not).

My routers don't have any extra packages installed (the list of packages is empty). I've configured LAN, WAN and SYNC (for high availability) interfaces. All seems to be working, but startup times / DNS resolver startup times are exorbitant for a router.

I had to tune my Hyper-V WAN NIC to disable "Large Send Offload Version 2" and "SoftwareRscEnable" because network downloads were like 10kbyte/sec-100mbyte/sec when the network is capable of much more. With these customizations download speeds are 150-200+ mbytes/sec. But the problem occurs regardless of these changes. I.e., it takes 10 seconds to start DNS resolver on a clean install (without any config customizations from my side).

johnpoz

@baah I just fired up my 2.6 vm, and yeah hit stopwatch when saw that message pop up and its about 10 seconds or so.

Keep in mind that just not starting a service - I believe it creates the chroot environment, and has to build the full config from the xml, and then all the files like the .conf and the host file, etc.

It has to check a bunch of other stuff that it uses to build the config, etc.

You would have to go through the unbound.inc to see all the stuff it does..

example
https://github.com/pfsense/pfsense/blob/master/src/etc/inc/unbound.inc

Could that be optimized for speed - maybe? is there something specific in there causing the slower start up - maybe as well.. I have not spent any time going through all what happens when unbound first starts on a boot..

But does it matter - I could see if it was 5 minutes or something where there might be a concern.. I think there is something that happens with the anchor file as well that could be delayed if no wan, etc.

What sort of start up time of unbound are you hoping for? 5 seconds, 2 seconds?

Gertjan

@baah

If DNSSEC is activated, a helper app ( unbound-anchor ) is started to retrieve the DNSSEC root key file first.

Try this for yourself :

/usr/bin/su -m unbound -c '/usr/local/sbin/unbound-anchor -a /tmp/key -F -v'

I've added the switches -F and -v for more verbose output.

Take note : after running "unbound-anchor -h" :
I presume that unbound-anchor does it's own resolving, using DNS root server hints (the IP addresses are hard coded in the executable so it can boot trap resolving itself as no DNS resolver is available yet on the system).
It's a modern app : it will use IPv6 first, and fall back to IPv4 if that doesn't worked out.
If you suspect IPv6 issues, add a "-4" here, right after the "-a", to force IPv4 usage.

edit :

Welcome to Netgate pfSense Plus 23.01-RELEASE...

No core dumps found.
...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/freeradius-3.0.25 /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.32/mach/CORE
32-bit compatibility ldconfig path:
done.
>>> Removing vital flag from php81... done.
External config loader 1.0 is now starting... nvd0p1 nvd0p2 nvd0p4
Launching the init system...Updating CPU Microcode...
CPU: Intel(R) Atom(TM) CPU C3338R @ 1.80GHz (1800.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506f1  Family=0x6  Model=0x5f  Stepping=1
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2294e283<FSGSBASE,TSCADJ,SMEP,ERMS,NFPUSG,MPX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA>
  Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0xc69<RDCL_NO,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
Done.
 done.
Initializing.................. done.
Starting device manager (devd)...done.
Loading configuration....done.
Updating configuration...done.
Checking config backups consistency.................................done.
Setting up extended sysctls...done.
Setting timezone...done.
Configuring loopback interface...done.
Starting syslog...done.
Starting Secure Shell Services...done.
Setting up interfaces microcode...done.
Configuring loopback interface...done.
Configuring WAN interface...done.
Configuring LAN interface...done.
Configuring IDRAC interface...done.
Configuring PORTAL interface...done.
Configuring CARP settings...done.
Syncing OpenVPN settings...done.
Configuring firewall......done.
Starting PFLOG...done.
Setting up gateway monitors...done.
Setting up static routes...done.
Setting up DNSs... <==== 3 seconds or so 
Starting DNS Resolver...done.
Synchronizing user settings...done.
Configuring CRON...done.
Bootstrapping clock...done.       <==== this took a couple of seconds
Starting NTP Server...done.
Starting webConfigurator...done.
Starting DHCP service...done.
Starting DHCPv6 service...done.
Configuring firewall......done.
Starting captive portal(CPZONE1)... done <==== this took 10 seconds or so, as several portal users were connected while I decide to restart .. 
Enabling voucher support... done <=== strange, voucher support is disabled
Generating RRD graphs...done.
Starting syslog...done.
Configuring filter for dynamic IPsec VPN hosts... done
Starting CRON... done.
 Starting package AWS VPC Wizard...done.
 Starting package IPsec Profile Wizard...done.
 Starting package Netgate Firmware Upgrade...done.
 Starting package acme...done.
 Starting package Cron...done.
 Starting package Notes...done.
 Starting package nut...done.
 Starting package System Patches...done.
 Starting package OpenVPN Client Export Utility...done.
 Starting package freeradius3...done.
 Starting package Shellcmd...
done.
 Starting package Avahi...done.
 Starting package Filer...done.
 Starting package Backup...done.
 Starting package pfBlockerNG-devel...done.
 Starting package OpenVPN Client Import Utility...done.
 Starting package Service Watchdog...done. <==== WTF : forgot about this one, have to remove it asap.
 Starting /usr/local/etc/rc.d/munin-node.sh...done.
 Starting /usr/local/etc/rc.d/pfb_dnsbl.sh...done.
 Starting /usr/local/etc/rc.d/pfb_filter.sh...done.
 Starting /usr/local/etc/rc.d/shutdown.nut.sh...done.
Netgate pfSense Plus 23.01-RELEASE amd64 Fri Feb 10 20:06:33 UTC 2023
Bootup complete

The entire reroot sequence : from kernel loaded to boot menu shown : 30 seconds ?

I've several pfSense packages, notably FreeRadius