DNS crashing every ~ 36 hours or so and unbound has to be restarted.

gawainxx

This post is deleted!

gawainxx

@Gertjan
Sorry, I had a derp moment and as accessing the wrong server.


##########################
# Unbound Configuration
##########################

##
# Server configuration
##
server:

chroot: /var/unbound
username: "unbound"
directory: "/var/unbound"
pidfile: "/var/run/unbound.pid"
use-syslog: yes
port: 53
verbosity: 1
hide-identity: yes
hide-version: yes
harden-glue: yes
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
do-daemonize: yes
module-config: "validator iterator"
unwanted-reply-threshold: 0
num-queries-per-thread: 4096
jostle-timeout: 200
infra-host-ttl: 900
infra-cache-numhosts: 10000
outgoing-num-tcp: 10
incoming-num-tcp: 10
edns-buffer-size: 4096
cache-max-ttl: 86400
cache-min-ttl: 0
harden-dnssec-stripped: yes
msg-cache-size: 4m
rrset-cache-size: 8m

num-threads: 2
msg-cache-slabs: 2
rrset-cache-slabs: 2
infra-cache-slabs: 2
key-cache-slabs: 2
outgoing-range: 4096
#so-rcvbuf: 4m
auto-trust-anchor-file: /var/unbound/root.key
prefetch: no
prefetch-key: no
use-caps-for-id: no
serve-expired: no
# Statistics
# Unbound Statistics
statistics-interval: 0
extended-statistics: yes
statistics-cumulative: yes

# TLS Configuration
tls-cert-bundle: "/etc/ssl/cert.pem"

# Interface IP(s) to bind to
interface-automatic: yes
interface: 0.0.0.0
interface: ::0

# Outgoing interfaces to be used

# DNS Rebinding
# For DNS Rebinding prevention
private-address: 10.0.0.0/8
private-address: ::ffff:a00:0/104
private-address: 172.16.0.0/12
private-address: ::ffff:ac10:0/108
private-address: 169.254.0.0/16
private-address: ::ffff:a9fe:0/112
private-address: 192.168.0.0/16
private-address: ::ffff:c0a8:0/112
private-address: fd00::/8
private-address: fe80::/10
# Set private domains in case authoritative name server returns a Private IP address
private-domain: "_msdcs.britannia2.local"
domain-insecure: "_msdcs.britannia2.local"
private-domain: "britannia2.local"
domain-insecure: "britannia2.local"


# Access lists
include: /var/unbound/access_lists.conf

# Static host entries
include: /var/unbound/host_entries.conf

# dhcp lease entries
include: /var/unbound/dhcpleases_entries.conf



# Domain overrides
include: /var/unbound/domainoverrides.conf


# Unbound custom options
server: 
# Allow plex to work over LAN
private-domain: "plex.direct"
# Configuration for Britannia2.local with the PDC of mordred.britannia2.local
local-data: "_ldap._tcp.your.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.pdc._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.gc._msdcs.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.gc._msdcs.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_ldap._tcp.30e36ab8-a6ac-4c64-85aa-0fbeb612a33b.domains._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "d4f866aa-a210-4c29-81a2-ebb256bdef7d._msdcs.britannia2.local 600 IN CNAME mordred.britannia2.local"
local-data: "_kerberos._tcp.dc._msdcs.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_ldap._tcp.dc._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_kerberos._tcp.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_kerberos._tcp.Default-First-Site-Name._sites.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_gc._tcp.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_gc._tcp.Default-First-Site-Name._sites.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_kerberos._udp.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_kpasswd._tcp.britannia2.local 600 IN SRV 0 100 464 mordred.britannia2.local"
local-data: "_kpasswd._udp.britannia2.local 600 IN SRV 0 100 464 mordred.britannia2.local"
local-data: "_ldap._tcp.ForestDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.ForestDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.DomainDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.DomainDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "britannia2.local 600 IN A 192.168.4.5"
local-data: "britannia2.local  600 IN A 192.168.4.5"
local-data: "gc._msdcs.britannia2.local 600 IN A 192.168.4.5"
local-data: "gc._msdcs.britannia2.local 600 IN A 192.168.4.5"
local-data: "ForestDnsZones.britannia2.local 600 IN A 192.168.4.5"
local-data: "ForestDnsZones.britannia2.local 600 IN A 192.168.4.5"
local-data: "DomainDnsZones.britannia2.local 600 IN A 192.168.4.5"
local-data: "DomainDnsZones.britannia2.local 600 IN A 192.168.4.5"


###
# Remote Control Config
###
include: /var/unbound/remotecontrol.conf

Gertjan

Looks pretty normal to me.

stephenw10

Do you have pfBlocker installer with DNS-BL enabled? I don't see it in the conf file but that would update the file potentially causing a problem.

Steve

Gertjan

@gawainxx said in DNS crashing every ~ 36 hours or so and unbound has to be restarted.:

config file

https://github.com/NLnetLabs/unbound/blob/e828d678bafb7ef0df32623f6883bc4bdc07dc5b/daemon/unbound.c#L664

The config file is actually ok /unbound.conf - this file named is prefixed with with chrooted dir.
The chroot went wrong ? => File system errors ?

gawainxx

Died again, help plox!

stephenw10

Need more info to help further. What's logged in the system log when it fails? Or just before it fails?

gawainxx

@stephenw10 I'll grab those logs for you in a bit once I'm able to access my network again , am currently remote and am locked out due to the issue.

My logs are divided by process within Splunk.
Is there a specific process that would have the most relevant log data?

gawainxx

Here are some logs, they are csvs renamed to .txt
Unbound logs from 10:40am - 3 PM
1576040675_650.txt
System" logs from 10:40am - 3 PM
1576040737_651.txt

I've also purged the DC related entries in my unbound config to see if that perhaps makes a change as I really only use that for labs/training stuff. Also adjusted my firewall rules so that I can access the router webUI from it, would have saved myself a lot of headache if I could have just restarted it via Ovpn.

Gertjan

unbound is stopped and restarted.

More logs are needed to see which process is doing this. It could also be a hardware event like a "LINK UP / LINK UP"

Btw : this "plunked" unbound log is close to totally unreadable : possible to see the original one ?
And while testing, can snort be send on a holiday ? What is snort protecting ?

Gertjan

@Gertjan said in DNS crashing every ~ 36 hours or so and unbound has to be restarted.:

File system errors ?

?

stephenw10

Yeah very hard to read that. You should export the snort logs separately and not log the main system log, that makes it much easier to see actual system events.
But anyway nothing seems to be logged there, not much to go on.

Running a filesystem check is probably a good idea.

Steve

gawainxx

@stephenw10
Unfortunately the system logs have already looped and only go as far back as this morning.

I've set up service watchdog to monitor the unbound process which will hopefully prevent the issue from causing extended outages while I work on getting it figured out. Snort is protecting my home network as well as a few miscellaneous things, mostly running for added security, I'm using one of the lighter pre-defined snort ruleset bundles.

I've exported the data from splunk in a raw format, perhaps that will be closer to the original?

Here are my logs from yesterday, the outage was around 10:56am, where there is an absolute absense of unbound log data until I had someone at home restart the server via console.

UnboundIssues_SystemLogs.txt
UnboundIssues_UnboundLogs.txt
UnboundIssues_SnortLogs.txt

stephenw10

Nothing logged but that file error seems like a permissions issue.

I would definitely run the file system check. I would consider just reinstalling and restoring, it's usually pretty quick.

Steve

gawainxx

This post is deleted!

gawainxx

@stephenw10

Thanks,
Here are the things i'm currently planning to do in order, moving to the next one if I see the service failure in the logs afterwords.

Gutting all non-critical code from my unbound.conf (Awaiting results on this currently).
SSHing to the router and running a filesystem check.
Toggling snort
Toggling Avahi
Toggling NUT
Reload and restore

Seem fair?

bmeeks

@gawainxx said in DNS crashing every ~ 36 hours or so and unbound has to be restarted.:

@stephenw10

Thanks,
Here are the things i'm currently planning to do in order, moving to the next one if I see the service failure in the logs afterwords.

Gutting all non-critical code from my unbound.conf (Awaiting results on this currently).

SSHing to the router and running a filesystem check.

Toggling snort

Toggling Avahi

Toggling NUT

Reload and restore

Seem fair?

Just an FYI. Snort and Unbound have absolutely nothing to do with each other in terms of Unbound starting or stopping. However, the DNSBL function of pfBlockerNG does rewrite the unbound.conf file and that can lead to Unbound issues.

While troubleshooting it is certainly prudent to stop Snort to remove that variable, but Snort running or not will have no impact on Unbound stopping and failing to restart.

gawainxx

That was my thought as well although another had suggested it may be related.

Installed packages are as follows
Avahi
notpng
nut
Openvpn-client-export
service_watchdog (New per this thread)
snort

gawainxx

@bmeeks

So far i've gone a week without any issues after pulling the domain server forwarding code from my unbound.conf file.

I would like to re implement the code or an alternative setup to obtain the same goal.
I have a domain controller, mordred.britannia2.local which I use for CBT labs as well as testing some misc group policies before proposing them at work.
I need systems on my server as well as test vlans to be able to do dns lookups for this system if they happen to be a domain member, i however only want domain specific DNS lookups going through it.

Could you please advise me on how to best accommodate this so that only domain specific traffic is forwarded to the DC?

I guess I could alternatively move the DC to one of my two test VLANS and add a static DNS record for it although that'd be less then ideal for me.

Here's the code I'd be looking to reimplement.
There were 3 redundant entries towards the bottom that I have since cleaned up.

server:

Allow plex to work over LAN

private-domain: "plex.direct"

Configuration for Britannia2.local with the PDC of mordred.britannia2.local

local-data: "_ldap._tcp.your.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.pdc._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.gc._msdcs.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.gc._msdcs.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_ldap._tcp.30e36ab8-a6ac-4c64-85aa-0fbeb612a33b.domains._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "d4f866aa-a210-4c29-81a2-ebb256bdef7d._msdcs.britannia2.local 600 IN CNAME mordred.britannia2.local"
local-data: "_kerberos._tcp.dc._msdcs.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_ldap._tcp.dc._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_kerberos._tcp.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_kerberos._tcp.Default-First-Site-Name._sites.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_gc._tcp.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_gc._tcp.Default-First-Site-Name._sites.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_kerberos._udp.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_kpasswd._tcp.britannia2.local 600 IN SRV 0 100 464 mordred.britannia2.local"
local-data: "_kpasswd._udp.britannia2.local 600 IN SRV 0 100 464 mordred.britannia2.local"
local-data: "_ldap._tcp.ForestDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.ForestDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.DomainDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.DomainDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "britannia2.local 600 IN A 192.168.4.5"
local-data: "gc._msdcs.britannia2.local 600 IN A 192.168.4.5"
local-data: "ForestDnsZones.britannia2.local 600 IN A 192.168.4.5"
local-data: "DomainDnsZones.britannia2.local 600 IN A 192.168.4.5"

bmeeks

So is the AD domain name "Britannia2.local"? If so, you simply need to add two entries to the Domain Override section of the DNS Resolver configuration page in pfSense.

Here is an example from my system. I have an internal AD domain and it contains two LAN subnets (192.168.10.0/24 and 192.168.20.0/24). In the example below, the 192.168.10.4 IP address is my AD domain controller.

Domain Override Example.png

So this tells Unbound that whenever someone attempts to lookup a host within the "themeeks.net" domain, it should forward the request to 192.168.10.4 (my internal AD domain controller) for resolution. Since I also wanted reverse IP lookups to work, I added the reverse pointer entries as well and set those to be forwarded to my AD domain controller, too.

For this to work, you must have your AD domain controller assigned a static IP address. That is really required anyway, though.

Don't try to put any of the AD records in Unbound! Just configure the domain override section for Unbound and let the AD DNS server handle all the Active Directory services lookups.