DNS crashing every ~ 36 hours or so and unbound has to be restarted.

gawainxx

Died again, help plox!

stephenw10

Need more info to help further. What's logged in the system log when it fails? Or just before it fails?

gawainxx

@stephenw10 I'll grab those logs for you in a bit once I'm able to access my network again , am currently remote and am locked out due to the issue.

My logs are divided by process within Splunk.
Is there a specific process that would have the most relevant log data?

gawainxx

Here are some logs, they are csvs renamed to .txt
Unbound logs from 10:40am - 3 PM
1576040675_650.txt
System" logs from 10:40am - 3 PM
1576040737_651.txt

I've also purged the DC related entries in my unbound config to see if that perhaps makes a change as I really only use that for labs/training stuff. Also adjusted my firewall rules so that I can access the router webUI from it, would have saved myself a lot of headache if I could have just restarted it via Ovpn.

Gertjan

unbound is stopped and restarted.

More logs are needed to see which process is doing this. It could also be a hardware event like a "LINK UP / LINK UP"

Btw : this "plunked" unbound log is close to totally unreadable : possible to see the original one ?
And while testing, can snort be send on a holiday ? What is snort protecting ?

Gertjan

@Gertjan said in DNS crashing every ~ 36 hours or so and unbound has to be restarted.:

File system errors ?

?

stephenw10

Yeah very hard to read that. You should export the snort logs separately and not log the main system log, that makes it much easier to see actual system events.
But anyway nothing seems to be logged there, not much to go on.

Running a filesystem check is probably a good idea.

Steve

gawainxx

@stephenw10
Unfortunately the system logs have already looped and only go as far back as this morning.

I've set up service watchdog to monitor the unbound process which will hopefully prevent the issue from causing extended outages while I work on getting it figured out. Snort is protecting my home network as well as a few miscellaneous things, mostly running for added security, I'm using one of the lighter pre-defined snort ruleset bundles.

I've exported the data from splunk in a raw format, perhaps that will be closer to the original?

Here are my logs from yesterday, the outage was around 10:56am, where there is an absolute absense of unbound log data until I had someone at home restart the server via console.

UnboundIssues_SystemLogs.txt
UnboundIssues_UnboundLogs.txt
UnboundIssues_SnortLogs.txt

stephenw10

Nothing logged but that file error seems like a permissions issue.

I would definitely run the file system check. I would consider just reinstalling and restoring, it's usually pretty quick.

Steve

gawainxx

This post is deleted!

gawainxx

@stephenw10

Thanks,
Here are the things i'm currently planning to do in order, moving to the next one if I see the service failure in the logs afterwords.

Gutting all non-critical code from my unbound.conf (Awaiting results on this currently).
SSHing to the router and running a filesystem check.
Toggling snort
Toggling Avahi
Toggling NUT
Reload and restore

Seem fair?

bmeeks

@gawainxx said in DNS crashing every ~ 36 hours or so and unbound has to be restarted.:

@stephenw10

Thanks,
Here are the things i'm currently planning to do in order, moving to the next one if I see the service failure in the logs afterwords.

Gutting all non-critical code from my unbound.conf (Awaiting results on this currently).

SSHing to the router and running a filesystem check.

Toggling snort

Toggling Avahi

Toggling NUT

Reload and restore

Seem fair?

Just an FYI. Snort and Unbound have absolutely nothing to do with each other in terms of Unbound starting or stopping. However, the DNSBL function of pfBlockerNG does rewrite the unbound.conf file and that can lead to Unbound issues.

While troubleshooting it is certainly prudent to stop Snort to remove that variable, but Snort running or not will have no impact on Unbound stopping and failing to restart.

gawainxx

That was my thought as well although another had suggested it may be related.

Installed packages are as follows
Avahi
notpng
nut
Openvpn-client-export
service_watchdog (New per this thread)
snort

gawainxx

@bmeeks

So far i've gone a week without any issues after pulling the domain server forwarding code from my unbound.conf file.

I would like to re implement the code or an alternative setup to obtain the same goal.
I have a domain controller, mordred.britannia2.local which I use for CBT labs as well as testing some misc group policies before proposing them at work.
I need systems on my server as well as test vlans to be able to do dns lookups for this system if they happen to be a domain member, i however only want domain specific DNS lookups going through it.

Could you please advise me on how to best accommodate this so that only domain specific traffic is forwarded to the DC?

I guess I could alternatively move the DC to one of my two test VLANS and add a static DNS record for it although that'd be less then ideal for me.

Here's the code I'd be looking to reimplement.
There were 3 redundant entries towards the bottom that I have since cleaned up.

server:

Allow plex to work over LAN

private-domain: "plex.direct"

Configuration for Britannia2.local with the PDC of mordred.britannia2.local

local-data: "_ldap._tcp.your.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.pdc._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.gc._msdcs.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.gc._msdcs.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_ldap._tcp.30e36ab8-a6ac-4c64-85aa-0fbeb612a33b.domains._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "d4f866aa-a210-4c29-81a2-ebb256bdef7d._msdcs.britannia2.local 600 IN CNAME mordred.britannia2.local"
local-data: "_kerberos._tcp.dc._msdcs.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_ldap._tcp.dc._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_kerberos._tcp.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_kerberos._tcp.Default-First-Site-Name._sites.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_gc._tcp.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_gc._tcp.Default-First-Site-Name._sites.britannia2.local 600 IN SRV 0 100 3268 mordred.britannia2.local"
local-data: "_kerberos._udp.britannia2.local 600 IN SRV 0 100 88 mordred.britannia2.local"
local-data: "_kpasswd._tcp.britannia2.local 600 IN SRV 0 100 464 mordred.britannia2.local"
local-data: "_kpasswd._udp.britannia2.local 600 IN SRV 0 100 464 mordred.britannia2.local"
local-data: "_ldap._tcp.ForestDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.ForestDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.DomainDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "_ldap._tcp.Default-First-Site-Name._sites.DomainDnsZones.britannia2.local 600 IN SRV 0 100 389 mordred.britannia2.local"
local-data: "britannia2.local 600 IN A 192.168.4.5"
local-data: "gc._msdcs.britannia2.local 600 IN A 192.168.4.5"
local-data: "ForestDnsZones.britannia2.local 600 IN A 192.168.4.5"
local-data: "DomainDnsZones.britannia2.local 600 IN A 192.168.4.5"

bmeeks

So is the AD domain name "Britannia2.local"? If so, you simply need to add two entries to the Domain Override section of the DNS Resolver configuration page in pfSense.

Here is an example from my system. I have an internal AD domain and it contains two LAN subnets (192.168.10.0/24 and 192.168.20.0/24). In the example below, the 192.168.10.4 IP address is my AD domain controller.

Domain Override Example.png

So this tells Unbound that whenever someone attempts to lookup a host within the "themeeks.net" domain, it should forward the request to 192.168.10.4 (my internal AD domain controller) for resolution. Since I also wanted reverse IP lookups to work, I added the reverse pointer entries as well and set those to be forwarded to my AD domain controller, too.

For this to work, you must have your AD domain controller assigned a static IP address. That is really required anyway, though.

Don't try to put any of the AD records in Unbound! Just configure the domain override section for Unbound and let the AD DNS server handle all the Active Directory services lookups.

johnpoz

Not a fan of this method - if your an AD shop, then all your AD clients should be pointing to your AD for dns, and should use that as your dhcp as well..

If you want pfsense to be able to resolve your AD clients, and or any other clients that are not members of your domain to be able to resolve your ad stuff, then sure you can use a domain override as you have shown.

Keep in mind that you will have to set this domain as private or your going to not resolve anything due to rebind protection. Or you would have to disable rebind protection across the board.

gawainxx

@bmeeks
That's how I swore it worked on one of my previous pfsense instances.. I wonder why it's having issues that are requiring me to throw stuff in the unbound.conf.

Without throwing that crap into the unbound config I was getting errors such as clients being unable to resolve _ldap._tcp.Default-First-Site-Name._sites.britannia2.local

Does my screenshot look correct?

bmeeks

@johnpoz said in DNS crashing every ~ 36 hours or so and unbound has to be restarted.:

Not a fan of this method - if your an AD shop, then all your AD clients should be pointing to your AD for dns, and should use that as your dhcp as well..

If you want pfsense to be able to resolve your AD clients, and or any other clients that are not members of your domain to be able to resolve your ad stuff, then sure you can use a domain override as you have shown.

Keep in mind that you will have to set this domain as private or your going to not resolve anything due to rebind protection. Or you would have to disable rebind protection across the board.

@johnpoz:
This is only for pfSense in my case so that log entries and hosts in the ARP table resolve to their actual names. All my LAN clients do indeed point to the AD controller for DNS and get their addresses via DHCP from there.

I could have also just disabled the DNS Resolver and simply used the DNS Forwarder and forwarded everything from pfSense back to the AD domain controller. Six of one and half-a-dozen of the other in a manner of speaking.

johnpoz

Yeah for pfsense to be able to resolve your AD stuff its great and sure works as you stated. You just left off the rebind info - if you forward to something in unbound - it will fall under rebind if rfc1918 is returned.

Not exactly sure what the OP is after - and setting up a domain override is fine.. Just not a fan of AD members pointing to anything other than their AD and using AD for dhcp... If your a AD shop - why would not just what is part of what your paying for anyway with your MS tax ;)

bmeeks

@gawainxx said in DNS crashing every ~ 36 hours or so and unbound has to be restarted.:

@bmeeks
That's how I swore it worked on one of my previous pfsense instances.. I wonder why it's having issues that are requiring me to throw stuff in the unbound.conf.

Without throwing that crap into the unbound config I was getting errors such as clients being unable to resolve _ldap._tcp.Default-First-Site-Name._sites.britannia2.local

Does my screenshot look correct?

No, you don't need to enter that host information. Simply put the AD domain name in the override and then optionally any reverse pointer records if you also want to be able to resolve IP addresses back to hosts.

Also remove that "_msdcs" entry. You do not need to let pfSense be where your AD clients try to find AD services! Let the AD domain controller handle all of that. You only would put AD overrides in pfSense if you wanted, as I did, certain logging information on pfSense to have resolved host names from your AD domain.