Dnsmasq daemon crashing after 2.2.5 update

it-marmalade

Hi all,

Yesterday we upgraded from 2.2.3 to 2.2.5.

Since then dnsmasq DNS Forwarder goes down every few hours (on average) effectively disabling all DNS for clients.

It comes back fine when started from Status -> Services but then crashes again.

I can't see much in logs :(

In /var/log/resolver.log

"dnsmasq[13346]: read /etc/hosts - 406 addresses"

which I'm guessing is fine.

In /var/log/system.log

"dhcpleases: Could not deliver signal HUP to process because its pidfile (/var/run/dnsmasq.pid) does not exist, No such process."

which appears to be a consequence rather then a root cause.

More info about my system:

FreeBSD xxxxxxxx 10.1-RELEASE-p24 FreeBSD 10.1-RELEASE-p24 #0 f27a67c(releng/10.1)-dirty: Wed Nov 4 16:18:37 CST 2015 root@pfs22-amd64-builder:/usr/obj.RELENG_2_2.amd64/usr/pfSensesrc/src.RELENG_2_2/sys/pfSense_SMP.10 amd64

I tried to determine dnsmasq version too and got as far as:

pkg info
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: N

I wasn't sure if it won't interfere with anything and decided not to take this risk.

Can anybody advise on the crashes / next troubleshooting steps?

Thanks
Adam

–-----------------------------------------------------------------------------------------------------

A bit more log info from the last crash (this time DNS service only ran for about 15 mins...):

Dec 9 13:22:39 dnsmasq[56966]: read /etc/hosts - 406 addresses
Dec 9 13:22:38 dnsmasq[56966]: read /etc/hosts - 271 addresses –---> ???
Dec 9 13:22:37 dnsmasq[56966]: read /etc/hosts - 406 addresses
Dec 9 13:22:37 dnsmasq[56966]: read /etc/hosts - 406 addresses
Dec 9 13:22:37 dnsmasq[56966]: using nameserver 8.8.4.4#53
Dec 9 13:22:37 dnsmasq[56966]: using nameserver 8.8.8.8#53
Dec 9 13:22:37 dnsmasq[56966]: ignoring nameserver 127.0.0.1 - local interface
Dec 9 13:22:37 dnsmasq[56966]: reading /etc/resolv.conf
Dec 9 13:22:37 dnsmasq[56966]: ignoring nameserver 10.79.1.1 - local interface
Dec 9 13:22:37 dnsmasq[56966]: compile time options: IPv6 GNU-getopt no-DBus i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect no-inotify
Dec 9 13:22:37 dnsmasq[56966]: started, version 2.75 cachesize 10000
Dec 9 12:53:20 dnsmasq[60000]: read /etc/hosts - 406 addresses –---> CRASHED AND DOWN FOR 30 MINS
Dec 9 12:53:20 dnsmasq[60000]: read /etc/hosts - 271 addresses –---> ???
Dec 9 12:53:10 dnsmasq[60000]: read /etc/hosts - 406 addresses

–-----------------------------------------------------------------------------------------------------

dnsmasq core dumped several times in dmesg:

pid 13346 (dnsmasq), uid 65534: exited on signal 11
pid 22787 (dnsmasq), uid 65534: exited on signal 11
pid 36791 (dnsmasq), uid 65534: exited on signal 11
pid 60000 (dnsmasq), uid 65534: exited on signal 11

iocon

I am seeing this same issue after updating from 2.2.4
My dnsmasq service is stopping every day or two. There are no specific logs entries indicating why.

This is not my main DNS server (it forwards to it however) so it doesn't get too much traffic.

cmb

'dnsmasq -v' will show the version. 2.2.5 has dnsmasq 2.75. 2.2.4 had 2.73. Not seeing any reports of crashes with newer dnsmasq versions, but that's the only thing that's changed so there must be something there.

You should see dnsmasq dying in the log somewhere if it's crashing. Finding a core file, if there is one, would be helpful. Run:

find / -name '*core'

and see what the results are. Adding log-queries to the advanced box for the DNS Forwarder may prove useful if it's something to do with a specific unusual query that's crashing it.

Pimmetje

I also have a issue with DNSMasq crashing after update to 2.2.5. Ill hold update for other devices till the cause is found.

We have basic DNS config with Domain Overrides and Host Overrides. Maybe a combination causes the issue. I can't find any entries in the log about crash or unexpected stop.

If i can enable extra debugging somewhere let me know.

Edit:
My dmesg also shows:
pid 23900 (dnsmasq), uid 65534: exited on signal 11
pid 5797 (dnsmasq), uid 65534: exited on signal 11
pid 74774 (dnsmasq), uid 65534: exited on signal 11

Edit2:
Also found a related ticket: https://redmine.pfsense.org/issues/5341

cmb

edit: removed original now-dated recommendation.

If you're seeing this, you can run:

pkg install dnsmasq-devel

to get dnsmasq 2.76test version, which several people have confirmed fixes this issue.

Note: 2.3 snapshots include 2.76test by default.

it-marmalade

Just to let everyone know that rolling back to 2.2.3 from 2.2.5 has fixed the dnsmasq issue.

I will probably wait for 2.2.6, see if the bug is addressed and try again.

Thanks
Adam

cmb

2.2.6 isn't going to be any different in that regard, as it still has the same dnsmasq version as 2.2.5 (the latest available release at this time, 2.75). Short of getting some feedback on the dnsmasq 2.76test1 binary I posted above, release will be the same dnsmasq 2.75 which will have the same issue. The only thing that's changed between 2.2.x versions is the version of dnsmasq itself.

It'd be good to know from someone who's seeing this issue if either the 2.76test1 version above, or replacing the dnsmasq binary with that from 2.2.3, has any impact.

it-marmalade

I really appreciate your help and effort cmb.
Saying that I'm very reluctant to try an unofficial test release on a live production system serving hundreds of users and records.

This sounds like a quite fundamental bug in an important feature.
I'm assuming it has a potential of finding its way into the next pfSense version as soon as it's released by FreeBSD community?

Which can be tracked here:

https://www.freebsd.org/cgi/ports.cgi?query=dnsmasq&stype=all

right?

cmb

As soon as dnsmasq releases a new version, it'll make its way to FreeBSD ports and we'll include it. But there isn't anything specific known about the issue, not something I've been able to replicate, so no telling whether or not the next dnsmasq will fix the problem. Or when the next dnsmasq release will happen. That's why I'd like to have someone who's seeing the issue try that 2.75test1 binary. It's unlikely to be any worse in circumstances where it's crashing already. If it's still an issue there, it's something that needs to be pursued with the dnsmasq folks.

Pimmetje

We used the version posted above for some time and we haven't had any problems with it anymore. Thanks for the help.

cmb

Pimmetje: thanks for the feedback!

I think this may be the same as what people are reporting in this bug ticket.

Hopefully dnsmasq 2.76 will be released soon and we can update it. Otherwise we'll update to a 2.76test version from git for 2.3 release.

aeble

Hi cmb

Thanks for supplying a workaround!

I do however have problems with starting the agent once it's in place, even though I followed your simple steps.

Do you have any suggestions to what might help me?

Thanks in advance!

/aeble

cmb

@aeble:

Hi cmb

Thanks for supplying a workaround!

I do however have problems with starting the agent once it's in place, even though I followed your simple steps.

Do you have any suggestions to what might help me?

There isn't much to it. Make sure you followed all the steps including the 'chmod +x' part. If it fails to start it should log something as to why in either the system log or resolver log, you see anything there?

Simon74

my DNS service crash daily on pfsense 2.2.6.
is there a solution ?

dwhacks

I have the same issue, only for me it happens only when I view a certain website http://www.diyaudio.com. After dnsmasq crashes, no other sites will load except for diyaudio, until i restart it.

I tried the workaround and dnsmasq wouldnt start from the GUI so I started it from the command line. I couldn't find any other info in the log except what has already been posted by others.

edit: I just tried to reproduce it now after doing the fix and then reverting and it doesnt seem to crash. It crashed every time I went to the site after the update to 2.2.6-RELEASE (i386) before though.

Patryn999

Just for other people:
I had this issue also, and using cmb's binary wouldn't start (I suspect due to it being built for a different architecture? I was running i386).

What solved it for me:
Install pkg (Package manager) and use this to install to 2.76test5 - it's really easy.
Restart the dnsmasq daemon so that it runs the new version (check log to see that the correct version started)
Problem solved.

We used to crash approx every 2 days, and started crashing about every 2 hours before I had to do something about it - now the DNS forwarder has been up for 6 days without issue.

necron

Any progress?
2.2.6 dnsmasq is still crashy once every few hours/days (depending on the load).

cmb

@necron:

Any progress?

There's a fix here for the few that are seeing it, with the dnsmasq 2.76test version. We'll update to dnsmasq 2.76 as soon as it's released, or to a prerelease version if necessary if they're not releasing 2.76 soon enough for our 2.3 release.

Cortexian

@Patryn999:

Just for other people:
I had this issue also, and using cmb's binary wouldn't start (I suspect due to it being built for a different architecture? I was running i386).

What solved it for me:
Install pkg (Package manager) and use this to install to 2.76test5 - it's really easy.
Restart the dnsmasq daemon so that it runs the new version (check log to see that the correct version started)
Problem solved.

We used to crash approx every 2 days, and started crashing about every 2 hours before I had to do something about it - now the DNS forwarder has been up for 6 days without issue.

Could you explain the steps you took in more detail? I've installed pkg via ssh into my pfSense box, but not sure where to go from there. When I try to get it to list packages it only lists itself as installed…

cmb

@Cortexian:

Could you explain the steps you took in more detail? I've installed pkg via ssh into my pfSense box, but not sure where to go from there. When I try to get it to list packages it only lists itself as installed…

Just run:

pkg install dnsmasq-devel