Netgate 2100 blocking? Spotify issue

[jrey]

@MikeHalsey said in Netgate 2100 blocking? Spotify issue:

when I first saw it I thought that's far too many restarts. So what might be causing them?

#1 pfblockerng with a list that is updating every time it runs and restarting unbound in combination with
#2 service_watchdog, seeing that pfblocker has cause unbound to restart and attempting to restart it constantly.

Beyond that it is likely not a "spotify" issue as such, but rather a set of speakers, that try to check in on everything and end up getting bad DNS replies, because well unbound is messed up.

I can/have simulated this in a lab setup with various speakers (not spotify, but others do the same thing), and just by rapidly restarting unbound for no good reason. Speakers (well not actually the speaker part, more like the phone home part) like to check in often, typically every song, so a series of bad DNS replies causes them to loose focus, and they stop.

I'd be looking at unbound restarting and why? start with the suggestions provided. if you know for sure it is not a particular list and not the watchdog; or even if it is @stephenw10 is spot on - crank up the logging on unbound.

[MikeHalsey]

@jrey I do have a, well if I'm honest, far too large list of blocks in pfBlockerNG and I took on board the warnings "don't add everything"... I didn't, just most of everything. But this problem with Spotify started long before I did any of that.

I too believe it's either a Spotify or an Amazon issue. There's a known issue with Spotify where speakers don't appear unless you, in the app, activate offline mode, then turn it off again. This is a two year+ known issue that a great many people have.

Amazon are also known to "save money" by throttling the bandwidth they give to their smart speakers to contact the mothership. I can see this causing DNS cutoffs for speakers that they just expect to be used for the occasional quick task or question, and not the way I use them.

But, if this restart of Unbound does fix things for me, then I suppose it'll do, so long as it's not opening up any security flaws.

[jrey]

@MikeHalsey

just so we are clear it is not the size of the lists - (unless you are loading so many as to cause a memory issue that can cause unbound issues as well) but generally it will be caused by a changed in a list causing unbound to restart in the first place. A list with 1 IP in it that changes every cycle will cause unbound to restart every cycle, just as much as 1 IP in a list of 100,000 will)

are you seeing memory issues ?

another place in the pfblockerng.log for example -- what do you see here ?

pfSense Table Stats
-------------------
table-entries hard limit   600000
Table Usage Count         140848

that is my table count and the bottom number has to be no more than half of the top number.
So without raising the top number (configuration) the bottom number in my case can be no more than just shy of 300000. That should be more than ample for most.

what is in your pfblockerng.log for these values? --- very large lists tend to push it over 1/2 limit edge too - so move the edge up (but not too big, better to review what the lists are actually doing for you)

System -> Advanced -> Firewall & NAT (part way down the screen)

I run typically 50-60 various clients behind a 2100 and unbound DNS-Reply returns about 85,000 queries per hour on a normal day.

[MikeHalsey]

@jrey Mine is set to 400000 and memory has never been an issue for me

[jrey]

@MikeHalsey

Sorry different memory and 2 possible variations: my bad

it will be php memory limit (not system memory) - you would have to check the logs .

and then this at 400,000 all your lists must be under 200,000 are they ? (you said you had a lot of large lists) -- "just most of everything." what do they add up to in the pfblockerng.log file.
"Maximum number of table entries for systems such as aliases, sshguard, snort, etc, combined."

[MikeHalsey]

@jrey How do I check these things? Many thanks

[jrey]

@MikeHalsey

I wasn't thinking the php issue, because it it would likely show up in an more obvious fashion -- usually something like

PHP Fatal error: Allowed memory size of ....

the other you can see in the pfblockerng.log file

Firewall -> pfBlockerNG -> Logs

select the pfblockerng.log file in the list - scroll to the end, then go back until you see the pfSense Table Stats section.. yours should say 400000 on the top number, based on the setting value you indicated it was set to, what is the bottom number ?

Edit: You will also see in the same log file - that part I quoted earlier about unbound restarting ..

[jrey]

@MikeHalsey

huh I just noticed according to your screen capture - you have no lists showing in the pfBlockerNG dashboard.

and your DNSBL is "unknown" --

there should be a summary there --- like this..

the bottom five on mine are all DNSBL related. (non have refreshed since Sep 1, no new data)
the others are the top are just other lists building alias tables.
either way I don't see this "Summary" in your screen capture.

just for reference what version of pfBlockerNG are you running ?

[Gertjan]

@MikeHalsey

Humm ... I missed the boat ...

From where I left :

Yeah, that pretty bad.
There are moment that unbound restarts every minute.
New info : you have many and/or big DNSBL lists.
What is your pfBlockerng setting :

?
Unbound mode - so unbound loads this pretty huge file at each startup

Or Python mode ? = way faster.

Anyway : your mission is, if you accept it : finding out why unbound restarts that often.

If presume you use KEA as the DHCP server (is that so ?), so it's not the dreaded dhcp_leases process that restarts unbound on every incoming DHCP lease or lease renew.
Is it an interface that goes down and up again, thus restarting all process, and unbound ?
Some other event ?

What I do understand now : your DNS experience is pretty bad as it is restarting all the time, and thanks to the big DNSBL lists it takes a long time to restart.
Result : most of the time, your DNS isn't working. It takes all day to 'stop' and 'start'.
And you were looking to restart it even more often ... (oh lol - like putting out the fire with a bucket of gasoline ^^)

[bmeeks]

Is perhaps DHCP client registration in DNS enabled on the firewall? If so, that can result in unbound restarts each time a DHCP client obtains or renews its lease. If enabled, that can contribute to a bunch of unbound DNS Resolver restarts. Combine that with large DNSBL lists and you could have a perfect storm essentially killing DNS resolution on the network for large intervals of time.

[SteveITS]

@bmeeks said in Netgate 2100 blocking? Spotify issue:

result in unbound restarts each time a DHCP client obtains or renews its lease

FWIW to all, this was/will be finally changed, in Kea, in 24.08.
https://www.netgate.com/blog/improvements-to-kea-dhcp

[stephenw10]

Yup, going to be so much better!

[Gertjan]

@bmeeks

Very true and I would have mentioned that potential issue right away.

But seeing this :

I've deducted that he is using KEA, and KEA should disable the dhcpleases process that restarts unbound on every ISC DHCP lease or lease renewal.
Let's be sure :

@MikeHalsey can you run :

ps ax | grep 'dhcpd.leases'

?

as if this return something like

97385  -  Is        0:00.02 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d brit-hotel-fumel.net -p /var/run/unbound.pid -u /var/unbound/dhcpleases_en
.......

then the case is solved.

[MikeHalsey]

@Gertjan said in Netgate 2100 blocking? Spotify issue:

ps ax | grep 'dhcpd.leases'

The result was...

8208 - Is 0:00.01 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d home.arpa -p /var/run/unbound.pid -u /var/unbound/dhcpleases_entries.conf -h /etc/hosts
86231 - S 0:00.01 sh -c ps ax | grep 'dhcpd.leases' 2>&1
86634 - S 0:00.01 grep dhcpd.leases

[Gertjan]

@MikeHalsey

Ok.
Call Houston.
You have a problem.

---

The solution :
Go here and select "ISC DHCP" :

and Save.

Now go to Services > DNS Resolver > General Settings

and locate

and remove the check from "DHCP Registration". This simple check, if set will activate the dhcpdleases process that restart unbound xx per hour.
Extra info : It's not checked by default ... and you can image why.

Save the new unbound settings.
Apply ( !! ) the new unbound settings.

Now, if you want to, you can go back to KEA : reverse the first step.
My advise : you dion't need to, ISC DHCP works very well.

Test phase :
You know how to check the unbound restarts.
Test during a couple of days.
You will notice the difference : DNS now behaves correcly.

---

and WTF : with KEA, dhcpdleaeses is still started ?? Netgate ?!!

[MikeHalsey]

@Gertjan Done, I'll let you know how it goes many thanks

[MikeHalsey]

@Gertjan Oh my god! Oh my god! Oh my god! Oh my god! Oh my god! That seems to have fixed it Spotify now seems to be responding to me immediately, all of the speakers are still there, and the music isn't stopping every 20 seconds like it was the last few days.

The constant restarts have stopped too. Here's hoping but it's looking good and I can't thank everybody who helped, enough 🫡

[MikeHalsey]

@Gertjan Actually, scrub that. It fixed it for all of 5 minutes then the problem started again

[MikeHalsey]

@Gertjan hmm... so far it's stable again, so I suppose that could have been cause by a pfBlocker update check. I'll run it for a day or three and report back 🫡

[Gertjan]

@MikeHalsey said in Netgate 2100 blocking? Spotify issue:

could have been cause by a pfBlocker update check

These can happen every hour max (I think).
Anyway, as most lists are updated maybe ones a week, or even less frequent-, I've set :

so pfBlockerng can restart unbound ones a week max.

Be ware : the list with : "who can restart unbound" isn't empty yet.

For example, when you hook up to a LAN or (actually any interface port) not a switch that is always powered on, but something else, every time you shut down this device, and the power it up again, the interface gets activated : this will restart unbound, and a lot of other processes as well.

Now you know all the reasons why unbound restarts (for 99,x %).

Btw : you've checked that the dhcpdleases process doesn't run anymore ?

Illustration :

Every dip in the stats is an unbound restart.
That's more often as ones a week, true, because it was the admin (me - so that's another reason why unbund restart : it's the admin ) doing so while testing settings so I can take screen captures to show them on the forum.
When I'm not there, my unbound doesn't restart anymore.

It will behave like this :