Netgate 2100 blocking? Spotify issue

[MikeHalsey]

@stephenw10 Hi, I just restarted the service and all the devices reappeared looks like I might need to automate this

[MikeHalsey]

@stephenw10 Just in case you're going to ask for them, here are screenshots of the DNS Resolver settings. Many thanks for all your help with this.

[stephenw10]

Ok, good result. Then it appears to be some DNS issue. Check the DNS logs when it's failing. Try to resolve the hosts (speakers) from one of the clients when it's failing.

[stephenw10]

Ah, looks like you have pfBlocker in the mix too. You might try running an update on it when Spotify is failing and see if that also allows it to start.

[MikeHalsey]

@stephenw10 As an interim anyway, do you know how I can configure a cron job to restart Unbound each day please?

[stephenw10]

You can use: /usr/local/sbin/pfSsh.php playback svc restart unbound

You have to use the full path like that in a cronjob.

[Gertjan]

@MikeHalsey said in Netgate 2100 blocking? Spotify issue:

do you know how I can configure a cron job to restart Unbound each day please?

Depending on how you've set up pfBlockerng, it already restarts regularly.

Restarting unbound, is a subject that is very known.
But the question is always the other way around : nobody wants that unbound restarts as you'll be loosing the cache - thus the overall DNS speed, and during the DNS restart, a you are using pfBlockerng with the 'unbound' mode instead of the python mode, is also slow(er) to restart. And during the restart : DNS errors, as it is not answering, all over the place.

You've shown your unbound (resolver) settings and you took some settings from 'default', may I draw your attention to the fact that de default settings are chosen by Netgate for a reason : they are the best.

Really, what you want, is something like this. Although the image shows the resolve's memory or cache usage, it also shows the restarts.
A perfect scenario, unbound shouldn't restart at all. It runs all the time. It's meant to run all the time.

And what if, for example, pfBlocker does it's thing like updating dnsbl and afterwards it restart unbound and at the same moment, your cron also restart unbound...
You've got yourself a nice race condition ... and a complete unknown issue, purely self inflected.

Also :

Do this :

and click on execute.
What did you saw ?

( you really want to restart the resolver even more then that ?? )

[MikeHalsey]

@stephenw10 I've set that up to restart at 5am each day and will report back if I still get the issue. Many thanks again

[MikeHalsey]

@Gertjan

[stephenw10]

Mmm, so it restarts multiple times already. You need to see what actually fixes the blocking issue when you run that. Or rather what's failing before you run it. Check the Unbound logs at that point. You may need to turn up the logging.

[MikeHalsey]

@stephenw10 So, you're saying wait until I can't see the devices and run this command again, then restart Unbound and run the command a second time to compare?

[jrey]

@MikeHalsey

that's a lot of unbound starts/restarts your are showing there -- why?

just a thought..

are you buy chance running service_watchdog to restart unbound if it stops -- don't (i've never had a need for service watchdog, not even installed)

pfblockerNG? - are you updating something in DNSBL that is updating hourly ? Several of those look like hourly, followed by chaos -- check the pfblockerng.log it will tell you in there when it has restarted unbound.

if the list of DNSBL is processing hourly, and something changes it will restart unbound.
service_watchdog will in that brief time see it is down and try to restart it.
that pid likely won't last long because it is already running, but it is detected as down again. etc etc in that condition it doesn't take very long for things to go south and stop.

unbound can and should run for days without issue. Mine hasn't had to restart since Sept 1. even though the DNSBL checks for updates regularly, none of those lists have changed, so nothing (unbound) needs restarting.

on the pfblocker dashboard if you are running it. you will see (whatever list / lists) you are running and when they were updated last example

and that is the last time unbound "started" here

and it won't restart again - until the next time DNSBL (list or list(s)) update.

pfblockerng.log will have entries like this

Saving DNSBL statistics... completed
Reloading Unbound Resolver (DNSBL python)
Stopping Unbound Resolver.
Unbound stopped in 2 sec.
Additional mounts (DNSBL python):
  No changes required.
Starting Unbound Resolver... completed [with a timestamp when it restarted]

It really goes down and up in "2 seconds" on my 2100 - and you should really never know it happened.

[MikeHalsey]

@jrey I am running a lot of lists in pfBlockerNG but they're not causing the problem with Spotify, as I only implemented them recently and the Spotify problems goes back more than a year to when I bought the 2100.

As regards other configuration, I'm a tech guy but my expertise lies elsewhere, so I leave this sort of thing in default config.

I agree though, when I first saw it I thought that's far too many restarts. So what might be causing them?

[stephenw10]

I would wait until it fails then check the Unbound logs. But you may need to turn up the logging level in Unbound to see what the clients are trying to resolve.

There is also a possibility that restarting Unbound triggers something else and that's actually what fixes the issue. That should be shown in the main system logs if so.

[jrey]

@MikeHalsey said in Netgate 2100 blocking? Spotify issue:

when I first saw it I thought that's far too many restarts. So what might be causing them?

#1 pfblockerng with a list that is updating every time it runs and restarting unbound in combination with
#2 service_watchdog, seeing that pfblocker has cause unbound to restart and attempting to restart it constantly.

Beyond that it is likely not a "spotify" issue as such, but rather a set of speakers, that try to check in on everything and end up getting bad DNS replies, because well unbound is messed up.

I can/have simulated this in a lab setup with various speakers (not spotify, but others do the same thing), and just by rapidly restarting unbound for no good reason. Speakers (well not actually the speaker part, more like the phone home part) like to check in often, typically every song, so a series of bad DNS replies causes them to loose focus, and they stop.

I'd be looking at unbound restarting and why? start with the suggestions provided. if you know for sure it is not a particular list and not the watchdog; or even if it is @stephenw10 is spot on - crank up the logging on unbound.

[MikeHalsey]

@jrey I do have a, well if I'm honest, far too large list of blocks in pfBlockerNG and I took on board the warnings "don't add everything"... I didn't, just most of everything. But this problem with Spotify started long before I did any of that.

I too believe it's either a Spotify or an Amazon issue. There's a known issue with Spotify where speakers don't appear unless you, in the app, activate offline mode, then turn it off again. This is a two year+ known issue that a great many people have.

Amazon are also known to "save money" by throttling the bandwidth they give to their smart speakers to contact the mothership. I can see this causing DNS cutoffs for speakers that they just expect to be used for the occasional quick task or question, and not the way I use them.

But, if this restart of Unbound does fix things for me, then I suppose it'll do, so long as it's not opening up any security flaws.

[jrey]

@MikeHalsey

just so we are clear it is not the size of the lists - (unless you are loading so many as to cause a memory issue that can cause unbound issues as well) but generally it will be caused by a changed in a list causing unbound to restart in the first place. A list with 1 IP in it that changes every cycle will cause unbound to restart every cycle, just as much as 1 IP in a list of 100,000 will)

are you seeing memory issues ?

another place in the pfblockerng.log for example -- what do you see here ?

pfSense Table Stats
-------------------
table-entries hard limit   600000
Table Usage Count         140848

that is my table count and the bottom number has to be no more than half of the top number.
So without raising the top number (configuration) the bottom number in my case can be no more than just shy of 300000. That should be more than ample for most.

what is in your pfblockerng.log for these values? --- very large lists tend to push it over 1/2 limit edge too - so move the edge up (but not too big, better to review what the lists are actually doing for you)

System -> Advanced -> Firewall & NAT (part way down the screen)

I run typically 50-60 various clients behind a 2100 and unbound DNS-Reply returns about 85,000 queries per hour on a normal day.

[MikeHalsey]

@jrey Mine is set to 400000 and memory has never been an issue for me

[jrey]

@MikeHalsey

Sorry different memory and 2 possible variations: my bad

it will be php memory limit (not system memory) - you would have to check the logs .

and then this at 400,000 all your lists must be under 200,000 are they ? (you said you had a lot of large lists) -- "just most of everything." what do they add up to in the pfblockerng.log file.
"Maximum number of table entries for systems such as aliases, sshguard, snort, etc, combined."

[MikeHalsey]

@jrey How do I check these things? Many thanks