ROKU Issues with pfBlockerNG (CBS All Access, PBS, CNET) not working

BBcan177 · Jun 15, 2017, 3:16 AM

In my case I had to whitelist pubads.g.doubleclick.net.

That domain also has a CNAME, so you might have to whitelist both of them… When you click on the "+" whitelist icon in the Alerts tab, it will automatically whitelist all CNAME options also.... but when you manually add the domain to the whitelist, you should run a similar drill command to add all CNAME options. I am adding the @8.8.8.8 so that you bypass DNSBL since that will show the DNSBL VIP address if its being blocked.

drill @8.8.8.8 pubads.g.doubleclick.net

;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 3058
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; pubads.g.doubleclick.net. IN A

;; ANSWER SECTION:
pubads.g.doubleclick.net. 86399 IN CNAME partnerad.l.doubleclick.net.
partnerad.l.doubleclick.net. 299 IN A 172.217.0.226

BBcan177 · Jun 15, 2017, 3:21 AM

@Xentrk:

Thank you Ramosel,

That gives me something to go on. I did see the entries
dw.cbsi.com
om.cbsi.com
pubads.g.doubleclick.net
b.scorecardresearch.com
in the DNSBL alerts and added them to the DNSBL Custom Domain Whitelist.

Using a AS number is a new one for me. I created a new IPv4 list called "Whitelist" and added the number 54113 in the Custom Address (es) field and checked the Enable Domain/AS box with List Action as Permit Both.

I then ran the Force Update. I still have the issue with the three channels. It is getting late in my time zone. I will take a break and pick it up tomorrow. I appreciate the tips.

Just keep in mind that when something is blocked by DNSBL, you can't whitelist or permit it by using IP firewall rules…

and vice-versa... They are two different processes...

BBcan177 · Jun 15, 2017, 3:36 AM

@Xentrk:

Using a AS number is a new one for me. I created a new IPv4 list called "Whitelist" and added the number 54113 in the Custom Address (es) field and checked the Enable Domain/AS box with List Action as Permit Both.

Also note that you shouldn't use "Permit Both" unless you configure the "Adv. Inbound Settings" so that any unsolicited Inbound from these IPs is limited to the Open Ports and Destination IPs….

Usually when there are no open WAN ports, you only need to use "Permit Outbound" so that the LAN has to initiate the request which will create a firewall state to allow those packets back thru the firewall (inbound)... ie: stateful firewall

Xentrk · Jun 15, 2017, 1:38 PM

Thank you everyone for your helpful replies and suggestions!

I removed the IPv4 whitelist I created for fastly.com AS number 54113. I then went back to the DNSBL Custom Domain Whitelist and added periods in front of the domain name:

.dw.cbsi.com
.om.cbsi.com
.pubads.g.doubleclick.net
.b.scorecardresearch.com

I then saved, and ran update. It works. CBS All Access, PBS and CNET are now working. Wooot!

Based on comments from reisender, I then removed all entries except for pubads.g.doubleclick.net. CBS All Access, PBS and CNET worked. So the solution is to whitelist pubads.g.doubleclick.net with a dot in front of it.

I am very grateful for the help and suggestions.

BBcan177 · Jun 15, 2017, 4:00 PM

Here are the potential domains:

grep "pubads.g.doubleclick.net" /var/db/pfblockerng/dnsblorig/*

/var/db/pfblockerng/dnsblorig/Cameleon.orig:127.0.0.1 pubads.g.doubleclick.net
/var/db/pfblockerng/dnsblorig/Cameleon.orig:127.0.0.1
securepubads.g.doubleclick.net
/var/db/pfblockerng/dnsblorig/MVPS.orig:0.0.0.0 pubads.g.doubleclick.net
/var/db/pfblockerng/dnsblorig/MVPS.orig:0.0.0.0 securepubads.g.doubleclick.net
/var/db/pfblockerng/dnsblorig/SWC.orig:#127.0.0.1 pubads.g.doubleclick.net #interferes with video on cwtv.com
/var/db/pfblockerng/dnsblorig/SWC.orig:127.0.0.1 securepubads.g.doubleclick.net
/var/db/pfblockerng/dnsblorig/hpHosts_ads.orig:127.0.0.1 pubads.g.doubleclick.net
/var/db/pfblockerng/dnsblorig/hpHosts_ads.orig:127.0.0.1 securepubads.g.doubleclick.net

Here is the CNAME entry:

grep "partnerad.l.doubleclick.net" /var/db/pfblockerng/dnsblorig/*

/var/db/pfblockerng/dnsblorig/Cameleon.orig:127.0.0.1 partnerad.l.doubleclick.net
/var/db/pfblockerng/dnsblorig/MVPS.orig:0.0.0.0 partnerad.l.doubleclick.net
/var/db/pfblockerng/dnsblorig/hpHosts_ads.orig:127.0.0.1 partnerad.l.doubleclick.net

I am not sure how prefixing the "period" to that domain removed any additional domains, as it wouldn't match the securepubads.g.doubleclick.net domain…

This command will show all sub-domains:

grep ".pubads.g.doubleclick.net" /var/db/pfblockerng/dnsblorig/*

and it shows no domains listed… Maybe there was another reason for this issue? Maybe the device still had that domain cached after you whitelisted it...

Either way glad that you got it working...

reisender · Jun 15, 2017, 4:09 PM

@BBcan177:

@reisender:

In my case I had to whitelist pubads.g.doubleclick.net.

That domain also has a CNAME, so you might have to whitelist both of them…

BBcan177, you are correct. I also have the CNAME you provided whitelisted since I used the "+" icon to add it to the whitelist. My whitelist does not have a "period" in front of pubads.

Do you have any idea why the Roku channels work normally when I block pubads.g.doubleclick.net when using piHole for DNS, but fail when blocking this domain using pfBlockerNG? It makes me think the channel doesn't depend on doubleclick.net since it works while blocked with piHole. Something is different with the way DNSBL is handling it. It isn't the end of the world to keep doubleclick whitelisted, but I prefer to block it if possible.

Xentrk · Jun 15, 2017, 5:47 PM

@reisender:

@BBcan177:

@reisender:

In my case I had to whitelist pubads.g.doubleclick.net.

That domain also has a CNAME, so you might have to whitelist both of them…

BBcan177, you are correct. I also have the CNAME you provided whitelisted since I used the "+" icon to add it to the whitelist. My whitelist does not have a "period" in front of pubads.

Do you have any idea why the Roku channels work normally when I block pubads.g.doubleclick.net when using piHole for DNS, but fail when blocking this domain using pfBlockerNG? It makes me think the channel doesn't depend on doubleclick.net since it works while blocked with piHole. Something is different with the way DNSBL is handling it. It isn't the end of the world to keep doubleclick whitelisted, but I prefer to block it if possible.

Thanks for the feedback. I made two changes today from my updates last night: 1 - removing the AS number entry 2 - adding periods to the whitelist entries. I then got it to work. So I assumed it was the periods in front that did the trick. I removed the period and saved. But I can't update DNSBL right now as an active cron job is running. I will report back and update with the results. You are right. The period should not be required.

EDIT: I can confirm the leading dot is not required in front of the domain name.

I should also report that I have two other routers running ASUS Merlin 380.66_2. On these routers I use an ad blocking solution https://www.ab-solution.info/ which also blacklists pubads.g.doubleclick.net. But CBS, PBS and CNET work fine on those routers with the Roku 4 and I don't have to whitelist the pubads.g.doubleclick.net site on that ad blocking solution.

BBcan177 · Jun 15, 2017, 5:49 PM

I can only assume that the Roku devices are attempting to connect to this domain via HTTPS, and are not timing out as expected… DNSBL uses port forwards to redirect the DNS blocked request to the lighttod webserver. So for HTTPS blocked domains, the browser or app will see that the Certificate of the DNSBL Webserver doesn't match the requested domain and it should immediately timeout and close the connection....

The next version of the package will have an option to utilize "0.0.0.0" instead of the DNSBL VIP... So that might address this issue for these particular situations. You could temporarily whitelist this domain in DNSBL and add a host override in Unbound for this domain and point it to "0.0.0.0"...

BBcan177 · Jun 15, 2017, 6:38 PM

@Xentrk:

I should also report that I have two other routers running ASUS Merlin 380.66_2. On these routers I use an ad blocking solution https://www.ab-solution.info/ which also blacklists pubads.g.doubleclick.net. But CBS, PBS and CNET work fine on those routers with the Roku 4 and I don't have to whitelist the pubads.g.doubleclick.net site on that ad blocking solution.

You can see from the website link that you provided above, that pixelserv is MITM'ing the domains by creating false certificates… obviously something that I want to avoid:

pixelserv-tls.add: Added option to purge auto-generated pixelserv-tls certificates.

reisender · Jun 15, 2017, 6:54 PM

Thanks for the explanation BBcan117. Your contribution to the pfSense community is huge! I look forward to the next version.

reisender · Jun 15, 2017, 9:19 PM

A quick follow-up. I followed BBCan117's suggestion to use a host override for pubads.g.doubleclick.net and the CNAME partnerads.l.doubleclick.net.

The Roku channels still work, and the ads are not appearing. Problem solved.

Xentrk · Jun 16, 2017, 4:21 AM

@reisender:

A quick follow-up. I followed BBCan117's suggestion to use a host override for pubads.g.doubleclick.net and the CNAME partnerads.l.doubleclick.net.

The Roku channels still work, and the ads are not appearing. Problem solved.

Does my host overrides look correct (see attachment)? This is a new one for me on pfSense.

Also, did you keep the Custom Domain Whitelist entry for pubads.g.doubleclick.net in the DNSBL Configuration screen?

Regards, Xentrk

reisender · Jun 16, 2017, 9:14 PM

Your host override isn't quite right. Make it look like the attached image. Keep the domains in your whitelist. I kept the whitelist for both domains. I'm not sure if DNSBL or the host override is processed first, but it seems to work this way.

Xentrk · Jun 17, 2017, 3:58 AM

@reisender:

Your host override isn't quite right. Make it look like the attached image. Keep the domains in your whitelist. I kept the whitelist for both domains. I'm not sure if DNSBL or the host override is processed first, but it seems to work this way.

Thanks you reisender and BBCan177 for the help. With the DNS Resolver change, I no longer see ads on those channels. I will need to watch more to make sure. A nice benefit for sure. I'm happy to get those channels working again though. I never thought it was my firewall blocking them for two months as they worked okay before the last part of April. I then saw on the Roku forums that others were having an issue as well. I posted the fix on the Roku forum site. Having access to TV in USA enhances my expat life here in the land of smiles.

Because of this issue, I learned features of pfBlockerNG and DNS Resolver that I was not aware of. I appreciate your help and educating me along the way! I am very grateful for the help you all provided.

Regards, Xen

BBcan177 · Jun 19, 2017, 9:37 PM

@Xentrk:

Thanks you reisender and BBCan177 for the help. With the DNS Resolver change, I no longer see ads on those channels. I will need to watch more to make sure. A nice benefit for sure. I'm happy to get those channels working again though. I never thought it was my firewall blocking them for two months as they worked okay before the last part of April. I then saw on the Roku forums that others were having an issue as well. I posted the fix on the Roku forum site. Having access to TV in USA enhances my expat life here in the land of smiles.

Because of this issue, I learned features of pfBlockerNG and DNS Resolver that I was not aware of. I appreciate your help and educating me along the way! I am very grateful for the help you all provided.

Regards, Xen

NP your welcome… ;)

Xentrk · Sep 7, 2017, 1:15 AM

@reisender:

Your host override isn't quite right. Make it look like the attached image. Keep the domains in your whitelist. I kept the whitelist for both domains. I'm not sure if DNSBL or the host override is processed first, but it seems to work this way.

reisender,

Is CBS all access still working for you? It stopped working for me last night. When I select a video to watch, I get a little spinning symbol for a few seconds followed by a black screen. I can watch live TV okay. It is just the on demand videos that are the issue. It works okay on my ASUS router using AB-Solution.info ad blocker. But I do see ads. Maybe it is time to pay the extra $$ for the ad free version.

Xentrk · Sep 10, 2017, 1:08 PM

@Xentrk:

@reisender:

Your host override isn't quite right. Make it look like the attached image. Keep the domains in your whitelist. I kept the whitelist for both domains. I'm not sure if DNSBL or the host override is processed first, but it seems to work this way.

reisender,

Is CBS all access still working for you? It stopped working for me last night. When I select a video to watch, I get a little spinning symbol for a few seconds followed by a black screen. I can watch live TV okay. It is just the on demand videos that are the issue. It works okay on my ASUS router using AB-Solution.info ad blocker. But I do see ads. Maybe it is time to pay the extra $$ for the ad free version.

I have it working again. I was unable to determine the domain or host file causing the issues. I did some testing with hosts file on my ASUS router using AB-Solution. I had issues when I went to higher levels of hosts files. I found the right combination that made it work on the ASUS. I replicated that on the pfSense. I started with this list:

http://someonewhocares.org/hosts/hosts
http://sysctl.org/cameleon/hosts
http://winhelp2002.mvps.org/hosts.txt
http://www.malekal.com/HOSTS_filtre/HOSTS.txt
http://www.malwaredomainlist.com/hostslist/hosts.txt
https://zeustracker.abuse.ch/blocklist.php?download=hostfile
http://www.hostsfile.org/Downloads/hosts.txt
http://www.securemecca.com/Downloads/hosts.txt
http://hosts-file.net/exp.txt
http://hosts-file.net/ad_servers.txt
http://hosts-file.net/emd.txt
http://hosts-file.net/hjk.txt
http://hosts-file.net/fsa.txt
http://hosts-file.net/grm.txt
http://hosts-file.net/psh.txt
http://hosts-file.net/mmt.txt
http://hosts-file.net/hfs.txt
http://hosts-file.net/pha.txt
http://hosts-file.net/wrz.txt
http://raw.githubusercontent.com/michaeltrimm/hosts-blocking/master/_hosts.txt

And narrowed it down to this list

https://adaway.org/hosts.txt
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext&useip=0.0.0.0
http://someonewhocares.org/hosts/zero/hosts
http://winhelp2002.mvps.org/hosts.txt
http://www.malwaredomainlist.com/hostslist/hosts.txt
http://hosts-file.net/ad_servers.txt
http://hosts-file.net/emd.txt
http://hosts-file.net/grm.txt
http://hosts-file.net/mmt.txt

Perhaps one of these days, I will take some the hosts files back in one by one until I determine which one caused me the grief.

Xentrk · Sep 15, 2017, 2:41 PM

The above solution was a false positive. It did not work. I ended up removing the Host Overrides in DNS Resolver to get it working. However, ads are now appearing. We'll, I am paying for the lower tier with ads. So I can live with it. Enjoying it ad free was nice while it lasted though.