DNS Puzzle

Bob.Dig

@provels This is AAAA, so it is IPv6. You have to watch out that DNS is not going different routes for IPv4 and IPv6. I disabled IPv6-DNS in DHCPv6 and RA. Although, source is IPv4 so this also doesn't seem to be the problem here... It is a puzzle for sure.

johnpoz

@provels there is no way that pfsense would ask 192.168.0.1 for dns if its resolving, and did not setup a domain override.. Are you doing some dns redirection? Or nat reflection and some client asking and its being reflected to your pihole.

You sure you just don't have some box on your network with a dupe IP of that 0.1 ? What I would do is sniff on your pihole - what mac address are you seeing that traffic from, or maybe you can just see that in your network tab on your pihole

Are you doing any source natting where if device on another segment gets routed through pfsense and you nat it to pfsense IP.. I do this to talk to my IP cameras because they point to a different gateway then pfsense because they are behind the nvr.

provels

@johnpoz @Bob-Dig Both DHCPv6 Server and RA are disabled. The FW is trying both IPv4 and v6.

My net is quite simple, it's like 1998... No DNS redirection, NAT reflection. I do use a couple HOSTS entries on a couple Windows boxes, but that should just blow past both machines.
No dupe IP because I'd probably be dead in the water w/o any gateway.
ARP table from the Pi below. The highlighted address is the LAN Bridge. It appears today's flood occurred between 07:33:46 and 07:33:47, hundreds of packets. Weird.

johnpoz

@provels that it odd that that mac shows up as FreeBSD Foundation? Don't think have ever seen that before. Are you running Bhyve, I think that might be a mac used in Bhyve??

Do you have any packages running on pfsense?

What box are you running pfsense on.. Can you post up an ifconfig output of that interface..

[24.11-RELEASE][admin@sg4860.home.arpa]/root: ifconfig igb0
igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN
        options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 00:08:a2:0c:e6:24
        inet 192.168.9.253 netmask 0xffffff00 broadcast 192.168.9.255
        inet6 fe80::208:a2ff:fe0c:e624%igb0 prefixlen 64 scopeid 0x1
        inet6 2001:470:<snipped> prefixlen 64
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[24.11-RELEASE][admin@sg4860.home.arpa]/root:

If you look through the full output of ifconfig - what shows up with that 58:9c:fc mac?

Are you running pfsense as VM in bhyve?

provels

@johnpoz It's running in hardware, on an Adlink Intel box, not VM . The MAC is the result of the bridge. When a bridge starts up it gets assigned a random virtual MAC by BSD. I've added that MAC into the bridge config so it's fixed. I run pfBlockerNG as the only DNS related thing. Doesn't seem to be a scheduled job, as it happened at 14:45 and 19:48 the day previous. Same flood to the same destination address.

bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN_BRIDGE0
        options=0
        ether 58:9c:fc:10:ff:88
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 2601:240:4e81:61a3:5a9c:fcff:fe10:ff88 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: ath1_wlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 22222
        member: ath0_wlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 33333
        member: rum0_wlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 370370
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=1<PERFORMNUD>

johnpoz

@provels so your running wifi bridged to your lan interface.

That could be problematic.. I have not spent any time with trying to use freebsd as wifi - just makes zero sense to me - freebsd and wifi are not a good fit..

Your problem could be related to that - not sure exactly how.. But what I can tell you is if your resolving - pfsense would have no way to even know to ask your pihole IP for dns. It uses roots to resolve, so unless something resolves to a NS it should ask to your pihole IP - unbound on pfsense would have no way or reason to ever send dns query to your pihole IP.

If I had to guess its client set to use pihole IP running through the bridge and for whatever reason the bridge mac is being used.. When talking to pihole - and pihole sees that at your pfsense IP.

You might want to look in the unbound status - this should list all the name servers that unbound knows to talk to for different domains or tlds, etc..,

example

Do you see your pihole IP listed in there? If not pfsense would have zero reason to ever send a dns query to your pihole IP, be it was pfsense itself looking for something, or some client asking unbound to resolve something. Unless you have something on pfsense that is set to use your pihole.. But I don't know what that could be?

You could look in resolv.conf

[24.11-RELEASE][admin@sg4860.home.arpa]/root: cat /etc/resolv.conf 
nameserver 127.0.0.1
search home.arpa
[24.11-RELEASE][admin@sg4860.home.arpa]/root:

In case what your showing in gui is not showing all of it?? Do you have the setting to let dhcp override dns?

Do you have any sort of vpn setup on pfsense? Where there could be another dns setup? That happens to point to your pihole IP?

Unless you have something else on pfsense running that can be told to query your pihole IP address? I can not think of anything that would do that.. And you say your not running any packages anyway, other than pfblocker.

If me I would prob sniff on pfsense on your bridge to see if you can see a client asking for what your seeing in the mass query.

edit:
if you don't want to scan through the gui output.. You could grep it for the IP of your pihole.

Example - I just picked a random IP I saw in the output

[24.11-RELEASE][admin@sg4860.home.arpa]/root: unbound-control -c /var/unbound/unbound.conf dump_infra | grep 23.61.199.65
23.61.199.65 60.3.103.in-addr.arpa. ttl 394 ping 24 var 120 rtt 504 rto 504 tA 0 tAAAA 0 tother 0 ednsknown 1 edns 0 delay 0 lame dnssec 0 rec 0 A 0 other 0
23.61.199.65 220.175.66.in-addr.arpa. ttl 688 ping 24 var 119 rtt 500 rto 500 tA 0 tAAAA 0 tother 0 ednsknown 1 edns 0 delay 0 lame dnssec 0 rec 0 A 0 other 0
[24.11-RELEASE][admin@sg4860.home.arpa]/root:

You could do that for the IP of your pihole..

If I do that on mine - you will see there is nothing that would talk to my pihole IP

[24.11-RELEASE][admin@sg4860.home.arpa]/root: unbound-control -c /var/unbound/unbound.conf dump_infra | grep 192.168.3.10
[24.11-RELEASE][admin@sg4860.home.arpa]/root:

provels

@johnpoz
Pihole IP not found in Status/Infra.

[24.11-RELEASE][root@fw.workgroup]/root: cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver ::1
search workgroup

Yeah, I know about Wi-Fi and FreeBSD, but it's just a lab. And actually, none of those adapters are in use regularly. The one I use is a free-standing AP. Has my phone, laptop, and thermostat on it.

But back to the weirdness, it's just that one external host that gets pinged backwards and floods. I wonder if the "smart" tv is smart enough to sniff for DNS servers? But why would .01 even look at .08?
It should just shrug it off. TV is wired, not Wi-Fi, address/DNS from DHCP. Weird. Done for tonight. Thanks for all the thoughts.

johnpoz

@provels said in DNS Puzzle:

Pihole IP not found in Status/Infra.

Well then its not pfsense (if all you have setup in dns is to talk to loopback) or anything asking unbound for dns that is causing it. So its something else that somehow is using the mac of your bridge you setup. Or something else running on pfsense talking to it directly.. Because the infra cache would show you all the NS that unbound has talked too.. if possible I would check that cache as soon after you see a mass query, they can fall off or if you have restarted unbound the infra cache would get flushed.

But if your not seeing it listed in the infra cache - then unbound didn't talk to it.

So you have tracked to a smart TV? Why would it be seen as your bridge mac? It is a wireless client and when pfsense bridges this traffic to your lan, its putting it on the wire as coming from the bridge mac?

Proxy Arp could do that, and have seen that in the past with like wireless extenders - but normally you should see the actual mac if on the same layer 2, even through a bridge.. Unless pfsense was routing that traffic - then your pihole could see traffic from some remote IP as the mac of the pfsense interface that sent the traffic too it.

But you should see the remote IP, just the mac of pfsense interface that routed it.. Unless pfsense was also natting the traffic.

provels

@johnpoz said in DNS Puzzle:

But if you're not seeing it listed in the infra cache - then unbound didn't talk to it.

Understood, agreed

So you have tracked to a smart TV?

Only from Googling the target hostname.
d1oxlq5h9kq8q5.cloudfront.net
All seem to ref Samsung TVs.

Why would it be seen as your bridge mac? It is a wireless client and when pfsense bridges this traffic to your lan, its putting it on the wire as coming from the bridge mac?

Yeah, no clue. But it is a wired client. I just cleared the log and will watch for it again. Thanks for all you input. I'll reply back when I see it again.

johnpoz

@provels wired client - wow that makes it weirder even - wireless ok something odd with the bridge and the wireless card in pfsense.

So the TV is on 1 side of the bridge and the pihole is on the other side? I mean if they are on the same side - then how would pfsense even be involved in the conversation, so there would be no way for your pihole to see the mac from the bridge. Is the pihole wired as well? And they are on different sides of the bridge?

Hmmm - wonder if mask could be wrong, and traffic is being sent to pfsense because TV thinks the IP its pointing to is on different network. I mean possible the TV could be trying to talk to say googledns or something - but you said you have no redirections or anything setup so even if it asked 8.8.8.8 pfsense wouldn't be sending that to the pihole. But if pfsense is seeing traffic to it with destination of IP of the pihole, could it maybe send that on, but from the mac of the bridge?

If you believes its the TV generating the dns queries - I would packet capture to validate that and see exactly what is going on, where is the traffic being sent, what IP and mac coming off the TV.

provels

@johnpoz

then how would pfsense even be involved in the conversation

Good question! TV and Pi are on the same flat net behind the bridge. Pi runs as a VM on a Windows server. Mask is right , TV gets DHCP. But would anyone be surprised if Google DNS was hard coded into the TV? After all, they try to make these things as stupid-proof as possible. I do attempt to block DoH in pfB. And I force all normal 53 DNS to the FW. But still, pfS shouldn't even know there is another DNS server on the net. As far as pfS knows, Pi is just another client. I'll watch it and post back if I have any revelations. Thanks again.

Bob.Dig

@provels said in DNS Puzzle:

And I force all normal 53 DNS to the FW

I think you have said you don't. So what do you do exactly.

@provels said in DNS Puzzle:

@Bob-Dig I only have one port forward, for OpenVPN, and that's not enabled unless I'm out of town.

I also would argue that this is not a port forward, just an open port.

johnpoz

@provels said in DNS Puzzle:

And I force all normal 53 DNS to the FW.

Yeah with @Bob-Dig here - you stated here that you don't do any redirection.

My net is quite simple, it's like 1998... No DNS redirection, NAT reflection.

If you redirect dns, and your TV is trying to talk to say 8.8.8.8 - where are you redirecting it too.. If the pihole than that explains your flood. If you were forwarding it to unbound, which resolves then it doesn't.. Even if redirected unbound would never ask pihole for anything unless you setup a domain override pointing to pihole, and you showed that is not happening when you stated you don't see the pihole IP in your infra cache. I haven't actually validated such a override would be in the infra cache - but I would think it would be.

Many iot devices these days hard code dns entries - I think it's horrible for them to do that, but yeah lots of them do.. The only way to try and stop that is with redirection, or just a block - and many of them will scream and holler and fail to move forward in a setup if they can't talk to their overlords via dns query to the outside.. Shoot some of them have started doing doh, which you can't really redirect and pita to block via whack-a-mole lists of known doh servers, even if you could like with the case with dot on its own port of 853, any sane client would be able to tell its been redirected because the cert being served wouldn't validate.

Good question! TV and Pi are on the same flat net behind the bridge.

Ok on the same side - because one side of your bridge is the wireless, and only 1 wired interface in the bridge.. So yeah same side makes sense - doh ;)

If that is the case then there is no freaking way pihole could ever see the mac of your bridge - unless you are actually redirecting traffic on pfsense and the TV is talking to something that would need to go through its gateway via the bridge connection on pfsense and your redirecting it to the pihole.

provels

@Bob-Dig I do, but the only client pfS's DNS should see is the Pihole.

i turned on the TV at 06:06 local and then saw this.

OK, so it looks like the TV is trying to go out the gateway and being bounced back to Pi. .74 is the TV, .01 is the pfS. Think that's it?

provels

@Bob-Dig said in DNS Puzzle:

I also would argue that this is not a port forward, just an open port.

I guess technically it is, since it's a random high port forwarded to 1194?

johnpoz

@provels that sure looks like the TV tried to directly talk to pihole, but then when it didn't get what it wanted seems like it says ok I will ask xyz out on the internet, which you then redirected.

Where exactly do you have that !pi_hole rule? source of ! pihole would be any IP that is not the pihole.. But that shows zero triggers with that 0/0 B -- but any other rule could allow that traffic.. What is your port forward - that looks like just a firewall rule that says hey any IP other than what IP is in the pihole allias going to the pihole alias ip(s) on dns_ports alias allow it.. As you can see with 0/0 there that rule has never triggered.. Which I don't see why it ever would since anything on your flat network wanting to talk to the pihole IP would never go through pfsense in the first place. What ports would you have in the dns_ports alias? Like I said you can not realistically redirect doh or dot (443 and 853) so creating an alias doesn't make a lot of sense.

Example here is a dns port forward redirection

Notice there is no 0/0 column - only a firewall rule would have that column..

johnpoz

@provels said in DNS Puzzle:

I guess technically it is, since it's a random high port forwarded to 1194?

Huh? A firewall rule that allows access to port 1194 on your wan interface isn't a port forward.. A port forward would be traffic hitting port X and being sent to port X or Y on different or same IP, etc.

What random high port are you talking about - the source port of some client wanting to talk to your vpn on 1194.. Yeah that is almost always going to be some random high port.. That you have no control over or ability to even know what it might be, etc.

provels

This is too much like work!!!

@johnpoz Yeah wrong rule. Here's the NAT rule. This was something I setup years ago and haven't had the need to look at since. I followed some recipe here on the forum.

Again, it's that Cloudfront host causing the flood and originates from the TV. Maybe best to just whitelist it. .74 def the TV.

@Bob-Dig Well, I thought I was port forwarding from the high port to 1194, but I guess I'm just passing the high port through. The OVPN server is disabled at present. Been a long time since I set this up, too!
Probably should have just left it at 1194 and NAT'd it at the FW if that's what I wanted.

johnpoz

@provels You have a few options.. Just block it talking to anything outbound for dns, doh is harder but not impossible, dot is easy with just a single port. Don't redirect it.

You can redirect it, and increase your flood number, or whitelist some of the stuff its looking for.. Problem with lots of these iot devices if they want to lookup xyz, and they don't get the answer they want or expect they can flood - like hey didn't get an answer, maybe if I just keep asking over and over again like every second maybe I will get an answer <rolleyes>. Maybe if I just ask faster will get an answer like 100 times in a second ;)

You could change the answer they get for what they are looking for, vs sending them nx or 0.0.0.0 - try sending them back loopback 127.0.0.1.. Also look to sending back different ttl vs what the default 2 seconds.. So if they do happen to have their own local cache - many iot devices don't.. Maybe if you give them some answer like loopback or all zeros with a ttl of an hour vs 2 seconds - they will only ask every hour vs every 2 seconds, etc.

Do some looking into what they are looking for - maybe some you might allow and not block, depends on what your wanting to stop?

Other option is just don't use internet on devices that flood like that - are you using the built in apps on the TV, or is this really some media stick.. Not all media sticks are as chatty and or easier to block specific ad fqdn, etc. that don't cause them to explode into a dns asking frenzy..

You could also just turn off the warnings of some of those rate limiting warnings in pihole - so sure it doesn't change what is going on, but it doesn't bother you warning you about something that you understand is going on..

provels

@johnpoz I do run a "Smart TV Blocklist" that's pretty aggressive; I have had to WL a few things to get the built-in app to work. I think the Pi default was 1000 hits before throttling, I dropped it to 250, thought I'd want to know if something went wild (and it did). I think I'll go ahead and WL that domain and leave it be. It's probably Samsung just wanting to know everything I've looked at today so they can squeeze their advertisers some more. Thanks again for all your input!