IoT Devices Not Using DNS from DCHP
-
Network design/info - including fiber and any other address spaces:
Actually, the fiber run to the barn is the short fiber run! We're in a rural area and the only option for good internet service is Starlink. With all the trees around, the nearest place I could put the dish for that is a 1,000' run from the house to the "Outpost," my joke name for the post out in the field with the Starlink dish and router on it. I have the dish hooked up to the Starlink router (in a temperature controlled box). It uses the 192.168.1.xxx address space. It has its own DHCP for it's LAN interface.My Netgate box with pfSense is in the house. I use the LAN and WAN interfaces only. WAN connects to the Starlink router through that 1,000' of fiber and the pfSense WAN is in the Starlink's LAN and gets its address from the Starlink's DHCP server. So I have a 2nd address space controlled by the Starlink router, but all that's on it is my pfSense WAN interface.
Both fiber runs are invisible to the systems on the network. For example, the pfSense WAN interface has a CAT5 ethernet cable connected to an ethernet/fiber converter, then there's the long fiber run, then the ethernet/fiber converter in the box with the Starlink router, then another CAT5 to the router. The converters don't even have IP addresses, they are completely transparent on a networking level. The only DHCP servers I have (unless some LInux system is doing something it shouldn't) are the ones on my Netgate box and the Starlink one. So as far as any fiber goes, it acts just like it's a long ethernet cable (except it's not prone to lightning strikes - which damaged some of my equipment before I used fiber).
Address space on my LAN
The address blocking is for me. The TL;DR version is that I have reading and perception issues and that, every now and then, I run into something that asks for an IP address and either won't work with a machine name, or won't allow me to enter a name instead of an address. While I can check the DHCP leases for those situations, to find a system, this makes it a lot less confusing for me.pfSense and the bogon DNS issue
I get that it's not pfSense that's causing the issue. I'm not sure how to continue tracking it down and finding out what's going on. I have discussed this with the openHASP people and they are sure it's something on my LAN. My understanding is that I can't capture packets being sent from one address to another (non-broadcast) - is that correct? So maybe the next step to find out what's going on is to do a capture for broadcasts on port 5353?I figure if it's somehow getting a bogon address for a DNS server, it has to be asking for a server AND something has to be responding. Maybe even the Starlink router? From what I've read, though, it seems like mDNS doesn't even ask for an actual server address. Somehow it's trying to get and being given a DNS server address. I don't get what's going on there.
-
@TangoOversway said in IoT Devices Not Using DNS from DCHP:
While I can check the DHCP leases for those situations, to find a system, this makes it a lot less confusing for me.
Which is why I always assign static to devices that are going to be on my network on a perm or semiperm basis.. If I had a device, I let it get dhcp, then setup a reservation for that mac and now that device will always be that specific IP give it. While converting over a large network to this model can be a pain.. But I do it whenever I add a new device or replace a device.. Like light bulbs..
Another option that is now maybe becoming viable is use of dhcp registration with kea.. Before this could be painful because anything to do with a lease would cause a restart of unbound - which could be problematic depending on number of devices, length of your lease.. Why I use reservation registration because that only happens when unbound starts, or you add a new record, etc.
example - here is a few of my lightbulbs..
The DX are all in the light above my dining room table.. The names are not in order - over the years I have changed some of them.. Had some issues power blowing them out.. Took me a while to figure that out ;) was the stupid dimmer switch on the wall providing power to the ficture.. Once I replaced that with just normal switch no more issues with bulbs ceasing to function. New ones I added I started with D1x vs just like D3.. So I knew which ones I had replaced..
I also isolate devices to their own segments - I have a segment for iot devices, like alexas and lightbulbs, thermostat, etc.. I have different one for my rokus, etc. Clients like phones and tablets are on their own segment, etc. etc.
As to capturing traffic that is not broadcast - no pfsense not going to "see" any traffic between devices to be able to capture it - your only going to see traffic sent to pfsense interface (ie to route out) or directly asking it something like dns, ntp, etc. If wireless is involved you might be able to sniff on your actual AP.. Some of the unifi AP supports this.. mine do..
Other option is to if you have switch that supports port span/mirroring - where you could create a span port to send all traffic seen son port x or x and y to port z.. And you have your sniff on port Z..
If your goal is troubleshooting mdns - while you can see a query on pfsense because it is sent to multicast address, you can't really tell if device saw the query or sent an answer because the answer would/could be directed to who asked - not a multicast response where everyone could see.
Not sure what your asking these guys - but again pfsense is not involved in devices on the same network - it just has no way to do such a thing - unless the traffic was flowing over pfsense in a bridge.. It just has nothing to do with clients A and B talking to each other - be it via broadcast or multicast or unicast, etc.. pfsense just not involved in the conversation at all.
Could it be something else like a switch doing some sort of ACL blocking multicast or IPs or Macs - sure.. But if that was the case you would of had to configure it, and should know about. Now wifi on the other hand might be doing something, that you might not be aware of, AP/Client isolation is a common issue with devices not being able to talk to each other or discovery protocols working. They could also have some function that seems useful that could be doing it, if your not aware of how that function works..
Those can cause all sorts of problems if you need multicast to work, etc. You could look to your APs if your devices are wireless or one of them are, etc.
Other option for sniffing would be on the actual devices in question.. Then you can validate that device A sees the query from B, and B sends the answer, etc. And then A sees that response, etc.
-
@johnpoz said in IoT Devices Not Using DNS from DCHP:
Which is why I always assign static to devices that are going to be on my network on a perm or semiperm basis.
Ah! So you're doing pretty much what I'm doing. In my case, I allow room for expansion. So if I'm doing "human use" computers in the 172.16.7.xxx range and, say, need the first 10 addresses that I know I'll use, I'll likely reserve 30 addresses for that, since computers tend to multiply like rabbits in the spring! So that's part of the reason I have such a large address space. There's also the way I subdivide it, like using 172.16.4.xxx for IoT only - and that's a large block I set aside, since so many new IoT products keep coming out.
@johnpoz said in IoT Devices Not Using DNS from DCHP:
If your goal is troubleshooting mdns - while you can see a query on pfsense because it is sent to multicast address, you can't really tell if device saw the query or sent an answer because the answer would/could be directed to who asked - not a multicast response where everyone could see.
I'm not so sure that it's mDNS that's the issue - but I do not get why something that is using mDNS is getting a DNS server address. Seems to me that unless mDNS is asking for a server, the system should not be getting one.
While I think what one of the openHASP developers has said, that the issue could very likely be something on my LAN, I do question why that device is even open to being told, "Use this DNS," unless it's asking for one.
@johnpoz said in IoT Devices Not Using DNS from DCHP:
Could it be something else like a switch doing some sort of ACL blocking multicast or IPs or Macs
My switches are unmanaged and I have never set them up to do anything special. I'm not even sure I can configure them. I seriously doubt Home Assistant is doing something like that without me setting it up (as you pointed out - I would have had to set it up). I do have some Linux servers, mainly for things like media/streaming service and a few NAS systems. I don't see how any of those could be sending out any DNS server info.
@johnpoz said in IoT Devices Not Using DNS from DCHP:
As to capturing traffic that is not broadcast - no pfsense not going to "see" any traffic between devices to be able to capture it - your only going to see traffic sent to pfsense interface (ie to route out) or directly asking it something like dns, ntp, etc.
Okay, that's a good point. If I could capture packets being sent from the ESP to the WAN, I could at least rule out any communication with something not on my LAN. But I think that would have already shown up in the capture I already did - wouldn't it?
@johnpoz said in IoT Devices Not Using DNS from DCHP:
Now wifi on the other hand might be doing something, that you might not be aware of, AP/Client isolation is a common issue with devices not being able to talk to each other or discovery protocols working.
I've been working my way through everything that I have on my LAN as I've been reading your post - not all in the order you wrote everything out. I think I can rule out my Linux servers. They're Debian and I don't see Debian including a service that would do this as a default service it would leave on. I don't see either NAS doing it, or my switches. I had not thought about wifi. I use 2 TP-Link Deco M9 units in my house (and 2 more in the barn, but this ESP is in my house) and I have never really felt they were solid. Mobile devices can be within 8' of them and report a weak signal. I've also had problems with their behavior when coming back up after a power outage. There were other incidents along the way that have bothered me. But the control interface is dead simple - too simple, really, since there are settings I have wanted to change, but can't. I don't trust device like that, and if I had known how limited the control app was, I wouldn't have bought the system. I'm going to look through that and see if I can find something on it that might misbehave.
-
@TangoOversway said in IoT Devices Not Using DNS from DCHP:
hat is using mDNS is getting a DNS server address.
Where are you seeing that - what you posted is how mdns works - its asking everyone if via the multicast address. 224.0.0.251 is everyone that would answer mdns.
eing sent from the ESP to the WAN
Why would your esp being sending traffic to the "wan" if it wants to talk to your devices on the local network? What does the esp device need the wan for to do openhasp?
The esp device is just going to talk to the MQTT broker.. Via the IP you put in..
http://www.steves-internet-guide.com/mqtt-works/
Are your esp devices on the pfsense wan or lan? Where is your broker? They should both be on the lan of pfsense.
-
@johnpoz said in IoT Devices Not Using DNS from DCHP:
Where are you seeing that - what you posted is how mdns works - its asking everyone if via the multicast address. 224.0.0.251 is everyone that would answer mdns.
Okay - I'm asking to rule out things. The things you've just helped me rule out may seem obvious to you, but I was asking about them just to be sure they're not a problem.
My IoT and ESP devices are on my LAN, on the "innermost circle," where the pfSense system is the DHCP and DNS. My broker is in the same LAN.
The only thing on the WAN side of pfSense is the Starlink router, and, of course, on the WAN side of that is the whole internet.
-
@TangoOversway what is not working exactly?
You stated in your OP, that " is not getting the info from my pfSense DHCP server to use the pfSense DNS"
Think you have gone down some rabbit hole because you don't understand how this works.. Your esp devices talking to your broker has zero need for dns to be honest.. You set the broker IP on the device. It talks to the broker.. You setup the client in the broker.. To know what info to get from it or send to it..
Or they discover each other? But normal dns, ie pfsense is not part of the equation.. That dns would be for when whatever is a dhcp client on your network needs to actually resolve a fqdn like google.com or something on your network host.yourlocaldomain.tld - none of which should be required for you to use your esp with your broker - that talk MQTT.
-
@johnpoz said in IoT Devices Not Using DNS from DCHP:
Think you have gone down some rabbit hole because you don't understand how this works.
That's about it. I've learned a LOT about this on this thread!
But even if the ESP is using mDNS and broadcasting, somehow it's still asking for a DNS server and something is giving it the bogon as a DNS server - right?
So, yes, the broker and the ESP find each other, IF I give the ESP the broker's IP address. I can't use the broker's name. (Or if I added ".local" on the broker's name, would that help, even if it's not my local domain?)
-
@TangoOversway said in IoT Devices Not Using DNS from DCHP:
something is giving it the bogon as a DNS server - right?
Show us this - like you did with your other packet capture.. Pfsense would never give out that IP, unless you specifically configured it.. Its not a valid IP.. Maybe you have a typo on your dhcp config where you can give out dns?
You could put fqdn or ip if the device where your putting it can resolve that name.. Again mdns is not normally something you put in as a fqdn - no I would not suggest putting brokername.local.. but if your esp device is using pfsense as its dns, you could put in broker.yourdomain.tld.. Where yourdomain.tld would be the domain pfsense is set for.. Default is home.arpa currently I believe but you would have to check your setup.
Then if you have setup pfsense to resolve the fqdn your going to put in.. broker.home.arpa for example, either via dhcp registration or dhcp reservation registration or with a host override.
Why would anyone want to use mdns when they have dns there to use.. I mean what is better asking 1 IP (your nameserver) directly via unicast hey give me the ip of something.domain.tld vs spewing noise on your whole network asking hey is your name something - let me know what your IP is..
unicast is like handing someone a note to ask them hey do you know billys phone number. Vs yelling in a crowded room - hey Billy whats your phone number.. You don't even know if billy is in the room. Now everyone knows your asking for billys phone number.. But they will not give it to you even if they know it..
Vs yelling asking for people phone numbers - isn't easier to go ask the doorman (dns) hey whats billys phone number. Since everyone should of let the doorman know hey if anyone is asking here is my phone number.
-
@johnpoz said in IoT Devices Not Using DNS from DCHP:
Show us this - like you did with your other packet capture.. Pfsense would never give out that IP, unless you specifically configured it.. Its not a valid IP.. Maybe you have a typo on your dhcp config where you can give out dns?
I'm not clear how to show that with packet capture. Just capture all packets going to and from this device and pfSense? And if it's a typo, wouldn't I be having this issue with other systems using that bogon as a DNS server?
@johnpoz said in IoT Devices Not Using DNS from DCHP:
Why would anyone want to use mdns when they have dns there to use.
My best guess, in this case, is because the project is for IoT devices and it's unknown what kind of LAN a device will go on, so they're using mDNS as a default. But that's only a guess. I'm not clear on the reasoning. Personally, I'm not a fan of multicasting for reasons similar to what you've mentioned: It means extra traffic on the LAN and often it's not needed. I did use it in ONE case in a Java program I was writing. The jar file was on Samba share on an embedded system and my clients had a very simple Java class on their desktops. They'd run that class on their desktops, it would do a multicast to find the embedded system. That was a one-time thing each time the program was run - find the system, then shut up. Once the system was found, the jar file could be used for the rest of the program. That made it easy for me to do updates on the jar file (since I could communicate with the embedded system) and they didn't have to do any configuration or setup other than put the one Java file on their desktop.
-
@TangoOversway said in IoT Devices Not Using DNS from DCHP:
wouldn't I be having this issue with other systems using that bogon as a DNS server?
Ding Ding Ding - you get a the cookie! Now your thinking!! Unless you setup a reservation specific for that clients mac address to hand him specific details - yup if pfsense was handing out that bogon IP to clients to use as their dns.. Then everyone using your dhcp server would get that IP for their dns.
As to showing - well for starters where did you get the idea that was happening in the first place - where did you see it?
As to using mdns - yup lots things use mdns, your phone more than likely is using mdns to find your printer.. But mdns again does not go to 253.x.x. mdns is ALWAYS to 224.0.0.251 which is a multicast address
But still again - what is not working with your esp and your broker?? Or your still down some rabbit hole about this 253.x address that you can't even show where you came up with that idea.. If your client was doing a query for that - then yeah you could could easy sniff for it.. either on the client itself, or since its for sure not on the devices via the IP it has - it would have to send it to its default router (pfsense) in some hope to get there.
-
When I try to just use the system name for MQTT on the ESP32, it won't find it with mDNS. Never. That's what got me going on the whole thing - why isn't it doing that? I've had one or two other systems with similar issues: They either don't allow me to enter a system name or don't work with one and need an address instead of a system name. The short version is trying to keep track of, look up, or remember numbers is a pain. Sometimes I have to use paper with a hole in it and read one quad at a time because of reading issues. So being able to use a name instead of an address makes things a lot easier for me. (Which is pretty close to why DNS was invented in the first place, isn't it? To make it easier to remember how to get to a specific system on the net?)
@johnpoz said in IoT Devices Not Using DNS from DCHP:
As to showing - well for starters where did you get the idea that was happening in the first place - where did you see it?
I didn't see anything in packets. But what I did see is that when the ESP was using mDNS, it would always end up with the bogon as its DNS server. I would think that's why it would not be able to find my system on the LAN - because, for some reason, even though it's supposedly using mDNS, it still gets a DNS server.
That leads to the issue I've brought up several times: Doesn't the ESP have to be asking for a DNS server to get one, even if it's a bad one (the bogon address)? The only thing on my LAN that should be handing out a DNS server address is my DHCP server, which should be using my pfSense firewall/DHCP server/DNS server.
I'm trying to figure out how this device ends up with a bogon DNS server instead of getting the DNS info through mDNS or from my DHCP server. It seems to me, if mDNS isn't working to get the broker system from the system name, AND that it's getting a bogon DNS server, something on the ESP is wrong.
Here's my logic: If it's using mDNS, it should be getting the needed information to contact my broker from the system name AND it shouldn't be using a normal DNS server. BUT - it's using a DNS server, even if it's a bogon, so it's somehow bypassing mDNS, even when it's enabled. With mDNS on, it's doing something it should NOT do.
Added to that, when I disable mDNS, it still gets that bogon DNS server somewhere around 10% of the time. It's not behaving reliably when it gets the right DNS server 90% of the time, and a bogon 10% of the time.
-
@TangoOversway said in IoT Devices Not Using DNS from DCHP:
My switches are unmanaged and I have never set them up to do anything special. I'm not even sure I can configure them.
What switches are you actually using? Are they able to be managed to create VLANs?
Doing so enables you to track and control the interactions you are trying and failing to do without VLANs.
I strongly recommend you configure VLANs to segregate and control your network.
-
@TangoOversway said in IoT Devices Not Using DNS from DCHP:
it would always end up with the bogon as its DNS server
Where did you see this?? In its resolv.conf? Where exactly are you seeing this 253 address?
Your esp doesn't even need dns to talk to the broker - be it normal dns or mdns.. Tell it what the broker Ip IS.. Does it now work? Or not?
We are going down some rabbit hole that you keep saying its using this 253.x address for dns - but have yet to show us where your seeing this dns being used?
If you show us where your seeing this - maybe we can figure out where its coming from.. But as you correctly pointed out if pfsense was handing it out via dhcp - then every dhcp client would get it, and your dns wouldn't work.
Unless you setup specific dhcp reservation for a specific client that is different than your normal dhcp settings.
-
The ESP system itself reports this:
That's what I've been saying - that somehow the ESP system is getting this bogon as a DNS server.
@johnpoz said in IoT Devices Not Using DNS from DCHP:
Your esp doesn't even need dns to talk to the broker - be it normal dns or mdns.. Tell it what the broker Ip IS.. Does it now work? Or not?
Yes, but, again, IP addresses (as opposed to names) are quite confusing to me and sometimes I have to use paper with a cutout for the numbers so I can read them one at a time to be sure I have the right address. I am also doing work on my network and have changed the broker from one machine to another and will probably be doing so 2-3 more times as I make other changes - so if I could use just the name, it would make this process a lot easier.
Also, since this ESP32 is getting this bogon as a DNS server (according to the ESP32 itself), where is that coming from?
-
@TangoOversway that is from a wifi connection.. You sure your wifi is not handing that out? Or it could be set on device itself.. Just because you use dhcp to get an IP doesn't mean you can't set a dns locally.
Have already gone over this - if pfsense was handing out that for dns in its dhcp server then every single client on your network would be getting it. Every one..
Unless you have setup a reservation for that device - and changed the settings.
On your pfsense do you have a dhcp reservation set for that mac?
Here look - here is my normal dhcp scope for my 192.168.3.0/24 network... I hand out all clients on this network dns of 192.168.3.10, by default it would hand out pfsense IP on that interface.
But in a reservation for a specific client, my pihole in this case I hand out a different dns server.. Pfsense IP 192.168.3.253
Do you have any such reservations (static mappings) setup for your esp box?
Lets see your dhcp server settings on pfsense, and if you show any static mappings at the bottom of the page
Also we already went over how to validate with a packet capture on pfsense - what IP it offers to the dhcp client.
You should see pfsense see the discover from the esp the offer and ack, etc.
-
@johnpoz Okay - just realized that even though I've assigned it an address. I have not, specifically, told it what to use for the DNS and gateway. It's getting the proper gateway. I think I forgot I have to specifically set the DNS in each DHCP entry. (I thought it was automatic.)
I'll test that out when I get back to that computer this afternoon.
-
@TangoOversway if you leave the dns blank - it will provide the IP address of pfsense interface.
See when I delete - it shows the default which is pfsense IP on that interface
See how its a gray color.
-
@johnpoz I thought that was how it worked - leave the DNS, search domain, or gateway blank, and it should use the default - and I have set them, in the DHCP server page, to the pfSense system. So, by transitive property, it should be getting the proper DNS server from the pfSense DHCP server. Otherwise, as we've agreed, other systems would be having this kind of issue.
-
@TangoOversway the only way this one client could be different is if you set a reservation for it and changed what it points to, or it is set locally on the device..
Any os can set the dns locally and still get IP from dhcp.. Even windows allows you to do that.
You can even do that on your phone, etc.
Or its not using pfsense dhcp and you some other dhcp handing out the same IP range, etc. Like dhcp on your wifi or something.
But yes if no reservation setup in pfsense for this mac address or every other mac address on your network having their own reservations.. And pfsense was handing out that IP for dns - everyone would get it.
That doesn't seem to show the IP it got dhcp from - but a simple sniff on pfsense would validate it handed out the info that client, or you could just look in pfsense leases table, etc.
And if you want to for sure validate pfsense is not handing out that IP you could do the packet capture of the dhcp transaction and look for yourself what pfsense offered.
-
@johnpoz You're going to find this extremely interesting. I have double checked all of this to be sure.
I know about the defaults and they're well labelled. If you don't set the DNS and gateway in the DHCP page, then it'll use the pfSense/DHCP/DNS address as a default. So if I've got it at 172.16.7.1, the DHCP server will provide that as the DNS server and gateway by default, even if I don't specify it.
I had not specified it in the DHCP configuration page, and had not specified it in the entry page for the ESP chip. (If I'm using a default, I don't explicitly specify it again "downline," since that just means I have to change it later if I ever change from the default.) So here's the DHCP Server configuration page:
Note that while it states that the pfSense unit acting as DHCP server will be used for the DNS and gateway, that it does not fill them in, ghosted. (It could be I need to update to the latest version for that.)And here's the DHCP page for the ESP32. Same thing. The default is not showing in gray:
So I went through and set the DNS and gateway in both pages and saved them, then restarted the DHCP for it to take effect. Then I rebooted the ESP32. I checked the info to get the new DNS info after the reboot:
FINALLY - it has the correct DNS server! I even waited a couple minutes to check to be sure it wasn't just something that had not been set yet after reboot and it was still correct, so I took a screenshot.
BUT - and this is the part that's just weird and interesting - I took that screenshot, was so glad it showed the right DNS, that I wrote up this post. Then, just to be sure I read it right, I went back and looked at that page again, and it's back to the bogon:
Somehow, between about 1-2 minutes after boot, and in the time it took to post the screen caps, it changed from the right DNS to the bogon.
