Thermostats going in and out
-
@stewart said in Thermostats going in and out:
his is a snippet from looking at one of the thermostats traffic. There are a lot of ARP requests looking for the firewall IP (192.168.30.1) and I don't see any replies but I don't know if that is normal or indicative of anything. I just don't see how it could be an issue on our end but the vendor insists that it is. I can't really capture on the WAN because all the devices are going to the same IP. I can't really see which request comes from which device so all I can capture is on the VLAN interface. It all goes over port 443 so the traffic isn't really any different than standard web browser traffic and connecting a laptop appears to work just fine.
Anyone have any ideas?Got to admit my Honeywell T5+ has been a little flakey as far as homekit is concerned for the past couple of months, sometimes it shows fine and other times it doesn't respond.
It's sat on my IOT network, I had to create a 2.4Ghz only SSID.
Hmmm...
https://twitter.com/Honeywell_Home/with_replies
-
I add just one more :
- Do you have a neighbour that uses an AP that transmits over the same radio channels, making communication close to impossible for some of your Wifi clients : your AC unit.
- Some other device like a leaking micro wave oven ?
- Etc etc.
@stewart said in Thermostats going in and out:
I can't really capture on the WAN because all the
WAN ??
capture on it's (v)LAN.
You can see the arp, DHCP and other traffic of every device.Over the Internet ?
These device have to call 'home' (some where on the Internet) -and your phone app connect also to this server ?
Why not a local server - keep you info where it belongs, inside your walls.Also, you could ping monitor all your AC units. **
They should reply 24/24. Except when the connection gets lost between the PfSence LAN NIC and the unit breaks.@NogBadTheBad : yeah. Some where in 2006 we've ordered some 35+ AC units, and 2 huge 'towers' outside to go to 100 % nuclear ( electricity here in France).
They told us : we have this new options like bleutooth, wifi etc - and I replied : Over my d**** body. I wanted the old, proven,; shielded RS485 that interconnects all that stuff.
Gadgets, toys and other stuff could use (can only use) wifi are all on the "don't care" SSID network.edit **
'abuse' the dpinger ( System > Routing > Gateways ) for that. The monitoring stats and graph are offered by the courtesy of pfSense ^^ -
This post is deleted! -
@gertjan said in Thermostats going in and out:
I add just one more :
- Do you have a neighbour that uses an AP that transmits over the same radio channels, making communication close to impossible for some of your Wifi clients : your AC unit.
- Some other device like a leaking micro wave oven ?
- Etc etc.
This is an isolated resort covering many acres. No other interference that would account for this.
@stewart said in Thermostats going in and out:
I can't really capture on the WAN because all the
WAN ??
capture on it's (v)LAN.
You can see the arp, DHCP and other traffic of every device.This is a capture on the VLAN interface. Capturing on the WAN is useful since it would show if any replies were coming back not making it to the VLAN and being blocked by the firewall.
Over the Internet ?
These device have to call 'home' (some where on the Internet) -and your phone app connect also to this server ?
Why not a local server - keep you info where it belongs, inside your walls.I didn't install the system, the AC company did. I don't even know if there is an on-premise system. Regardless, that's not the issue. I'm trying to see if there is any other way that I'm not thinking of to track down what's going on with the way it is.
Also, you could ping monitor all your AC units. **
They should reply 24/24. Except when the connection gets lost between the PfSence LAN NIC and the unit breaks.As far as I can tell the thermostats never lose connection. I've not tried pinging but all the packet captures I've done show all the units transmitting and receiving.
@NogBadTheBad : yeah. Some where in 2006 we've ordered some 35+ AC units, and 2 huge 'towers' outside to go to 100 % nuclear ( electricity here in France).
They told us : we have this new options like bleutooth, wifi etc - and I replied : Over my d**** body. I wanted the old, proven,; shielded RS485 that interconnects all that stuff.
Gadgets, toys and other stuff could use (can only use) wifi are all on the "don't care" SSID network.Unfortunately, that isn't an option from what I know. The units still need to connect back to the network somehow.
edit **
'abuse' the dpinger ( System > Routing > Gateways ) for that. The monitoring stats and graph are offered by the courtesy of pfSense ^^I can look into this. Hadn't thought of that.
-
@bmeeks said in Thermostats going in and out:
The first question to always ask in this scenario is "what changed?"
Isn't that always the question? Nobody will admit that something has changed until you find it and they suddenly "remember".
- Did the firewall get updated?
Nothing that would correlate to this. I haven't moved it to 2.5.0 yet.
- Did some new firmware get pushed to the thermostats?
No idea. Neither does their support.
- Did the software on the server they connect to change?
No idea. Neither does their support.
- Did you change ISP?
Nope.
- Are there any packages installed on pfSense? If so, do any of them block stuff?
I've disabled both Suricata blocking (which would show in the firewall logs) and pfBlocker. No change.
- Has something in the physical network changed such as a new switch, or some new component was added?
Nope.
If it was working fine and then suddenly stopped, it's pretty obvious something changed in the environment. Finding that "something" is the game now.
You know, it's not a game I'm fond of playing but I find myself on the court all too often. Thanks for the feedback.
-
Will the vendor share this cell phone app with you for troubleshooting? It would be interesting to actually watch what is happening within the app's reporting while at the same time capturing traffic on the VLAN and WAN interfaces.
Some kind of WiFi interference is certainly a possibility as @Gertjan mentioned. Although you would not expect it to cover an entire typical resort as that's a large area, maybe some employee brought in their own WiFi AP and stuck it somewhere? One thing to check, if your AP infrastructure provides that info, is what WiFi sources are showing up and their signal levels. For example, in Ubiquiti APs the controller software can show you other non-affiliated transmitters whose signal is being received by the APs and at what level. But on the other hand, if interference was the issue you would really expect that to impact all WiFi on site (thus web browsing and everything else would suck).
Lastly, based on that Twitter link @NogBadTheBad provided, it would make you suspect Honeywell might have an issue on their end ???
I think many of these old-school companies trying to step up in the Internet world have difficulty. I have a pair of Lennox Wi-Fi thermostats that call home to a Lennox server. You can then connect to the Lennox server site via your phone to see and control your system. That process works maybe 65% of the time at best. The remote end server disappears fairly regularly for hours or a few days, then it's back. I suspect they just don't put enough capacity in their backend to handle the traffic from thousands and thousands of thermostats scattered all over everywhere.
-
Just one word about this, as I want you to think 'out of the box' (I'm not saying your setup is wrong, as I can't tell ) :
@stewart said in Thermostats going in and out:
This is an isolated resort covering many acres. No other interference that would account for this.
If you receive 3G/4G/5G on your premisses, everybody who has a cell phone can 'share' the data connection - and his phone becomes an Wifi AP.
All phones are not equal. Some just open the radio and start blasting away.
Not very good for a connection of the (your !) local AP and AC unit close by.@stewart said in Thermostats going in and out:
I can look into this.
System > Routing > Gateways :
On the PORTAL interface I have AP's like 192.168.2.2 192.168.2.3 192.168.2.4 etc.Check :
so that dpinger doesn't take action when it start loosing the ping.
It's the monitoring that you want. See : Status > Monitoring, select Quality (left) and the name you gave to it (right). -
I have 2 Honeywell thermostats in the house that had been performing flawlessly for a few years. Recently however there began some connectivity issues, red warning screens on the app, changes to settings not being acted upon, etc. I thought it was on Honeywell’s end and still believe that. Maybe they’ve been going through some unannounced upgrades. I hope that’s all it is. Very disruptive though.
-
We use IgniteNet APs so we can see all of the devices connected, how long they are connected for and gaps in connection time. This is what I can see from one of the units:
As you can see, always connected with regular traffic. -
Found something, maybe
Firewall logs show some oddities:
Not sure what to make of this.The rule that triggered this action is:
@5(1000000103) block drop in log inet all label "Default deny rule IPv4"
-
The obfuscation of the IP addresses makes it a little hard to follow. So I assume the 199.62.xx.xx address is your WAN IP and the 97.xxx is the Honeywell server end ??
If correct, then are the IP addresses in each block/pass sequence identical? That is weird. It's like the state is not working or something maybe.
Do you have any sort of multi-WAN configuration? I'm assuming "no" since you didn't mention it.
Edit: wait a minute... looking at those logs again is really confusing. Why is 443 the Source port? I would think that is the destination port, and there would be a random source port.
-
just a guess - you may try to set Firewall Optimization Options to conservativ, if not already set:
-
@bmeeks said in Thermostats going in and out:
The obfuscation of the IP addresses makes it a little hard to follow. So I assume the 199.62.xx.xx address is your WAN IP and the 97.xxx is the Honeywell server end ??
97.x.x.x is the WAN port. 199.62.x.x is the Honeywell server.
If correct, then are the IP addresses in each block/pass sequence identical? That is weird. It's like the state is not working or something maybe.
Weird indeed! Notice that it's like it the WAN receives 2 packets each time, the first is blocked while the second is allowed.
Do you have any sort of multi-WAN configuration? I'm assuming "no" since you didn't mention it.
Nope.
Edit: wait a minute... looking at those logs again is really confusing. Why is 443 the Source port? I would think that is the destination port, and there would be a random source port.
The thermostats connect out from the 97.x.x.x IP using a random port to 199.x.x.x on port 443. This is the reply back so it would come from 443 back to the random port.
-
@pete35 said in Thermostats going in and out:
just a guess - you may try to set Firewall Optimization Options to conservativ, if not already set:
I've set it, so we'll see how it goes.
-
@stewart said in Thermostats going in and out:
@bmeeks said in Thermostats going in and out:
The obfuscation of the IP addresses makes it a little hard to follow. So I assume the 199.62.xx.xx address is your WAN IP and the 97.xxx is the Honeywell server end ??
97.x.x.x is the WAN port. 199.62.x.x is the Honeywell server.
If correct, then are the IP addresses in each block/pass sequence identical? That is weird. It's like the state is not working or something maybe.
Weird indeed! Notice that it's like it the WAN receives 2 packets each time, the first is blocked while the second is allowed.
Do you have any sort of multi-WAN configuration? I'm assuming "no" since you didn't mention it.
Nope.
Edit: wait a minute... looking at those logs again is really confusing. Why is 443 the Source port? I would think that is the destination port, and there would be a random source port.
The thermostats connect out from the 97.x.x.x IP using a random port to 199.x.x.x on port 443. This is the reply back so it would come from 443 back to the random port.
Ah, okay. So I had the IP addresses backwards.
The double replies are indeed strange. Perhaps @pete35's solution will work. That setting will stretch out the state table entry expiration times. Still strange, though, that two replies seem to come back at essentially the same time (or certainly within one second of each other as the logger shows the times as the same due to its one-second resolution).
-
@bmeeks I had to undo it. Just got a call that for about the last hour or so calls have been going straight to voicemail intermittently. Looks like it broke the VOIP at the location.
