Honeywell HVAC Gateway SPI Compatibility Help
-
My HVAC system is setup with a Honeywell Prestige IAQ and uses an IOT gateway hub to connect the system to Honeywell servers. I currently connect to the gateway via ethernet to a Unifi switch connected to the LAN subnet/DHCP server on pfsense.
My LAN clients all function as expected, except for this one device. I even bought a second HVAC hub thinking the hardware was defective. The problem was present on this one as well. The hub connects to it's servers as expected using either DHCP or a DHCP reservation. It stays connected for about 30 min then disconnects. During the time it is connected, I can ping it using it's private IP. Once it disconnects and goes red, ping shows host unreachable.
This is what the DCHP server log is showing.
Jul 11 08:18:52 dhcpd 75199 DHCPOFFER on 192.168.92.200 to b8:2c:a0:56:55:fa via lagg0.4091
Jul 11 08:18:45 dhcpd 75199 DHCPOFFER on 192.168.92.200 to b8:2c:a0:56:55:fa via lagg0.4091
Jul 11 08:18:39 dhcpd 75199 DHCPOFFER on 192.168.92.200 to b8:2c:a0:56:55:fa via lagg0.4091
Jul 11 08:18:35 dhcpd 75199 DHCPOFFER on 192.168.92.200 to b8:2c:a0:56:55:fa via lagg0.4091
Jul 11 08:18:33 dhcpd 75199 DHCPOFFER on 192.168.92.200 to b8:2c:a0:56:55:fa via lagg0.4091
Jul 11 08:18:33 dhcpd 75199 DHCPRELEASE of 192.168.92.200 from b8:2c:a0:56:55:fa via lagg0.4091 (not found)
Jul 11 07:21:33 dhcpd 75199 DHCPACK on 192.168.92.200 to b8:2c:a0:56:55:fa via lagg0.4091
Jul 11 07:21:33 dhcpd 75199 DHCPREQUEST for 192.168.92.200 (192.168.92.1) from b8:2c:a0:56:55:fa via lagg0.4091I contacted support for the Honeywell hub and was sent a document that includes the following, which I think may be the underlying cause.
Stateful Packet Inspection / Proxy services may affect connectivity / communication
The only thing I can think of to try is creating rules that change the state type to one that is compatible. This is an area I have not needed to venture into, thus the ticket. I would like to understand what the issues is, if this can be figured out on my end, and what I can do to mitigate it. I have no hope of meaningful support from the Honeywell side, so the forum is my only option.
Since pfSense uses SPI for the firewall, I am wondering if something is going on that I can figure out using the logs. I am also afraid I am chasing my tail and maybe there was some sort of firmware change on the hub that has induced my connection issues. It is also IOT and that tends to represent badness.
The gateway connects to one of three urls about every 5 seconds.
tccprod01.honeywell.com
tccprod02.honeywell.com
tccprod03.honeywell.comHere is a LAN rule I setup to log hub connections. The hub's private ip is 192.168.92.200.
Jul 11 08:02:42 LAN Honeywell TCC URLs Test (1657545636) 192.168.92.200:63421 199.62.84.151:443 TCP:S
Jul 11 08:02:37 LAN Honeywell TCC URLs Test (1657545636) 192.168.92.200:61166 199.62.84.153:443 TCP:S
Jul 11 08:02:32 LAN Honeywell TCC URLs Test (1657545636) 192.168.92.200:59162 199.62.84.152:443 TCP:S
Jul 11 08:02:27 LAN Honeywell TCC URLs Test (1657545636) 192.168.92.200:61505 199.62.84.151:443 TCP:S
Jul 11 08:02:22 LAN Honeywell TCC URLs Test (1657545636) 192.168.92.200:64286 199.62.84.153:443 TCP:S
Jul 11 08:02:17 LAN Honeywell TCC URLs Test (1657545636) 192.168.92.200:55048 199.62.84.152:443 TCP:S
Jul 11 08:02:12 LAN Honeywell TCC URLs Test (1657545636)Here is the states summary
192.168.92.200 41 tcp 41 1
States Table
LAN tcp 192.168.92.200:54670 -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:57265 (192.168.92.200:54670) -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:53542 -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:9080 (192.168.92.200:53542) -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:58184 -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:14945 (192.168.92.200:58184) -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:51024 -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
WAN tcp xxx.xxx.xxx.xxx:60744 (192.168.92.200:51024) -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
LAN tcp 192.168.92.200:50940 -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:11157 (192.168.92.200:50940) -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:57335 -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:13094 (192.168.92.200:57335) -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:52679 -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:33741 (192.168.92.200:52679) -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:58033 -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
WAN tcp xxx.xxx.xxx.xxx:61701 (192.168.92.200:58033) -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
LAN tcp 192.168.92.200:59480 -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:58257 (192.168.92.200:59480) -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:59126 -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
WAN tcp xxx.xxx.xxx.xxx:9098 (192.168.92.200:59126) -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
LAN udp 192.168.92.200:53733 -> 192.168.92.1:53 SINGLE:MULTIPLE 1 / 1 69 B / 85 B
LAN tcp 192.168.92.200:64770 -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
WAN tcp xxx.xxx.xxx.xxx:22264 (192.168.92.200:64770) -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
LAN tcp 192.168.92.200:61931 -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:24019 (192.168.92.200:61931) -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:60205 -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:6443 (192.168.92.200:60205) -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:56862 -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:21696 (192.168.92.200:56862) -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:49638 -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:62129 (192.168.92.200:49638) -> 199.62.84.153:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:61157 -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:15082 (192.168.92.200:61157) -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
LAN tcp 192.168.92.200:63934 -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
WAN tcp xxx.xxx.xxx.xxx:37174 (192.168.92.200:63934) -> 199.62.84.151:443 FIN_WAIT_2:FIN_WAIT_2 5 / 5 260 B / 256 B
LAN tcp 192.168.92.200:52879 -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:54369 (192.168.92.200:52879) -> 199.62.84.152:443 FIN_WAIT_2:FIN_WAIT_2 5 / 4 260 B / 216 B
WAN tcp xxx.xxx.xxx.xxx:64655 (192.168.92.200:52197) -> 199.62.84.152:443 -
@jpvonhemel
2Q:- What's your DHCP lifespan set to?
- What's the Firewall Optimization Options set to? (System->Advanced->Firewall)
-
Thank you for responding to my post. Here are the answers.
DCHP Lease Time is the default 7200 seconds/120 minutes.
Firewall Opimization Options are set to Normal/Default.I want to add that I decided to see what would happen if I used a wifi-ethernet bridge in between pfsense and my honeywell hub. The result is it has not disconnected in over an hour. Somehow the bridge is creating a new mac address for the Gateway. Only the last three octets match the proper hardware mac address of the Gateway, which makes me question how it is connecting at all, since I register the gateway hardware mac with Honeywell.
-
@jpvonhemel OK so at 120 minutes the default "we are going to re-lease this IP" is 30 minutes. I would do a pcap on your Honeywall HVAC device to be sure.
Also change to "Conservative" on the options option to be sure... the packets you're showing above suggest that it's dropping the sate on the pf side and the SPI is refusing to open new requests.
-
I changed the firewall optimization to conservative and put the hub back on the unifi switch.
I am reading this may be a DHCP issue, and a separate state issue. I am thinking I need to run the packet capture on UDP 67 for the Honeywell gateway IP with zero for the count at power up, to capture packets until it disconnects and I can no longer ping it?
I am trying to understand what may be going on. It sounds like a DCHP request can include a specific time up to the maximum lease time designated in the server. If the request does not specifiy a lease length, it gets 120 minutes based on the default setting. I am not quite understanding the 30 minute re-lease time default. Betting it is part of the standard and not a user setting. Do you mean when there is 30 minutes left on the lease, or about 3/4 of the time has elapsed?
*Default lease time
Controls how long a lease will last when a client does not request a specific lease length. Specified in seconds, default value is 7200 seconds (2 hours)Maximum lease time
Limits a requested lease length to a stated maximum amount of time. Specified in seconds, default value is 86400 seconds (1 day).*Thank you very much,
Jerold -
@jpvonhemel You're spot on with the idea on the pcap.
I would make sure if you're on a thin client to verify you have a few GB just in case there's a lot of data recorded (unlikely to be the case but ... if there is and you only have a 8GB you'll likely fill your storage and lock your system up).
Regardless of the length of the lease... clients will always attempt to renew their IPs at the half-of half-life time.
So half-life of 120 is 60, and half of that is 30.
You'll see packet capture where it says, basically, "no, you're not ready to renew yet, shut up".
But I suggest you capture ALL traffic, not just UDP 67. You can have a system send a ping every 15-30 seconds instead of one/second to keep the capture entries lower.
Then you can view the details in Wireshark after downloading the file. -
I hadn't seen any disconnects for nearly 24 hours, then looked over and it was disconnected. The change to conservative did seem to help my problem. I power cycled the hub and I am running a pcap for all packets with the hub ip, using a ping script running on my pc for every 15 min.
Should I let it run until the hub disconnects and loses the dhcp lease? Since it was up for 24 hours, I am wondering if I should switch my firewall optimization to normal, just in hopes of getting it to disconnect more quickly. Let me know your thoughts and I can change it if needed
Thank you,
Jerold
-
@jpvonhemel are you doing a regular ping to the Honeywell to make sure it's online? Or just randomly checking?
I would honestly go to Honeywell and ask them what's up with this hardware and why it can't play well with others :-)
But that's the former LVT in me coming out.
-
I hear you. While it has the Honeywell branding on the hardware, it is "developed" and "supported" by a company called Resideo. I reached out to their support. I received a "pfWhat" when I told them what my router was, along with this document of best practices (picture below). Beyond that, they were not very forthcoming with support for my very outlying issue. The hardware is ethernet only and it is not configurable via a web browser. The model is THM6000R7001, in case someone does a search.
My networking skills are above average for the average consumer, but I don't have the necessary skills to figure out what the root problem is. I guess it is narrowed down to DHCP or some sort of non-standard TCP communication. The only port it seems to use is 53 and 443, so the traffic is encrypted. I figured I would post to document in case another pfSense user has the same issue and I also figured there may be a solution I did not know of. I also figured I may be able to learn a bit more about pfSense in the process.
When I built my home, the hvac contractor talked me out of an ecobee infavor of the Honeywell system and we have too much invested into it to pull it out after only a few years.
I have a pcap running now in the event it goes down again. It has been up for nearly 24 hours, so it may stb at anytime. I have an old SG-2440 I can use with it. By the way, what is LVT?
-
The Honeywell gateway has stayed connected for several days since the change to conservative on firewall optimization options (System->Advanced->Firewall). Thank you for your help. I am still curious what LVT is. Thanks
-
@jpvonhemel You're welcome!
LVT is a low-voltage technician. I spent most of the 2010s doing burglar, video surveillance, and wireless internet (MESH and AP) installations and support.
The current hardware is vastly better than what we had even five years ago when I left the field but manufacturers still don't have a clue usually about TCP vs. UDP.