pfSense Blocking Roborock app

kahodges1721

@bmeeks sorry I should have been more clear. When I say bypass I mean I took pfsense out of the loop. Exactly how the network diagram shows above. The WiFi for the NETGEAR was disabled so no chance of bypassing the APs but I agree it makes sense. Unfortunately that wasn’t the case here. The biggest red flag to me is that nothing changed in my setup except a possible update to the app. I won’t pretend to know how it all works but if the update brought a change to the server settings on their end I’m curious if I make a pfsense machine and take it to a buddies house and hook it up if I get the same results.

While you were out I did try to get fancy with it and connected the NETGEAR after the pfsense in the chain. Added some port forward rules for the router activated the WiFi on the NETGEAR and nothing still same result. I expected that but I think I’ve reached the point where I just am out of things to try. I even started just going through all the documentation and trying random settings just to see what would happen haha.

kahodges1721

@rcoleman-netgate understood! My bad! Luckily this is my first time this deep into the logs to solve a problem!

bmeeks

@kahodges1721:
I am rapidly running out of ideas. I'm not there actually touching the firewall nor able to see all the configuration, but if you say that other devices plugged into the network switch shown in your diagrams work fine through pfSense, and if your phone- when connected through the UniFi AP shown in your diagram- can surf the web and connect to social media sites with pfSense in the path, then all of that together indicates the firewall configuration is fine.

Things that the scenario I described validate are:

DHCP scope is correct and clients are getting the proper subnet mask and gateway
DNS Resolver is up and running and properly resolving hostnames to IP addresses
The default firewall rules are in place that allow LAN clients to go anywhere.

Is there any other software method available besides a phone app? Is there something you could download and install on a Windows PC or a Mac. That would take the phone app out of the picture.

kahodges1721

@bmeeks This gave me a new idea. With Home assistant I can run the integration for the roborock. This asks for the app credentials. I would assume it's making the same "call home". The home assistant is running on a VM and is part of my unraid server. The unraid server itself has an IP of .235 while the Home assistant VM is .238. I ran the integration while doing a packet capture. Ill put the results below. This has the same failure but could possible be compared to the iPhone packet capture to narrow things down a little. For reference the path of the Unraid PC is Modem -> pfSense -> 24port Switch -> UnraidPC -> VM instance of Home assistant. Ill also attached a screenshot showing all the other connected devices working except Roborocks failure. There seems to be a ton of back and forth between 192.168.1.1 and the HA instance at .238 not sure what this indicates

08:44:11.110372 IP 192.168.1.1.443 > 192.168.1.50.57565: tcp 0
08:44:11.110374 IP 192.168.1.50.57565 > 192.168.1.1.443: tcp 95
08:44:11.110379 IP 192.168.1.1.443 > 192.168.1.50.57565: tcp 0
08:44:11.110379 IP 192.168.1.50.57565 > 192.168.1.1.443: tcp 98
08:44:11.110383 IP 192.168.1.1.443 > 192.168.1.50.57565: tcp 0
08:44:11.110509 IP 192.168.1.50.57565 > 192.168.1.1.443: tcp 76
08:44:11.110514 IP 192.168.1.1.443 > 192.168.1.50.57565: tcp 0
08:44:11.110732 IP 192.168.1.1.443 > 192.168.1.50.57565: tcp 1460
08:44:11.110840 IP 192.168.1.1.443 > 192.168.1.50.57565: tcp 963
08:44:11.110903 IP 192.168.1.1.443 > 192.168.1.50.57565: tcp 372
08:44:11.111067 IP 192.168.1.50.57565 > 192.168.1.1.443: tcp 0
08:44:11.111071 IP 192.168.1.50.57565 > 192.168.1.1.443: tcp 0
08:44:11.111282 IP 192.168.1.1.443 > 192.168.1.50.57565: tcp 1460
08:44:11.111491 IP 192.168.1.50.57565 > 192.168.1.1.443: tcp 0
08:44:11.111602 IP 192.168.1.50.57565 > 192.168.1.1.443: tcp 0
08:44:11.260634 IP 35.213.232.93.443 > 192.168.1.50.57519: tcp 0
08:44:11.260786 IP 192.168.1.50.57519 > 35.213.232.93.443: tcp 0
08:44:11.281915 IP 192.168.1.108.44715 > 44.195.125.201.443: tcp 33
08:44:11.307143 IP 44.195.125.201.443 > 192.168.1.108.44715: tcp 31
08:44:11.310780 IP 192.168.1.108.44715 > 44.195.125.201.443: tcp 0
08:44:11.440024 IP 192.168.1.235.59185 > 239.255.255.250.1900: UDP, length 101
08:44:11.452702 IP 192.168.1.238.46140 > 192.168.1.1.2189: tcp 0
08:44:11.452704 IP 192.168.1.238.46126 > 192.168.1.1.2189: tcp 0
08:44:11.452763 IP 192.168.1.1.2189 > 192.168.1.238.46140: tcp 0
08:44:11.452770 IP 192.168.1.1.2189 > 192.168.1.238.46126: tcp 0
08:44:11.452783 IP 192.168.1.238.46154 > 192.168.1.1.2189: tcp 0
08:44:11.452784 IP 192.168.1.238.46156 > 192.168.1.1.2189: tcp 0
08:44:11.452802 IP 192.168.1.1.2189 > 192.168.1.238.46154: tcp 0
08:44:11.452803 IP 192.168.1.1.2189 > 192.168.1.238.46156: tcp 0
08:44:11.452808 IP 192.168.1.238.46170 > 192.168.1.1.2189: tcp 0
08:44:11.452819 IP 192.168.1.1.2189 > 192.168.1.238.46170: tcp 0
08:44:11.452948 IP 192.168.1.238.46174 > 192.168.1.1.2189: tcp 0
08:44:11.452997 IP 192.168.1.1.2189 > 192.168.1.238.46174: tcp 0
08:44:11.453032 IP 192.168.1.238.46154 > 192.168.1.1.2189: tcp 0
08:44:11.453033 IP 192.168.1.238.46156 > 192.168.1.1.2189: tcp 0
08:44:11.453040 IP 192.168.1.238.46140 > 192.168.1.1.2189: tcp 0
08:44:11.453061 IP 192.168.1.238.46126 > 192.168.1.1.2189: tcp 0
08:44:11.453089 IP 192.168.1.238.46170 > 192.168.1.1.2189: tcp 0
08:44:11.453388 IP 192.168.1.238.46174 > 192.168.1.1.2189: tcp 0
08:44:11.453533 IP 192.168.1.238.46156 > 192.168.1.1.2189: tcp 315
08:44:11.453546 IP 192.168.1.1.2189 > 192.168.1.238.46156: tcp 0
08:44:11.453622 IP 192.168.1.238.46140 > 192.168.1.1.2189: tcp 313
08:44:11.453629 IP 192.168.1.1.2189 > 192.168.1.238.46140: tcp 0
08:44:11.453764 IP 192.168.1.238.46140 > 192.168.1.1.2189: tcp 284
08:44:11.453765 IP 192.168.1.238.46126 > 192.168.1.1.2189: tcp 317
08:44:11.453771 IP 192.168.1.1.2189 > 192.168.1.238.46140: tcp 0
08:44:11.453778 IP 192.168.1.1.2189 > 192.168.1.238.46126: tcp 0
08:44:11.453805 IP 192.168.1.238.46156 > 192.168.1.1.2189: tcp 288
08:44:11.453811 IP 192.168.1.1.2189 > 192.168.1.238.46156: tcp 0
08:44:11.453840 IP 192.168.1.238.46154 > 192.168.1.1.2189: tcp 319
08:44:11.453848 IP 192.168.1.1.2189 > 192.168.1.238.46154: tcp 0
08:44:11.453860 IP 192.168.1.1.2189 > 192.168.1.238.46140: tcp 512
08:44:11.453868 IP 192.168.1.1.2189 > 192.168.1.238.46140: tcp 0
08:44:11.453889 IP 192.168.1.238.46170 > 192.168.1.1.2189: tcp 298
08:44:11.453897 IP 192.168.1.1.2189 > 192.168.1.238.46170: tcp 0
08:44:11.453907 IP 192.168.1.1.2189 > 192.168.1.238.46156: tcp 518
08:44:11.453912 IP 192.168.1.1.2189 > 192.168.1.238.46156: tcp 0
08:44:11.453966 IP 192.168.1.238.46154 > 192.168.1.1.2189: tcp 296
08:44:11.453972 IP 192.168.1.1.2189 > 192.168.1.238.46154: tcp 0
08:44:11.454002 IP 192.168.1.1.2189 > 192.168.1.238.46154: tcp 534
08:44:11.454007 IP 192.168.1.1.2189 > 192.168.1.238.46154: tcp 0
08:44:11.454013 IP 192.168.1.238.46126 > 192.168.1.1.2189: tcp 292
08:44:11.454017 IP 192.168.1.1.2189 > 192.168.1.238.46126: tcp 0
08:44:11.454017 IP 192.168.1.238.46170 > 192.168.1.1.2189: tcp 267
08:44:11.454020 IP 192.168.1.1.2189 > 192.168.1.238.46170: tcp 0
08:44:11.454033 IP 192.168.1.238.46156 > 192.168.1.1.2189: tcp 0
08:44:11.454100 IP 192.168.1.1.2189 > 192.168.1.238.46170: tcp 586
08:44:11.454107 IP 192.168.1.238.46174 > 192.168.1.1.2189: tcp 305
08:44:11.454108 IP 192.168.1.1.2189 > 192.168.1.238.46170: tcp 0
08:44:11.454115 IP 192.168.1.1.2189 > 192.168.1.238.46174: tcp 0
08:44:11.454116 IP 192.168.1.238.46174 > 192.168.1.1.2189: tcp 281
08:44:11.454119 IP 192.168.1.1.2189 > 192.168.1.238.46174: tcp 0
08:44:11.454122 IP 192.168.1.238.46140 > 192.168.1.1.2189: tcp 0
08:44:11.454147 IP 192.168.1.1.2189 > 192.168.1.238.46126: tcp 529
08:44:11.454152 IP 192.168.1.1.2189 > 192.168.1.238.46126: tcp 0
08:44:11.454192 IP 192.168.1.1.2189 > 192.168.1.238.46174: tcp 520
08:44:11.454196 IP 192.168.1.1.2189 > 192.168.1.238.46174: tcp 0
08:44:11.454234 IP 192.168.1.238.46154 > 192.168.1.1.2189: tcp 0
08:44:11.454371 IP 192.168.1.238.46174 > 192.168.1.1.2189: tcp 0
08:44:11.454389 IP 192.168.1.238.46170 > 192.168.1.1.2189: tcp 0
08:44:11.454392 IP 192.168.1.238.46126 > 192.168.1.1.2189: tcp 0
08:44:11.454497 IP 192.168.1.238.46156 > 192.168.1.1.2189: tcp 0
08:44:11.454510 IP 192.168.1.1.2189 > 192.168.1.238.46156: tcp 0
08:44:11.454900 IP 192.168.1.238.46140 > 192.168.1.1.2189: tcp 0
08:44:11.454901 IP 192.168.1.238.46154 > 192.168.1.1.2189: tcp 0
08:44:11.454909 IP 192.168.1.1.2189 > 192.168.1.238.46140: tcp 0
08:44:11.454913 IP 192.168.1.1.2189 > 192.168.1.238.46154: tcp 0
08:44:11.455389 IP 192.168.1.238.46174 > 192.168.1.1.2189: tcp 0
08:44:11.455389 IP 192.168.1.238.46170 > 192.168.1.1.2189: tcp 0
08:44:11.455395 IP 192.168.1.1.2189 > 192.168.1.238.46174: tcp 0
08:44:11.455396 IP 192.168.1.1.2189 > 192.168.1.238.46170: tcp 0
08:44:11.455398 IP 192.168.1.238.46126 > 192.168.1.1.2189: tcp 0
08:44:11.455404 IP 192.168.1.1.2189 > 192.168.1.238.46126: tcp 0
08:44:11.493205 IP 192.168.1.1.1900 > 192.168.1.235.59185: UDP, length 363
08:44:11.576010 IP 54.175.191.203.443 > 192.168.1.238.53800: tcp 0
08:44:11.576799 IP 192.168.1.238.53800 > 54.175.191.203.443: tcp 0
08:44:11.711770 IP 192.168.1.50.54401 > 162.254.192.87.27024: tcp 113
08:44:11.730535 IP 192.168.1.245.59178 > 34.117.13.189.443: tcp 296
08:44:11.755106 IP 34.117.13.189.443 > 192.168.1.245.59178: tcp 0

kahodges1721

Ok So In testing the integration I went to the github page. Looks like I am not the only one with the issue. Here is a link to the problem someone else is having. I would bet anything he is also using a FreeBSD router. The plot thickens...

Similar issue found on github integration with Home Assistant

bmeeks

@kahodges1721 said in pfSense Blocking Roborock app:

Ok So In testing the integration I went to the github page. Looks like I am not the only one with the issue. Here is a link to the problem someone else is having. I would bet anything he is also using a FreeBSD router. The plot thickens...

Similar issue found on github integration with Home Assistant

Read through that chain on Github. I lean towards an app update being the problem, but why it works via Wi-Fi with the Netgear in place, but fails with pfSense and OPNsense in place, is a mystery. Might be subtle differences in how states are handed between the Netgear firewall engine and FreeBSD. State timeouts can be configured in pfSense. Perhaps setting the firewall optimization to "Conservative" might help. That option is here:

kahodges1721

@bmeeks I gave the change a try and its a no go. I agree that it must be a change on the side of roborock. I guess the reason I am concerned is the population of users with pfSense and roborock devices might not be large enough to warrent a change from Roborock. Meaning I am left either leaving pfSense which I really dont want to do or get a different robot which this one is by far the best I have used so I dont want to do that either! I keep going back to the fact that the netgear works. This has to give us some clues. At first I considered that Roborock flagged my IP or something bc it didnt like the integration through HA but that cant be the case since it works through the Netgear. The only thing left is there is some difference either in DNS handling or firewall rules that is blocking the communication. With pfSense being so open to the user vs a standard router I feel there is a fighting chance to find the setting and configure it. Hopefully posting in the github brings more users with issues to light.

kahodges1721

Some good information learned from the github link that I thought would be worth sharing here for ideas.

I believe the apps (Mi Home and Roborock) work slightly different. One thing I noticed is the Mi Home app connects using both their servers and locally to the device whereas the Roborock app connects to the device through their cloud MQTT service on port 8883 (in addition to TCP connections on 443). I don't believe the Mi Home app ever used MQTT since I explicitly had to allow that on my firewall to get the Roborock app to work. I haven't used pfSense or OPNsense for a while but I wonder if there's something weird when it comes to MQTT?

I also noticed my Sophos XG logs shows the MQTT connections with the Roborock MQTT broker as "SSL Traffic over Non-SSL Ports".

kahodges1721

usiot.roborock.com is apparently the culprit. I got a message on the other forum from the developer of the integration and that’s the site that is being used for the api. It seems pfsense will not resolve it. I’ve never had this issue so not sure why or what to do to fix it but it at least pin points the problem. I’m curious if it resolves on anyone else running pfsense

Jarhead

@kahodges1721
It resolved here just fine.

kahodges1721

@jarhead yes it shows “resolved“ on mine also but when I try to navigate to it I don’t get anything. I may be using the wrong terminology here. Basically I type in the aggress into a web browser and can’t reach it. Navigating to the other api euiot.roborock.com seems to not get blocked and opens a white page. I also ran a trace route below is what I got.

10.157.0.1 8.118 ms 7.113 ms 7.503 ms
2 209.196.177.66 7.205 ms 7.788 ms 8.731 ms
3 207.255.30.160 25.315 ms 24.719 ms 29.381 ms
4 99.83.89.162 28.368 ms
99.83.68.84 23.760 ms 24.622 ms
5 * * *
6 * * *
7 52.93.29.98 31.090 ms
52.93.29.92 33.040 ms
52.93.29.82 25.631 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *

kahodges1721

This is what I get with the USIOT and Euiot address

pics of difference

Sorry for using the links but I can’t get to the computer right now and linking an image from my phone I couldn’t figure out how

bmeeks

The usiot.roborock.com hostname resolves fine on my pfSense box:

You said "usiot.roborock.com" in your first post, but then in the follow up with the link to Imgur you show both usiot and euiot. Which one is correct? Do both resolve properly on your firewall?

Go to DIAGNOSTICS > DNS Lookup in the pfSense menu and enter both hostnames, one at the time, hitting Lookup between attempts. Post back the output from the results window as I did above.

kahodges1721

@bmeeks both resolve fine According to the test. I guess I was assuming it wasn’t resolving since when I type them into the web browser the US gives the server stopped responding error and the EU works just fine. Switching to LTE both work just fine and display a blank white page as expected.

Both are used for the api according to the developer of the integration. I focused only on the U.S. bc that seems to be the culprit of the issue

This is the post that lead to the testing:

Starting with the basics. Is your router resolving DNS euiot.roborock.com? (Which is the default base url to discover the real api to be used based on your account country). And, if your account from US, is it also solving usiot.roborock.com ?

kahodges1721

@bmeeks dns lookup results

dns lookup results

bmeeks

@kahodges1721:
Are you running pfSense CE or pfSense Plus? Those have different versions of the unbound DNS resolver, and the version in the current pfSense Plus (1.15.0) has some issues sometimes chasing CNAME records and/or resolving IPv6 stuff. IPv6 does not appear to figure in your problem, though, since those domains resolve to IPv4 addresses.

I also need to ask one more time. When you say you reinstalled pfSense and chose the defaults, does that mean all of the following statements are true?

You installed fresh from either an ISO image or USB memstick image and took all the offered defaults during installation, changing nothing other than probably answering "No" for anything IPv6 related.
You did absolutely nothing to DNS. You did not type in any DNS server IP addresses anyplace at any time during or after the install procedure. DNS is 100% stock, out-of-the-box.
You answered "Yes" during the installation when prompted whether to enable DHCP on the LAN. You took any offered defaults for that configuration as well.
You did NOT restore any kind of previous configuration backup either during the install procedure nor after from the DIAGNOSTICS menu.
You neither added to nor edited any of the firewall rules that came as part of the default out-of-the-box install.
You have not installed (or restored) any add-on packages.

I want to make sure the firewall is truly in a default configuration, because that usually just works for everything.

I can successfully navigate to a web page at both hostnames you provided (usiot and euiot). I get a blank, white page in both instances which I assume is expected as that is really an API for authentication and not for HTML display.

I would also try really basic things like can you ping both host names from the DIAGNOSTICS menu of pfSense? Try using the ping -s 1460 option use a maximum size ICMP packet.

kahodges1721

@bmeeks that’s interesting it worked for you and disappointing bc I was hoping it wouldn’t work for anyone on pfsense haha hmm…

As far as defaults yes it was a fresh download of pfsense CE. All defaults were selected with DHCP on lan. I did enter a dns address of 8.8.8.8 for primary and 8.8.4.4 for secondary. This is also the dns addresses used on the NETGEAR. Nothing else was touched at all. Tested the app and it failed. I then upgraded it to pfsense plus which is what I typically run and same result. I haven’t found a way to install plus directly for some reason so maybe installing CE first is causing something? But what still doesn’t make sense is why it broke in the first place. Nothing about dns or pfsense was touched at all in the past like 6-8 months. I’ve had the same settings for a while and just let it go.

bmeeks

@kahodges1721 said in pfSense Blocking Roborock app:

I did enter a dns address of 8.8.8.8 for primary and 8.8.4.4 for secondary.

Remove those. You do not need them. Out of the box unbound is configured to resolve in pfSense. That means it talks to the root DNS servers and walks down the tree to find IP addresses for hostnames. Also that is quite a bit more "private" than handing Google all of your DNS traffic so they can log where you go and target you with ads and otherwise profile you. Putting in those servers is not "default" . They likely are not the problem here, but that is an unnecessary change to pfSense. Leave the DNS boxes under SYSTEM > GENERAL SETUP blank. Also go back and make sure the DNS server boxes in the DHCP server configuration page are blank. That way, DHCP will hand your clients pfSense as their DNS server (which is generally what you want).

One other thing, make sure the DNS Server Override box is unchecked on the SYSTEM > GENERAL SETUP page!

kahodges1721

@bmeeks again I can’t thank you enough for the help. Unfortunately I had to call it quits for the day and get ready for work tomorrow. I will give it a go again tomorrow after work and update

bmeeks

Your DNS configuration in pfSense is needlessly complex. You have 4 name servers listed. You should only have 127.0.0.1 showing there as the default configuration for pfSense uses unbound running in resolver mode. Your DNS configuration is NOT default. You said earlier you did a fresh install and had everything set for defaults.

Have you enabled "Forwarding" under SERVICES > DNS Resolver?

Have you checked the DNS Server Override box under SYSTEM > General Setup?

What are those two 173.44.120.88 and 173.44.120.89 DNS servers? Are they perhaps from your ISP? You certainly do not want those in the mix in my opinion.

Your DNS setup is NOT default! This could be a source of problems.

Notice in my screenshot for resolving the usiot.roborock.com host there is only a single DNS server listed. It's 127.0.0.1, which is the local unbound daemon running in resolver mode on my firewall. You need to do what I mentioned earlier:

Go to SYSTEM > General Setup and remove all the DNS server IP addresses you entered there (I think you said they were 8.8.8.8 and 8.8.4.4). There should be nothing at all in those DNS server boxes. Leave them blank.
Be sure the DNS Server Override checkbox immediately below on the same page is unchecked (as in cleared or not checked).
Go to the DHCP Server settings under SERVICES > DHCP and remove anything you typed into the DNS server settings there to be served to clients. Make sure all the DNS boxes are empty. That way, DHCP will serve clients the pfSense firewall's LAN IP as the DNS server they are to use. That is what you want.
Go to SERVICES > DNS Resolver > General Settings and uncheck everything on that page except two boxes. Those are Enabled at the top, and DNSSEC about halfway down. Make sure both of those are checked.

After you make all these checks/changes, I would reboot the firewall to be 100% sure the new settings take effect. After rebooting the firewall, toggle your phone's the Wi-Fi off and back on so it will pick up the new DNS settings.

When you make multiple DNS servers available, your clients do not use them in the order you list them. Instead, clients randomly choose one and will continue to use just that one until it fails to respond. Only then will they try another one in the list. This is especially true of Windows machines. This can cause a really confusing mess if the various DNS servers return different results for some reason such as a misconfiguration on one of them. Stuff can then work on one client but fail with another on the same LAN because they choose different DNS servers from the available list to ask for name resolution. The DNS Lookup utility on pfSense will on purpose try every configured DNS server. But normal clients don't behave that way.