21.05 blocking TiVo connections for unknown reasons
-
@sydgarrett said in 21.05 blocking TiVo connections for unknown reasons:
I've pretty well stripped things down
18 installed packages is hardly "stripped down". Check the System logs. Check your firewall logs. Power off the Tivo, start a packet capture on the Tivo IP address and then take a look and see what's really going on. If you don't know your way around Wireshark, post your trace here with any public IP details obscured.
-
By checking DNS I mean looking at what you have setup there, especially in the advanced/custom options (e.g. from pfBlocker if you have DNSBL or things like that). It's possible something is allowing one DNS lookup to succeed but blocking the hostname of the next one.
I ran a packet capture of one of mine doing a network connection back to TiVo and it made several connections out to different servers. Most on port 80 and then one on port 8081, though I only saw one DNS request and it was only used for the very first port 80 connection.
-
@kom said in 21.05 blocking TiVo connections for unknown reasons:
18 installed packages is hardly "stripped down".
:) Well, I meant to say that I had stripped out anything that I thought could have an impact.
System logs have nothing regarding any of the TiVo IP addresses. Firewall logs -- same thing.
I did a packet capture around one of the TiVo's attempt to connect and went through it with WireShark. I don't see anything unusual out of it but would love to have more sets of eyes on it. If ya'll see anything, I'd love to hear it.
OK -- attempted to upload package capture .cap file but the forum software doesn't recognize it as an allowable file type. Suggestions for loading the .cap file?
Sorry for the newbie question.
-
Thanks for the help.
I only have 1.1.1.1 and 8.8.8.8 in as DNS servers. I also checked the box to allow the DNS Server list to be overridden by DHCP on WAN. DNS uses the local 127.0.0.1 and then falls back to the remote.
So, the resulting list is
127.0.0.1
209.18.47.62 (provided by Spectrum)
209.18.47.61 (provided by Spectrum)
1.1.1.1
8.8.8.8I also see one request for DNS, then requests to multiple servers on 80 and then one on 8080 and one on 8081.
-
@sydgarrett I believe you can select the lines then ctrl-c/ctrl-v them into a text file. From there, you can find & replace any public IP details.
Example:
1 0.000000 192.168.88.10 192.168.88.1 HTTP 573 POST /ifstats.php HTTP/1.1 (application/x-www-form-urlencoded) 2 0.000111 192.168.88.1 192.168.88.10 TCP 66 80 → 58784 [ACK] Seq=1 Ack=508 Win=511 Len=0 TSval=4108081792 TSecr=2254862057 3 0.000608 192.168.88.10 192.168.88.1 HTTP 648 POST /getstats.php HTTP/1.1 (application/x-www-form-urlencoded) 4 0.000755 192.168.88.1 192.168.88.10 TCP 66 80 → 58788 [ACK] Seq=1 Ack=583 Win=510 Len=0 TSval=870658763 TSecr=2254862058 5 0.035698 192.168.88.1 192.168.88.10 HTTP/JSON 683 HTTP/1.1 200 OK , JavaScript Object Notation (application/json) 6 0.035959 192.168.88.10 192.168.88.1 TCP 66 58784 → 80 [ACK] Seq=508 Ack=618 Win=524 Len=0 TSval=2254862093 TSecr=4108081828 7 0.390496 192.168.88.1 192.168.88.10 HTTP 656 HTTP/1.1 200 OK (text/html) 8 0.390773 192.168.88.10 192.168.88.1 TCP 66 58788 → 80 [ACK] Seq=583 Ack=591 Win=501 Len=0 TSval=2254862448 TSecr=870659153 9 1.001156 192.168.88.10 192.168.88.1 HTTP 510 GET /widgets/widgets/pfblockerng.widget.php?getNewWidget=1625105937799 HTTP/1.1 10 1.001253 192.168.88.1 192.168.88.10 TCP 66 80 → 58784 [ACK] Seq=618 Ack=952 Win=511 Len=0 TSval=4108082794 TSecr=2254863058 11 1.001270 192.168.88.10 192.168.88.1 HTTP 573 POST /ifstats.php HTTP/1.1 (application/x-www-form-urlencoded) 12 1.001308 192.168.88.1 192.168.88.10 TCP 66 80 → 58788 [ACK] Seq=591 Ack=1090 Win=511 Len=0 TSval=870659764 TSecr=2254863058 13 1.035307 192.168.88.1 192.168.88.10 HTTP/JSON 684 HTTP/1.1 200 OK , JavaScript Object Notation (application/json) 14 1.035569 192.168.88.10 192.168.88.1 TCP 66 58788 → 80 [ACK] Seq=1090 Ack=1209 Win=501 Len=0 TSval=2254863093 TSecr=870659797 -
Thanks. Here it is
Hopefully someone sees something in there that I'm missing.
-
@sydgarrett That looks normal to me, or at least nothing jumps out as being the problem. The two ends are talking just fine. The flow is what @jimp said it was for his units. Does the Tivo have any logs that might show what's wrong? You might have to bite the bullet and start disabling or removing packages to rule those out.
-
208 17.280483 10.0.1.153 208.73.181.202 TCP 74 49802 → 8080 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3450530 TSecr=0 WS=64 209 17.280579 208.73.181.202 10.0.1.153 TCP 54 8080 → 49802 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 213 17.503194 10.0.1.153 208.73.181.98 TCP 74 48909 → 8081 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3450753 TSecr=0 WS=64 214 17.503279 208.73.181.98 10.0.1.153 TCP 54 8081 → 48909 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0Something is immediately rejecting the 8080 and 8081 connections. The TiVo sends a SYN and immediately gets back a reset (RST+ACK) when it should be getting a SYN+ACK.
You can capture on the WAN to see if that's really coming from the remote end, but it is far more likely that's local. It's hitting a rule somewhere set to reject (not block).
-
@jimp said in 21.05 blocking TiVo connections for unknown reasons:
Something is immediately rejecting the 8080 and 8081 connections. The TiVo sends a SYN and immediately gets back a reset (RST+ACK) when it should be getting a SYN+ACK.
You can capture on the WAN to see if that's really coming from the remote end, but it is far more likely that's local. It's hitting a rule somewhere set to reject (not block).
Wow. I completely missed that. That matches with what the TiVo's were complaining about.
I did a capture on the WAN and verified that the RST isn't coming from the remote end. I see all the other traffic to that TiVo server. However, I don't see the port 8080 or 8081 exchanges. So, I don't see what is generating the RSTs in pfSense.
I have only the basic LAN rules.
I've deleted and/or disabled all my WAN rules just to figure this out.
As you can see from the post above, I don't think I have any packages loaded that would be doing filtering.
Ya'll have been extremely helpful. Any ideas on where to look next?
Thanks
-
What's on the Floating tab?
-
@jimp
Nothing on the Floating tab
-
Do you maybe have a port forward matching 8080 / 8081 that is matching it somehow?
Or maybe there is something hiding in the ruleset from another package. Post the contents of
/tmp/rules.debug, see if it has anything unusual in there. -
@jimp
No port forwarding and only Automatic Outbound NAT rules.Here is the rules.debug
-
OK -- new information here. I know that "correlation doesn't equal causation" but maybe this will give someone else a clue.
I noticed that there were a bunch of miniupnpd errors in the Routing Logs.
So, I restarted the miniupnpd process. As long as that process is in the "Listening for NAT-PMP/PCP traffic on port 5351" state, the TiVo's can SUCCESSFULLY connect. As soon as the "ioctl(dev, DIOCGETADDRS, ...): Device busy" errors start showing up again (after about 4 or 5 minutes), the TiVo's go back to the state where the 8080 and 8081 transactions receive a RST response.
If I disable UPnP & NAT-PMP, the connections run just fine. As soon as I re-enable UPnP & NAT-PMP, we are back to our original state.
Doesn't make ANY sense to me given that the TiVo's don't use UPnP & NAT-PMP (as far as I know and/or can tell). So, how could that be blocking things and why does it get the multiple "Device Busy" errors?
I can live without it for a few days but I really would like to re-enable it for some other things I have on my network.
Thoughts?
-
@sydgarrett Perhaps this?
-
Curious, do you see anything in the UPnP status when the TiVo can't connect?
What if you try this command from the shell at the same time:
pfSsh.php playback pfanchordrill -
@kom said in 21.05 blocking TiVo connections for unknown reasons:
@sydgarrett Perhaps this?
Yep. That sounds like the same thing. Also explains why this all started around the time of the upgrade to 21.05.
-
Nothing in the status for UPnP.
This is interesting
And, of course, those go away when the miniupnpd is restarted
I'm not following what those rules are there for but you clearly found our culprit for what is blocking 8080 and 8081
-
So despite the error, something on your local network is setting up block rules via UPnP for 8080, 8081, and 8082 which are rejecting all that traffic.
So UPnP is letting it happen, but ultimately it's being done by whatever device is adding those rules to UPnP.
Some kind of device with a web interface, which may limit your search when tracking it down.
Anything server-like on your local network that uses those ports for a web-based GUI to manage it?
-
Yeah, lot's of devices with web based GUIs. Let me spend some time tracking that down.
However, all of those "played well together" and I had no issues until about a month ago when the Tivo's started complaining. Is it possible that the rules are left from some intermediate state when the UPnP dies?
