21.05 blocking TiVo connections for unknown reasons
-
@sydgarrett I believe you can select the lines then ctrl-c/ctrl-v them into a text file. From there, you can find & replace any public IP details.
Example:
1 0.000000 192.168.88.10 192.168.88.1 HTTP 573 POST /ifstats.php HTTP/1.1 (application/x-www-form-urlencoded) 2 0.000111 192.168.88.1 192.168.88.10 TCP 66 80 → 58784 [ACK] Seq=1 Ack=508 Win=511 Len=0 TSval=4108081792 TSecr=2254862057 3 0.000608 192.168.88.10 192.168.88.1 HTTP 648 POST /getstats.php HTTP/1.1 (application/x-www-form-urlencoded) 4 0.000755 192.168.88.1 192.168.88.10 TCP 66 80 → 58788 [ACK] Seq=1 Ack=583 Win=510 Len=0 TSval=870658763 TSecr=2254862058 5 0.035698 192.168.88.1 192.168.88.10 HTTP/JSON 683 HTTP/1.1 200 OK , JavaScript Object Notation (application/json) 6 0.035959 192.168.88.10 192.168.88.1 TCP 66 58784 → 80 [ACK] Seq=508 Ack=618 Win=524 Len=0 TSval=2254862093 TSecr=4108081828 7 0.390496 192.168.88.1 192.168.88.10 HTTP 656 HTTP/1.1 200 OK (text/html) 8 0.390773 192.168.88.10 192.168.88.1 TCP 66 58788 → 80 [ACK] Seq=583 Ack=591 Win=501 Len=0 TSval=2254862448 TSecr=870659153 9 1.001156 192.168.88.10 192.168.88.1 HTTP 510 GET /widgets/widgets/pfblockerng.widget.php?getNewWidget=1625105937799 HTTP/1.1 10 1.001253 192.168.88.1 192.168.88.10 TCP 66 80 → 58784 [ACK] Seq=618 Ack=952 Win=511 Len=0 TSval=4108082794 TSecr=2254863058 11 1.001270 192.168.88.10 192.168.88.1 HTTP 573 POST /ifstats.php HTTP/1.1 (application/x-www-form-urlencoded) 12 1.001308 192.168.88.1 192.168.88.10 TCP 66 80 → 58788 [ACK] Seq=591 Ack=1090 Win=511 Len=0 TSval=870659764 TSecr=2254863058 13 1.035307 192.168.88.1 192.168.88.10 HTTP/JSON 684 HTTP/1.1 200 OK , JavaScript Object Notation (application/json) 14 1.035569 192.168.88.10 192.168.88.1 TCP 66 58788 → 80 [ACK] Seq=1090 Ack=1209 Win=501 Len=0 TSval=2254863093 TSecr=870659797
-
Thanks. Here it is
Hopefully someone sees something in there that I'm missing.
-
@sydgarrett That looks normal to me, or at least nothing jumps out as being the problem. The two ends are talking just fine. The flow is what @jimp said it was for his units. Does the Tivo have any logs that might show what's wrong? You might have to bite the bullet and start disabling or removing packages to rule those out.
-
208 17.280483 10.0.1.153 208.73.181.202 TCP 74 49802 → 8080 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3450530 TSecr=0 WS=64 209 17.280579 208.73.181.202 10.0.1.153 TCP 54 8080 → 49802 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 213 17.503194 10.0.1.153 208.73.181.98 TCP 74 48909 → 8081 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3450753 TSecr=0 WS=64 214 17.503279 208.73.181.98 10.0.1.153 TCP 54 8081 → 48909 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
Something is immediately rejecting the 8080 and 8081 connections. The TiVo sends a SYN and immediately gets back a reset (RST+ACK) when it should be getting a SYN+ACK.
You can capture on the WAN to see if that's really coming from the remote end, but it is far more likely that's local. It's hitting a rule somewhere set to reject (not block).
-
@jimp said in 21.05 blocking TiVo connections for unknown reasons:
Something is immediately rejecting the 8080 and 8081 connections. The TiVo sends a SYN and immediately gets back a reset (RST+ACK) when it should be getting a SYN+ACK.
You can capture on the WAN to see if that's really coming from the remote end, but it is far more likely that's local. It's hitting a rule somewhere set to reject (not block).
Wow. I completely missed that. That matches with what the TiVo's were complaining about.
I did a capture on the WAN and verified that the RST isn't coming from the remote end. I see all the other traffic to that TiVo server. However, I don't see the port 8080 or 8081 exchanges. So, I don't see what is generating the RSTs in pfSense.
I have only the basic LAN rules.
I've deleted and/or disabled all my WAN rules just to figure this out.
As you can see from the post above, I don't think I have any packages loaded that would be doing filtering.
Ya'll have been extremely helpful. Any ideas on where to look next?
Thanks
-
What's on the Floating tab?
-
@jimp
Nothing on the Floating tab
-
Do you maybe have a port forward matching 8080 / 8081 that is matching it somehow?
Or maybe there is something hiding in the ruleset from another package. Post the contents of
/tmp/rules.debug
, see if it has anything unusual in there. -
@jimp
No port forwarding and only Automatic Outbound NAT rules.Here is the rules.debug
-
OK -- new information here. I know that "correlation doesn't equal causation" but maybe this will give someone else a clue.
I noticed that there were a bunch of miniupnpd errors in the Routing Logs.
So, I restarted the miniupnpd process. As long as that process is in the "Listening for NAT-PMP/PCP traffic on port 5351" state, the TiVo's can SUCCESSFULLY connect. As soon as the "ioctl(dev, DIOCGETADDRS, ...): Device busy" errors start showing up again (after about 4 or 5 minutes), the TiVo's go back to the state where the 8080 and 8081 transactions receive a RST response.
If I disable UPnP & NAT-PMP, the connections run just fine. As soon as I re-enable UPnP & NAT-PMP, we are back to our original state.
Doesn't make ANY sense to me given that the TiVo's don't use UPnP & NAT-PMP (as far as I know and/or can tell). So, how could that be blocking things and why does it get the multiple "Device Busy" errors?
I can live without it for a few days but I really would like to re-enable it for some other things I have on my network.
Thoughts?
-
@sydgarrett Perhaps this?
-
Curious, do you see anything in the UPnP status when the TiVo can't connect?
What if you try this command from the shell at the same time:
pfSsh.php playback pfanchordrill
-
@kom said in 21.05 blocking TiVo connections for unknown reasons:
@sydgarrett Perhaps this?
Yep. That sounds like the same thing. Also explains why this all started around the time of the upgrade to 21.05.
-
Nothing in the status for UPnP.
This is interesting
And, of course, those go away when the miniupnpd is restarted
I'm not following what those rules are there for but you clearly found our culprit for what is blocking 8080 and 8081
-
So despite the error, something on your local network is setting up block rules via UPnP for 8080, 8081, and 8082 which are rejecting all that traffic.
So UPnP is letting it happen, but ultimately it's being done by whatever device is adding those rules to UPnP.
Some kind of device with a web interface, which may limit your search when tracking it down.
Anything server-like on your local network that uses those ports for a web-based GUI to manage it?
-
Yeah, lot's of devices with web based GUIs. Let me spend some time tracking that down.
However, all of those "played well together" and I had no issues until about a month ago when the Tivo's started complaining. Is it possible that the rules are left from some intermediate state when the UPnP dies?
-
Doubtful, but I suppose it's possible. UPnP isn't dying, though, it's getting an error of some sort when attempting to probe addresses.
It's also possible that whatever is adding those rules had an update which enabled UPnP support to do what it's doing.
-
Googling the labels on the rules it appears they come from a QNAP device.
-
OK -- that helps. I've got a QNAP TS-653A on the network and it looks like it had a firmware update about the same time as all this started happening.
Now why it would be adding those rules is another question.
You have been a BIG help on this. Thanks so much!
-
From what I saw, QNAP has the ability to setup UPnP rules to allow outside access automatically. Seems dangerous to me for a NAS, but some people find it convenient. Probably a knob in there somewhere to shut that down.