21.05 blocking TiVo connections for unknown reasons
-
208 17.280483 10.0.1.153 208.73.181.202 TCP 74 49802 → 8080 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3450530 TSecr=0 WS=64 209 17.280579 208.73.181.202 10.0.1.153 TCP 54 8080 → 49802 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 213 17.503194 10.0.1.153 208.73.181.98 TCP 74 48909 → 8081 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3450753 TSecr=0 WS=64 214 17.503279 208.73.181.98 10.0.1.153 TCP 54 8081 → 48909 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0Something is immediately rejecting the 8080 and 8081 connections. The TiVo sends a SYN and immediately gets back a reset (RST+ACK) when it should be getting a SYN+ACK.
You can capture on the WAN to see if that's really coming from the remote end, but it is far more likely that's local. It's hitting a rule somewhere set to reject (not block).
-
@jimp said in 21.05 blocking TiVo connections for unknown reasons:
Something is immediately rejecting the 8080 and 8081 connections. The TiVo sends a SYN and immediately gets back a reset (RST+ACK) when it should be getting a SYN+ACK.
You can capture on the WAN to see if that's really coming from the remote end, but it is far more likely that's local. It's hitting a rule somewhere set to reject (not block).
Wow. I completely missed that. That matches with what the TiVo's were complaining about.
I did a capture on the WAN and verified that the RST isn't coming from the remote end. I see all the other traffic to that TiVo server. However, I don't see the port 8080 or 8081 exchanges. So, I don't see what is generating the RSTs in pfSense.
I have only the basic LAN rules.
I've deleted and/or disabled all my WAN rules just to figure this out.
As you can see from the post above, I don't think I have any packages loaded that would be doing filtering.
Ya'll have been extremely helpful. Any ideas on where to look next?
Thanks
-
What's on the Floating tab?
-
@jimp
Nothing on the Floating tab
-
Do you maybe have a port forward matching 8080 / 8081 that is matching it somehow?
Or maybe there is something hiding in the ruleset from another package. Post the contents of
/tmp/rules.debug, see if it has anything unusual in there. -
@jimp
No port forwarding and only Automatic Outbound NAT rules.Here is the rules.debug
-
OK -- new information here. I know that "correlation doesn't equal causation" but maybe this will give someone else a clue.
I noticed that there were a bunch of miniupnpd errors in the Routing Logs.
So, I restarted the miniupnpd process. As long as that process is in the "Listening for NAT-PMP/PCP traffic on port 5351" state, the TiVo's can SUCCESSFULLY connect. As soon as the "ioctl(dev, DIOCGETADDRS, ...): Device busy" errors start showing up again (after about 4 or 5 minutes), the TiVo's go back to the state where the 8080 and 8081 transactions receive a RST response.
If I disable UPnP & NAT-PMP, the connections run just fine. As soon as I re-enable UPnP & NAT-PMP, we are back to our original state.
Doesn't make ANY sense to me given that the TiVo's don't use UPnP & NAT-PMP (as far as I know and/or can tell). So, how could that be blocking things and why does it get the multiple "Device Busy" errors?
I can live without it for a few days but I really would like to re-enable it for some other things I have on my network.
Thoughts?
-
@sydgarrett Perhaps this?
-
Curious, do you see anything in the UPnP status when the TiVo can't connect?
What if you try this command from the shell at the same time:
pfSsh.php playback pfanchordrill -
@kom said in 21.05 blocking TiVo connections for unknown reasons:
@sydgarrett Perhaps this?
Yep. That sounds like the same thing. Also explains why this all started around the time of the upgrade to 21.05.
-
Nothing in the status for UPnP.
This is interesting
And, of course, those go away when the miniupnpd is restarted
I'm not following what those rules are there for but you clearly found our culprit for what is blocking 8080 and 8081
-
So despite the error, something on your local network is setting up block rules via UPnP for 8080, 8081, and 8082 which are rejecting all that traffic.
So UPnP is letting it happen, but ultimately it's being done by whatever device is adding those rules to UPnP.
Some kind of device with a web interface, which may limit your search when tracking it down.
Anything server-like on your local network that uses those ports for a web-based GUI to manage it?
-
Yeah, lot's of devices with web based GUIs. Let me spend some time tracking that down.
However, all of those "played well together" and I had no issues until about a month ago when the Tivo's started complaining. Is it possible that the rules are left from some intermediate state when the UPnP dies?
-
Doubtful, but I suppose it's possible. UPnP isn't dying, though, it's getting an error of some sort when attempting to probe addresses.
It's also possible that whatever is adding those rules had an update which enabled UPnP support to do what it's doing.
-
Googling the labels on the rules it appears they come from a QNAP device.
-
OK -- that helps. I've got a QNAP TS-653A on the network and it looks like it had a firmware update about the same time as all this started happening.
Now why it would be adding those rules is another question.
You have been a BIG help on this. Thanks so much!
-
From what I saw, QNAP has the ability to setup UPnP rules to allow outside access automatically. Seems dangerous to me for a NAS, but some people find it convenient. Probably a knob in there somewhere to shut that down.
-
Yeah, I shut down access outside my network a long time ago. Digging through things to see what might be setting up those rules.
-
@sydgarrett said in 21.05 blocking TiVo connections for unknown reasons:
Yeah, I shut down access outside my network a long time ago. Digging through things to see what might be setting up those rules.
Found it. Even though I had shut down access, there was still an option in there to configure the router using UPnP that had NOT been disabled. Don't remember that option being there in the past but it has been a LONG time since I set up the server.
Thanks SO much for your help on this. I consider this resolved (at least until the next update of something on the network :) )
Thanks!
