21.05 blocking TiVo connections for unknown reasons
-
Hi folks.
Relative PfSense newbie here but fairly familiar with networking.Been using SG-3100 for about a year with no problems after initial setup. However, about 3 weeks ago, all three of my TiVo's starting complaining about being unable to make connections to the TiVo servers. Although I can't verify the exact date, this was about the time I upgraded to 21.05-Release.
After debugging on the TiVo side, the TiVo port test fails stating that ports 8080 and 8081 are being blocked. These were working fine up until about 3 weeks ago.
I verified that I have no Firewall rules blocking those ports. As a test, I stripped out ALL of my rules with no change to the TiVo symptoms. I have zero entries in the FireWall logs for the TiVo IP addresses.
I swapped out my SG-3100 for an old Netgear router and the TiVo's connect just fine. So, I verified that it is nothing else in my network causing the problem.
I disabled pfBlockerNG and eventually just deleted it -- still no change to the TiVo's connectability.
I did a packet capture on one of the TiVo's during its attempt to connect. I see nothing in there which indicates an error or a blocking. I see multiple connections with port 8080 but no port 8081. It successfully gets the correct time, validates the account, gets the service level, does a bunch of housekeeping transactions and then it just stops. There is probably something in there which would give a clue but I'm not familiar with what they are trying to do to figure it out. I do see a bunch of transmissions marked as "bad checksum" throughout the log.
The only other issue I'm having on the network is that I lost the ability to run Speedtest.net. The apps on my iphone and Apple TV as well as the app on my PC and the webpage ALL either just hang or give an error of "unable to connect". I tested those while I had swapped out the SG-3100 for the Netgear router and they worked just fine with the Netgear router.
I'm at a loss for what to look at next on this. Any pointers would be greatly appreciated.
-
I'm running 21.05 on my edge and have two TiVo devices, a Bolt Vox and an older Premiere XL. Both are phoning home and getting data as needed.
I do see a bunch of transmissions marked as "bad checksum" throughout the log.
That's probably from hardware checksum offloading and is completely normal.
How are the TiVo devices connecting? Wired? Wireless bridge? MoCA?
I can't see this being a problem in pfSense since it's working here, but it may be something in your configuration.
pfBlocker was a good place to start, but also check other packages, your DNS setup, things like that.
-
Thanks for the help and the suggestions.
All TiVo's are wired. However, as a test, I connected one of them via wifi with no change in results. I also used my phone as a hotspot and connected via that wifi and the TiVo connected just fine.
The DNS test on the TiVo's passed so I'm no sure what you are suggesting to look at in the DNS setup.
As far as other packages are concerned, I've pretty well stripped things down. The only packages I have left are
apcupsd
arping
arpwatch
Avahi
aws-wizard (no VPNs configured)
bandwidthd
Cron
darkstat
iperf
ipsec-profile-wizrd
mailreport
mtr-nox11
net-snmp
nmap
RRD_Summary
Service_Watchdog
sudo
System_Patches (PHP patch for 21.05 installed)I would agree that it seems to be something in my configuration. However, I haven't changed anything in the configuration in months (until I started debugging this problem).
-
@sydgarrett said in 21.05 blocking TiVo connections for unknown reasons:
I've pretty well stripped things down
18 installed packages is hardly "stripped down". Check the System logs. Check your firewall logs. Power off the Tivo, start a packet capture on the Tivo IP address and then take a look and see what's really going on. If you don't know your way around Wireshark, post your trace here with any public IP details obscured.
-
By checking DNS I mean looking at what you have setup there, especially in the advanced/custom options (e.g. from pfBlocker if you have DNSBL or things like that). It's possible something is allowing one DNS lookup to succeed but blocking the hostname of the next one.
I ran a packet capture of one of mine doing a network connection back to TiVo and it made several connections out to different servers. Most on port 80 and then one on port 8081, though I only saw one DNS request and it was only used for the very first port 80 connection.
-
@kom said in 21.05 blocking TiVo connections for unknown reasons:
18 installed packages is hardly "stripped down".
:) Well, I meant to say that I had stripped out anything that I thought could have an impact.
System logs have nothing regarding any of the TiVo IP addresses. Firewall logs -- same thing.
I did a packet capture around one of the TiVo's attempt to connect and went through it with WireShark. I don't see anything unusual out of it but would love to have more sets of eyes on it. If ya'll see anything, I'd love to hear it.
OK -- attempted to upload package capture .cap file but the forum software doesn't recognize it as an allowable file type. Suggestions for loading the .cap file?
Sorry for the newbie question.
-
Thanks for the help.
I only have 1.1.1.1 and 8.8.8.8 in as DNS servers. I also checked the box to allow the DNS Server list to be overridden by DHCP on WAN. DNS uses the local 127.0.0.1 and then falls back to the remote.
So, the resulting list is
127.0.0.1
209.18.47.62 (provided by Spectrum)
209.18.47.61 (provided by Spectrum)
1.1.1.1
8.8.8.8I also see one request for DNS, then requests to multiple servers on 80 and then one on 8080 and one on 8081.
-
@sydgarrett I believe you can select the lines then ctrl-c/ctrl-v them into a text file. From there, you can find & replace any public IP details.
Example:
1 0.000000 192.168.88.10 192.168.88.1 HTTP 573 POST /ifstats.php HTTP/1.1 (application/x-www-form-urlencoded) 2 0.000111 192.168.88.1 192.168.88.10 TCP 66 80 → 58784 [ACK] Seq=1 Ack=508 Win=511 Len=0 TSval=4108081792 TSecr=2254862057 3 0.000608 192.168.88.10 192.168.88.1 HTTP 648 POST /getstats.php HTTP/1.1 (application/x-www-form-urlencoded) 4 0.000755 192.168.88.1 192.168.88.10 TCP 66 80 → 58788 [ACK] Seq=1 Ack=583 Win=510 Len=0 TSval=870658763 TSecr=2254862058 5 0.035698 192.168.88.1 192.168.88.10 HTTP/JSON 683 HTTP/1.1 200 OK , JavaScript Object Notation (application/json) 6 0.035959 192.168.88.10 192.168.88.1 TCP 66 58784 → 80 [ACK] Seq=508 Ack=618 Win=524 Len=0 TSval=2254862093 TSecr=4108081828 7 0.390496 192.168.88.1 192.168.88.10 HTTP 656 HTTP/1.1 200 OK (text/html) 8 0.390773 192.168.88.10 192.168.88.1 TCP 66 58788 → 80 [ACK] Seq=583 Ack=591 Win=501 Len=0 TSval=2254862448 TSecr=870659153 9 1.001156 192.168.88.10 192.168.88.1 HTTP 510 GET /widgets/widgets/pfblockerng.widget.php?getNewWidget=1625105937799 HTTP/1.1 10 1.001253 192.168.88.1 192.168.88.10 TCP 66 80 → 58784 [ACK] Seq=618 Ack=952 Win=511 Len=0 TSval=4108082794 TSecr=2254863058 11 1.001270 192.168.88.10 192.168.88.1 HTTP 573 POST /ifstats.php HTTP/1.1 (application/x-www-form-urlencoded) 12 1.001308 192.168.88.1 192.168.88.10 TCP 66 80 → 58788 [ACK] Seq=591 Ack=1090 Win=511 Len=0 TSval=870659764 TSecr=2254863058 13 1.035307 192.168.88.1 192.168.88.10 HTTP/JSON 684 HTTP/1.1 200 OK , JavaScript Object Notation (application/json) 14 1.035569 192.168.88.10 192.168.88.1 TCP 66 58788 → 80 [ACK] Seq=1090 Ack=1209 Win=501 Len=0 TSval=2254863093 TSecr=870659797 -
Thanks. Here it is
Hopefully someone sees something in there that I'm missing.
-
@sydgarrett That looks normal to me, or at least nothing jumps out as being the problem. The two ends are talking just fine. The flow is what @jimp said it was for his units. Does the Tivo have any logs that might show what's wrong? You might have to bite the bullet and start disabling or removing packages to rule those out.
-
208 17.280483 10.0.1.153 208.73.181.202 TCP 74 49802 → 8080 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3450530 TSecr=0 WS=64 209 17.280579 208.73.181.202 10.0.1.153 TCP 54 8080 → 49802 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 213 17.503194 10.0.1.153 208.73.181.98 TCP 74 48909 → 8081 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3450753 TSecr=0 WS=64 214 17.503279 208.73.181.98 10.0.1.153 TCP 54 8081 → 48909 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0Something is immediately rejecting the 8080 and 8081 connections. The TiVo sends a SYN and immediately gets back a reset (RST+ACK) when it should be getting a SYN+ACK.
You can capture on the WAN to see if that's really coming from the remote end, but it is far more likely that's local. It's hitting a rule somewhere set to reject (not block).
-
@jimp said in 21.05 blocking TiVo connections for unknown reasons:
Something is immediately rejecting the 8080 and 8081 connections. The TiVo sends a SYN and immediately gets back a reset (RST+ACK) when it should be getting a SYN+ACK.
You can capture on the WAN to see if that's really coming from the remote end, but it is far more likely that's local. It's hitting a rule somewhere set to reject (not block).
Wow. I completely missed that. That matches with what the TiVo's were complaining about.
I did a capture on the WAN and verified that the RST isn't coming from the remote end. I see all the other traffic to that TiVo server. However, I don't see the port 8080 or 8081 exchanges. So, I don't see what is generating the RSTs in pfSense.
I have only the basic LAN rules.
I've deleted and/or disabled all my WAN rules just to figure this out.
As you can see from the post above, I don't think I have any packages loaded that would be doing filtering.
Ya'll have been extremely helpful. Any ideas on where to look next?
Thanks
-
What's on the Floating tab?
-
@jimp
Nothing on the Floating tab
-
Do you maybe have a port forward matching 8080 / 8081 that is matching it somehow?
Or maybe there is something hiding in the ruleset from another package. Post the contents of
/tmp/rules.debug, see if it has anything unusual in there. -
@jimp
No port forwarding and only Automatic Outbound NAT rules.Here is the rules.debug
-
OK -- new information here. I know that "correlation doesn't equal causation" but maybe this will give someone else a clue.
I noticed that there were a bunch of miniupnpd errors in the Routing Logs.
So, I restarted the miniupnpd process. As long as that process is in the "Listening for NAT-PMP/PCP traffic on port 5351" state, the TiVo's can SUCCESSFULLY connect. As soon as the "ioctl(dev, DIOCGETADDRS, ...): Device busy" errors start showing up again (after about 4 or 5 minutes), the TiVo's go back to the state where the 8080 and 8081 transactions receive a RST response.
If I disable UPnP & NAT-PMP, the connections run just fine. As soon as I re-enable UPnP & NAT-PMP, we are back to our original state.
Doesn't make ANY sense to me given that the TiVo's don't use UPnP & NAT-PMP (as far as I know and/or can tell). So, how could that be blocking things and why does it get the multiple "Device Busy" errors?
I can live without it for a few days but I really would like to re-enable it for some other things I have on my network.
Thoughts?
-
@sydgarrett Perhaps this?
-
Curious, do you see anything in the UPnP status when the TiVo can't connect?
What if you try this command from the shell at the same time:
pfSsh.php playback pfanchordrill -
@kom said in 21.05 blocking TiVo connections for unknown reasons:
@sydgarrett Perhaps this?
Yep. That sounds like the same thing. Also explains why this all started around the time of the upgrade to 21.05.
