bypassing VPN for amazon/netflix troubleshooting

[norm1320]

Have been working for a while now to get 3 media devices to work with netflix and amazon while still running my vpn for everything else LANside. The two Rokus seem to be working most of the time, but the media PC is having major problems.

Current Setup:
Firewall Rules
Standard anti-lockout rule
192.168.10.31 LAN to WAN (Media PC)
192.168.10.60 LAN to WAN (Roku 1)
192.168.10.61 LAN to WAN (Roku 2)
catchall rule LAN to VPN

General setup
2 DNS lookup addresses assigned to WAN (standard OpenDNS addresses)
2 DNS lookup addresses assigned to VPN as provided by NordVPN
DNS address over-ride disallowed

Outbound NAT rules
Hybrid rules
WAN 127.0.0.0/8 * * 500(ISAKMP) WAN address * static
WAN 127.0.0.0/8 * * * WAN address * random
WAN ::1/128 * * 500(ISAKMP) WAN address * static
WAN ::1/128 * * * WAN address * random
WAN 192.168.10.0/24 * * 500(ISAKMP) WAN address * static
NORNVPN 192.168.10.0/24 * * * NORDVPN address * random
WAN 192.168.10.0/24 * * * WAN address * random

Symptoms:
For the two Rukos, they seem to work well with both Amazon and Netflix most of the time, but always seem to take longer than they used to before loading the videos.

For the PC, Amazon usually times out before making a connection (even to the main site). When it does connect it fails to load any thumnails of the videos in the Prime Video library but it does load the tooltips so if I take the time to hover over each title I can see what it is. If I then attempt to view a video it will go to the loading screen but will eventually time out. These symptoms seem to be more or less at random but in the end it will fail to actually load the video every time.

Netflix on the PC seems to have more consistent symptoms. After a reboot of both the PfSense box and the PC it will connect to Netflix fine (although somewhat slowly) and play a video, but only one. After playing that one video it will time out attempting to connect to Netflix at all and will continue to fail in this manner until doing another reboot of both machines.

If I disable the 192.168.10.31 firewall rule and allow the PC to go through the VPN it will then be able to easily and quickly connect to and navigate both sites, but of course will error out when trying to play a video (although it does sometimes still work with Netflix.)

Connecting to a site like whatsmyipaddress.com does correctly show my ISP address when the 192.168.10.31 rule is active and the VPN address when it is disabled.

Request for the community:
I am at a loss for where to look for properly diagnosing this situation. If anyone out there knows what setting I have wrong or can point me in the right direction for what I need to be looking at, please let me know.

Thanks in advance

edit: I should add that any other website I've tried going to works just fine. This seems to only involve sites that disallow using VPNs.

[oggydoggy]

@norm1320
I had similar issues with TV units and their apps. On your lan rule for the 192.168.10.31 going to the wan, on the left of the rule under states click on the reading there (which should not be zero if the unit is active) and it will open the states page. With my issue when the app on the tv was started this was full of links trying to get a dns resolution IE x.x.x.x:53, and not getting resolution and it failing.
In my instance it was cured by clicking services->DNS Resolver and ticking DNS Query forwarding. Also on same page make sure Network interfaces and outgoing network interfaces have ALL selected.
The states on the lan page when clicked only shows the states passing through that rule only so if its sitting at zero bytes then another rull above it or below it is actually handling the traffic. Same goes for when you look at the states and there are ports active but none are coming from the IP you want in that rule, so the rule is not doing what you want.
IE I created a rule that would forward OpenVPN on UDP from my main pc, so if I started the vpn app on the pc it would by pass the pfsense lan->vpn catchall rule. No point in a vpn though a vpn. It didn't work. Looking at the states for the rule, there are states from my pc but the port its sending from my pc to the pfsense turns out to be a random high number one and only the incoming port from the vpn has the OpenVPN std port. So I changed the rule and put the OpenVPN port on the destination port and set the source port to any. Then the rule activated. If there had been no states in the rule and the device is on and I expect traffic I would look at the states and start at the catchall and open it and then filter by the ip I expect the traffic from.
States which are at zero every day could mean you have a rule which doesn't do what you wanted and could be deleted, however it could be a device which is rarely turned on or rarely does that particular connecting.

[norm1320]

I don't see any failing DNS traffic in the state table for that rule. Checked the DNS Resolver settings anyway and DNS Query Forwarding is enabled.

[oggydoggy]

In the states page you will not see traffic blocked, rather when the connection first starts there should be a flurry of multiple dns states seen and when you refresh they disappear as they have got their answer. If they are hanging around for over a few seconds then they are not getting resolved. Not saying this is your issue.
Look in firewall logs at dns logs?
Go to diagnostics->states and filter for the device ip your looking for. Do all the seen states triggering on the lan interface go to the wan? they should.
Is roku rule all ports allowed including udp? certainly dns is udp.
Do you see traffic from the roku devices on any of the states going out the vpn? IE some port set to trigger a rule that sends them out the vlan and other out the wan. If these are being sent to the same destination then the destination will not like the two IP addresses.

What you describe as having is roughly correct. So it could be some of the detail behind these causing issues. Are there static routes and do these compete with the rule routing and send the traffic in a different interface than expected? I believe Lan traffic hits the lan rules then the static routes. So just creating a rule to route traffic the your chosen interface may be overruled by the static routes.
All issues I have seen.

Disable all your static routes and disable all the lan rules except the one your testing including the catch all , its easy to do and undo. Then check logs for traffic and get a look at just the one roku.

[norm1320]

Well, I haven't made any changes since I made the original post, however it all seems to be working quite well now. Perhaps something needed to propagate out or clear out of memory or something, but I don't see how since I restarted both machines and manually cleared the state tables multiple times before making that post. Since I doubt that I will be able to find the source of an issue that is currently absent I will simply make a note of what I have found in answer to the latest questions in the hopes that it might help someone in the future. I will keep an eye out for any suggestions that people make for future reference and come back with an update if things stop working again.

Yes, all the states triggering on the LAN interface seem to be going to the WAN at least for now.

The rules for the Rokus and the media PC do allow all ports including udp.

I have not seem any traffic from any of these 3 devices going out to the VPN.

I do not have any static routes set up for anything, however every LANside device on the network does have a static IP address if you think that might be a factor. (This is a home network, so only 25 devices currently connected) I have IPs 192.168.10.100-250 for transients like guest cellphones to be assigned as needed through DHCP.

If the problems return I will try disabling other LAN rules as my next troubleshooting step, but I don't think it would be a productive diagnostic step while everything seems to be working.

One last piece of information that might be useful for others who come across this: The last thing I did before making the original post (and so the most likely factor in things starting to work if it was in fact just something that needed time to propagate) was to switch the outbound NAT mode from manual to hybrid. It does not appear that PfSense made any changes when I did that, but perhaps I missed something there.

Anyway, thanks for the help. Fingers crossed that it is actually fixed and not just temporarily working.