Basic dual WAN policy-based routing setup doesn't work

[qfixxx]

Hello,
Running 2.5.2. I've got a dual wan scenario (Frontier and Spectrum) and single LAN. Frontier is the default gateway for the configured gateway group. I need to route all traffic our of the default Frontier WAN interface which works without issue. The only policy I need to apply is to send any streaming traffic destined for watch.spectrum.net out of the spectrum wan interface. This would seem to be easily achieved by creating an alias for watch.spectrum.net and then creating a LAN-based rule that specifically used the spectrum gateway. No matter what I do, traffic destined for watch.spectrum.net always routes out of the Frontier interface. While the documentation specifically calls out using an alias for CDN-derived content in a policy-based routing scenario is not recommended due to the transient nature of IP address assignments on the streaming side, it should work briefly, if maintaining the alias lookup was the issue, correct? Any suggestions?

[pete35]

Recently i saw the same problem with my pfsense 2.5.2. I created a static route which worked flawlessly. Give it a try.
But you may also try to reset all states or reboot the pfsense. Sometimes it works, if the rule catches the traffic.

[johnpoz]

@qfixxx said in Basic dual WAN policy-based routing setup doesn't work:

Any suggestions?

Lets see you rules on lan. They are processed in order, top down first rule to trigger wins.. So if you have some rule that allows the traffic before your policy route - then it would go out the default gateway, etc.

Also as mentioned - once a state is created, rule would use that state to be allowed. When you create new rules that might block, or policy route specific traffic. You need to make sure that any states for that traffic have been removed. Either closed on their own, timed out, or you killed them, etc.

I also see that is hosted per cdn.

;; QUESTION SECTION:
;watch.spectrum.net.            IN      A

;; ANSWER SECTION:
watch.spectrum.net.     3600    IN      CNAME   blue.watch.spectrum.net.
blue.watch.spectrum.net. 3600   IN      CNAME   k8s-nginx-nginxalb-bf946806b4-1335036550.us-east-1.elb.amazonaws.com.
k8s-nginx-nginxalb-bf946806b4-1335036550.us-east-1.elb.amazonaws.com. 3600 IN A 34.194.70.174
k8s-nginx-nginxalb-bf946806b4-1335036550.us-east-1.elb.amazonaws.com. 3600 IN A 52.2.131.75
k8s-nginx-nginxalb-bf946806b4-1335036550.us-east-1.elb.amazonaws.com. 3600 IN A 3.224.10.42

So yeah you could have disconnect from what pfsense knows of the IPs, and where your client might be going. Especially if using different dns - client using doh for example vs asking pfsense for the dns. Or client using some hard coded dns vs what you hand it via dhcp or tell it to use.

[qfixxx]

@johnpoz Thanks. Posting a screenshot of LAN rules and trace route. I've reset the state, but it still refuses to honor the rule. The Frontier interface is set as the default and both Frontier, and Spectrum interfaces are in a gateway group. I've also set Aliases Hostname Resolve Internal to 30 seconds. Granted, that's lower than I would normally set it, but wanted to see if lookups were part of the issue.

[johnpoz]

@qfixxx not sure why you think a traceroute should work there? Do it from your client on your lan.

There is a hidden rule that allows the firewall to go anywhere, which parsed before your interface rules.

pass out  inet all keep state allow-opts tracker 1000012115 label "let out anything IPv4 from firewall host itself"

And lan rules you posted - isn't showing what actual gateway you se for the policy route. Valid test would be from a client on the lan doing the trace - this would also allow you to see what IP the client is trying to go to - and if it exists in the alias table.

[qfixxx]

@johnpoz I guess I mistakenly assumed that setting the source to 'LAN' on the trace route would generate traffic that would be covered by the policy. I think what had happened was the alias had not kept pace with a change in IP. Resetting the state got me going out the Spectrum interface but now I'm going to have to debug what other URLs/IPs maybe involved in subsequent calls to watch.spectrum.net as it's still failing, but now I just need to dig into the additional URLs.

[johnpoz]

@qfixxx said in Basic dual WAN policy-based routing setup doesn't work:

have to debug what other URLs/IPs maybe involved in subsequent calls to watch.spectrum.net

Yeah that can be PITA ;)

A sniff of all the traffic when force all its traffic out the correct gateway and it works can be helpful.