Routing a service to non-default WAN
-
@mik256 Is this for inbound or outbound? You can control outbound in pfSense, and that's normally policy routing. Why/how was that not working...?
https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html#configuring-firewall-rules-for-policy-routing
You can control that by rule matching, as in, the source IP or the destination IP.
Do the remote endpoints with dynamic IPs have a hostname you can use in an alias? And, one IP or is it a CDN with multiple IPs that rotate?
-
@SteveITS
Thanks for quick reply!
It is outgoing what is causing issues. I have tried policy routing using IP, port, etc. all combinations. There's one remote endpoint with 1.2.3.4 dynamic from ISP Seems like static routes always win/precedes policy routing. If I use static route to 1.2.3.4 using WAN2GW, it works ok. If policy routing WANGW with default gw set to WAN1GW, it always uses WAN1GW for ike replies as well as testing with traceroute -p 500 -s WAN2IP.
It is ridiculous, WAN2 is only interface for IPSec, ike initiations come to WAN2, but pfsense just always replies with WAN1. -
I have read every topic on forums regarding this issue. It seems you can not use policy routing for traffic comming from the firewall itself (like ipsec response). I don't feel like putting another pfsense box just to have a service wokring on non-default WAN. Is there really not a solution to that?
-
@mik256 said in Routing a service to non-default WAN:
It is ridiculous, WAN2 is only interface for IPSec, ike initiations come to WAN2, but pfsense just always replies with WAN1
This is only the case with wrong dual WAN configuration.
Do you use a floating rule or interface group rule to permit IPSec access?
You have to state a rule allowing it on the respective interface tab and ensure, that this rule is applied.Note, that floating and interface group rules have precedence over interface rules.
-
@viragomann
Are you saying that policy rules should work for handling the traffic from pfsense itself (like ipsec)?
Becuase I tried almost everyting and also find a lot of posts here on the forum and elsewhere where the issue seem basically the same:
https://forum.netgate.com/topic/8332/ipsec-on-opt1-wan2
https://forum.netgate.com/topic/116745/policy-based-routing-for-traffic-from-pfsense-itself/4and here it is even you who states something telling me policy routing is not the way it could be done:
https://forum.netgate.com/topic/180384/how-to-setup-an-interface-specific-default-gatewayThanks!
-
@mik256
If I understood correctly, you're talking about response traffic to incoming requests.
This should not need any policy or static routing, but should rather be routed properly automatically. -
@viragomann
Yes, that would be logic, but tcpdump shows pfsense still uses WAN1 for replies to ipsec initiations comming from WAN2:( -
@mik256
Did you obey my above suggestions? -
Which pfSense version are you running?
You cannot policy route traffic from the firewall itself as you said. But this should not be a policy routing issue. It may be a reply-to issue if the 2nd WAN is not being assigned with a gateway in the usual way.
-
@stephenw10
Thanks! So many people keep telling I have not set policy routing properly, finally someone confirmed this is not the way.
My wan2 is interface with ip 192.168.40.3 and gw set to 192.168.40.1 (this is another router with port forwarding). Wan1 is default gw. I have tried using gw groups etc with no success. What could be wrong with the wan2 setup? -
Is the gateway actually defined on the interface? That's what configures it as a WAN with reply-to and route-to on rules.
Also I assume IPSec connections are failing because clients see traffic from the wrong IP?
-
@stephenw10
I am on pfsense 2.7.2-release.
Yes ipsec failing because replies are sent from wrong WAN. If I make a static route to the remote ipsec endpoint, ipsec connects immediately and everything works.This is my WAN2 interface, it has gateway associated:
this is my P1:
Pls help! Thanks
-
@viragomann said in Routing a service to non-default WAN:
@mik256
Did you obey my above suggestions?
Of course I did. I ended with every imaginable policy rule just to make sure I did not omit anything.. No luck:
-
Looks OK.
Try to connect to it and check the states. See what's actually being opened there.Make sure wherever you're connecting from is not in the routing table as a static route.
You can also check the ruleset in /tmp/rules.debug to make sure it has the reply-to tags on the rule.
-
@mik256 said in Routing a service to non-default WAN:
Did you obey my above suggestions?
Of course I did. I ended with every imaginable policy rule just to make sure I did not omit anything.. No luck:
But the suggestion was to not use floating rules or interface group rules for incoming connections!
This disables reply-to.
-
Oh, and you should not have a gateway on those inbound rules on PE3. That's invalid and can do weird stuff.
You shouldn't actually need any rules to pass the IPSec traffic. It's passed bu automatically added rules unless you have disabled them.
-
@stephenw10
no reply to besides the one that i set up to reach pfsense over ssh from home
cat /tmp/rules.debug | grep 193
pass in quick on $WAN reply-to ( hn1 'WAN1GW' ) inet proto tcp from '193.x.x.x to 'WAN1' port 22 ridentifier 1728499701 flags S/SA keep state label "USER_RULE: ssh home" label "id:1728499701"inbound rules on PE3 cleared (I knew it was bad, bud was too desperate to try everyting lol)
-
@viragomann said in Routing a service to non-default WAN:
But the suggestion was to not use floating rules or interface group rules for incoming connections!
This disables reply-to.
Would you please explain? I deleted all policy rules from all interfaces and just left those floating rules. They have direction out and PE3GW. That should target outgoing connection.. or not?
There is likely something intefering with reply-to, but I just couldn't see what. I see on tcpdump, that packets are being sent back to the initiator with correct IP (of PE3), but on the interface of WAN1. -
Any rules that can apply to more than one interface cannot use reply-to tags because it cannot know which interface to reply on.
That means any rules applied to interface groups or any floating rules. So if you have either of those and they are passing the IPSec traffic it would break reply-to.
-
@mik256
I was talking about incoming connections only, as mentioned above. reply-to only applies to incoming traffic.As @stephenw10 mentioned, the reply-to tagging requires, that the interface has a gateway stated in the interface settings.
The reply-to tag contains the respective gateway, where the reply should be sent to. This tag is added to an incoming connection by the firewall rule, which is allowing it. But this requires, that the rule the gateway is unique.
Floating and group rules can be applied to multiple gateway. Hence pfSense doesn't add the reply-to for connections permitted by these rules.
