Routing a service to non-default WAN
-
@viragomann
Are you saying that policy rules should work for handling the traffic from pfsense itself (like ipsec)?
Becuase I tried almost everyting and also find a lot of posts here on the forum and elsewhere where the issue seem basically the same:
https://forum.netgate.com/topic/8332/ipsec-on-opt1-wan2
https://forum.netgate.com/topic/116745/policy-based-routing-for-traffic-from-pfsense-itself/4and here it is even you who states something telling me policy routing is not the way it could be done:
https://forum.netgate.com/topic/180384/how-to-setup-an-interface-specific-default-gatewayThanks!
-
@mik256
If I understood correctly, you're talking about response traffic to incoming requests.
This should not need any policy or static routing, but should rather be routed properly automatically. -
@viragomann
Yes, that would be logic, but tcpdump shows pfsense still uses WAN1 for replies to ipsec initiations comming from WAN2:( -
@mik256
Did you obey my above suggestions? -
Which pfSense version are you running?
You cannot policy route traffic from the firewall itself as you said. But this should not be a policy routing issue. It may be a reply-to issue if the 2nd WAN is not being assigned with a gateway in the usual way.
-
@stephenw10
Thanks! So many people keep telling I have not set policy routing properly, finally someone confirmed this is not the way.
My wan2 is interface with ip 192.168.40.3 and gw set to 192.168.40.1 (this is another router with port forwarding). Wan1 is default gw. I have tried using gw groups etc with no success. What could be wrong with the wan2 setup? -
Is the gateway actually defined on the interface? That's what configures it as a WAN with reply-to and route-to on rules.
Also I assume IPSec connections are failing because clients see traffic from the wrong IP?
-
@stephenw10
I am on pfsense 2.7.2-release.
Yes ipsec failing because replies are sent from wrong WAN. If I make a static route to the remote ipsec endpoint, ipsec connects immediately and everything works.This is my WAN2 interface, it has gateway associated:
this is my P1:
Pls help! Thanks
-
@viragomann said in Routing a service to non-default WAN:
@mik256
Did you obey my above suggestions?
Of course I did. I ended with every imaginable policy rule just to make sure I did not omit anything.. No luck:
-
Looks OK.
Try to connect to it and check the states. See what's actually being opened there.Make sure wherever you're connecting from is not in the routing table as a static route.
You can also check the ruleset in /tmp/rules.debug to make sure it has the reply-to tags on the rule.
-
@mik256 said in Routing a service to non-default WAN:
Did you obey my above suggestions?
Of course I did. I ended with every imaginable policy rule just to make sure I did not omit anything.. No luck:
But the suggestion was to not use floating rules or interface group rules for incoming connections!
This disables reply-to.
-
Oh, and you should not have a gateway on those inbound rules on PE3. That's invalid and can do weird stuff.
You shouldn't actually need any rules to pass the IPSec traffic. It's passed bu automatically added rules unless you have disabled them.
-
@stephenw10
no reply to besides the one that i set up to reach pfsense over ssh from home
cat /tmp/rules.debug | grep 193
pass in quick on $WAN reply-to ( hn1 'WAN1GW' ) inet proto tcp from '193.x.x.x to 'WAN1' port 22 ridentifier 1728499701 flags S/SA keep state label "USER_RULE: ssh home" label "id:1728499701"inbound rules on PE3 cleared (I knew it was bad, bud was too desperate to try everyting lol)
-
@viragomann said in Routing a service to non-default WAN:
But the suggestion was to not use floating rules or interface group rules for incoming connections!
This disables reply-to.
Would you please explain? I deleted all policy rules from all interfaces and just left those floating rules. They have direction out and PE3GW. That should target outgoing connection.. or not?
There is likely something intefering with reply-to, but I just couldn't see what. I see on tcpdump, that packets are being sent back to the initiator with correct IP (of PE3), but on the interface of WAN1. -
Any rules that can apply to more than one interface cannot use reply-to tags because it cannot know which interface to reply on.
That means any rules applied to interface groups or any floating rules. So if you have either of those and they are passing the IPSec traffic it would break reply-to.
-
@mik256
I was talking about incoming connections only, as mentioned above. reply-to only applies to incoming traffic.As @stephenw10 mentioned, the reply-to tagging requires, that the interface has a gateway stated in the interface settings.
The reply-to tag contains the respective gateway, where the reply should be sent to. This tag is added to an incoming connection by the firewall rule, which is allowing it. But this requires, that the rule the gateway is unique.
Floating and group rules can be applied to multiple gateway. Hence pfSense doesn't add the reply-to for connections permitted by these rules. -
@stephenw10
I disabled all floating rules. Besides no interface groups have been set up. But the issue persists - pfsense replies with correct IP but on wrong interface :( -
@viragomann
I think I don't have any of these now. I rid of policy rules completely, have no floating rules or interface groups. However I am not sure - the "IPv4 Upstream gateway" on the PE3 interface should be enough for reply-to WAN2 on WAN2GW? You talked about incoming firewall rules - do I need to set up some of these and if yes, could you give some example (like do I need to put anything in the advanced->gateway?).
Thank you very much, I spent so many hours trying to get the ipsec on secondary wan routing that I really don't want to give up setting up another pfsense with just the WAN2 interface to get it working lol. -
@mik256 said in Routing a service to non-default WAN:
But the issue persists - pfsense replies with correct IP but on wrong interface
The only reason for this, I can think of, is that both WAN gateways are on the same layer 2. Is this the case?
However I am not sure - the "IPv4 Upstream gateway" on the PE3 interface should be enough for reply-to WAN2 on WAN2GW?
Yes, it needs the gateway in the interface setting and a rule on the respective interface.
No floating or group rule must matching the incoming traffic, because this rules have precedence.You talked about incoming firewall rules - do I need to set up some of these
@stephenw10 said, IPSec access is allowed allowed automatically if an instance is enabled. I don't know.
But if you enable the logging of the default deny rule (in the log settings) and there is nothing to see regarding IPSec in the logs then, while packets are coming in, it might work without a user rule. -
Just to be clear PE3 is the second WAN interface here right?
Is the subnet conflicting with the primary WAN? Or gateway?
Does other traffic work correctly from PE3? Can you ping out from it for example?
