Routing a service to non-default WAN

mik256

@stephenw10
Thanks! So many people keep telling I have not set policy routing properly, finally someone confirmed this is not the way.
My wan2 is interface with ip 192.168.40.3 and gw set to 192.168.40.1 (this is another router with port forwarding). Wan1 is default gw. I have tried using gw groups etc with no success. What could be wrong with the wan2 setup?

stephenw10

Is the gateway actually defined on the interface? That's what configures it as a WAN with reply-to and route-to on rules.

Also I assume IPSec connections are failing because clients see traffic from the wrong IP?

mik256

@stephenw10
I am on pfsense 2.7.2-release.
Yes ipsec failing because replies are sent from wrong WAN. If I make a static route to the remote ipsec endpoint, ipsec connects immediately and everything works.

This is my WAN2 interface, it has gateway associated:

this is my P1:

Pls help! Thanks

mik256

@viragomann said in Routing a service to non-default WAN:

@mik256
Did you obey my above suggestions?
Of course I did. I ended with every imaginable policy rule just to make sure I did not omit anything.. No luck:

stephenw10

Looks OK.
Try to connect to it and check the states. See what's actually being opened there.

Make sure wherever you're connecting from is not in the routing table as a static route.

You can also check the ruleset in /tmp/rules.debug to make sure it has the reply-to tags on the rule.

viragomann

@mik256 said in Routing a service to non-default WAN:

@mik256

Did you obey my above suggestions?

Of course I did. I ended with every imaginable policy rule just to make sure I did not omit anything.. No luck:

But the suggestion was to not use floating rules or interface group rules for incoming connections!

This disables reply-to.

stephenw10

Oh, and you should not have a gateway on those inbound rules on PE3. That's invalid and can do weird stuff.

You shouldn't actually need any rules to pass the IPSec traffic. It's passed bu automatically added rules unless you have disabled them.

mik256

@stephenw10
no reply to besides the one that i set up to reach pfsense over ssh from home
cat /tmp/rules.debug | grep 193
pass in quick on $WAN reply-to ( hn1 'WAN1GW' ) inet proto tcp from '193.x.x.x to 'WAN1' port 22 ridentifier 1728499701 flags S/SA keep state label "USER_RULE: ssh home" label "id:1728499701"

inbound rules on PE3 cleared (I knew it was bad, bud was too desperate to try everyting lol)

mik256

@viragomann said in Routing a service to non-default WAN:

But the suggestion was to not use floating rules or interface group rules for incoming connections!

This disables reply-to.

Would you please explain? I deleted all policy rules from all interfaces and just left those floating rules. They have direction out and PE3GW. That should target outgoing connection.. or not?
There is likely something intefering with reply-to, but I just couldn't see what. I see on tcpdump, that packets are being sent back to the initiator with correct IP (of PE3), but on the interface of WAN1.

stephenw10

Any rules that can apply to more than one interface cannot use reply-to tags because it cannot know which interface to reply on.

That means any rules applied to interface groups or any floating rules. So if you have either of those and they are passing the IPSec traffic it would break reply-to.

viragomann

@mik256
I was talking about incoming connections only, as mentioned above. reply-to only applies to incoming traffic.

As @stephenw10 mentioned, the reply-to tagging requires, that the interface has a gateway stated in the interface settings.
The reply-to tag contains the respective gateway, where the reply should be sent to. This tag is added to an incoming connection by the firewall rule, which is allowing it. But this requires, that the rule the gateway is unique.
Floating and group rules can be applied to multiple gateway. Hence pfSense doesn't add the reply-to for connections permitted by these rules.

mik256

@stephenw10
I disabled all floating rules. Besides no interface groups have been set up. But the issue persists - pfsense replies with correct IP but on wrong interface :(

mik256

@viragomann
I think I don't have any of these now. I rid of policy rules completely, have no floating rules or interface groups. However I am not sure - the "IPv4 Upstream gateway" on the PE3 interface should be enough for reply-to WAN2 on WAN2GW? You talked about incoming firewall rules - do I need to set up some of these and if yes, could you give some example (like do I need to put anything in the advanced->gateway?).
Thank you very much, I spent so many hours trying to get the ipsec on secondary wan routing that I really don't want to give up setting up another pfsense with just the WAN2 interface to get it working lol.

viragomann

@mik256 said in Routing a service to non-default WAN:

But the issue persists - pfsense replies with correct IP but on wrong interface

The only reason for this, I can think of, is that both WAN gateways are on the same layer 2. Is this the case?

However I am not sure - the "IPv4 Upstream gateway" on the PE3 interface should be enough for reply-to WAN2 on WAN2GW?

Yes, it needs the gateway in the interface setting and a rule on the respective interface.
No floating or group rule must matching the incoming traffic, because this rules have precedence.

You talked about incoming firewall rules - do I need to set up some of these

@stephenw10 said, IPSec access is allowed allowed automatically if an instance is enabled. I don't know.
But if you enable the logging of the default deny rule (in the log settings) and there is nothing to see regarding IPSec in the logs then, while packets are coming in, it might work without a user rule.

stephenw10

Just to be clear PE3 is the second WAN interface here right?

Is the subnet conflicting with the primary WAN? Or gateway?

Does other traffic work correctly from PE3? Can you ping out from it for example?

mik256

@viragomann

The only reason for this, I can think of, is that both WAN gateways are on the same layer 2. Is this the case?

They are not on the same layer2:
WAN1 interface has public IP 2.3.4.5 assigned and is directly connected to the ISP router.
WAN2 interface has 192.168.40.3 and it is behind another router NAT.

Yes, it needs the gateway in the interface setting and a rule on the respective interface.

Both interfaces have gateways assigned in their interface settings (1.2.3.4 for WAN1 and 192.168.40.1 for WAN2).

@stephenw10 said, IPSec access is allowed allowed automatically if an instance is enabled. I don't know.
But if you enable the logging of the default deny rule (in the log settings) and there is nothing to see regarding IPSec in the logs then, while packets are coming in, it might work without a user rule.

I have tried disabling ipsec service and mimic with socat, no luck either.

Can you please think of something else breaking reply-to mechanism?

mik256

@stephenw10 said in Routing a service to non-default WAN:

Just to be clear PE3 is the second WAN interface here right?

Is the subnet conflicting with the primary WAN? Or gateway?

Does other traffic work correctly from PE3? Can you ping out from it for example?

Yes, PE3 is the second WAN. I can ping from it, see traffic comming from its public ip on remote endpoints (if I create static rule). There's no coflict in WANs networks - WAN1 has public ip and is directly connected to the isp's router, PE3 (the second WAN with 192.168.40.3) is behind another router (192.168.40.1) with NAT.

I have made this test:

disabled ipsec in pfsense
run socat on pfsense:

[2.7.2-RELEASE][admin@pfSense-aav1]/root: /usr/local/bin/socat -v UDP-LISTEN:500,bind=192.168.40.3 UDP-SENDTO:VPS-PUBLIC-IP:500
aa

run netcat on VPS:

nc -u PE3-PUBLIC-IP 500

run tcpdump on the VPS:

root@vmi1032933:~# tcpdump -i eth0 -nn udp port 500
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:18:05.726660 IP VPS-PUBLIC-IP.53642 > PE3-PUBLIC-IP.500: [|isakmp]
09:18:05.753419 IP WAN1-PUBLIC-IP.1599 > VPS-PUBLIC-IP.500: [|isakmp]

Then I tried again with policy rules (with PE3GW), again same result. The VPS sends packet to WAN2 and pfsense replies with WAN1 instead.
Am I doing something wrong? What could be breaking the reply-to functionality?
Thank you so much!

viragomann

@mik256 said in Routing a service to non-default WAN:

root@vmi1032933:~# tcpdump -i eth0 -nn udp port 500
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:18:05.726660 IP VPS-PUBLIC-IP.53642 > PE3-PUBLIC-IP.500: [|isakmp]
09:18:05.753419 IP WAN1-PUBLIC-IP.1599 > VPS-PUBLIC-IP.500: [|isakmp]

The second packet doesn't seem to be a response to the first one, but an additional connection.

On which interface of pfSense to you see this?
I'm not expect to see this on PE3 / WAN2. Otherwise you've probably configured the outbound NAT wrongly.

And also this controverts your statement above:

pfsense replies with correct IP but on wrong interface :(

Now what? Wrong IP or wrong interface?

mik256

@viragomann said in Routing a service to non-default WAN:

The second packet doesn't seem to be a response to the first one, but an additional connection.

On which interface of pfSense to you see this?
I'm not expect to see this on PE3 / WAN2. Otherwise you've probably configured the outbound NAT wrongly.

You are correct. There's a different port number being used. Maybe that's how socat works. I think ipsec didn't behave like this, but I will find out.

And also this controverts your statement above:

pfsense replies with correct IP but on wrong interface :(

Now what? Wrong IP or wrong interface?

It's still the same, sorry for the confusion. On pfsense tcpdump shows WAN2 IP is used for reply, but using tcpdump on WAN1 interface. If I do packet capture on WAN2, I don't see any reply being send at all.

I tried with http (pfsense web ui) and there reply-to worked fine: wget from a host to WAN2 received html from WAN2 public ip. Maybe reply-to fails just for udp?

viragomann

@mik256 said in Routing a service to non-default WAN:

Maybe that's how socat works. I think ipsec didn't behave like this, but I will find out.

I'd expect that a device sends a response to the source IP and port it found in the request packet.
So this capture doesn't seem to be suitable to troubleshoot this for me.

Maybe reply-to fails just for udp?

Just make a test with a custom rule for IPSec.
Add a rule to WAN2 allowing IPSec traffic to the interface IP (if the service is running on pfSense itself), enable logging and state a unique description.
Custom rules should override automatic rule and should hence be applied.

Try to connect from remote and check in the firewall log if the rule was applied.

Reply-to only affects to replies. If pfSense itself initiates a connection it will follow the routing table.
So if you want to use a certain gateway you will have to set the routes respectively.
However, if I remember correctly, it's possible to policy route traffic with a floating rule for direction 'out'.