Routing a service to non-default WAN

mik256

@stephenw10
no reply to besides the one that i set up to reach pfsense over ssh from home
cat /tmp/rules.debug | grep 193
pass in quick on $WAN reply-to ( hn1 'WAN1GW' ) inet proto tcp from '193.x.x.x to 'WAN1' port 22 ridentifier 1728499701 flags S/SA keep state label "USER_RULE: ssh home" label "id:1728499701"

inbound rules on PE3 cleared (I knew it was bad, bud was too desperate to try everyting lol)

mik256

@viragomann said in Routing a service to non-default WAN:

But the suggestion was to not use floating rules or interface group rules for incoming connections!

This disables reply-to.

Would you please explain? I deleted all policy rules from all interfaces and just left those floating rules. They have direction out and PE3GW. That should target outgoing connection.. or not?
There is likely something intefering with reply-to, but I just couldn't see what. I see on tcpdump, that packets are being sent back to the initiator with correct IP (of PE3), but on the interface of WAN1.

stephenw10

Any rules that can apply to more than one interface cannot use reply-to tags because it cannot know which interface to reply on.

That means any rules applied to interface groups or any floating rules. So if you have either of those and they are passing the IPSec traffic it would break reply-to.

viragomann

@mik256
I was talking about incoming connections only, as mentioned above. reply-to only applies to incoming traffic.

As @stephenw10 mentioned, the reply-to tagging requires, that the interface has a gateway stated in the interface settings.
The reply-to tag contains the respective gateway, where the reply should be sent to. This tag is added to an incoming connection by the firewall rule, which is allowing it. But this requires, that the rule the gateway is unique.
Floating and group rules can be applied to multiple gateway. Hence pfSense doesn't add the reply-to for connections permitted by these rules.

mik256

@stephenw10
I disabled all floating rules. Besides no interface groups have been set up. But the issue persists - pfsense replies with correct IP but on wrong interface :(

mik256

@viragomann
I think I don't have any of these now. I rid of policy rules completely, have no floating rules or interface groups. However I am not sure - the "IPv4 Upstream gateway" on the PE3 interface should be enough for reply-to WAN2 on WAN2GW? You talked about incoming firewall rules - do I need to set up some of these and if yes, could you give some example (like do I need to put anything in the advanced->gateway?).
Thank you very much, I spent so many hours trying to get the ipsec on secondary wan routing that I really don't want to give up setting up another pfsense with just the WAN2 interface to get it working lol.

viragomann

@mik256 said in Routing a service to non-default WAN:

But the issue persists - pfsense replies with correct IP but on wrong interface

The only reason for this, I can think of, is that both WAN gateways are on the same layer 2. Is this the case?

However I am not sure - the "IPv4 Upstream gateway" on the PE3 interface should be enough for reply-to WAN2 on WAN2GW?

Yes, it needs the gateway in the interface setting and a rule on the respective interface.
No floating or group rule must matching the incoming traffic, because this rules have precedence.

You talked about incoming firewall rules - do I need to set up some of these

@stephenw10 said, IPSec access is allowed allowed automatically if an instance is enabled. I don't know.
But if you enable the logging of the default deny rule (in the log settings) and there is nothing to see regarding IPSec in the logs then, while packets are coming in, it might work without a user rule.

stephenw10

Just to be clear PE3 is the second WAN interface here right?

Is the subnet conflicting with the primary WAN? Or gateway?

Does other traffic work correctly from PE3? Can you ping out from it for example?

mik256

@viragomann

The only reason for this, I can think of, is that both WAN gateways are on the same layer 2. Is this the case?

They are not on the same layer2:
WAN1 interface has public IP 2.3.4.5 assigned and is directly connected to the ISP router.
WAN2 interface has 192.168.40.3 and it is behind another router NAT.

Yes, it needs the gateway in the interface setting and a rule on the respective interface.

Both interfaces have gateways assigned in their interface settings (1.2.3.4 for WAN1 and 192.168.40.1 for WAN2).

@stephenw10 said, IPSec access is allowed allowed automatically if an instance is enabled. I don't know.
But if you enable the logging of the default deny rule (in the log settings) and there is nothing to see regarding IPSec in the logs then, while packets are coming in, it might work without a user rule.

I have tried disabling ipsec service and mimic with socat, no luck either.

Can you please think of something else breaking reply-to mechanism?

mik256

@stephenw10 said in Routing a service to non-default WAN:

Just to be clear PE3 is the second WAN interface here right?

Is the subnet conflicting with the primary WAN? Or gateway?

Does other traffic work correctly from PE3? Can you ping out from it for example?

Yes, PE3 is the second WAN. I can ping from it, see traffic comming from its public ip on remote endpoints (if I create static rule). There's no coflict in WANs networks - WAN1 has public ip and is directly connected to the isp's router, PE3 (the second WAN with 192.168.40.3) is behind another router (192.168.40.1) with NAT.

I have made this test:

disabled ipsec in pfsense
run socat on pfsense:

[2.7.2-RELEASE][admin@pfSense-aav1]/root: /usr/local/bin/socat -v UDP-LISTEN:500,bind=192.168.40.3 UDP-SENDTO:VPS-PUBLIC-IP:500
aa

run netcat on VPS:

nc -u PE3-PUBLIC-IP 500

run tcpdump on the VPS:

root@vmi1032933:~# tcpdump -i eth0 -nn udp port 500
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:18:05.726660 IP VPS-PUBLIC-IP.53642 > PE3-PUBLIC-IP.500: [|isakmp]
09:18:05.753419 IP WAN1-PUBLIC-IP.1599 > VPS-PUBLIC-IP.500: [|isakmp]

Then I tried again with policy rules (with PE3GW), again same result. The VPS sends packet to WAN2 and pfsense replies with WAN1 instead.
Am I doing something wrong? What could be breaking the reply-to functionality?
Thank you so much!

viragomann

@mik256 said in Routing a service to non-default WAN:

root@vmi1032933:~# tcpdump -i eth0 -nn udp port 500
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:18:05.726660 IP VPS-PUBLIC-IP.53642 > PE3-PUBLIC-IP.500: [|isakmp]
09:18:05.753419 IP WAN1-PUBLIC-IP.1599 > VPS-PUBLIC-IP.500: [|isakmp]

The second packet doesn't seem to be a response to the first one, but an additional connection.

On which interface of pfSense to you see this?
I'm not expect to see this on PE3 / WAN2. Otherwise you've probably configured the outbound NAT wrongly.

And also this controverts your statement above:

pfsense replies with correct IP but on wrong interface :(

Now what? Wrong IP or wrong interface?

mik256

@viragomann said in Routing a service to non-default WAN:

The second packet doesn't seem to be a response to the first one, but an additional connection.

On which interface of pfSense to you see this?
I'm not expect to see this on PE3 / WAN2. Otherwise you've probably configured the outbound NAT wrongly.

You are correct. There's a different port number being used. Maybe that's how socat works. I think ipsec didn't behave like this, but I will find out.

And also this controverts your statement above:

pfsense replies with correct IP but on wrong interface :(

Now what? Wrong IP or wrong interface?

It's still the same, sorry for the confusion. On pfsense tcpdump shows WAN2 IP is used for reply, but using tcpdump on WAN1 interface. If I do packet capture on WAN2, I don't see any reply being send at all.

I tried with http (pfsense web ui) and there reply-to worked fine: wget from a host to WAN2 received html from WAN2 public ip. Maybe reply-to fails just for udp?

viragomann

@mik256 said in Routing a service to non-default WAN:

Maybe that's how socat works. I think ipsec didn't behave like this, but I will find out.

I'd expect that a device sends a response to the source IP and port it found in the request packet.
So this capture doesn't seem to be suitable to troubleshoot this for me.

Maybe reply-to fails just for udp?

Just make a test with a custom rule for IPSec.
Add a rule to WAN2 allowing IPSec traffic to the interface IP (if the service is running on pfSense itself), enable logging and state a unique description.
Custom rules should override automatic rule and should hence be applied.

Try to connect from remote and check in the firewall log if the rule was applied.

Reply-to only affects to replies. If pfSense itself initiates a connection it will follow the routing table.
So if you want to use a certain gateway you will have to set the routes respectively.
However, if I remember correctly, it's possible to policy route traffic with a floating rule for direction 'out'.

mik256

@viragomann said in Routing a service to non-default WAN:

Add a rule to WAN2 allowing IPSec traffic to the interface IP (if the service is running on pfSense itself), enable logging and state a unique description.

Sounds like what I did first before started with policy routing, NAT etc. to get it working.

Meanwhile, I really don't know what did the trick, but it started working somehow as expected.
Such a relief after so many hours:)
Thank you so much guys!!

stephenw10

So you added a rule back to pass the IPSec traffic on PE3 overriding the auto-rules?

mik256

@stephenw10

My setup now is this: no policy rules, no static rules, no manual NAT rules. Default GW is WAN1, ipsec listens on WAN2. I have manually opened udp/500 and udp/4500 on WAN2 (no gw set). I think, that was my setup in the first shot. When I tried to disable these 2 rules, it stopped working even though I can't see any denied packets in firewall log. When I added the static routing to the remote ip (still without rules to pass udp/500 and udp/4500 on WAN2), it works.

So it seems that to get pfsense work correctly as initiator on non-default wan you either need to:
set a static routing rule to remote endpoint with the non-default wan
or
set udp/500 and udp/4500 permit rules on the non-default wan
otherwise pfsense sends replies with the default wan instead.

stephenw10

When pfSense is the initiator it should be the route-to rule that sends traffic via the WAN2 gateway when it's sourced from the WAN2 IP.

When the other side initiates reply-to should apply.

However the auto rules do not appear to add reply-to. Which is interesting.

mik256

@stephenw10 said in Routing a service to non-default WAN:

When the other side initiates reply-to should apply.

However the auto rules do not appear to add reply-to. Which is interesting.

Pfsense is the responder in my case. Yes, it confused me a lot :) I was about to introduce some wild workarounds. So glad we got it working! Thanks a lot to both of you guys.