Routing a service to non-default WAN
-
@viragomann
I think I don't have any of these now. I rid of policy rules completely, have no floating rules or interface groups. However I am not sure - the "IPv4 Upstream gateway" on the PE3 interface should be enough for reply-to WAN2 on WAN2GW? You talked about incoming firewall rules - do I need to set up some of these and if yes, could you give some example (like do I need to put anything in the advanced->gateway?).
Thank you very much, I spent so many hours trying to get the ipsec on secondary wan routing that I really don't want to give up setting up another pfsense with just the WAN2 interface to get it working lol. -
@mik256 said in Routing a service to non-default WAN:
But the issue persists - pfsense replies with correct IP but on wrong interface
The only reason for this, I can think of, is that both WAN gateways are on the same layer 2. Is this the case?
However I am not sure - the "IPv4 Upstream gateway" on the PE3 interface should be enough for reply-to WAN2 on WAN2GW?
Yes, it needs the gateway in the interface setting and a rule on the respective interface.
No floating or group rule must matching the incoming traffic, because this rules have precedence.You talked about incoming firewall rules - do I need to set up some of these
@stephenw10 said, IPSec access is allowed allowed automatically if an instance is enabled. I don't know.
But if you enable the logging of the default deny rule (in the log settings) and there is nothing to see regarding IPSec in the logs then, while packets are coming in, it might work without a user rule. -
Just to be clear PE3 is the second WAN interface here right?
Is the subnet conflicting with the primary WAN? Or gateway?
Does other traffic work correctly from PE3? Can you ping out from it for example?
-
The only reason for this, I can think of, is that both WAN gateways are on the same layer 2. Is this the case?
They are not on the same layer2:
WAN1 interface has public IP 2.3.4.5 assigned and is directly connected to the ISP router.
WAN2 interface has 192.168.40.3 and it is behind another router NAT.Yes, it needs the gateway in the interface setting and a rule on the respective interface.
Both interfaces have gateways assigned in their interface settings (1.2.3.4 for WAN1 and 192.168.40.1 for WAN2).
@stephenw10 said, IPSec access is allowed allowed automatically if an instance is enabled. I don't know.
But if you enable the logging of the default deny rule (in the log settings) and there is nothing to see regarding IPSec in the logs then, while packets are coming in, it might work without a user rule.I have tried disabling ipsec service and mimic with socat, no luck either.
Can you please think of something else breaking reply-to mechanism?
-
@stephenw10 said in Routing a service to non-default WAN:
Just to be clear PE3 is the second WAN interface here right?
Is the subnet conflicting with the primary WAN? Or gateway?
Does other traffic work correctly from PE3? Can you ping out from it for example?
Yes, PE3 is the second WAN. I can ping from it, see traffic comming from its public ip on remote endpoints (if I create static rule). There's no coflict in WANs networks - WAN1 has public ip and is directly connected to the isp's router, PE3 (the second WAN with 192.168.40.3) is behind another router (192.168.40.1) with NAT.
I have made this test:
- disabled ipsec in pfsense
- run socat on pfsense:
[2.7.2-RELEASE][admin@pfSense-aav1]/root: /usr/local/bin/socat -v UDP-LISTEN:500,bind=192.168.40.3 UDP-SENDTO:VPS-PUBLIC-IP:500 aa- run netcat on VPS:
nc -u PE3-PUBLIC-IP 500- run tcpdump on the VPS:
root@vmi1032933:~# tcpdump -i eth0 -nn udp port 500 listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 09:18:05.726660 IP VPS-PUBLIC-IP.53642 > PE3-PUBLIC-IP.500: [|isakmp] 09:18:05.753419 IP WAN1-PUBLIC-IP.1599 > VPS-PUBLIC-IP.500: [|isakmp]Then I tried again with policy rules (with PE3GW), again same result. The VPS sends packet to WAN2 and pfsense replies with WAN1 instead.
Am I doing something wrong? What could be breaking the reply-to functionality?
Thank you so much! -
@mik256 said in Routing a service to non-default WAN:
root@vmi1032933:~# tcpdump -i eth0 -nn udp port 500
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:18:05.726660 IP VPS-PUBLIC-IP.53642 > PE3-PUBLIC-IP.500: [|isakmp]
09:18:05.753419 IP WAN1-PUBLIC-IP.1599 > VPS-PUBLIC-IP.500: [|isakmp]The second packet doesn't seem to be a response to the first one, but an additional connection.
On which interface of pfSense to you see this?
I'm not expect to see this on PE3 / WAN2. Otherwise you've probably configured the outbound NAT wrongly.And also this controverts your statement above:
pfsense replies with correct IP but on wrong interface :(
Now what? Wrong IP or wrong interface?
-
@viragomann said in Routing a service to non-default WAN:
The second packet doesn't seem to be a response to the first one, but an additional connection.
On which interface of pfSense to you see this?
I'm not expect to see this on PE3 / WAN2. Otherwise you've probably configured the outbound NAT wrongly.You are correct. There's a different port number being used. Maybe that's how socat works. I think ipsec didn't behave like this, but I will find out.
And also this controverts your statement above:
pfsense replies with correct IP but on wrong interface :(
Now what? Wrong IP or wrong interface?
It's still the same, sorry for the confusion. On pfsense tcpdump shows WAN2 IP is used for reply, but using tcpdump on WAN1 interface. If I do packet capture on WAN2, I don't see any reply being send at all.
I tried with http (pfsense web ui) and there reply-to worked fine: wget from a host to WAN2 received html from WAN2 public ip. Maybe reply-to fails just for udp?
-
@mik256 said in Routing a service to non-default WAN:
Maybe that's how socat works. I think ipsec didn't behave like this, but I will find out.
I'd expect that a device sends a response to the source IP and port it found in the request packet.
So this capture doesn't seem to be suitable to troubleshoot this for me.Maybe reply-to fails just for udp?
Just make a test with a custom rule for IPSec.
Add a rule to WAN2 allowing IPSec traffic to the interface IP (if the service is running on pfSense itself), enable logging and state a unique description.
Custom rules should override automatic rule and should hence be applied.Try to connect from remote and check in the firewall log if the rule was applied.
Reply-to only affects to replies. If pfSense itself initiates a connection it will follow the routing table.
So if you want to use a certain gateway you will have to set the routes respectively.
However, if I remember correctly, it's possible to policy route traffic with a floating rule for direction 'out'. -
@viragomann said in Routing a service to non-default WAN:
Add a rule to WAN2 allowing IPSec traffic to the interface IP (if the service is running on pfSense itself), enable logging and state a unique description.
Sounds like what I did first before started with policy routing, NAT etc. to get it working.
Meanwhile, I really don't know what did the trick, but it started working somehow as expected.
Such a relief after so many hours:)
Thank you so much guys!! -
So you added a rule back to pass the IPSec traffic on PE3 overriding the auto-rules?
-
My setup now is this: no policy rules, no static rules, no manual NAT rules. Default GW is WAN1, ipsec listens on WAN2. I have manually opened udp/500 and udp/4500 on WAN2 (no gw set). I think, that was my setup in the first shot. When I tried to disable these 2 rules, it stopped working even though I can't see any denied packets in firewall log. When I added the static routing to the remote ip (still without rules to pass udp/500 and udp/4500 on WAN2), it works.
So it seems that to get pfsense work correctly as initiator on non-default wan you either need to:
set a static routing rule to remote endpoint with the non-default wan
or
set udp/500 and udp/4500 permit rules on the non-default wan
otherwise pfsense sends replies with the default wan instead. -
When pfSense is the initiator it should be the route-to rule that sends traffic via the WAN2 gateway when it's sourced from the WAN2 IP.
When the other side initiates reply-to should apply.
However the auto rules do not appear to add reply-to. Which is interesting.
-
@stephenw10 said in Routing a service to non-default WAN:
When the other side initiates reply-to should apply.
However the auto rules do not appear to add reply-to. Which is interesting.
Pfsense is the responder in my case. Yes, it confused me a lot :) I was about to introduce some wild workarounds. So glad we got it working! Thanks a lot to both of you guys.
