Routing a service to non-default WAN
-
@stephenw10 said in Routing a service to non-default WAN:
Just to be clear PE3 is the second WAN interface here right?
Is the subnet conflicting with the primary WAN? Or gateway?
Does other traffic work correctly from PE3? Can you ping out from it for example?
Yes, PE3 is the second WAN. I can ping from it, see traffic comming from its public ip on remote endpoints (if I create static rule). There's no coflict in WANs networks - WAN1 has public ip and is directly connected to the isp's router, PE3 (the second WAN with 192.168.40.3) is behind another router (192.168.40.1) with NAT.
I have made this test:
- disabled ipsec in pfsense
- run socat on pfsense:
[2.7.2-RELEASE][admin@pfSense-aav1]/root: /usr/local/bin/socat -v UDP-LISTEN:500,bind=192.168.40.3 UDP-SENDTO:VPS-PUBLIC-IP:500 aa- run netcat on VPS:
nc -u PE3-PUBLIC-IP 500- run tcpdump on the VPS:
root@vmi1032933:~# tcpdump -i eth0 -nn udp port 500 listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 09:18:05.726660 IP VPS-PUBLIC-IP.53642 > PE3-PUBLIC-IP.500: [|isakmp] 09:18:05.753419 IP WAN1-PUBLIC-IP.1599 > VPS-PUBLIC-IP.500: [|isakmp]Then I tried again with policy rules (with PE3GW), again same result. The VPS sends packet to WAN2 and pfsense replies with WAN1 instead.
Am I doing something wrong? What could be breaking the reply-to functionality?
Thank you so much! -
@mik256 said in Routing a service to non-default WAN:
root@vmi1032933:~# tcpdump -i eth0 -nn udp port 500
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:18:05.726660 IP VPS-PUBLIC-IP.53642 > PE3-PUBLIC-IP.500: [|isakmp]
09:18:05.753419 IP WAN1-PUBLIC-IP.1599 > VPS-PUBLIC-IP.500: [|isakmp]The second packet doesn't seem to be a response to the first one, but an additional connection.
On which interface of pfSense to you see this?
I'm not expect to see this on PE3 / WAN2. Otherwise you've probably configured the outbound NAT wrongly.And also this controverts your statement above:
pfsense replies with correct IP but on wrong interface :(
Now what? Wrong IP or wrong interface?
-
@viragomann said in Routing a service to non-default WAN:
The second packet doesn't seem to be a response to the first one, but an additional connection.
On which interface of pfSense to you see this?
I'm not expect to see this on PE3 / WAN2. Otherwise you've probably configured the outbound NAT wrongly.You are correct. There's a different port number being used. Maybe that's how socat works. I think ipsec didn't behave like this, but I will find out.
And also this controverts your statement above:
pfsense replies with correct IP but on wrong interface :(
Now what? Wrong IP or wrong interface?
It's still the same, sorry for the confusion. On pfsense tcpdump shows WAN2 IP is used for reply, but using tcpdump on WAN1 interface. If I do packet capture on WAN2, I don't see any reply being send at all.
I tried with http (pfsense web ui) and there reply-to worked fine: wget from a host to WAN2 received html from WAN2 public ip. Maybe reply-to fails just for udp?
-
@mik256 said in Routing a service to non-default WAN:
Maybe that's how socat works. I think ipsec didn't behave like this, but I will find out.
I'd expect that a device sends a response to the source IP and port it found in the request packet.
So this capture doesn't seem to be suitable to troubleshoot this for me.Maybe reply-to fails just for udp?
Just make a test with a custom rule for IPSec.
Add a rule to WAN2 allowing IPSec traffic to the interface IP (if the service is running on pfSense itself), enable logging and state a unique description.
Custom rules should override automatic rule and should hence be applied.Try to connect from remote and check in the firewall log if the rule was applied.
Reply-to only affects to replies. If pfSense itself initiates a connection it will follow the routing table.
So if you want to use a certain gateway you will have to set the routes respectively.
However, if I remember correctly, it's possible to policy route traffic with a floating rule for direction 'out'. -
@viragomann said in Routing a service to non-default WAN:
Add a rule to WAN2 allowing IPSec traffic to the interface IP (if the service is running on pfSense itself), enable logging and state a unique description.
Sounds like what I did first before started with policy routing, NAT etc. to get it working.
Meanwhile, I really don't know what did the trick, but it started working somehow as expected.
Such a relief after so many hours:)
Thank you so much guys!! -
So you added a rule back to pass the IPSec traffic on PE3 overriding the auto-rules?
-
My setup now is this: no policy rules, no static rules, no manual NAT rules. Default GW is WAN1, ipsec listens on WAN2. I have manually opened udp/500 and udp/4500 on WAN2 (no gw set). I think, that was my setup in the first shot. When I tried to disable these 2 rules, it stopped working even though I can't see any denied packets in firewall log. When I added the static routing to the remote ip (still without rules to pass udp/500 and udp/4500 on WAN2), it works.
So it seems that to get pfsense work correctly as initiator on non-default wan you either need to:
set a static routing rule to remote endpoint with the non-default wan
or
set udp/500 and udp/4500 permit rules on the non-default wan
otherwise pfsense sends replies with the default wan instead. -
When pfSense is the initiator it should be the route-to rule that sends traffic via the WAN2 gateway when it's sourced from the WAN2 IP.
When the other side initiates reply-to should apply.
However the auto rules do not appear to add reply-to. Which is interesting.
-
@stephenw10 said in Routing a service to non-default WAN:
When the other side initiates reply-to should apply.
However the auto rules do not appear to add reply-to. Which is interesting.
Pfsense is the responder in my case. Yes, it confused me a lot :) I was about to introduce some wild workarounds. So glad we got it working! Thanks a lot to both of you guys.
-
@mik256 lol, again same issue:) this time with openvpn. Tried all common tricks: policy routing, nat, playing with firewall rules with and without specifying gateway, but openvpn just keeps replying on default wan. Why the hell i just can't make it reply on the interface the request came from:( btw can't really see what policy routing (setting gw in fw rule) could be good for, when the routing takes place before that.
-
So for remote clients connecting to pfSense?
The same thing applies, reply-to should normally pass replies back out of the same WAN as long as the rule passing it in one the specific WAN interface.
However openvpn has some additional setup options for multiwan that can be used instead. So how is it configured?
-
@stephenw10
Yes, remote clients connecting to pfsense. OpenVPN server is listening on LAN (only this interface, there is a port forwarding from another firewall). Tried TCP/UDP. Tcpdump shows requests from clients arrive on hn0 (LAN), but replies are sent from hn1 (WAN), while they need to send back to the firewall.Besides that, I didn't find any additional setup options which might be relevant in openvpn server settings, which do you mean?
Thanks
-
@stephenw10
I found how to deal with that: instead of using LAN for openvpn server, I used a new VLAN interface which has default gateway set to the port forwarding firewall. Nothing else needed to be changed.Seems like having a gatway on interface pfsense treats this interface differently a correctly sends replies.
-
@stephenw10 said in Routing a service to non-default WAN:
Is the gateway actually defined on the interface? That's what configures it as a WAN with reply-to and route-to on rules.
Yup, exactly.
Reply-to only applies to WAN type interfaces. -
@stephenw10
cool! my bad I didn't read this carefully. thanks -
No worries. I remember hitting that issue myself when I first setup pfSense with multiwan. Too long ago to mention!
Until you realise how pfSense determines what is a WAN interface and what that triggers it can easily seem like magic.
