Traffic from internet through IPSEC VTI not returning the same way

[adrianesqq]

@konstanti Approach with PBR is not working, as @jimp mentioned "this functionality doesn't work with VTI currently."
I will try with NAT on vti.

[Konstanti]

@adrianesqq
Not to be confused with the reply-to and route-to

route-to
The route-to option routes the packet to the specified interface
with an optional address for the next hop. When a route-to rule
creates state, only packets that pass in the same direction as the
filter rule specifies will be routed in this way. Packets passing
in the opposite direction (replies) are not affected and are routed

Reply-to
The reply-to option is similar to route-to, but routes packets that
pass in the opposite direction (replies) to the specified inter-
face. Opposite direction is only defined in the context of a state
entry, and reply-to is useful only in rules that create state. It
can be used on systems with multiple external connections to route
all outgoing packets of a connection through the interface the
incoming connection arrived through (symmetric routing enforce-
ment).
For example
192.168.1.96/32 - your virtual ip server1
443 - source port
tun200 - ip vti gw2 (next gateway, not by default)

[adrianesqq]

@konstanti Is this "Firewall rule" with advanced settings with specified gateway? I do have a rule like this from red box on picture (rule on vlan5 interface: from server1:443 to any:port any with gateway2), but somehow traffic is still directed to WAN (gw1).

How to create this rule which you're talking about? Did I forget about some setting? Could you point me to documentation about this?

[Konstanti]

@adrianesqq this rule should be the first in the list

Rulesets on the Interface tabs are evaluated on a first match basis by pfSense. This means that reading the ruleset for an interface from top to bottom, the first rule that matches will be the one used by the firewall. Evaluation stops after reaching this match and then the firewall takes the action specified by that rule. Always keep this in mind when creating new rules, especially when crafting rules to restrict traffic. The most permissive rules should be toward the bottom of the list, so that restrictions or exceptions can be made above them.
https://www.netgate.com/docs/pfsense/book/firewall/firewall-fundamentals.html

https://www.netgate.com/docs/pfsense/routing/directing-traffic-with-policy-routing.html

[adrianesqq]

@konstanti
This rule wasn't first and was with specified source port 443. It worked when packets originated from server1. And despite this rule it pushed packets originated from internet ware pushed back to WAN (gw1).
I will check this without port specified.
And if it will not work I will gather configuration in more details and will post this.

[Konstanti]

@adrianesqq show rules on vlan5 interface

[adrianesqq]

@konstanti
I will collect detailed info about config, firewall and post it.

[Konstanti]

@adrianesqq
I tested it myself.
It seems that this option will not work
It only works for outgoing connections
Jump was right
Sorry for the mistake!
Remains the only option with NAT OUTBOUND
This scheme works fine

[adrianesqq]

@konstanti
Thank you for help. Solution with nat is working.
The solution is:

1. Add SNAT on GW2
2. Add static routing on GW1 back to GW2

I also tried to replace VTI with GRE over ipsec tunnel.
Result is the same as in case of VTI.
Packets do not return to GRE interface.

@jimp Do you plan to add/repair return-to functionality? Can I post request somewhere?

[jimp]

@adrianesqq said in Traffic from internet through IPSEC VTI not returning the same way:

Do you plan to add/repair return-to functionality? Can I post request somewhere?

It's not up to us, it's broken in FreeBSD/pf

[e37921]

Has this been resolved in freebsd yet?

Thanks

[jimp]

No

[kevindd992002]

@Konstanti @adrianesqq

Can you help me out with the same issue I'm having here?

I basically do the same thing. I have a port forward rule and an outbound NAT rule on Site A (GW2 in your example) so that I can reach a server in Site B (GW1 in your example) through Site A's public IP and for the return traffic from the server to return to Site A's IPsec interface. Doing a packet capture in Site A's IPsec interface, I do see the return traffic making it until that point but its destination IP is somehow not translated back to the original external public source IP so I don't see any return traffic on Site A's WAN interface.

How did you get this working with outbound NAT? I'm scratching my head for a few weeks now as I'm not getting any help :(

[e37921]

@kevindd992002
The solution for me was to use OPENVPN. Netgate informed me that there are some limitations in the free bsd software that cause this issue. I deleted my IPsec vpn and then built the site to site vpn using open vpn in pfsense. All my policy based routing worked fine after the switch.

I am new to pfsense and was used to using IPsec vpn's with other firewalls. I had never used open vpn therefore I started with IPsec. Open vpn is very simple to setup and works well. I am using it for site to site vpn with pfsense on both sides as well as for mobile vpn.

Hope this helps you.

Eric

[kevindd992002]

@e37921 said in Traffic from internet through IPSEC VTI not returning the same way:

@kevindd992002
The solution for me was to use OPENVPN. Netgate informed me that there are some limitations in the free bsd software that cause this issue. I deleted my IPsec vpn and then built the site to site vpn using open vpn in pfsense. All my policy based routing worked fine after the switch.

I am new to pfsense and was used to using IPsec vpn's with other firewalls. I had never used open vpn therefore I started with IPsec. Open vpn is very simple to setup and works well. I am using it for site to site vpn with pfsense on both sides as well as for mobile vpn.

Hope this helps you.

Eric

Yeah, it's the other way around for me. Everything was working fine with OpenVPN but I needed to switch to IPsec because the bandwidth I'm getting with OpenVPN is limited by my hardware (APU2C4).

I consulted @jimp and confirmed that the workaround for this issues is this. The only caveat for setting that is it breaks policy-based IPsec tunnels, so if you use both route-based and policy-based then that would be a problem. As I'm using only route-based IPsec then I'm good. I'm testing now.