Traffic from internet through IPSEC VTI not returning the same way

[adrianesqq]

Hi,
I have problem with network traffic. Packets not returning the same way. Setup described below.

I have:

1. router (GW1 -pfsense 2.4.4) with one gateway.
2. vps server with centos (GW2)
   Both are connected with IPSEC vti.
   Green is incomming traffic and red is outgoing.
   
   Despite the firewall rule on Vlan5 interface packets not returning to gw2. I need to maintain source IP.
   Any suggestions?

[jimp]

pf reply-to doesn't work with VTI currently. Not something we can help. So to get traffic back from the server to the client, gw1 must have a route back to the client address over VTI.

[adrianesqq]

@jimp Thanks for reply.
I replaced VTI with ipsec tunnel ipv4. I put on top of this GRE. Doesn't reply-to work on GRE?

[jimp]

I believe so, but I haven't tested it in a while. Easy check in the firewall rules for the GRE interface.

[adrianesqq]

@jimp
It looks like rule on GRE interface is not returning also.
Rule below:
from * port * to Server1 port 443.

[Konstanti]

@adrianesqq Hey
As an option to use manual outbound NAT on the VTI interface, GW2 is still on port 443
(together with port forwarding on the wan interface)

source : ANY
interface: VTI
destination: server1
DST port: 443

[adrianesqq]

Hey, @konstanti
Packets from internet through ipsec will be translated on interface vti on GW1 and then forwarded to Server1. Then Server1 will return packets to vti interface. If it is that what you ment?
Sad thing is that with this approach, the source address will not be preserved. But it may work. I'll try this, a bird in the hand is worth two in the bush.

[Konstanti]

@adrianesqq
External ip->wan gw2:443 (port forwarding to server1:443) ->Internal IP (vti gw2 nat outbound):443->Server1:443->Internal IP (vti GW2) -> wan GW2 -> External IP

If the source address is important to you, then you can consider creating a virtual ip address on the server and forward ports to it . Then it will be possible on GW1 vlan5 interface to create a rule that all packets from this virtual ip will get to the VTI tunnel.

External ip->wan:443 (port forwarding to virtual ip server1:443)-> virtual ip server1:443 ->external ip-> PBR rule (vlan5 gw1 all traffic from the virtual ip goes through the VTI gateway) -> vti gw2->wan gw2 -> External IP

[adrianesqq]

@konstanti Approach with PBR is not working, as @jimp mentioned "this functionality doesn't work with VTI currently."
I will try with NAT on vti.

[Konstanti]

@adrianesqq
Not to be confused with the reply-to and route-to

route-to
The route-to option routes the packet to the specified interface
with an optional address for the next hop. When a route-to rule
creates state, only packets that pass in the same direction as the
filter rule specifies will be routed in this way. Packets passing
in the opposite direction (replies) are not affected and are routed

Reply-to
The reply-to option is similar to route-to, but routes packets that
pass in the opposite direction (replies) to the specified inter-
face. Opposite direction is only defined in the context of a state
entry, and reply-to is useful only in rules that create state. It
can be used on systems with multiple external connections to route
all outgoing packets of a connection through the interface the
incoming connection arrived through (symmetric routing enforce-
ment).
For example
192.168.1.96/32 - your virtual ip server1
443 - source port
tun200 - ip vti gw2 (next gateway, not by default)

[adrianesqq]

@konstanti Is this "Firewall rule" with advanced settings with specified gateway? I do have a rule like this from red box on picture (rule on vlan5 interface: from server1:443 to any:port any with gateway2), but somehow traffic is still directed to WAN (gw1).

How to create this rule which you're talking about? Did I forget about some setting? Could you point me to documentation about this?

[Konstanti]

@adrianesqq this rule should be the first in the list

Rulesets on the Interface tabs are evaluated on a first match basis by pfSense. This means that reading the ruleset for an interface from top to bottom, the first rule that matches will be the one used by the firewall. Evaluation stops after reaching this match and then the firewall takes the action specified by that rule. Always keep this in mind when creating new rules, especially when crafting rules to restrict traffic. The most permissive rules should be toward the bottom of the list, so that restrictions or exceptions can be made above them.
https://www.netgate.com/docs/pfsense/book/firewall/firewall-fundamentals.html

https://www.netgate.com/docs/pfsense/routing/directing-traffic-with-policy-routing.html

[adrianesqq]

@konstanti
This rule wasn't first and was with specified source port 443. It worked when packets originated from server1. And despite this rule it pushed packets originated from internet ware pushed back to WAN (gw1).
I will check this without port specified.
And if it will not work I will gather configuration in more details and will post this.

[Konstanti]

@adrianesqq show rules on vlan5 interface

[adrianesqq]

@konstanti
I will collect detailed info about config, firewall and post it.

[Konstanti]

@adrianesqq
I tested it myself.
It seems that this option will not work
It only works for outgoing connections
Jump was right
Sorry for the mistake!
Remains the only option with NAT OUTBOUND
This scheme works fine

[adrianesqq]

@konstanti
Thank you for help. Solution with nat is working.
The solution is:

1. Add SNAT on GW2
2. Add static routing on GW1 back to GW2

I also tried to replace VTI with GRE over ipsec tunnel.
Result is the same as in case of VTI.
Packets do not return to GRE interface.

@jimp Do you plan to add/repair return-to functionality? Can I post request somewhere?

[jimp]

@adrianesqq said in Traffic from internet through IPSEC VTI not returning the same way:

Do you plan to add/repair return-to functionality? Can I post request somewhere?

It's not up to us, it's broken in FreeBSD/pf

[e37921]

Has this been resolved in freebsd yet?

Thanks

[jimp]

No