SG-3100 - routing all internet access over IPSEC tunnel

[stephenw10]

Ah nice!

[ccb056]

Both my laptop and my phone can ping google.com through the ipsec tunnel, packet captures below

pings from phone.zip

pings from laptop.zip

Here is the traceroute from my laptop without going through the tunnel:

Here is the traceroute from my laptop going through the tunnel, but using local dns:

And this is what it looks like going through the tunnel and using remote dns:

Here are some photos of the configs on both routers:

Note, the browser on the laptop seems to load the google.com webpage, but the phone cannot browse to any webpage

What do I need to change to get this working properly?

[Derelict]

None of that really gives us anything to go on.

How are you selecting traffic to go over the tunnel? Policy routing or the routing table? How is that set up?

How are the two devices configured for both IP routing and DNS?

Not sure why you're showing us traceroutes and mentioning DNS. Use DNS tools to troubleshoot DNS and traceroute to troubleshoot routing.

It looks like you are routing through the tunnel and out the charter WAN so everything including outbound NAT there is working.

[Derelict]

@ccb056 No idea why you're making it so complicated. Just policy route everything from hosts_tunneled out the VTI interface, get that working, then worry about whatever it is you're trying to do with DNS.

[ccb056]

I am selecting traffic to go over the tunnel using this LAN rule on the local router:

Here are the routing tables on the local and remote routers:

Not everything is working.
The laptop is very spotty when accessing webpages, it can access google right now, but not yahoo, msn, etc.

The phone cannot access anything.

[stephenw10]

So you have three sites connected all connected together there? A quick diagram would be very useful.

Could there be some route asymmetry here?
Do you see blocked TCP traffic in the firewall logs at either site when you try to open a failing page?

There is no reply-to added to traffic from VTI interfaces so the system routing is always used.

Steve

[ccb056]

Here is a diagram of the topology:

And here is a youtube video showing what I'm experiencing:
https://www.youtube.com/watch?v=pUI7dhAYaYM

[stephenw10]

Ok, do you see any blocked TCP traffic in the logs at either site though?

The video stream is probably UDP and pings which you said were working are ICMP so...

Steve

[ccb056]

Here's a packet capture log from the interfaces on both the local and remote routers during an sftp transfer (tcp/22)

sftp file transfer packet capture.zip

[stephenw10]

Hmm, so some of those seem OK, the local VTI cap for example.

How exactly were those taken? Not all at the same time I assume? And did the transfer fail every time?

Steve

[ccb056]

One large download to the laptop was started over sftp.

Then the packet captures were taken sequentially, first on the local router, and then on the remote router, during the single file transfer.

After the packet captures were taken, the transfer was manually stopped.

What I found interesting was there was traffic on the wan side of the local router that appeared to be going directly to the sftp server. I expected this to all be inside the IPSEC tunnel.

[Derelict]

What I found interesting was there was traffic on the wan side of the local router that appeared to be going directly to the sftp server. I expected this to all be inside the IPSEC tunnel.

If that is the case (I have not looked at the captures) you are not routing/policy routing the traffic into the IPsec correctly.

[gabacho4]

@Derelict I'm having this very issue with two SG-5100s using routed IPSEC. Would you be able to explain your solution a little bit more so that I can understand what I need to do on my end.

// I worked with a guy doing exactly this. He first tried this:
LAN <-> VTI (OB NAT) <-> VTI <-> WAN (OB NAT) <-> Internet
He changed it to this because of this limitation on NAT on the VTI interface:
LAN <-> VTI <-> VTI <-> WAN (OB NAT) <-> Internet
Setting the Outbound NAT on the WAN to NAT the LAN source addresses.
All works fine. I think he said he's getting 500Mbps or more over it.
You just have to make sure the WAN side has a gateway and a route back to the LAN network over the VTI.//

I'm self taught and could use a slight hand holding on this one. Was super disappointed to encounter the issue using the routed IPSEC option but your info suggests current limitations can be overcome until the fix is made in FreeBSD. Appreciate any help you might be willing to provide.

[stephenw10]

What exactly do you have configured now? What works? What doesn't work?

The VTI interfaces have some limitations compared to other interface types, there is no reply-to feature due to where firewall rules are applied. Some NAT also cannot work because of that.
As long as you avoid those it should work.

Steve

[gabacho4]

@stephenw10 appreciate the response. This thread (see link below) sums up my issues and Derelict later responded referring me back to this thread. I’m just trying to understand the solution he provides. Just looking for some explanation of the steps he outlines.

Original thread I started:

https://forum.netgate.com/topic/141613/can-i-route-internet-traffic-from-site-b-through-site-a-via-ipsec-vti

[Derelict]

Does this help?

ETA: better resolution and firewall rule box.

[gabacho4]

@Derelict Most certainly. That's exactly how I have mine set up but am very glad to know this is how people smarter than I would do things. The biggest issue I suffer from is the 2xx ms latency due to distance between endpoints. Really do appreciate the extra help on this. Time is valuable and you let me have a little of yours.

[Derelict]

@ngoehring123 Yeah. Can't help you with the latency. Glad it helped.