SG-3100 - routing all internet access over IPSEC tunnel

[ccb056]

Is there a bug in redmine for this?

[Derelict]

Has to be fixed upstream in freebsd.

[stephenw10]

If you're configured as expected though the outbound NAT is on the SG-3100 WAN not actually on the VTI interface. But I have a feeling this also hits it somehow. Try it and see.

You can run a packet capture from the webgui in Diag > Packet Capture. Filter by protocol ICMP and whatever the external IP is you're pinging.

Steve

[Derelict]

This post is deleted!

[ccb056]

Is there a FreeBSD bug report?

I can't seem to find one:
https://bugs.freebsd.org/bugzilla/buglist.cgi?order=Importance&query_format=advanced&short_desc=ipsec%20nat&short_desc_type=allwordssubstr

I did find this pfsense bug:
https://redmine.pfsense.org/issues/9230

And this opnsense thread:
https://github.com/opnsense/core/issues/440

[stephenw10]

That pfSense bug report is probably the closest thing. That is NATing over a VTI interface.

The Opnsense bug is refering to policy based IPSec which also has NAT limitations but is when you're hitting here specifically.

Were you able to run that test and get the pcaps?

Steve

[Derelict]

I worked with a guy doing exactly this. He first tried this:

LAN <-> VTI (OB NAT) <-> VTI <-> WAN (OB NAT) <-> Internet

He changed it to this because of this limitation on NAT on the VTI interface:

LAN <-> VTI <-> VTI <-> WAN (OB NAT) <-> Internet

Setting the Outbound NAT on the WAN to NAT the LAN source addresses.

All works fine. I think he said he's getting 500Mbps or more over it.

You just have to make sure the WAN side has a gateway and a route back to the LAN network over the VTI.

I believe he used policy routing to send the internet traffic from certain local hosts over the VTI, not a 0.0.0.0/0 route. Both methods should work, but if I was going to try the route method I would probably adopt the OpenVPN strategy and add these instead of 0.0.0.0/0: 0.0.0.0/1 128.0.0.0/1 so your existing default route stays unmolested.

[stephenw10]

Ah nice!

[ccb056]

Both my laptop and my phone can ping google.com through the ipsec tunnel, packet captures below

pings from phone.zip

pings from laptop.zip

Here is the traceroute from my laptop without going through the tunnel:

Here is the traceroute from my laptop going through the tunnel, but using local dns:

And this is what it looks like going through the tunnel and using remote dns:

Here are some photos of the configs on both routers:

Note, the browser on the laptop seems to load the google.com webpage, but the phone cannot browse to any webpage

What do I need to change to get this working properly?

[Derelict]

None of that really gives us anything to go on.

How are you selecting traffic to go over the tunnel? Policy routing or the routing table? How is that set up?

How are the two devices configured for both IP routing and DNS?

Not sure why you're showing us traceroutes and mentioning DNS. Use DNS tools to troubleshoot DNS and traceroute to troubleshoot routing.

It looks like you are routing through the tunnel and out the charter WAN so everything including outbound NAT there is working.

[Derelict]

@ccb056 No idea why you're making it so complicated. Just policy route everything from hosts_tunneled out the VTI interface, get that working, then worry about whatever it is you're trying to do with DNS.

[ccb056]

I am selecting traffic to go over the tunnel using this LAN rule on the local router:

Here are the routing tables on the local and remote routers:

Not everything is working.
The laptop is very spotty when accessing webpages, it can access google right now, but not yahoo, msn, etc.

The phone cannot access anything.

[stephenw10]

So you have three sites connected all connected together there? A quick diagram would be very useful.

Could there be some route asymmetry here?
Do you see blocked TCP traffic in the firewall logs at either site when you try to open a failing page?

There is no reply-to added to traffic from VTI interfaces so the system routing is always used.

Steve

[ccb056]

Here is a diagram of the topology:

And here is a youtube video showing what I'm experiencing:
https://www.youtube.com/watch?v=pUI7dhAYaYM

[stephenw10]

Ok, do you see any blocked TCP traffic in the logs at either site though?

The video stream is probably UDP and pings which you said were working are ICMP so...

Steve

[ccb056]

Here's a packet capture log from the interfaces on both the local and remote routers during an sftp transfer (tcp/22)

sftp file transfer packet capture.zip

[stephenw10]

Hmm, so some of those seem OK, the local VTI cap for example.

How exactly were those taken? Not all at the same time I assume? And did the transfer fail every time?

Steve

[ccb056]

One large download to the laptop was started over sftp.

Then the packet captures were taken sequentially, first on the local router, and then on the remote router, during the single file transfer.

After the packet captures were taken, the transfer was manually stopped.

What I found interesting was there was traffic on the wan side of the local router that appeared to be going directly to the sftp server. I expected this to all be inside the IPSEC tunnel.

[Derelict]

What I found interesting was there was traffic on the wan side of the local router that appeared to be going directly to the sftp server. I expected this to all be inside the IPSEC tunnel.

If that is the case (I have not looked at the captures) you are not routing/policy routing the traffic into the IPsec correctly.

[gabacho4]

@Derelict I'm having this very issue with two SG-5100s using routed IPSEC. Would you be able to explain your solution a little bit more so that I can understand what I need to do on my end.

// I worked with a guy doing exactly this. He first tried this:
LAN <-> VTI (OB NAT) <-> VTI <-> WAN (OB NAT) <-> Internet
He changed it to this because of this limitation on NAT on the VTI interface:
LAN <-> VTI <-> VTI <-> WAN (OB NAT) <-> Internet
Setting the Outbound NAT on the WAN to NAT the LAN source addresses.
All works fine. I think he said he's getting 500Mbps or more over it.
You just have to make sure the WAN side has a gateway and a route back to the LAN network over the VTI.//

I'm self taught and could use a slight hand holding on this one. Was super disappointed to encounter the issue using the routed IPSEC option but your info suggests current limitations can be overcome until the fix is made in FreeBSD. Appreciate any help you might be willing to provide.