Access Internet Through VPC Using IPSec VPN Tunnel

tman222 · Oct 11, 2018, 1:14 AM

Hi all,

I'm trying to configure hosts on one my LAN subnets (that is connected via an IPSec tunnel to a Google Cloud VPC) how to route their internet bound traffic through the tunnel to an instance on the other side (in the VPC network) that is setup to be NAT Gateway. In other words, I would like to set things up so that it looks like that my internet traffic is coming from the external IP of the NAT Gateway on the other side as opposed my local internet (WAN) connection. Essentially I'm more or less trying to do what is illustrated here:

https://www.netgate.com/docs/pfsense/vpn/ipsec/routing-internet-traffic-through-a-site-to-site-ipsec-vpn.html

However, instead of having a pfSense box on the other side to work with I have a Google Cloud VPC and NAT Gateway. Is it possible to route traffic this way so that boxes on the local subnet pass traffic through the IPSec tunnel to the NAT Gateway where it gets routed further into the internet?

If so, how would I set it up? Here is some configuration info:

Local and Remote Networks:
Local Subnet: 192.168.40.1/24
Remote NAT Gateway: 10.85.5.5

Dynamic Routing via BGP:
Local: 169.254.40.2
Remote: 169.254.40.1

I currently have a working IPSec tunnel and can pass traffic between 192.168.40.1/24 and 10.85.5.0/20, and a also have working BGP session between 169.254.40.1 and 169.254.40.2. However, I"m stuck trying to figure out how to route internet bound traffic to 10.85.5.5.

Thanks in any help you can provide.

tman222 · Oct 11, 2018, 1:20 AM

Well, I"m still pretty much stuck on this. After doing some more thinking an experimenting, would any of the following put me the right direction:

Would it make sense to have the remote BGP peer (Google Cloud) advertise the 0.0.0.0/0 net to my pfSense as well besides just the 10.85.5.0/20 subnet? In other words, the entry would be 0.0.0.0/0 with next hop being 169.254.40.1.
Do I need to setup any virtual IP's or Outbound NAT Rules?
How would I need to adjust my Phase 2 IPSec rules? Do I just change the remote network to 0.0.0.0/0 to route all traffic through the IPSec tunnel?
Is what I"m trying to accomplish difficult to do with dynamic routing and would be more straightforward with routed IPSec (VTI)?

Thank again for all your help, I really appreciate it.

jimp · Oct 11, 2018, 12:48 PM

You'll need VTI for any of that to work properly with Internet traffic and BGP.

Then you just need to figure out how to nudge the Google side to send you a default route via BGP.

tman222 · Oct 11, 2018, 2:03 PM

Hi @jimp - thanks so much for responding.

So I am one step closer (I think). I was able to setup VTI interface for pfSense and for the transit network I used the BGP addresses, i.e. 169.254.40.1 and 169.254.40.2. I can now pass traffic between the remote VPC and my local subnets, but I still don't have a way to access the internet through the VPC. I can create a custom BGP route for the Google Cloud router to advertise, i.e. something like this:

Destination: 0.0.0.0/0 Next Hop: 169.254.40.1

When I do that, however, it's unfortunately not sufficient and I'm not able to reach any external hosts from my local subnet. Maybe the issue is that the traffic form my local net (192.168.40.1/24) is not being routed/forwarded on Google's end?

From what I understand there are two options to access the internet from the VPC:

I can create a cloud gateway which will be become the default NAT Gateway for all VPC VM instances that only have local IP addresses.
I can create a new instance that will act as a NAT Gateway itself for all VPC instances that do not have external access (only have an internal IP).

So here is now where I"m stuck:

If I create the cloud gateway I think the traffic arriving from my local subnet needs to arrive at the VPC looking like it came form the VPC subnet, e.g. traffic coming from 10.85.5.5/20 so it can be further forwarded. Could I accomplish this with and outbound NAT Rule?
If I use an instance as a NAT Gateway, how would I setup a rule/route/gateway so that internet bound traffic from my local subnet reaches that NAT Gateway instance, which will then forward for it?

Thanks in advance for all your help, I really appreciate it.

jimp · Oct 11, 2018, 2:56 PM

You can check with packet captures to see if the traffic is going the way you expect. The state table can offer some clues as well. It's possible that you are sending the traffic to the far side and then it dies there. Maybe it doesn't know on the Google end to do outbound NAT for you.

tman222 · Oct 11, 2018, 5:10 PM

Hi @jimp - you are right the traffic does reach the other side, but then doesn't go any further.

Is there anything I can do my side to make sure the traffic reaches the NAT Gateway instance on the other side (GCP)? Or does, it all hinge on being able to create a rule on the other side (GCP) on what to do once the traffic leaves the funnel and arrives on the GCP side?

Thanks again.

jimp · Oct 11, 2018, 5:16 PM

You are delivering the traffic to the other end, beyond that it's up to the other end to route it where it needs to go. Once you've handed it off down that pipe you can't do anything else to influence its routing.

In normal cases I might suggest doing NAT to the endpoint address but since it's APIPA/Link-Local that would hurt more than it would help.

tman222 · Oct 11, 2018, 6:08 PM

Thanks @jimp .

Well, bummer. I don't think this is going to be possible since I don't really have any way to edit the routing parameters for the VPN gateway (Cloud VPN) on the other side.

I suppose to make everything more straightforward, I could just install pfSense on a GCP compute instance and go from there. I saw this guide out on the net, but is there an official installation available as well on how-to available for Google Cloud?

https://blog.kylemanna.com/cloud/pfsense-on-google-cloud/

Thanks again.