Amazon Virtual Private Cloud (VPC) VPN

[ieee1394]

This very topic has come up at least a couple of times in this forum over the past year or so but there's still no definitive guide on how to use Pfsense 2.x to setup a hardware based VPN connection to an Amazon Virtual Private Cloud. The required configuration is particularly problematic because the AWS VPC requires what appears to be a bridge across the IPsec tunnel. The PFsense then needs to connect to a BGP neighbor across that "bridged" tunnel to receive route updates for public/private cloud networks configured in the VPC. Making this even more challenging (!) is the fact that the architects specify that TWO IPsec tunnels should be setup for redundancy (the endpoints are different for each tunnel, the bridge network is different, and the BGP neighbor addresses are different).

The following is an oft-cited example of how to set things up but I have yet to make that configuration work, and other users report the same on the AWS forums (or basically just give up in silence):

http://seattleit.net/blog/pfSense-IPSec-VPN-Gateway-with-Amazon-Virtual-Private-Cloud-BGP-Routing

another thread here:

http://forum.pfsense.org/index.php/topic,35331.0.html

The challenge spans across all areas here:

1. Typically the networks at each end of the tunnel are different and non-overlapping. For instance, routing traffic from our local LAN (192.168.2.0/24) over IPsec to our datacenter LAN (192.168.9.0/24) works without a hitch. AWS specifies the required VPC VPN tunnel like so:

169.254.255.1/30 --> 169.254.255.2/30

The following is a link to Amazon's Network Admin documentation:

http://docs.amazonwebservices.com/AmazonVPC/latest/NetworkAdminGuide/

An important point from the documentation with regard to the 169.254.x.x/30 network:

Bind tunnel to logical interface (route-based VPN) – Your gateway must support the ability to bind the IPsec tunnel to a logical interface. The logical interface contains an IP address used to establish BGP peering to the virtual private gateway. This logical interface should perform no additional encapsulation (e.g., GRE, IP in IP).

2. From what I've read, the PFsense can't send it's own (locally generated) traffic across an IPsec tunnel. Kind of problematic since Pfsense needs to run OpenBGP and connect to its neighbor across the tunnel. The solution to this problem is noted in the seattle.it link above: bind the local 169.254.255.2 address to the WAN interface as a Virtual IP (an "alias"), then create a static route that associates all traffic across 169.254.255.0/30 should be sent to the WAN interface. But doesn't this create a new problem because now we're saying that all traffic for that network is local to the WAN interface instead of routing traffic to the BGP neighbor at 169.254.255.1 across the tunnel itself???

3. It's not clear whether the Pfsense can handle redundant tunnels as described. I seem to have no problem establishing both IPsec tunnels (I don't see what's weird about that part since the endpoints are unique for each tunnel) but I have yet to find any example where someone actually considers setting up both tunnels on a PFsense device. The tunnels both come up for me regardless if I'm using an IP address or FQDN to identify the local endpoint. NOTE: There's a lot of chatter in the AWS forums about IPsec indicated as "UP" on the local end but marked as "DOWN" on the AWS VPC end, I believe this is related to BGP not being able to route and that AWS marks the connection status based on the presence of successful BGP traffic.

4. Then there's the part about configuring BGP to listen on multiple interfaces. Easily accomplished by bypassing the GUI and editing the raw config I guess.

I think for the sake of clarity it is best to just skip the notion of supporting a redundant tunnel for now. That leaves the question of figuring out how to configure just one tunnel (Phase 1, Phase 2), Virtual IPs, static routes, BGP, firewall rules if needed.

The seattleit example raises some questions. First of all, it appears that the user assigned an additional IP address to the WAN interface instead of just using the existing IP address. He then created a gateway for that additional IP, and then referred to that gateway in the static route. If this is true, why can't the original IP on the WAN interface be used? Also, the part that's left out is whether or not the additional IP is then referenced as the Customer Gateway IP in the AWS VPC configuration or if it's just used locally for internal routing purposes.

Then what about the firewall rules. Doesn't IPsec on Pfsense automatically create internal firewall rules to pass traffic?

It would be great to get a working configuration documented in its entirety. Not sure how seattleit got their config to work, perhaps there are some additional sysctl vars or other configuration params that had been adjusted as well but weren't documented. Perhaps the fact that I already have another IPsec tunnel configured (and working happily connected to a Cisco PIX at the far end). I've spent many days trying to get a working config up and running but have had no success. It's clearly a routing issue related to the 169.254.x.x network since the tunnel appears to come up fine but BGP fails to connect, and I can't ping the remote BGP neighbor (yes, even if specifying the source address on the pfsense itself…presumably I'd add another Phase 2 entry to support pinging from the local 192.168.x.x network).

And there you have it. Hopefully one of the pfsense VPN gurus can shed some light on this?

[Serjaydee]

Yeouch! You guys aren't the only who is having this problem when using VPN services, I've also read in some forums about this and only some of them are also able to give solution to it. I would just like to share though, there's this new VPN and it's considered as the Best VPN for China that you might want to use for accessing inaccessible websites too.

[ieee1394]

Spam?

I came across a good blog post that summarizes the configuration required (in general, not for pfsense):

http://blog.akquinet.de/2011/11/11/connecting-to-amazon-vpc/

And provides a handy diagram:

I still see the challenge for pfsense arising from the need to configure a bridged network (in the example: 169.254.254.0/30 and 169.254.254.4/30) over the tunnel. As an example, the local BGP daemon needs to listen on 169.254.254.1 and the remote (BGP neighbor) on 169.254.254.2. The two problems are: 1) I've read that pfsense can't support routing traffic over an IPSec tunnel unless the local and remote networks are different; 2) pfsense can't route it's own local traffic (in this case, traffic from its own BGP server) over an IPSec tunnel for which it acts as the local endpoint.

By the number of views to this thread (over 230 to date) I can see this is a popular problem in search of a working solution. I originated the thread on the AWS forum (https://forums.aws.amazon.com/thread.jspa?threadID=79263&tstart=25) but like here have failed to garner any responses.

[davidfine]

I have figured this out. The seattleit guide comes close but there are a couple gotchas. I'll document this soon, feel free to contact me if you want to chat about it.

[abonstu]

hangin out for your documented amazon<–> pfsense guide dude!

[davidfine]

Attached are screenshots of a working Amazon VPC configuration in pfSense. Note, we aren't able to create a second Phase2 connection to our VPC subnet, or the tunnel won't come up. But it works fine with one entry, and since the tunnels both go over the same WAN connection I don't think we loose any redundancy. Also, you'll need to change the MTU on your servers within the VPC or else you'll have a very slow connection and dropped packets. I believe the correct value was 1432, could have been 1396. A little bit too low is always better than too high, you can install the iperf package to verify you've got it right.
This config took a weekend to figure out, I hope this saves someone the trouble!

[abonstu]

wow… thats awesome - thanks for taking the time to document those screens!

i've actually got two pfsense routers in different locations:

• the first has a /29 frame route resulting in a config practically identical to your example - this connection is working perfectly

• the second router only has a single public IP address and this is where i am tearing my hair out - i cant seem to get the VPN up - the Amazon tunnel status just says PHASE_2_SUCCESS

im sure its a routing issue but i cant see where im going wrong.

• the ipsec config is just using my WAN interface

• i shouldnt need a gateway or static route as my WAN is the only gateway i have, the tunnel subnet (169.*) should just go straight out the default gateway (right? :-/)

any ideas on what im missing here? naturally ive tried all sorts of things, including manually configuring a static route anyway but i cant seem to get connected.


[Bob_L]

Thanks to this thread I was able to get this up and running without an undue amount of frustration. A couple of observations now that it's running:

1. I didn't find any errors in the SeattleIT doco published last Spring; but, I did find that I was awash in dotted decimal addresses in this config.  Two tunnels - each with an external link and an inside link - each link with a local address and a VPC address - throw in the subnet of the VPC LAN and you have a lot of moving parts.  What worked for me is that I drew up a worksheet where I made a grid where one axis had each of these addresses and the other had: The address Amazon assigned when I ordered the VPN - the address SeattleIT used in their doco - where in pfSense it is defined (gateway, virtual ip, etc) - where in pfSense it is used (Phase 1 remote gateway, Phase 2 local subnet, OpenBGP neighbor, static route, etc).  Armed with that worksheet I stepped through the SeattleIT step-by-step and confirmed each step with the screen shots above and had good success.

2. I tried and tried to get BGP to distribute its routes into the routing table only to read in the pfSense book that IPSec doesn't use the routing table to decide what packets to forward to the tunnel. It uses the Phase 2 settings that are configured in the IPSec section. I'm disappointed that we have this nifty dynamic routing protocol, BGP, that we went through all this effort to get up and running and IPSec doesn't even use the routes that it produces.

[abonstu]

we were setting up two amazon vpn's (one for each office) and i could get it going at one of the locations but not the other. the only difference between the two sites was that one had a single static public address whereas the other had a /29 framed route - i couldnt get the single address endpoint to get past PHASE_2_CONNECT.

after two days i ended up throwing in the towel and setting up an EC2 OpenVPN server, had the entire thing sorted in 30 minutes.

perhaps im just ignorant but ipsec always seems like a massive unnecessary hassle to me. there are so many aspects to configure, it takes ages to connect and rarely are the performance benefits of a hardware VPN worth it (in my line of work anyway) - not to mention the savings of avoiding the amazon VPC ipsec tunnel costs.

give me a VPN higher up the network stack that uses a single port to connect and has identical configuration across all platforms. its really given me a new appreciation for the simplicity and power of openvpn.

[shan1_ansari]

@abonstu:

wow… thats awesome - thanks for taking the time to document those screens!

i've actually got two pfsense routers in different locations:

• the first has a /29 frame route resulting in a config practically identical to your example - this connection is working perfectly

• the second router only has a single public IP address and this is where i am tearing my hair out - i cant seem to get the VPN up - the Amazon tunnel status just says PHASE_2_SUCCESS

im sure its a routing issue but i cant see where im going wrong.

• the ipsec config is just using my WAN interface

• i shouldnt need a gateway or static route as my WAN is the only gateway i have, the tunnel subnet (169.*) should just go straight out the default gateway (right? :-/)

any ideas on what im missing here? naturally ive tried all sorts of things, including manually configuring a static route anyway but i cant seem to get connected.

That make two of us, I think this is the same Binding issue. I gave up now and recommended to terminated on Cisco Router.