AWS VPN BGP - Routing
-
@stev said in AWS VPN BGP - Routing:
Thanks for the suggestions.
- I have verified that the routes are propagating as they show up on both sides of the VPN.
- I believe everything is setup correctly in terms of the VPC subnets and associated ACLs. I worked with AWS support who also reviewed those settings.
Thanks for the video, I went through that section and tweaked one setting but it didn't seem to change anything.
Hi @stev -- hmmm, that's interesting. A couple more questions:
- Are you using VTI by chance? If not, it may be worth a try: https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/ipsec-routed.html
- Does the other side work, i.e. can you access (ping) your machines from AWS?
Hope this helps.
-
Yes, that seems like a good test. Ping out from an instance in AWS and (if the ping fails) run a packet capture in pfSense.
You should see those packets arriving on the IPSec interface even if pfSense then drops that.
If you are not using VTI you would just be sending between the two subnets directly with policy based IPSec, BGP doesn't come into it. How exactly is your IPSec configured?
Steve
-
I just looked back and found the thread that I started about setting all this up using GCP (instead of AWS):
https://forum.netgate.com/topic/136509/routed-ipsec-vti-and-google-cloud
Hopefully the information contained within can help you, perhaps in particular this link.
https://www.1strategy.com/blog/2017/08/29/tutorial-using-pfsense-as-a-vpn-to-your-vpc/
-
We have been working on this in support and currently believe it's a missing setting in AWS somewhere.
Steve
-
Did you find a fix for this?
I am experiencing identical behaviour to this.
v2.5.0
IPSec up, BGP peers up, routes exchanged.
route show for my AWS VPC gives next-hop of the local 169.x.
Can ping the remote 169.x peer ips using the 169.x source from the firewall but can not ping host in VPC when sourcing lan.
The traffic just seems to blackhole, no firewall logs, nothing received from AWS and nothing getting there.
I suppose a tcpdump is my next line of attack but it's becoming tiresome.
This is peculiar because using a static VPN configuration I can connect into the hosts just fine, so am aware this is NOT an AWS security rules issue.
I'm finding it frustrating because I want to connect the ipsec site to site with bgp to take advantage of AWS Transit Gateway's and its ECMP features (Not to mention having much fewer routes to configure in the cloud VPCs and locally).
-
Looks like you fixed this with VTI in your other thread?
Steve
-
@wstocker Unfortunately I never found a resolution to this issue. I ended up switching to a PA-220 which worked as expected.
-
@stev said in AWS VPN BGP - Routing:
P
Hey,
Yup VTI fixed this for me, although it’s unclear to me why this wouldn’t work with the configuration as described (that we both had tried)
-
BGP over policy based IPSec is just asking for trouble IMO. It's possible to make it work but you need P2s to cover all traffic being routed and the BGP traffic itself. Since AWS only allows 2 P2s on each P1 that's often a problem.
Steve
-
I got this working, but it took a solid 6 hours or so of tweaking to finally get things going. Mostly, for posterity...
First, it's a good idea to set up a static IPSec config, just to get all of the AWS bits working properly (RTB, SG, ACL, etc.). When that's working, remove the static route(s) in the RTB to the VGW. Make sure to set up the RTB to propagate routes. Kill the static VPN.
Then, create the new BGP VPN, and set up the two VTIs. A critical step is to set up both the VTI interfaces and gateways (it's easy to forget). For the gateways, leave the gateway IP blank, as you won't be able to enter the IP anyway. It will show as "dynamic" for a bit, and eventually will show the correct tunnel IP.
Finally, FRR BGP, which needs way more docs/examples to get working properly. Nowhere does anything mention having to define some form of route map, but without that, you won't get routes to set up - they'll just be discarded. (Admittedly, this was my first foray into BGP). Set up both neighbors (VTI tunnel IPs). It seems like the Route Map field should be required, or at least, have a tooltip of some sort - you need a policy defined that will allow the routes to be exchanged.
One symptom of failing to set up VTI interfaces/gateways is rejected BGP traffic - firewall rules won't help, but those messages should go away once the gateways are defined and up.
Once everything is working, you should see the two tunnels up in AWS, and show "X BGP ROUTES" (where X > 0). That may take a bit, as AWS doesn't update things very often, but you'll also see the routes show up in FRR BGP status.
Now you should be able to take down the primary VTI on pfSense and within a few secs, traffic will fail over to the other VTI, and recover after the downed VTI is back.
-
P.S. I take it back - you may need firewall rules for IPSec to allow BGP traffic. You can create them from the firewall logs if you see blocked BGP traffic on IPSec.
