AWS VPN BGP - Routing

stev

I think it might be hopeless. I found a bit of FreeBSD code that drops packets in that range.

I don’t know why pinging the IP works, but routing through it apparently does not. I wish FreeBSD, pfSense and AWS were on the same page.

There were others who apparently got AWS VPN with BGP and pfSense to work but maybe they were using an older version?

https://raw.githubusercontent.com/freebsd/freebsd/master/sys/netinet/ip_input.c

/* RFC 3927 2.7: Do not forward datagrams for 169.254.0.0/16. */
if (IN_LINKLOCAL(ntohl(ip->ip_dst.s_addr))) {
IPSTAT_INC(ips_cantforward);
m_freem(m);
return;
}

Also a mail
https://lists.freebsd.org/pipermail/freebsd-stable/2010-June/057180.html

stephenw10

If you have set that value in the pfSense config it should work. There are many people using that, we had to put that in just for this.

However, when I try to ping my 10.2.0.0/16 AWS server from the router it never works.

Where are you trying to ping it from?

If you ping it from something that actually has the APIPA address it might not work as that's unroutable but as long as it's just using it as a transport network to route over I would expect it to.

Try setting the source IP to something other than APIPA if you're able to.

Steve

stev

Thanks for your reply!

The source IP is my workstation on the 10.4.0.0/16 LAN 10.4.0.123. The only APIPA addresses involved are the BGP peer addresses AWS required.

stev

I tried replacing FRR BGP with OpenBGP, but it didn't make a difference.

Here is a log of my attempts:

First I try pinging a server in AWS (it fails):

[2.4.4-RELEASE][me@router-1.whatever.com]/home/me: ping 10.2.1.52
PING 10.2.1.52 (10.2.1.52): 56 data bytes
^C
--- 10.2.1.52 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

I then check to see what the next hop would be, it is the BGP peer on AWS' side.

[2.4.4-RELEASE][me@router-1.whatever.com]/home/me: route show 10.2.1.52
   route to: 10.2.1.52
destination: 10.2.0.0
       mask: 255.255.0.0
    gateway: 169.254.47.141
        fib: 0
  interface: lagg0.4090
      flags: <UP,GATEWAY,DONE,PROTO1>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

I verify that I am able to ping that IP. It takes 30ms round trip which indicates to me it is in fact on the AWS side.

[2.4.4-RELEASE][me@router-1.whatever.com]/home/me: ping 169.254.47.141
PING 169.254.47.141 (169.254.47.141): 56 data bytes
64 bytes from 169.254.47.141: icmp_seq=0 ttl=254 time=28.924 ms
64 bytes from 169.254.47.141: icmp_seq=1 ttl=254 time=28.888 ms
64 bytes from 169.254.47.141: icmp_seq=2 ttl=254 time=28.770 ms
64 bytes from 169.254.47.141: icmp_seq=3 ttl=254 time=28.716 ms
64 bytes from 169.254.47.141: icmp_seq=4 ttl=254 time=28.774 ms
64 bytes from 169.254.47.141: icmp_seq=5 ttl=254 time=28.863 ms
^C
--- 169.254.47.141 ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 28.716/28.823/28.924/0.074 ms

I try doing a traceroute to our AWS server. It fails, not even the first hop responds which is really confusing to me.

[2.4.4-RELEASE][me@router-1.whatever.com]/home/me: traceroute 10.2.1.29
traceroute to 10.2.1.29 (10.2.1.29), 64 hops max, 40 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * *^C

If anyone has any suggestions as to what else I might try to debug this it would be hugely appreciated.

tman222

Hi @stev -

I'm not convinced that the 169.254.x.x usage is the issue here. I setup a routed IPSec connection on Google Cloud using FRR and BGP a little while back and also had to use the 169.254.x.x. IP range for the BGP neighbors. This ended up working fine.

Couple questions for you:

Can you confirm that the BGP session is working and that routes are being exchanged (i.e. you can see the routing tables update on both the AWS side and pfSense)?
Can you confirm that your local subnet has been granted proper access to your VPC subnet? That is, check your VPC firewall settings, e.g., security groups and Network ACL's, etc.

Also here is a video that might be helpful -- there is a mention of AWS around minute 66:

https://www.youtube.com/watch?v=4IlKcB17rWk

Hope this helps

stev

Thanks for the suggestions.

I have verified that the routes are propagating as they show up on both sides of the VPN.
I believe everything is setup correctly in terms of the VPC subnets and associated ACLs. I worked with AWS support who also reviewed those settings.

Thanks for the video, I went through that section and tweaked one setting but it didn't seem to change anything.

tman222

@stev said in AWS VPN BGP - Routing:

Thanks for the suggestions.

I have verified that the routes are propagating as they show up on both sides of the VPN.

I believe everything is setup correctly in terms of the VPC subnets and associated ACLs. I worked with AWS support who also reviewed those settings.

Thanks for the video, I went through that section and tweaked one setting but it didn't seem to change anything.

Hi @stev -- hmmm, that's interesting. A couple more questions:

Are you using VTI by chance? If not, it may be worth a try: https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/ipsec-routed.html
Does the other side work, i.e. can you access (ping) your machines from AWS?

Hope this helps.

stephenw10

Yes, that seems like a good test. Ping out from an instance in AWS and (if the ping fails) run a packet capture in pfSense.

You should see those packets arriving on the IPSec interface even if pfSense then drops that.

If you are not using VTI you would just be sending between the two subnets directly with policy based IPSec, BGP doesn't come into it. How exactly is your IPSec configured?

Steve

tman222

I just looked back and found the thread that I started about setting all this up using GCP (instead of AWS):

https://forum.netgate.com/topic/136509/routed-ipsec-vti-and-google-cloud

Hopefully the information contained within can help you, perhaps in particular this link.

https://www.1strategy.com/blog/2017/08/29/tutorial-using-pfsense-as-a-vpn-to-your-vpc/

stephenw10

We have been working on this in support and currently believe it's a missing setting in AWS somewhere.

Steve

wstocker

Did you find a fix for this?

I am experiencing identical behaviour to this.

v2.5.0

IPSec up, BGP peers up, routes exchanged.

route show for my AWS VPC gives next-hop of the local 169.x.

Can ping the remote 169.x peer ips using the 169.x source from the firewall but can not ping host in VPC when sourcing lan.

The traffic just seems to blackhole, no firewall logs, nothing received from AWS and nothing getting there.

I suppose a tcpdump is my next line of attack but it's becoming tiresome.

This is peculiar because using a static VPN configuration I can connect into the hosts just fine, so am aware this is NOT an AWS security rules issue.

I'm finding it frustrating because I want to connect the ipsec site to site with bgp to take advantage of AWS Transit Gateway's and its ECMP features (Not to mention having much fewer routes to configure in the cloud VPCs and locally).

stephenw10

Looks like you fixed this with VTI in your other thread?

Steve

stev

@wstocker Unfortunately I never found a resolution to this issue. I ended up switching to a PA-220 which worked as expected.

wstocker

@stev said in AWS VPN BGP - Routing:

P

Hey,

Yup VTI fixed this for me, although it’s unclear to me why this wouldn’t work with the configuration as described (that we both had tried)

stephenw10

BGP over policy based IPSec is just asking for trouble IMO. It's possible to make it work but you need P2s to cover all traffic being routed and the BGP traffic itself. Since AWS only allows 2 P2s on each P1 that's often a problem.

Steve

MushyMiddle

I got this working, but it took a solid 6 hours or so of tweaking to finally get things going. Mostly, for posterity...

First, it's a good idea to set up a static IPSec config, just to get all of the AWS bits working properly (RTB, SG, ACL, etc.). When that's working, remove the static route(s) in the RTB to the VGW. Make sure to set up the RTB to propagate routes. Kill the static VPN.

Then, create the new BGP VPN, and set up the two VTIs. A critical step is to set up both the VTI interfaces and gateways (it's easy to forget). For the gateways, leave the gateway IP blank, as you won't be able to enter the IP anyway. It will show as "dynamic" for a bit, and eventually will show the correct tunnel IP.

Finally, FRR BGP, which needs way more docs/examples to get working properly. Nowhere does anything mention having to define some form of route map, but without that, you won't get routes to set up - they'll just be discarded. (Admittedly, this was my first foray into BGP). Set up both neighbors (VTI tunnel IPs). It seems like the Route Map field should be required, or at least, have a tooltip of some sort - you need a policy defined that will allow the routes to be exchanged.

One symptom of failing to set up VTI interfaces/gateways is rejected BGP traffic - firewall rules won't help, but those messages should go away once the gateways are defined and up.

Once everything is working, you should see the two tunnels up in AWS, and show "X BGP ROUTES" (where X > 0). That may take a bit, as AWS doesn't update things very often, but you'll also see the routes show up in FRR BGP status.

Now you should be able to take down the primary VTI on pfSense and within a few secs, traffic will fail over to the other VTI, and recover after the downed VTI is back.

MushyMiddle

P.S. I take it back - you may need firewall rules for IPSec to allow BGP traffic. You can create them from the firewall logs if you see blocked BGP traffic on IPSec.