AWS VPN BGP - Routing

stev

Hi,

I'm new to pfSense and I'm trying to setup an AWS Transit Gateway VPN with BGP. Unfortunately, AWS does not provide config settings for pfSense with BGP (only without), or a configuration tutorial. I've been looking at tutorials and trying to figure it out with AWS support but no luck.

The sticking point right now seems to be that their BGP server (169.254.47.141) is trying to contact my BGP (169.254.47.142) but its hitting a firewall rule "Block IPv4 link-local (1000108731)".

I have tried:
Setting up a virtual IP of 169.254.47.142/32
Setting up a route for 169.254.47.142/32 off the WAN connection used by the VPN
Adding a rule to the IPSEC interface using the 'easy add' button.

It's still blocking the request. I understand that range is special, but its the only range that AWS will allow. I cannot ping or attach to port 179 from my workstation on the LAN, but I don't know if that's something weird with that range and my Windows workstation.

How can I allow the connection to go through?

Any help would be greatly appreciated... I've been stuck on this for days now.

Steve

Rico

Interfaces > WAN > uncheck Block bogon networks

-Rico

stev

Thanks Rico.

I tried that, but the requests are still being blocked. I enabled both bogon and private then rebooted the router just to be sure it took.

stephenw10

Bogons won't help there it's blocking APIPA as it should. APIPA subnets should never be used like that, it violates rfc3927 . But, Amazon so.....

https://forum.netgate.com/post/534884

Steve

stev

Thanks!

The BGP traffic seems to be flowing now.

stev

So things got weirder...

The BGP traffic is flowing. The routes show up, but it doesn’t work.

I do a route show for a 10.2.0.0/16 server that lives in AWS. It says the next hop is the AWS BGP neighbor which has a APIPA address.

I cannot ping that IP address from my Windows workstation. I believe it says General Failure, like Windows just refuses because it’s APIPA?

I CAN ping that IP address from the pfSense router and the replies take about 30ms so I think it’s really talking to the AWS BGP neighbor. However, when I try to ping my 10.2.0.0/16 AWS server from the router it never works. When I traceroute it shows no responses along the way (not even the BGP neighbor that had responded to a ping previously).

So I’m at a loss. I know APIPA is weird... when I try to ping a thing and the next hop is an APIPA does FreeBSD just drop the packet, but when I directly ping an APIPA address it works?

After making the recommended config change it no longer looks like the firewall is blocking it.

I’m really stuck. If anybody has any ideas I’d really appreciate it.

stev

I think it might be hopeless. I found a bit of FreeBSD code that drops packets in that range.

I don’t know why pinging the IP works, but routing through it apparently does not. I wish FreeBSD, pfSense and AWS were on the same page.

There were others who apparently got AWS VPN with BGP and pfSense to work but maybe they were using an older version?

https://raw.githubusercontent.com/freebsd/freebsd/master/sys/netinet/ip_input.c

/* RFC 3927 2.7: Do not forward datagrams for 169.254.0.0/16. */
if (IN_LINKLOCAL(ntohl(ip->ip_dst.s_addr))) {
IPSTAT_INC(ips_cantforward);
m_freem(m);
return;
}

Also a mail
https://lists.freebsd.org/pipermail/freebsd-stable/2010-June/057180.html

stephenw10

If you have set that value in the pfSense config it should work. There are many people using that, we had to put that in just for this.

However, when I try to ping my 10.2.0.0/16 AWS server from the router it never works.

Where are you trying to ping it from?

If you ping it from something that actually has the APIPA address it might not work as that's unroutable but as long as it's just using it as a transport network to route over I would expect it to.

Try setting the source IP to something other than APIPA if you're able to.

Steve

stev

Thanks for your reply!

The source IP is my workstation on the 10.4.0.0/16 LAN 10.4.0.123. The only APIPA addresses involved are the BGP peer addresses AWS required.

stev

I tried replacing FRR BGP with OpenBGP, but it didn't make a difference.

Here is a log of my attempts:

First I try pinging a server in AWS (it fails):

[2.4.4-RELEASE][me@router-1.whatever.com]/home/me: ping 10.2.1.52
PING 10.2.1.52 (10.2.1.52): 56 data bytes
^C
--- 10.2.1.52 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

I then check to see what the next hop would be, it is the BGP peer on AWS' side.

[2.4.4-RELEASE][me@router-1.whatever.com]/home/me: route show 10.2.1.52
   route to: 10.2.1.52
destination: 10.2.0.0
       mask: 255.255.0.0
    gateway: 169.254.47.141
        fib: 0
  interface: lagg0.4090
      flags: <UP,GATEWAY,DONE,PROTO1>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

I verify that I am able to ping that IP. It takes 30ms round trip which indicates to me it is in fact on the AWS side.

[2.4.4-RELEASE][me@router-1.whatever.com]/home/me: ping 169.254.47.141
PING 169.254.47.141 (169.254.47.141): 56 data bytes
64 bytes from 169.254.47.141: icmp_seq=0 ttl=254 time=28.924 ms
64 bytes from 169.254.47.141: icmp_seq=1 ttl=254 time=28.888 ms
64 bytes from 169.254.47.141: icmp_seq=2 ttl=254 time=28.770 ms
64 bytes from 169.254.47.141: icmp_seq=3 ttl=254 time=28.716 ms
64 bytes from 169.254.47.141: icmp_seq=4 ttl=254 time=28.774 ms
64 bytes from 169.254.47.141: icmp_seq=5 ttl=254 time=28.863 ms
^C
--- 169.254.47.141 ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 28.716/28.823/28.924/0.074 ms

I try doing a traceroute to our AWS server. It fails, not even the first hop responds which is really confusing to me.

[2.4.4-RELEASE][me@router-1.whatever.com]/home/me: traceroute 10.2.1.29
traceroute to 10.2.1.29 (10.2.1.29), 64 hops max, 40 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * *^C

If anyone has any suggestions as to what else I might try to debug this it would be hugely appreciated.

tman222

Hi @stev -

I'm not convinced that the 169.254.x.x usage is the issue here. I setup a routed IPSec connection on Google Cloud using FRR and BGP a little while back and also had to use the 169.254.x.x. IP range for the BGP neighbors. This ended up working fine.

Couple questions for you:

Can you confirm that the BGP session is working and that routes are being exchanged (i.e. you can see the routing tables update on both the AWS side and pfSense)?
Can you confirm that your local subnet has been granted proper access to your VPC subnet? That is, check your VPC firewall settings, e.g., security groups and Network ACL's, etc.

Also here is a video that might be helpful -- there is a mention of AWS around minute 66:

https://www.youtube.com/watch?v=4IlKcB17rWk

Hope this helps

stev

Thanks for the suggestions.

I have verified that the routes are propagating as they show up on both sides of the VPN.
I believe everything is setup correctly in terms of the VPC subnets and associated ACLs. I worked with AWS support who also reviewed those settings.

Thanks for the video, I went through that section and tweaked one setting but it didn't seem to change anything.

tman222

@stev said in AWS VPN BGP - Routing:

Thanks for the suggestions.

I have verified that the routes are propagating as they show up on both sides of the VPN.

I believe everything is setup correctly in terms of the VPC subnets and associated ACLs. I worked with AWS support who also reviewed those settings.

Thanks for the video, I went through that section and tweaked one setting but it didn't seem to change anything.

Hi @stev -- hmmm, that's interesting. A couple more questions:

Are you using VTI by chance? If not, it may be worth a try: https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/ipsec-routed.html
Does the other side work, i.e. can you access (ping) your machines from AWS?

Hope this helps.

stephenw10

Yes, that seems like a good test. Ping out from an instance in AWS and (if the ping fails) run a packet capture in pfSense.

You should see those packets arriving on the IPSec interface even if pfSense then drops that.

If you are not using VTI you would just be sending between the two subnets directly with policy based IPSec, BGP doesn't come into it. How exactly is your IPSec configured?

Steve

tman222

I just looked back and found the thread that I started about setting all this up using GCP (instead of AWS):

https://forum.netgate.com/topic/136509/routed-ipsec-vti-and-google-cloud

Hopefully the information contained within can help you, perhaps in particular this link.

https://www.1strategy.com/blog/2017/08/29/tutorial-using-pfsense-as-a-vpn-to-your-vpc/

stephenw10

We have been working on this in support and currently believe it's a missing setting in AWS somewhere.

Steve

wstocker

Did you find a fix for this?

I am experiencing identical behaviour to this.

v2.5.0

IPSec up, BGP peers up, routes exchanged.

route show for my AWS VPC gives next-hop of the local 169.x.

Can ping the remote 169.x peer ips using the 169.x source from the firewall but can not ping host in VPC when sourcing lan.

The traffic just seems to blackhole, no firewall logs, nothing received from AWS and nothing getting there.

I suppose a tcpdump is my next line of attack but it's becoming tiresome.

This is peculiar because using a static VPN configuration I can connect into the hosts just fine, so am aware this is NOT an AWS security rules issue.

I'm finding it frustrating because I want to connect the ipsec site to site with bgp to take advantage of AWS Transit Gateway's and its ECMP features (Not to mention having much fewer routes to configure in the cloud VPCs and locally).

stephenw10

Looks like you fixed this with VTI in your other thread?

Steve

stev

@wstocker Unfortunately I never found a resolution to this issue. I ended up switching to a PA-220 which worked as expected.

wstocker

@stev said in AWS VPN BGP - Routing:

P

Hey,

Yup VTI fixed this for me, although it’s unclear to me why this wouldn’t work with the configuration as described (that we both had tried)