AWS VPN BGP - Routing



  • Hi,

    I'm new to pfSense and I'm trying to setup an AWS Transit Gateway VPN with BGP. Unfortunately, AWS does not provide config settings for pfSense with BGP (only without), or a configuration tutorial. I've been looking at tutorials and trying to figure it out with AWS support but no luck.

    The sticking point right now seems to be that their BGP server (169.254.47.141) is trying to contact my BGP (169.254.47.142) but its hitting a firewall rule "Block IPv4 link-local (1000108731)".

    I have tried:
    Setting up a virtual IP of 169.254.47.142/32
    Setting up a route for 169.254.47.142/32 off the WAN connection used by the VPN
    Adding a rule to the IPSEC interface using the 'easy add' button.

    It's still blocking the request. I understand that range is special, but its the only range that AWS will allow. I cannot ping or attach to port 179 from my workstation on the LAN, but I don't know if that's something weird with that range and my Windows workstation.

    How can I allow the connection to go through?

    Any help would be greatly appreciated... I've been stuck on this for days now.

    Steve


  • LAYER 8 Rebel Alliance

    Interfaces > WAN > uncheck Block bogon networks

    -Rico



  • Thanks Rico.

    I tried that, but the requests are still being blocked. I enabled both bogon and private then rebooted the router just to be sure it took.


  • Netgate Administrator

    Bogons won't help there it's blocking APIPA as it should. APIPA subnets should never be used like that, it violates rfc3927. But, Amazon so.....

    https://forum.netgate.com/post/534884

    Steve



  • Thanks!

    The BGP traffic seems to be flowing now.



  • So things got weirder...

    The BGP traffic is flowing. The routes show up, but it doesn’t work.

    I do a route show for a 10.2.0.0/16 server that lives in AWS. It says the next hop is the AWS BGP neighbor which has a APIPA address.

    I cannot ping that IP address from my Windows workstation. I believe it says General Failure, like Windows just refuses because it’s APIPA?

    I CAN ping that IP address from the pfSense router and the replies take about 30ms so I think it’s really talking to the AWS BGP neighbor. However, when I try to ping my 10.2.0.0/16 AWS server from the router it never works. When I traceroute it shows no responses along the way (not even the BGP neighbor that had responded to a ping previously).

    So I’m at a loss. I know APIPA is weird... when I try to ping a thing and the next hop is an APIPA does FreeBSD just drop the packet, but when I directly ping an APIPA address it works?

    After making the recommended config change it no longer looks like the firewall is blocking it.

    I’m really stuck. If anybody has any ideas I’d really appreciate it.



  • I think it might be hopeless. I found a bit of FreeBSD code that drops packets in that range.

    I don’t know why pinging the IP works, but routing through it apparently does not. I wish FreeBSD, pfSense and AWS were on the same page.

    There were others who apparently got AWS VPN with BGP and pfSense to work but maybe they were using an older version?

    https://raw.githubusercontent.com/freebsd/freebsd/master/sys/netinet/ip_input.c

    /* RFC 3927 2.7: Do not forward datagrams for 169.254.0.0/16. */
    if (IN_LINKLOCAL(ntohl(ip->ip_dst.s_addr))) {
    IPSTAT_INC(ips_cantforward);
    m_freem(m);
    return;
    }

    Also a mail
    https://lists.freebsd.org/pipermail/freebsd-stable/2010-June/057180.html


  • Netgate Administrator

    If you have set that value in the pfSense config it should work. There are many people using that, we had to put that in just for this.

    However, when I try to ping my 10.2.0.0/16 AWS server from the router it never works.

    Where are you trying to ping it from?

    If you ping it from something that actually has the APIPA address it might not work as that's unroutable but as long as it's just using it as a transport network to route over I would expect it to.

    Try setting the source IP to something other than APIPA if you're able to.

    Steve



  • Thanks for your reply!

    The source IP is my workstation on the 10.4.0.0/16 LAN 10.4.0.123. The only APIPA addresses involved are the BGP peer addresses AWS required.



  • I tried replacing FRR BGP with OpenBGP, but it didn't make a difference.

    Here is a log of my attempts:

    First I try pinging a server in AWS (it fails):

    [2.4.4-RELEASE][me@router-1.whatever.com]/home/me: ping 10.2.1.52
    PING 10.2.1.52 (10.2.1.52): 56 data bytes
    ^C
    --- 10.2.1.52 ping statistics ---
    3 packets transmitted, 0 packets received, 100.0% packet loss
    

    I then check to see what the next hop would be, it is the BGP peer on AWS' side.

    [2.4.4-RELEASE][me@router-1.whatever.com]/home/me: route show 10.2.1.52
       route to: 10.2.1.52
    destination: 10.2.0.0
           mask: 255.255.0.0
        gateway: 169.254.47.141
            fib: 0
      interface: lagg0.4090
          flags: <UP,GATEWAY,DONE,PROTO1>
     recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
           0         0         0         0      1500         1         0
    

    I verify that I am able to ping that IP. It takes 30ms round trip which indicates to me it is in fact on the AWS side.

    [2.4.4-RELEASE][me@router-1.whatever.com]/home/me: ping 169.254.47.141
    PING 169.254.47.141 (169.254.47.141): 56 data bytes
    64 bytes from 169.254.47.141: icmp_seq=0 ttl=254 time=28.924 ms
    64 bytes from 169.254.47.141: icmp_seq=1 ttl=254 time=28.888 ms
    64 bytes from 169.254.47.141: icmp_seq=2 ttl=254 time=28.770 ms
    64 bytes from 169.254.47.141: icmp_seq=3 ttl=254 time=28.716 ms
    64 bytes from 169.254.47.141: icmp_seq=4 ttl=254 time=28.774 ms
    64 bytes from 169.254.47.141: icmp_seq=5 ttl=254 time=28.863 ms
    ^C
    --- 169.254.47.141 ping statistics ---
    6 packets transmitted, 6 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 28.716/28.823/28.924/0.074 ms
    

    I try doing a traceroute to our AWS server. It fails, not even the first hop responds which is really confusing to me.

    [2.4.4-RELEASE][me@router-1.whatever.com]/home/me: traceroute 10.2.1.29
    traceroute to 10.2.1.29 (10.2.1.29), 64 hops max, 40 byte packets
     1  * * *
     2  * * *
     3  * * *
     4  * * *
     5  * * *
     6  * * *
     7  * * *
     8  * * *
     9  * * *
    10  * * *
    11  * * *
    12  * * *
    13  * * *
    14  * * *
    15  * * *
    16  * *^C
    

    If anyone has any suggestions as to what else I might try to debug this it would be hugely appreciated.



  • Hi @stev -

    I'm not convinced that the 169.254.x.x usage is the issue here. I setup a routed IPSec connection on Google Cloud using FRR and BGP a little while back and also had to use the 169.254.x.x. IP range for the BGP neighbors. This ended up working fine.

    Couple questions for you:

    1. Can you confirm that the BGP session is working and that routes are being exchanged (i.e. you can see the routing tables update on both the AWS side and pfSense)?
    2. Can you confirm that your local subnet has been granted proper access to your VPC subnet? That is, check your VPC firewall settings, e.g., security groups and Network ACL's, etc.

    Also here is a video that might be helpful -- there is a mention of AWS around minute 66:

    Youtube Video

    Hope this helps



  • Thanks for the suggestions.

    1. I have verified that the routes are propagating as they show up on both sides of the VPN.
    2. I believe everything is setup correctly in terms of the VPC subnets and associated ACLs. I worked with AWS support who also reviewed those settings.

    Thanks for the video, I went through that section and tweaked one setting but it didn't seem to change anything.



  • @stev said in AWS VPN BGP - Routing:

    Thanks for the suggestions.

    1. I have verified that the routes are propagating as they show up on both sides of the VPN.
    2. I believe everything is setup correctly in terms of the VPC subnets and associated ACLs. I worked with AWS support who also reviewed those settings.

    Thanks for the video, I went through that section and tweaked one setting but it didn't seem to change anything.

    Hi @stev -- hmmm, that's interesting. A couple more questions:

    1. Are you using VTI by chance? If not, it may be worth a try: https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/ipsec-routed.html
    2. Does the other side work, i.e. can you access (ping) your machines from AWS?

    Hope this helps.


  • Netgate Administrator

    Yes, that seems like a good test. Ping out from an instance in AWS and (if the ping fails) run a packet capture in pfSense.

    You should see those packets arriving on the IPSec interface even if pfSense then drops that.

    If you are not using VTI you would just be sending between the two subnets directly with policy based IPSec, BGP doesn't come into it. How exactly is your IPSec configured?

    Steve



  • I just looked back and found the thread that I started about setting all this up using GCP (instead of AWS):

    https://forum.netgate.com/topic/136509/routed-ipsec-vti-and-google-cloud

    Hopefully the information contained within can help you, perhaps in particular this link.

    https://www.1strategy.com/blog/2017/08/29/tutorial-using-pfsense-as-a-vpn-to-your-vpc/


  • Netgate Administrator

    We have been working on this in support and currently believe it's a missing setting in AWS somewhere.

    Steve