OpenVPN site-to-site and FRR OSPF with 3 pfSenses



  • I have made a test set up with three pfSense virtual machines. Call them test1, test2, test3. Test1 runs an OpenVPN peer to peer SSL/TLS server for test2 to connect as a client. Similarly, test2 has a server for test3 and test3 has a server for test 1. This all works fine, all three are interconnected, each firewall can ping the others' LAN address. The VPNs use the net30 topology, but I also tried the subnet way with the same outcome.
    I then install the frr package on all three firewalls. I enable FRR and OSPF on the three firewalls, and add the OpenVPN interfaces to the OSPF interfaces section as well as the local LAN, so that's three interfaces on each firewall. The OSPF status shows that each can see its neighbors. I can ping each firewall's LAN address from the other two. I then stop the VPN between test1 and test2. The OSPF notices and now shows a route from test1 to test2 via the test1 to test3 VPN. However, when I ping test2 from test1 or vice versa, I get no reply. I was hoping that the ping would be relayed from test1 to test2 via the VPNs on test3. Here are some details.

    test1 LAN 192.168.128.1/24 WAN 192.168.127.216
    test2 LAN 192.168.129.1/24 WAN 192.168.127.218
    test3 LAN 192.168.130.1/24 WAN 192.168.127.221
    test2 to test1 VPN tunnel 10.127.1.0/30
    test3 to test2 VPN tunnel 10.127.2.0/30
    test1 to test3 VPN tunnel 10.127.3.0/30

    Here are more details for test1, the other two are set up in a similar fashion.

    test1 FRR zebra.conf
    ##################### DO NOT EDIT THIS FILE! ######################
    ###################################################################

    This file was created by an automatic configuration generator.

    The contents of this file will be overwritten without warning!

    ###################################################################
    password ZEBRA
    log syslog

    Null Routes

    Static Routes

    ip route 192.168.128.0/24 em1

    Accept Filters

    ip prefix-list ACCEPTFILTER permit any
    route-map ACCEPTFILTER permit 10
    match ip address prefix-list ACCEPTFILTER
    ip protocol ospf route-map ACCEPTFILTER

    test1 FRR ospfd.conf
    ##################### DO NOT EDIT THIS FILE! ######################
    ###################################################################

    This file was created by an automatic configuration generator.

    The contents of this file will be overwritten without warning!

    ###################################################################
    password ZEBRA
    log syslog
    interface ovpnc2
      ip ospf authentication-key ZEBRA
    interface ovpns1
      ip ospf authentication-key ZEBRA
    interface em1
      ip ospf authentication-key ZEBRA

    router ospf
      ospf router-id 192.168.127.216
      redistribute connected route-map DNR
      redistribute static route-map DNR
      passive-interface em1
      network 10.127.3.0/30 area 0.0.0.0
      network 10.127.1.0/30 area 0.0.0.0
      network 192.168.128.0/24 area 0.0.0.0
    area 0.0.0.0 authentication
    area 0.0.0.0 authentication
    area 0.0.0.0 authentication
    access-list dnr-list permit any
    route-map DNR permit 10
      match ip address dnr-list

    Here's the status on test1 when the test2 to test1 VPN is down.
    test1 OSPF General
    OSPF Routing Process, Router ID: 192.168.127.216
    Supports only single TOS (TOS0) routes
    This implementation conforms to RFC2328
    RFC1583Compatibility flag is disabled
    OpaqueCapability flag is disabled
    Initial SPF scheduling delay 0 millisec(s)
    Minimum hold time between consecutive SPFs 50 millisec(s)
    Maximum hold time between consecutive SPFs 5000 millisec(s)
    Hold time multiplier is currently 1
    SPF algorithm last executed 2m18s ago
    Last SPF duration 195 usecs
    SPF timer is inactive
    LSA minimum interval 5000 msecs
    LSA minimum arrival 1000 msecs
    Write Multiplier set to 20
    Refresh timer 10 secs
    This router is an ASBR (injecting external routing information)
    Number of external LSA 8. Checksum Sum 0x0003a1e3
    Number of opaque AS LSA 0. Checksum Sum 0x00000000
    Number of areas attached to this router: 1
    Area ID: 0.0.0.0 (Backbone)
      Number of interfaces in this area: Total: 3, Active: 3
      Number of fully adjacent neighbors in this area: 1
      Area has simple password authentication
      SPF algorithm executed 8 times
      Number of LSA 3
      Number of router LSA 3. Checksum Sum 0x000174d4
      Number of network LSA 0. Checksum Sum 0x00000000
      Number of summary LSA 0. Checksum Sum 0x00000000
      Number of ASBR summary LSA 0. Checksum Sum 0x00000000
      Number of NSSA LSA 0. Checksum Sum 0x00000000
      Number of opaque link LSA 0. Checksum Sum 0x00000000
      Number of opaque area LSA 0. Checksum Sum 0x00000000

    test1 OSFP Routes
    ============ OSPF network routing table ============
    N    192.168.128.0/24      [100] area: 0.0.0.0
                              directly attached to em1
    N    192.168.129.0/24      [120] area: 0.0.0.0
                              via 10.127.3.1, ovpnc2
    N    192.168.130.0/24      [110] area: 0.0.0.0
                              via 10.127.3.1, ovpnc2

    ============ OSPF router routing table =============
    R    192.168.127.218      [20] area: 0.0.0.0, ASBR
                              via 10.127.3.1, ovpnc2
    R    192.168.127.221      [10] area: 0.0.0.0, ASBR
                              via 10.127.3.1, ovpnc2

    ============ OSPF external routing table ===========
    N E2 10.127.2.1/32        [10/20] tag: 0
                              via 10.127.3.1, ovpnc2
    N E2 10.127.2.2/32        [20/20] tag: 0
                              via 10.127.3.1, ovpnc2
    N E2 10.127.3.2/32        [10/20] tag: 0
                              via 10.127.3.1, ovpnc2
    N E2 192.168.127.0/24      [10/20] tag: 0
                              via 10.127.3.1, ovpnc2

    In this last section, it's clear that the OSPF thinks it can get to both 192.168.130.0/24 and 192.168.129.0/24 via ovpnc2 which is the VPN to test3.
    I'm a newbie to OSFP, so I expect I've done something daft, any pointers gratefully received.
    Thanks for reading.



  • Update. I think I fixed it. But I don't really understand it properly. I'd appreciate it if anyone can explain what's happening!
    I had to add some rules to Outbound NAT. On each pfSense I added a rule for all OpenVPN tunnel IP addresses (10.127.0.0/16 in my case) sources on the WAN to translate to the WAN interface address. This got the ping working via the third pfSenses during VPN outages. I then also added a rule for all IP addresses sourced from the LAN on the OpenVPN interface to translate to the OpenVPN interface address. In my example above, this meant NATing 192.168.128.0/24 on test1, 192.168.129.0/24 on test 2 etc. Now it works. If I set a ping going from one pfSense's lan to another, and I stop the VPN between the two, the pings get re-routed via the third pfSense. A few pings get lost while it's swapping, but this is what I wanted! Back of the net!
    Anyway, as I said, if anyone can explain what's happening here, that would be great. I won't mark this [SOLVED] just yet until I'm sure I've done this correctly.
    p.s. Don't you love it when things start to work just before pub o'clock!