Apinger and OpenVPN: Gateway down after OpenVPN client service restart



  • Hi,

    when i restart the OpenVPN client service, which has an interface assigned, the correspondig gateway is going down and never comes up again. The same holds true, if the openVPN server is restartet and the clients reconnect.

    In both cases the openVPN client reconnects, however the gateway stays down.

    The gateway for the OpenVPN client has a manuelly set monitoring IP. This works on startup, after a reboot of the server. Another way to get the gateway up again is to restart apinger.

    In combination with Gateway Groups, the failover or load balancing options are not working as the gateway is still down.

    This behavour has version 2.1 and the newest 2.1.1-PRERELEASE (i386) built on Wed Mar 26 13:50:29 EDT 2014.

    Any suggestions?



  • as a workaround , and if failover is good enough, you could remove gateway-group and use quagga-ospf to do dynamic routing for you.

    no more gateway monitoring, just simple failover by adding/removing routes when a vpn goes up or down



  • Hi,

    I tried this, but had several problems with Quagga OSPF and the OpenVPN tun interface. Still I have serveral questions:

    1. Is it possible to use a tun interface with three or more subnets connecting to a central OpenVPN server AND get OSPF working?

    2. My current OpenVPN server configuration uses the tun interface and the option: "Topology subnet". This allows the OSPF client/servers to see each other. (Seems like broadcasts are working in that mode… but I'm not sure... ) However, they do not "syncronize".  They jump between init, ExStart and Exchange state...

    3. Is there a how to or more information to run OSPF over a tun interface? It seems to work with one other OSPF client/server, but nor with three or more...

    kind regards



  • yes i have a couple of those sites setup that way.

    draw a network map, so i can understand what you are trying to do



  • Hi,

    well it's pretty standard (and I'm not good in ascii art, therefore I try to explain it in words): I have mote than three sites to connect (for simplicity only three).  These are in the subnets:

    192.168.0.0/24
    192.168.1.0/24
    192.168.2.0/24

    At the moment they are connected to one central OpenVPN server in the internet. The OpenVPN server subnet is 10.0.8.0/24.

    We would like to have some more reliablility as sometimes the server in the internet has downtimes. Therefore, we would like a secound OpenVPN server in the internet with the subnet 10.0.7.0/24.

    These two OpenVPN connections are already established and working. However, the "dynamic" routing is the problem, when the primary server goes down. (And later on cames up again.) The gateway group solution seems not to work very good on the openVPN client interfaces.

    Therefore, I followed your sugestion und installed Quagga OSPF. But there seems to be a problem with OpenVPN tun interfaces, as they provide no real boradcasting. However, we would like to stay with the tun interfaces. So first question, has anybody such a setup running with tun interfaces and OSPF?

    Secound question, how would the config look like with tap interfaces?

    Just give a short reply, where you need more infomations.

    Thanks and kind regards



  • 1. don't use TAP if you can avoid it … it will seriously damage performance. OSPF will/should/can work fine on TUN. i run all my tunnels this way, with ospf

    2. i don't know, see 1

    i don't think you ospf problems has got anything to do with tun or tap.
    below you will something that could get you on your way (i think).
    be careful with automatic NAT ... chances are high that this will mess things up when you have interfaces assigned to your openvpn's and attempt the setup below.

    enjoy




  • Hi,

    first thanks! The graphic helped a lot. I understand completly why this is working.

    However, what we tried is not to use single OpenVPN Tunnels between the 20 sides we have. This would require 20 tunnels on each server and two tunnels on each side.

    Therefore, I would like to use a "star" network. So each side connects to the two servers. The two server just host one OpenVPN Server network where just all sides connect to.

    And now my question. Is OSPF capable of negotiating such star situations? I thoughed yes…

    However, using tun interfaces without any other config options, results in no connection between the OSPF client/servers. Using "topology subnet" results in OSPF clients/servers seeing each other. But they can not negotiate... Here is an exemplary log:

    Mar 28 09:04:22 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3: Initial DBD from Slave, ignoring.
    Mar 28 09:04:22 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 Negotiation fails.
    Mar 28 09:04:22 ospfd[71882]: AdjChg: Nbr 10.0.3.3 on ovpnc1:10.0.8.3: Exchange -> Loading (ExchangeDone)
    Mar 28 09:04:22 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 Negotiation fails.
    Mar 28 09:04:22 ospfd[71882]: Link State Request received from 10.0.1.3: Neighbor state is ExStart, packet discarded.
    Mar 28 09:04:22 ospfd[71882]: AdjChg: Nbr 10.0.3.3 on ovpnc1:10.0.8.3: Loading -> Full (LoadingDone)
    Mar 28 09:04:22 ospfd[71882]: LSA[Type5:0.0.0.0]: Not originate AS-external-LSA for default
    Mar 28 09:04:22 ospfd[71882]: nsm_change_state(10.0.3.3, Loading -> Full): scheduling new router-LSA origination
    Mar 28 09:04:22 ospfd[71882]: Point-to-Point link has more than 1 neighobrs.
    Mar 28 09:04:22 ospfd[71882]: Link State Update: Neighbor[10.0.1.3] state ExStart is less than Exchange
    Mar 28 09:04:22 ospfd[71882]: Link State Update: Neighbor[10.0.1.3] state ExStart is less than Exchange
    Mar 28 09:04:23 ospfd[71882]: Link State Acknowledgment: Neighbor[10.0.1.3] state ExStart is less than Exchange
    Mar 28 09:04:27 ospfd[71882]: AdjChg: Nbr 10.0.3.3 on ovpnc1:10.0.8.3: Full -> ExStart (SeqNumberMismatch)
    Mar 28 09:04:27 ospfd[71882]: nsm_change_state(10.0.3.3, Full -> ExStart): scheduling new router-LSA origination
    Mar 28 09:04:27 ospfd[71882]: Point-to-Point link has more than 1 neighobrs.
    Mar 28 09:04:27 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3: Initial DBD from Slave, ignoring.
    Mar 28 09:04:27 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 Negotiation done (Master).
    Mar 28 09:04:27 ospfd[71882]: AdjChg: Nbr 10.0.1.3 on ovpnc1:10.0.8.3: ExStart -> Exchange (NegotiationDone)
    Mar 28 09:04:27 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 MS-bit mismatch.
    Mar 28 09:04:27 ospfd[71882]: AdjChg: Nbr 10.0.1.3 on ovpnc1:10.0.8.3: Exchange -> ExStart (SeqNumberMismatch)
    Mar 28 09:04:27 ospfd[71882]: Packet[DD]: Neighbor 10.0.3.3 Negotiation done (Slave).
    Mar 28 09:04:27 ospfd[71882]: AdjChg: Nbr 10.0.3.3 on ovpnc1:10.0.8.3: ExStart -> Exchange (NegotiationDone)
    Mar 28 09:04:27 ospfd[71882]: Link State Update: Neighbor[10.0.1.3] state ExStart is less than Exchange
    Mar 28 09:04:27 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 Negotiation fails.
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3: Initial DBD from Slave, ignoring.
    Mar 28 09:04:28 ospfd[71882]: AdjChg: Nbr 10.0.3.3 on ovpnc1:10.0.8.3: Exchange -> Loading (ExchangeDone)
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 Negotiation done (Master).
    Mar 28 09:04:28 ospfd[71882]: AdjChg: Nbr 10.0.1.3 on ovpnc1:10.0.8.3: ExStart -> Exchange (NegotiationDone)
    Mar 28 09:04:28 ospfd[71882]: AdjChg: Nbr 10.0.3.3 on ovpnc1:10.0.8.3: Loading -> ExStart (SeqNumberMismatch)
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 MS-bit mismatch.
    Mar 28 09:04:28 ospfd[71882]: AdjChg: Nbr 10.0.1.3 on ovpnc1:10.0.8.3: Exchange -> ExStart (SeqNumberMismatch)
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 Negotiation fails.
    Mar 28 09:04:28 ospfd[71882]: Link State Request received from 10.0.1.3: Neighbor state is ExStart, packet discarded.
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3: Initial DBD from Slave, ignoring.
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 Negotiation fails.
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3: Initial DBD from Slave, ignoring.
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.3.3 Negotiation fails.
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.3.3 Negotiation done (Slave).
    Mar 28 09:04:28 ospfd[71882]: AdjChg: Nbr 10.0.3.3 on ovpnc1:10.0.8.3: ExStart -> Exchange (NegotiationDone)
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 Negotiation fails.
    Mar 28 09:04:28 ospfd[71882]: AdjChg: Nbr 10.0.3.3 on ovpnc1:10.0.8.3: Exchange -> Loading (ExchangeDone)
    Mar 28 09:04:28 ospfd[71882]: Packet[DD]: Neighbor 10.0.1.3 Negotiation fails.
    Mar 28 09:04:28 ospfd[71882]: Link State Request received from 10.0.1.3: Neighbor state is ExStart, packet discarded.
    Mar 28 09:04:28 ospfd[71882]: AdjChg: Nbr 10.0.3.3 on ovpnc1:10.0.8.3: Loading -> Full (LoadingDone)
    Mar 28 09:04:28 ospfd[71882]: LSA[Type5:0.0.0.0]: Not originate AS-external-LSA for default
    Mar 28 09:04:28 ospfd[71882]: nsm_change_state(10.0.3.3, Loading -> Full): scheduling new router-LSA origination
    Mar 28 09:04:28 ospfd[71882]: Point-to-Point link has more than 1 neighobrs.
    Mar 28 09:04:28 ospfd[71882]: Link State Update: Neighbor[10.0.1.3] state ExStart is less than Exchange
    Mar 28 09:04:28 ospfd[71882]: Link State Update: Neighbor[10.0.1.3] state ExStart is less than Exchange
    Mar 28 09:04:28 ospfd[71882]: Link State Acknowledgment: Neighbor[10.0.1.3] state ExStart is less than Exchange

    Any idea if such an scenario is possible and how?

    Kind regards



  • i think the problem can be found here:

    Mar 28 09:04:22    ospfd[71882]: Point-to-Point link has more than 1 neighobrs.
    

    the pfSense quagga implementation doesn't have the ability in the webgui to set the interface to point-to-multipoint (i think this might solve your issue)

    what you could try is editing the config manually on your "central servers" ( /var/etc/quagga/ospfd.conf )

    — Interface Command: ip ospf network (broadcast|non-broadcast|point-to-multipoint|point-to-point) 
    

    then restarting the the ospf service on status–>services.  (don't make config changes from the webgui, because this will overwrite your manual changes)
    If that seems to solve the problem, then you could ask the pfsense-package-maintainer for the quagga package to add this ability to the webgui. (contact Jimp)

    oh yea: don't blame me if a nuclear explosion occurs after applying my suggestions ;)



  • The option point-to-multipoint results in:

    
        Neighbor ID Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
    10.0.2.3          1 ExStart/DROther   35.823s 10.0.8.2        ovpnc1:10.0.8.4          0     0     0
    10.0.3.3          1 ExStart/DROther   30.806s 10.0.8.3        ovpnc1:10.0.8.4          0     0     0
    
    
    
    Mar 28 14:50:59 	ospfd[16029]: *** sendmsg in ospf_write failed to 10.0.8.3, id 0, off 0, len 52, interface ovpnc1, mtu 1500: Network is unreachable
    Mar 28 14:50:59 	ospfd[16029]: *** sendmsg in ospf_write failed to 10.0.8.2, id 0, off 0, len 52, interface ovpnc1, mtu 1500: Network is unreachable
    Mar 28 14:51:04 	ospfd[16029]: *** sendmsg in ospf_write failed to 10.0.8.3, id 0, off 0, len 52, interface ovpnc1, mtu 1500: Network is unreachable
    Mar 28 14:51:04 	ospfd[16029]: *** sendmsg in ospf_write failed to 10.0.8.2, id 0, off 0, len 52, interface ovpnc1, mtu 1500: Network is unreachable
    
    

    In my opinion this results from the ospf client/server trying to send multicasts (or broadcasts) that are not supported by tun.



  • looks more like a NAT/route issue to me. have you disabled NAT for the openvpn interfaces ?

    or

    perhaps you need to add the point-to-multipoint on the site(s) aswell ? i'm just speculating ::> see the cisco example here: http://ccieblog.co.uk/ospf/ospf-point-to-multipoint-network



  • Hi,

    just enabled OSPF on the branch sides and not on the server. Used only one OpenVPN server and network.

    When using "ip ospf network broadcast" on all branch sides the error message goes away:

    Point-to-Point link has more than 1 neighobrs.
    

    But this message stays:

    ospfd[16029]: *** sendmsg in ospf_write failed to 10.0.8.3, id 0, off 0, len 52, interface ovpnc1, mtu 1500: Network is unreachable
    

    However, I can ping 10.0.8.3 from all interfaces including ovpnc1…

    Any more ideas? I'm about to give up....

    kind regards



  • have made a quick attempt to get ospf working with a similar openvpn tls/ssl setup in VM - and failed :(

    after some digging with openvpn "topology subnet" i did manage to get ospf to get to a full-link state … but routing failed.
    perhaps jimp or ermal or any other core dev could shed some light on this.


  • Rebel Alliance Developer Netgate

    Moved this to Packages since it's specific to OSPF and not specific to 2.1.1

    @heper:

    have made a quick attempt to get ospf working with a similar openvpn tls/ssl setup in VM - and failed :(

    after some digging with openvpn "topology subnet" i did manage to get ospf to get to a full-link state … but routing failed.
    perhaps jimp or ermal or any other core dev could shed some light on this.

    OSPF requires tun mode to use /30 tunnel networks, or tap mode for a shared instance with multiple clients.

    I've not seen it work in tun mode w/topology subnet.


Log in to reply