OpenVPN Dynamic Routing iroute and OSPF via CCD files [SOLVED]

clahti · Nov 6, 2014, 9:47 PM

Hello all:

We are users of OpenVPN, where we have sensors running linux in the field connecting to multiple OVPN instances in our datacenters. These servers run OSPF and everything works as expected, a client can randomly choose from one of twelve OVPN server targets and all this works very well. My problem is that we have some devices in the field which serve as "backhauls" to other networks of sensors, such as

sensornetwork <-> –-- ovpn-client-backhaul ---- <-> ---------- ovpn-server ----------
192.168.1.0/24 <-> 192.168.1.1(LAN) 10.10.10.5(TUN) <-> 10.10.10.1(TUN) 10.200.20.1(LAN)

So we need servers in datacenter trust 10.200.20.0/24 network to see the LAN 192.168.1.0/24 network behind this backhaul. Enter route and iroute, push a static IP to the backhaul via CCD file on one of the openvpn servers, this all works perfectly fine as expected. The problem is I need to add redundancy to these backhaul devices, similar to the single connections being able to round-robin choose a server at random. It seems that you have to have a route command present in the ovpn server config on startup for an iroute CCD entry to work at all, so with only one openvpn server this works perfectly. This completely falls apart when you enter that same route on all 12 openvpn server config files because this ends up giving OSPF a route to the 192.168.1.0/24 network via the "potential" TUN interfaces on all 12 ovpn-servers. So when the client chooses one of the 12 targets, OSPF has 12 equal cost metric routes to the remote network but at that moment only ONE is a valid path. What I really need is when an iroute comes up, insert the kernel route to match AT THAT TIME so OSPF picks up the correct entry. When the client disconnects then the kernel route should be removed. I have researched this for days and tried everything, the only suggestion I could find was to use TAP instead of TUN, but no concrete working example.

I have network diagrams that better explain what I am trying to do, the root question is, how do I add/modify/delete the kernel route at client connection time? Why does iroute need a preexisting route command in the server file at startup time? It seems to me when iroute is fired THAT would be the appropriate time to insert the kernel route, not a server start time. Does anyone have a solution?

I hope my question is clear, I am in the freenode IRC channel #pfsense right now as well.

Thanks!

/Christian

clahti · Nov 24, 2014, 5:56 PM

I am not one to give up easily, I wrote a custom script to prune the routing table. To implement, download the cron and filer package, then save this script as /usr/local/bin/ovpnroutefix.php. This polls active iroutes and prunes kernel routes that are not valid, and fixes routes for active iroutes. I am not going to go into details on how OSPF is setup, you should get the idea This is INCREDIBLY hackish but works for me:


#!/usr/local/bin/php
#begin logging
logline("===begin scan","");

#get iroute definitions
$defiroutes = get_defiroutes();
logline("defiroute table", $defiroutes);

#get current iroute table, pass the OVPN port numbers to scan for
$servers = array('1198','1199');
$iroutes = get_iroutes($servers);
logline("active iroute table", $iroutes);

#loop through potential defiroutes
#if an active iroute exists for entry, check the kernel routing table and fix route gateway if necessary
#if an active iroute does not exist for entry, remove from kernel routing table
foreach($defiroutes as $defiroute) {
    logline($defiroute, "evaluating $defiroute");
    $evalroute = explode(':',$defiroute);

    if (in_array($defiroute, $iroutes)) {
        #we have active iroute, let's fix routing table if necessary
        logline($defiroute, "has an active iroute match");

        #let's see if the kernel route is already correct
        if(match_route($evalroute[0],$evalroute[1])) {
            #gateway is correct, do nothing
            logline($defiroute, "active iroute, kernel route for network correct, no table update");
        } else {
            #gateway mismatch, need to fix
            kill_route($evalroute[0]);
            logline($defiroute, "active iroute, removed mismatched kernel route");
            #re-add with correct gateway
            add_route($evalroute[0], $evalroute[1]);
            logline($defiroute, "active iroute, added corrected kernel route");
        }
    } else {
        #we have no active iroute, remove from routing table if necessary
        logline($defiroute, "no active iroute match");
        if(match_route($evalroute[0],$evalroute[1])) {
            #a route exists for this gateway, we should kill it. OSPF will insert from other gateway if active
            kill_route($evalroute[0]);
            logline($defiroute, "inactive iroute, removed kernel route");
        } else {
            #no route for this, except perhaps an OSPF route to other gateway, but that is ok
            logline($defiroute, "inactive iroute, no kernel route for network, no table update");
        }
    }
}
logline("===end scan ","");

#===========================================================================
function get_defiroutes() {
#===========================================================================
    #get potential iroute definitions into array
    $defiroutes = array();
    $csccmd = rtrim(shell_exec("/bin/ls /var/etc/openvpn-csc/"));
    $csclist = preg_split('/[\s]+/', $csccmd);
    foreach ($csclist as $cscfile) {
        $file = fopen("/var/etc/openvpn-csc/$cscfile", "r");
        $iroute = "";
        $gateway = "";
        while(!feof($file)){
            $line = fgets($file);
            # look for gateway or iroute entry
            if (strpos($line, 'iroute') !== FALSE) {
                $elems = preg_split('/[\s]+/', $line);
                $iroute = $elems[1];
            }
            if (strpos($line, 'ifconfig-push') !== FALSE) {
                $elems = preg_split('/[\s]+/', $line);
                #need to fix the gateway, on this side it is .2 not .1
                $gateway = preg_replace('/.1$/', '.2', $elems[2]);
            }
        }
        fclose($file);
        if((strlen($iroute) > 0) && (strlen($gateway) > 0)) {
            $irouteval = rtrim($iroute) . "/24:" . rtrim($gateway);
            array_push ($defiroutes,$irouteval);
        }
    }
    return $defiroutes;
}

#===========================================================================
function get_iroutes($servers) {
#===========================================================================
    global $defiroutes;
    $iroutes = array();
    foreach ($servers as $server) {
        #first determine correct router file to get socket and gateway
        $filecmd = "/usr/bin/grep 'lport $server' /var/etc/openvpn/*conf | /usr/bin/awk -F: '{print $1}'";
        $cmdresult = rtrim(shell_exec($filecmd));
        $patterns[0] = '/\/var\/etc\/openvpn\//';
        $patterns[1] = '/.conf/';
        $ovpn_server = preg_replace($patterns, '', $cmdresult);

        #construct the socket and router file strings
        $socketfile = '/var/etc/openvpn/' . $ovpn_server . '.sock';
        $routerfile = '/tmp/' . $ovpn_server . '_router';
        $routerfile = str_replace('server','ovpns',$routerfile);
        $routercmd = "/bin/cat $routerfile";

        #get gateway and open socket
        $gateway = rtrim(shell_exec($routercmd));
        $socket = "unix://$socketfile";
        $errval;
        $errstr;

        #stream socket to get active iroutes
        $fp = @stream_socket_client($socket, $errval, $errstr, 1);
        if ($fp) {
            stream_set_timeout($fp, 1);
            #send our status request
            fputs($fp, "status 2\n");

            #recv all response lines
            while (!feof($fp)) {
                #read the next line
                $line = fgets($fp, 1024);
                $info = stream_get_meta_data($fp);
                if ($info['timed_out'])
                    break;

                #parse header list line
                if (strstr($line, "HEADER"))
                    continue;

                #parse end of output line
                if (strstr($line, "END") || strstr($line, "ERROR"))
                    break;

                #parse routing table lines
                if (strstr($line, "ROUTING_TABLE")) {
                    $elems = explode(",", $line);
                    $irouteval = $elems[1] . ':' . $gateway;
                    if (in_array($irouteval, $defiroutes)) {
                        array_push($iroutes,$irouteval);
                    }
                }
            }
            #cleanup
            fclose($fp);
        }
    }
    return $iroutes;
}

#===========================================================================
function kill_route($network) {
#===========================================================================
    #removes passed network from kernel routing table
    while($x <= 2) {
        $kroutecmd = "/sbin/route del -net $network";
        $cmdresult = rtrim(shell_exec($kroutecmd));
        sleep(3);
        $x++;
    }
    return;
}

#===========================================================================
function add_route($network,$gateway) {
#===========================================================================
    #adds route for network for passed gateway
    $kroutecmd = '/sbin/route add -net ' . $network . ' ' . $gateway;
    $cmdresult = rtrim(shell_exec($kroutecmd));
    return;
}

#===========================================================================
function match_route($network,$gateway) {
#===========================================================================
    #returns true if a kernel route matches passed network/gateway or false
    #get current kernel route
    $kroute = get_route($network);
    return ($gateway == $kroute[1]);
}

#===========================================================================
function get_route($network) {
#===========================================================================
    #returns current route as array
    #array[0] = network
    #array[1] = gateway
    $kroutecmd = rtrim(shell_exec("/usr/bin/netstat -rn | grep $network | tail -1"));
    if($kroutecmd) {
        $elems = preg_split('/[\s]+/', $kroutecmd);
        $retval[0] = $elems[0];
        $retval[1] = $elems[1];
    } else { 
        $retval = array('','');
    }
    return $retval;
}

#===========================================================================
function logline($var,$message) {
#===========================================================================
    $logfile = "/var/log/ovpnroutefix.log";
    if(is_array($message)) {
        $str = implode (", ", $message);
        $message = rtrim($str, ',');
    }
    $line = date('Y-m-d-H:i:s') . " [$var] " . $message . "\n";
    #self prune log if greater than 500000 bytes
    $logcmd = "/bin/ls -al /var/log/ovpnroutefix.log | awk '{print $5}'";
    $logsize = shell_exec($logcmd);
    if($logsize > 5000000) {
        file_put_contents($logfile, $line, LOCK_EX);
    } else {
        file_put_contents($logfile, $line, FILE_APPEND | LOCK_EX);
    }
}
?>

clahti · Nov 24, 2014, 5:58 PM

some additional assumptions, this scans the /var/etc/openvpn-csc file for custom client overrides. It is expecting both an ifconfig and an iroute directive in these files to work. You need both, the first pushes a "static" IP to the client so you can reference an iroute behind that interface.