Openvpn - quagga ospf - mesh



  • Hi all,

    I've been playing around a bit at work and tried to setup something like the attached drawing.
    I've stumbled upon a complication.

    For some reason quagga-ospf "publishes" the tunnel-routes to ALL members, this while i don't think i've configured it like that.
    so for example:

    • Site-C has routes for the tunnel network between A & B

    • Site-C has routes for the tunnel network between B & C

    • Site-C Allready has routes for the tunnel network between A & C - Even before you start the openvpn-service that connects A&C

    This is an issue because you cannot start an openvpn-client/server when the tunnel-network-subnet is a route allready in the routing table.
    The route in the routing table is being "pushed" from the other end of the ospf network.
    I don't see the need for ospf to publish remote tunnel-networks to each member, but i could be mistaken.

    Can someone duplicate this behavior? Is this intended behaviour ?
    Is there any way to solve this? If yes, how ? =)

    Kind Regards,

    Jeroen




  • bump


  • Rebel Alliance Developer Netgate

    Some fixes for this were put into 2.0.3 already, but you can try adding the following to your advanced options:

    persist-remote-ip; persist-local-ip;
    


  • sorry for the late responds but i was on holliday

    the persist-remote-ip nor local-ip works …
    it seems something drastically has changed in what routes quagga distributes ... i'm starting to have issues with 2.0.3 release and dual-vpn failover between two sites.
    for whatever reason the tunnel-networks are being distributed and it is screwing things up; this cause the last openvpn deamon to be unable to start because the route is allready there

    syslog:

    Apr 16 23:21:52 	kernel: ovpnc1: link state changed to DOWN
    Apr 16 23:21:52 	kernel: ifa_add_loopback_route: insertion failed
    Apr 16 23:21:52 	kernel: ovpnc1: link state changed to UP
    

    ovpnlog:

    Apr 16 23:21:52 	openvpn[7315]: Exiting
    Apr 16 23:21:52 	openvpn[7315]: FreeBSD ifconfig failed: external program exited with error status: 1
    Apr 16 23:21:52 	openvpn[7315]: /sbin/ifconfig ovpnc1 192.168.99.2 192.168.99.1 mtu 1500 netmask 255.255.255.255 up
    Apr 16 23:21:52 	openvpn[7315]: TUN/TAP device /dev/tun1 opened
    Apr 16 23:21:52 	openvpn[7315]: LZO compression initialized
    Apr 16 23:21:52 	openvpn[7315]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
    Apr 16 23:21:52 	openvpn[7315]: OpenVPN 2.2.2 amd64-portbld-freebsd8.1 [SSL] [LZO2] [eurephia] built on Apr 2 2013
    

    route is allready there, distributed by quagga from the primary openvpn link between the 2 sites.

    O>* 192.168.99.2/32 [110/100] via 192.168.88.2, ovpns2, 00:05:33
    ``` <– 192.168.88.0 being the primary ovpn-network
    
    please advice
    
    Also note that this exact setup has worked before without any problem ... this started happening after update from 2.0.2 --> 2.0.3-prerelease


  • i've looked into this a bit further and found that in the ospfd.conf file on the pfsense's the tunnel networks are there. This is unwanted and unspecified in the webgui.
    they are probably automagically by a script on the background.
    I'd like someone who is more knowledgeable then me to investigate this.

    [# This file was created by the pfSense package manager.  Do not edit!
    
    password *****
    interface ovpnc1
      ip ospf cost 100
      ip ospf dead-interval 20
    interface ovpns2
      ip ospf cost 50
      ip ospf dead-interval 20
    interface ovpns5
    interface ovpnc6
      ip ospf authentication-key ******
    
    router ospf
      ospf router-id 10.10.10.1
      network 192.168.88.0/30 area 0.0.0.1    <--tunnel-network
      network 192.168.99.0/30 area 0.0.0.1    <--tunnel-network
      network 192.168.222.0/30 area 0.0.0.1
      network 192.168.224.0/30 area 0.0.0.1
      network 10.10.10.0/24 area 0.0.0.1
      network 10.10.100.0/24 area 0.0.0.1
      network 192.168.77.0/24 area 0.0.0.1
      network 10.10.44.0/24 area 0.0.0.1
      network 192.168.1.0/24 area 0.0.0.1
    /code]
    
    



  • Rebel Alliance Developer Netgate

    The network statement is how it's indicated to quagga that it should use that interface for ospf. I'm not sure there is a way to exclude that and still have it attach to the interface properly.

    I'd love to be proven wrong though, if you figure out a way around that, I'd be happy to fix the package to work that way.


  • Rebel Alliance Developer Netgate



  • i've tried some manual editing the confs without success.

    Still this setup has worked since 2.0 beta builds up till a couple of weeks ago … the only option i currently have is to start the vpn tunnels and start quagga afterwards.

    This is very troubling because one wrong click and i have to drive 40miles to fix it :(
    I hope someone could find a way to solve this.

    Thanks in advance and kind regards.



  • @jimp
    would this patch cause any issue's on 2.0.3 ? it's 5 months old so i figured i'd better ask before trying ;)
    This appears to be exactly the thing i'm experiencing. Is there an easy fix for the last comment by Johan braeken ?

    http://redmine.pfsense.org/issues/2712

    thanks in advance


  • Rebel Alliance Developer Netgate

    I believe I'd already put that into 2.0.3.



  • Thanks for the heads up about the patch jimp.
    For whatever reason it does not seem to work in all scenarios on 2.0.3

    I've attached a new screenshot; hopefully it contains some clue's to figure out how to get this working properly.
    note that my ovpn instances have all been assigned to interfaces and that quagga binds itself to the interfaces and not the ovpn itself.

    Thanks in advance.



  • Rebel Alliance Developer Netgate

    I have an idea on how to fix this, but since I can't replicate it locally, I need a guinea pig to test the fix. Any takers?

    With the most current version of the Quagga OSPF package installed, install the System Patches package and apply this patch:

    http://files.pfsense.org/jimp/patches/quagga-tun-route-fix.patch
    Path Strip: 1
    Base: /
    Ignore Whitespace: checked

    Then edit/save the Quagga OSPF Settings.



  • patch fetch failed on webgui. When going to that link with browser i get a 403 - Forbidden ; probably needs a 6 somewhere in the chmod ;)

    i'll gladly volunteer as i am the one who keeps bothering you with this :D


  • Rebel Alliance Developer Netgate

    ok try now



  • i haven't had time to fully look into it, but i'm having some odd preliminary results.

    when enabling patch on PF1 it stops finding ANY neighbours. yet no errors in quagga webgui. Reverting the patch makes it work again.
    i'll try rebooting PF1 after closing time in the next couple of days to see if that resolves it.

    when enabling patch on PF2 everything seems to remain functional.

    I've also setup another site, lets call it PF3. I've setup a vpn to PF1 and a seperate one to PF2.
    without the patch quagga see's both members, with the patch it only see's 1 member. (not sure if this is intended behaviour or not)

    i'll investigate further when i find some spare time …


  • Rebel Alliance Developer Netgate

    What OpenVPN role do each of those have? Client? Server?

    What did the OSPF status look like on the ones that did work compared to the ones that didn't? Did it show as being attached to the interface?



  • Hi,

    @heper:

    i haven't had time to fully look into it, but i'm having some odd preliminary results.

    i'll investigate further when i find some spare time …

    we need this patch also … I tested it and found out that ospfd.conf wrote for server and client same line for OpenVPN client and server.

    network 172.16.4.1/32 area 192.168.6.0

    Normally it must be:

    • server side:
        network 172.16.4.2/32 area 192.168.6.0

    • client side:
        network 172.16.4.1/32 area 192.168.6.0

    => I think /guess there is a logical error in your network mask decision because the OpenVPN mask is /30 and not /32 in Openvpn config (but on ovpn interface later)

            if ($subnet == 32) {
    

    but I'm not sure.

    Bests


  • Rebel Alliance Developer Netgate

    The mask selection isn't the issue, there's probably a bit of a logic error in the client or server detection, I just haven't had a chance to go back and hack on it. Probably a very simple fix.



  • @jimp:

    What OpenVPN role do each of those have? Client? Server?

    What did the OSPF status look like on the ones that did work compared to the ones that didn't? Did it show as being attached to the interface?

    It seems WITH patch that only the ovpn-client neighbour is found. without patch both are found.
    for more info see pdf's attached. As noted before i do have on 1 point in my mashup of vpn's where no members are found when patch is enabled, this while there are both client as server instances.

    if you require more logs/screenshots/… please let me know



  • ok, next guess ^^

    http://blog.milkfarmsoft.com/2007/11/ip2long-32-bit64-bit/
    http://forums.phpfreaks.com/topic/254568-bitwise-not-wip2long-32-bit-vs-64-bit-os/

    still "buggy"…

    => + $baselong = ip2long32($ip) & ip2long("255.255.255.252");

    must be better
    => + $baselong = ip2long32($ip) & ip2long32("255.255.255.252");

    I try to check it later when our network is not used…

    Must it be the p2p IP/32 or could we not easy add the OpenVPN "network /30" to quagga ?


  • Rebel Alliance Developer Netgate

    It used /30 before, that's the whole problem.

    Let me know if the ip2long32 change works. I still haven't had time to go back and try it.



  • Hi,

    I modified the patch and seems that the missing (typo) 32 in the function name is the problem…
    Actually I'm running OpenVPN tunnel with quagga routing searching for my next problem ^^
    (Nagios checks won't reach/get response when routing over OpenVPN tunnel)



  • mmh, actually it won't work again…

    Perhaps because I need a "separated" OpenVPN tunnel which is bound to an interface (opt12) ...
    We have OpenVPN client networks... and one bridge network and want allow all traffic going between the (transfer) bridge network but not from/to the client networks...


  • Rebel Alliance Developer Netgate

    ok I fixed the line in the patch to have ip2long32. I'm not sure about your other question, that may need its own thread since it may not be related to the problem that initiated this discussion.



  • Hi yes, could be perhaps another thread/problem…
    Only the idea / nice to have that this patch can fix both scenarios if possible ;)

    I tested before and got stable OpenVPN tunnel... with quagga routing/traffic.

    Then I setup like this to filter traffic separately per tunnel:
    http://doc.pfsense.org/index.php/Can_I_filter_OpenVPN_traffic%3F#Assign_OpenVPN_interface_as_OPT

    but now get wrong IP in server back and quagga is disables on my server interface again:
      network 172.16.4.1/32 area 192.168.6.0



  • @reiner030 so you have found that it works on a tunnel but stops working when you assign an interface to the tunnel ?

    I just confirmed that members are can only be found on ovpn-client side of the tunnel.

    situation:

    ovpns_A  –-----interwebs--------- ovpnc_A
    ovpnc_B  -------interwebs--------- ovpns_B

    if patch is enabled on both ends: no members found by "left" nor "right"
    if patch is enabled on the "left" side but not the "right": 1 member is found on the B-connection
    if patch is enabled on the "right" side but not the "left": 1 member is found on the A-connection

    do note that all vpn's have been assigned an interface and quagga is bound on the interfaces

    hope this makes sense somehow ;)



  • mmh, not sure ..
    Yesterday it works for me but perhaps only because my server side was not restarted and my manually fix in the quagga config was still available.
    Ssince yesterday evening it won't work again as written so it could be the "old" problem and not my bridge try…

    Make sense when taking a look onto interfaces... the OpenVPN tunnel => Interface assigning makes no difference to my interface definition itself:

    ovpns5: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
    options=80000 <linkstate>inet6 fe80::225:90ff:fe26:1f22%ovpns5 prefixlen 64 scopeid 0x6e
    inet 172.16.4.1 –> 172.16.4.2 netmask 0xffffffff
    nd6 options=43 <performnud,accept_rtadv>Opened by PID 3002

    so perhaps for 1st fix it's help to apply the patch only to client side ? I try it at evening again…</performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast>



  • Hello - i think i have same problem (http://forum.pfsense.org/index.php?topic=60942.new;topicseen#new) like in this tread - i try to do failover OPENVPN site-to-site tunnel.
    I found some instructions on the forum - for setting up the VPN failover scheme. And no one else had similar problems - so that Openvpn demo fails to start- why not all users have this problem?
    I tried to apply patches - it did not help me. Is there a 100% solution to this problem?
    I use pfsense 2.0.2
    thanks.



  • @tomas1

    no solution at the moment. i hope the developers find some spare time to look into this further.

    it has worked in the past.
    2.0.1 with openospf worked for sure ; 2.0.1 in a mixed environment with quagga & openospf also worked.
    around the time i've updated my site's to 2.0.2 - ALL - with quagga i've started to notice some odd behaviour but didn't have the time to worry about it.



  • Hi,

    our setup is working this way…

    @Reiner030:

    so perhaps for 1st fix it's help to apply the patch only to client side ? I try it at evening again…

    server side without patch:
    [2.0.3-RELEASE][root@fw1.jws1.local]/root(71): ifconfig ovpns5
    ovpns5: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
    options=80000 <linkstate>inet6 fe80::225:90ff:fe26:1f22%ovpns5 prefixlen 64 scopeid 0x6e
    inet 172.16.4.1 –> 172.16.4.2 netmask 0xffffffff
    nd6 options=43 <performnud,accept_rtadv>Opened by PID 57530
    [2.0.3-RELEASE][root@fw1.jws1.local]/root(72): grep 172.16 /var/etc/quagga/ospfd.conf
      network 172.16.4.0/30 area 192.168.6.0
      access-list dnr-list deny 172.16.4.0/24

    client side with patch:
    [2.1-BETA1][root@fw1.zws8.local]/root(1): ifconfig ovpnc1
    ovpnc1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
    options=80000 <linkstate>inet6 fe80::225:90ff:fe7a:c489%ovpnc1 prefixlen 64 scopeid 0x44
    inet 172.16.4.2 –> 172.16.4.1 netmask 0xffffffff
    nd6 options=1 <performnud>Opened by PID 30259
    [2.1-BETA1][root@fw1.zws8.local]/root(2): grep 172.16 /var/etc/quagga/ospfd.conf
      network 172.16.4.1/32 area 192.168.6.0
      access-list dnr-list deny 172.16.4.0/24

    so I guess that on both sides it would be better to have not this patch part:

    +function quagga_fix_ovpn_peer_ip(&$ip, &$subnet) {
    +	if ($subnet == 32) {
    +		$baselong = ip2long32($ip) & ip2long32("255.255.255.252");
    +		$ip1 = long2ip32($baselong + 1);
    +		$ip2 = long2ip32($baselong + 2);
    +		if (substr($realif, 4, 1) == "c") {
    +			$ip = $ip2;
    +		} else {
    +			$ip = $ip1;
    +		}
    +	}
    +}
    +
    

    but this one:

    
    +function quagga_fix_ovpn_peer_ip(&$ip, &$subnet) {
    +	if ($subnet == 32) {
    +		$ip = ip2long32($ip) & ip2long32("255.255.255.252");
    +	}
    +}
    +
    

    which can be easily embedded in above if-then-else construct instead of own function if it works ;)</performnud></linkstate></up,pointopoint,running,multicast></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast>



  • Hi Eveyone,
    I will be setting something like this up very soon. (Two sites both with dual WAN/Carp)

    Is this problem fixed with or without the patch?
    thanks in advance.
    vito



  • for me it's still working with patch on client side, no patch on server side…  and since I saw no update for the package itself...



  • it works for redundant vpn's between 2 sites.

    it does not work when making a "triangle" between 3 sites.
    I still hope someday some brilliant mind will come up with a solution for  this ;)


  • Rebel Alliance Developer Netgate

    Quagga works for doing many sites. I've seen some with half a dozen or more sites on the same "area" doing redundant OSPF+OpenVPN.



  • I figure the rush to 2.1 is over now and am wondering if there are plans to
    improve the way quagga behaves. As it still distributes tunnel networks this causes issues with mesh'd sites.



  • for completion of this thread/documentation:

    1. final pfSense 2.1 works fine without the Quagga OSPF Patch offered by Jimp in this threat.
    2. Quagga OSPF recognizes its neighbour over OpenVPN only if you use a peer2peer network /30.
          No idea why for instance a /24 won't work (I see on both sides HELO packages, but no Quagga OSPF response).

    Bests


  • Rebel Alliance Developer Netgate

    If you want to use a /24 you can, but it requires using TAP.

    Using a /24 with topology subnet appears to work but for some reason … doesn't. Switching to tap it works fine.

    The tunnel networks being distributed is fixed in the newest version of the Quagga package, either check the "accept filter" button on the tunnel interfaces, or add them manually to the main list with 'accept filter' checked.



  • Jimp,
    I still see this problem, even with the "Accept Filter" checked.
    The routes don't show up in the "Quagga OSPF Database" section anymore, but they absolutely still show up in the "Quagga OSPF Router Database", and in the "Quagga OSPF Routes".
    This means they are still distributed to the other OSPF clients, at which point, if you have a "triangle" topology, OpenVPN breaks on one leg during ifconfig because it sees a route to the other end of its private tunnel.



  • @althornin:

    Jimp,
    I still see this problem, even with the "Accept Filter" checked.
    The routes don't show up in the "Quagga OSPF Database" section anymore, but they absolutely still show up in the "Quagga OSPF Router Database", and in the "Quagga OSPF Routes".
    This means they are still distributed to the other OSPF clients, at which point, if you have a "triangle" topology, OpenVPN breaks on one leg during ifconfig because it sees a route to the other end of its private tunnel.

    you can sort of fix it by manually adding the tunnel IP to the "disable acceptance" list in the global-settings tab of the quagga service.

    for example you have a tunnel net of 192.168.22.1/24:
    on the server end add to "subnet to route': 192.168.22.1/32  (check disable acceptance)
    on the client end add to "subnet to route': 192.168.22.2/32  (check disable acceptance)

    do this for all the tunnel subnets on both ends and you won't have the problems with ifconfig.
    i know it's a hassle, but its the only way i know, to get the job done

    enjoy


  • Rebel Alliance Developer Netgate

    If you keep all of your tunnel networks in a close range you can add a manual accept filter for the entire larger subnet which includes the smaller tunnel networks. For example if you have 192.168.22.0/30, 192.168.22.4/30, 192.168.22.8/30 and so on for tunnel networks, then you can setup an accept filter for 192.168.22.0/24 and I believe that should work OK.


Log in to reply