OpenOSPF + PFsense 2, OSPF interface route gets deleted (w/ workaround)



  • Anyone have some insight?
    I would appreciate any and all comments!

    What's happening is if I reboot PF2, the network used to negotiate OSPF stuff (192.168.1.0/28) gets deleted (the route gets deleted).
    When PF2 is finished booting back up, there is an exchange of OSPF information, but it either fails or loops (pasted below).
    All OSPF summary information is taken from PF1.


    I created 2 PFsense systems w/ the following configuration:

    |

    PF1:

    WAN:
    Interface: em0_vlan2056
    IP: 10.197.252.1/16
    Gateway: 10.197.254.254
    Description: Internet access, Web management

    LAN:
    Interface: em0
    IP: 192.168.1.1/28
    Description: OSPF Negotiations

    OPT1:
    Interface: em0_vlan11
    IP: 192.168.11.1/24
    Description: Test network for OSPF

    |

    PF2:

    WAN:
    Interface: em0_vlan2056
    IP: 10.197.252.2/16
    Gateway: 10.197.254.254
    Description: Internet access, Web management

    LAN:
    Interface: em0
    IP: 192.168.1.2/28
    Description: OSPF Negotiations

    OPT1:
    Interface: em0_vlan12
    IP: 192.168.12.1/24
    Description: Test network for OSPF

    |


    PF1 information when PF1 and PF2 are turned on fresh.
    OSPF looks good, … although if you compare the kernel routing table to what OpenBSDs looks like, you can see that OSPF is replacing/clobbering the 192.168.1.0/24 route to the physical NIC with a route to the adjacent PFsense machine (192.168.1.2 in this case).

    |

    OpenOSPFd Summary
    Router ID: 0.0.0.1
    Uptime: 00:01:14
    RFC1583 compatibility flag is disabled
    SPF delay is 1 sec(s), hold time between two SPFs is 5 sec(s)
    Number of external LSA(s) 4
    Number of areas attached to this router: 1

    Area ID: 0.0.0.0
     Number of interfaces in this area: 1
     Number of fully adjacent neighbors in this area: 1
     SPF algorithm executed 3 time(s)
     Number LSA(s) 3

    OpenOSPFd Neighbors
    ID              Pri State        DeadTime Address         Iface     Uptime
    0.0.0.2         1   FULL/DR      00:00:39 192.168.1.2     em0       00:00:30

    OpenOSPFd FIB
    flags: * = valid, O = OSPF, C = Connected, S = Static
    Flags  Prio Destination          Nexthop
    *S        0 0.0.0.0/0            10.197.254.254
    *C        0 10.197.0.0/16        link#7
    *C        0 10.197.252.1/32      link#3
    *C        0 127.0.0.1/8          link#0
    *C        0 127.0.0.1/32         link#3
    *C        0 192.168.1.0/28       link#1
    *O       16 192.168.1.0/28       192.168.1.1
    *C        0 192.168.1.1/32       link#3
    *C        0 192.168.12.0/24      link#9
    *C        0 192.168.12.1/32      link#3
    *O       16 192.168.13.0/24      192.168.1.2
    *O       16 192.168.13.1/32      192.168.1.2

    OpenOSPFd RIB
    Destination          Nexthop           Path Type    Type      Cost    Uptime
    0.0.0.2              192.168.1.2       Intra-Area   Router    10      00:00:22
    192.168.1.0/28       192.168.1.1       Intra-Area   Network   10      00:00:29
    192.168.13.0/24      192.168.1.2       Type 1 ext   Network   110     00:00:22
    192.168.13.1/32      192.168.1.2       Type 1 ext   Network   110     00:00:22

    OpenOSPFd Interfaces
    Interface   Address            State  HelloTimer Linkstate  Uptime    nc  ac
    em0         192.168.1.1/28     BCKUP  00:00:09   active     00:01:14   1   1

    OpenOSPFD Database
                   Router Link States (Area 0.0.0.0)

    Link ID         Adv Router      Age  Seq#       Checksum
    0.0.0.1         0.0.0.1         30   0x80000002 0x5716
    0.0.0.2         0.0.0.2         31   0x80000002 0x5515

    Net Link States (Area 0.0.0.0)

    Link ID         Adv Router      Age  Seq#       Checksum
    192.168.1.2     0.0.0.2         31   0x80000001 0x7087

    Type-5 AS External Link States

    Link ID         Adv Router      Age  Seq#       Checksum
    192.168.12.0    0.0.0.1         74   0x80000001 0x2d4d
    192.168.12.1    0.0.0.1         74   0x80000001 0x2356
    192.168.13.0    0.0.0.2         71   0x80000001 0x1c5c
    192.168.13.1    0.0.0.2         71   0x80000001 0x1265

    |
    |

    $ netstat -rn
    Routing tables

    Internet:
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            10.197.254.254     UGS         2      237 em0_vl
    10.197.0.0/16      link#7             U           0        0 em0_vl
    10.197.252.1       link#7             UHS         0        0    lo0
    127.0.0.1          link#3             UH          0      139    lo0
    192.168.1.0/28     192.168.1.1        U           0        7    em0
    192.168.1.1        link#1             UHS         0        0    lo0
    192.168.12.0/24    link#9             U           0        0 em0_vl
    192.168.12.1       link#9             UHS         0        0    lo0
    192.168.13.0/24    192.168.1.2        UG          0        0    em0
    192.168.13.1/32    192.168.1.2        UG          0        0    em0

    |


    PF1 information after PF2 was rebooted and I waited a few minutes.
    Take special note of the kernel routing table (notice how 192.168.1.0/24 is missing).

    |

    OpenOSPFd Summary
    Router ID: 0.0.0.1
    Uptime: 00:18:47
    RFC1583 compatibility flag is disabled
    SPF delay is 1 sec(s), hold time between two SPFs is 5 sec(s)
    Number of external LSA(s) 2
    Number of areas attached to this router: 1

    Area ID: 0.0.0.0
     Number of interfaces in this area: 1
     Number of fully adjacent neighbors in this area: 0
     SPF algorithm executed 1 time(s)
     Number LSA(s) 1

    OpenOSPFd Neighbors
    ID              Pri State        DeadTime Address         Iface     Uptime
    0.0.0.2         1   EXSTA/BCKUP  00:00:36 192.168.1.2     em0       -

    OpenOSPFd FIB
    flags: * = valid, O = OSPF, C = Connected, S = Static
    Flags  Prio Destination          Nexthop
    *S        0 0.0.0.0/0            10.197.254.254
    *C        0 10.197.0.0/16        link#7
    *C        0 10.197.252.1/32      link#3
    *C        0 127.0.0.1/8          link#0
    *C        0 127.0.0.1/32         link#3
    *C        0 192.168.1.1/32       link#3
    *C        0 192.168.12.0/24      link#9
    *C        0 192.168.12.1/32      link#3

    OpenOSPFd RIB
    Destination          Nexthop           Path Type    Type      Cost    Uptime

    OpenOSPFd Interfaces
    Interface   Address            State  HelloTimer Linkstate  Uptime    nc  ac
    em0         192.168.1.1/28     DR     00:00:06   active     00:18:47   1   1

    OpenOSPFD Database
                   Router Link States (Area 0.0.0.0)

    Link ID         Adv Router      Age  Seq#       Checksum
    0.0.0.1         0.0.0.1         1127 0x80000001 0x36b3

    Type-5 AS External Link States

    Link ID         Adv Router      Age  Seq#       Checksum
    192.168.12.0    0.0.0.1         1127 0x80000001 0x2d4d
    192.168.12.1    0.0.0.1         1127 0x80000001 0x2356

    |
    |

    $ netstat -rn
    Routing tables

    Internet:
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            10.197.254.254     UGS         3      938 em0_vl
    10.197.0.0/16      link#7             U           0        0 em0_vl
    10.197.252.1       link#7             UHS         0        0    lo0
    127.0.0.1          link#3             UH          0      139    lo0
    192.168.1.1        link#1             UHS         0        0    lo0
    192.168.12.0/24    link#9             U           0        0 em0_vl
    192.168.12.1       link#9             UHS         0        0    lo0

    |

    Here is a bit of ospfd -v -d after PF2 is up after having been rebooted (this just loops forever),
    recv_db_description: dupe from ID 0.0.0.2
    recv_db_description: seq num mismatch, bad flags
    nbr_fsm: event SEQ_NUM_MISMATCH resulted in action RESET_DD and changing state for neighbor ID 0.0.0.2 from EXCHG to EXSTA
    nbr_fsm: event NEGOTIATION_DONE resulted in action SNAPSHOT and changing state for neighbor ID 0.0.0.2 from EXSTA to SNAP
    nbr_fsm: event SNAPSHOT_DONE resulted in action SNAPSHOT_DONE and changing state for neighbor ID 0.0.0.2 from SNAP to EXCHG
    recv_db_description: dupe from ID 0.0.0.2

    And sometimes you'll see the "failed to form adjacency" (if PF1 is starting fresh and PF2 is missing its route (related to router priority?)).

    And this is the ospfd.conf file taken from PF1,

    [2.0-RC3][admin@pfospf1.localdomain]/usr/local/etc(2): cat ospfd.conf

    This file was created by the pfSense package manager.  Do not edit!

    router-id 0.0.0.1
    no redistribute 10.197.0.0/16
    no redistribute 192.168.1.0/28 # does this even do anything???
    redistribute connected
    redistribute static
    area 0.0.0.0 {
           interface em0
    }

    • changed the vLAN id to 12 for PF2 (typo)


  • What pfSense version is this?



  • Version is:

    2.0-RC3 (i386)
    built on Tue Aug 16 07:25:43 EDT 2011

    Thanks!



  • Just as an observation that I would like to share that shows a difference between OpenBSD and FreeBSD and their respective kernel routing table,

    This is one of my OpenBSD test machines.
    It's running OpenOSPF w/o any issues.
    This is the routing table when everything is running,

    Routing tables

    Internet:
    Destination        Gateway            Flags  Refs      Use  Mtu  Prio Iface
    default            10.197.254.254    UGS        0      33    -    8 vlan0
    10/8              link#4            UC        2        0    -    4 vlan0
    10.197/16          192.168.11.2      UG        0        0    -    32 vlan1
    10.197.254.253    00:21:9b:fc:03:3e  UHLc      1    1518    -    4 vlan0
    10.197.254.254    link#4            UHLc      1        0    -    4 vlan0
    127/8              127.0.0.1          UGRS      0        0 33160    8 lo0
    127.0.0.1          127.0.0.1          UH        1      557 33160    4 lo0
    192.168.11/24      link#5            UC        3        0    -    4 vlan1
    192.168.11/24      192.168.11.3      UGSP      0        0    -    8 vlan1
    192.168.11/24      192.168.11.1      UG        0        0    -    32 vlan1
    192.168.11.1      00:50:56:96:00:89  UHLc      1        0    -    4 lo0
    192.168.11.2      00:50:56:96:00:90  UHLc      3    99241    -    4 vlan1
    192.168.11.3      link#5            UHLc      1        0    -    4 vlan1
    192.168.12/24      link#6            UC        0        0    -    4 vlan2
    192.168.13/24      192.168.11.2      UG        0        0    -    32 vlan1
    224/4              127.0.0.1          URS        0        0 33160    8 lo0

    Notice how there are multiple entries for 192.168.11.0/24?
    I can even add more; here's an example:

    route add -net 192.168.11/24 192.168.11.5

    add net 192.168.11/24: gateway 192.168.11.5

    That route gets added to the routing table w/o errors.
    In FreeBSD, this would error out with the message, "route already in table".

    Just an observation that I wanted to share. I don't understand their respective network stacks to make any claim on how it should or should not work, but I thought it was interesting, none-the-less.



  • Try using no redistribute local



  • @ermal:

    Try using no redistribute local

    ospfd -v -d -f ospfd.conf

    ospfd.conf:8: unknown redistribute type
    em0 has index 1

    I tried adding "no distribute local" (which isn't in any of the documentation), and that's the error message I got.
    Do you mean to ask what happens if I don't distribute connected and static routes?
    If so, it's the same result.



  • Workaround To This Issue

    Before I created the original post, I found a way to get OpenOSPF working between PFsense machines that I will now share.
    I'm not happy about having to come up with a workaround, and would love any insight from others who are using FreeBSD/PFsense w/ OpenOSPF, and what their experiences are like.

    In the above example, 192.168.1.0/28 is used by PF1 and PF2 to exchange OSPF information.
    Create an IP alias where the network overlaps the existing IP+network.
    For example, on PF1, I created an IP alias of 192.168.1.11/24.
    Ensure that 192.168.1.0/24 is not distributed (no redistribute 192.168.1.0/24).
    OpenOSPF will only kill the 192.168.1.0/28 route, and the machines will still be able to negotiate correctly since they'll be able to talk over 192.168.1.0/24 when 192.168.1.0/28 is gone.



  • Because I believe this problem doesn't really have a lot to do w/ PFsense, I created a blog post about this specific issue.
    I would still love for any others to share their experiences with OpenOSPF.

    http://ouliakk.blogspot.com/2011/08/using-openospfd-with-freebsd-78.html


Locked