OpenOSPF + PFsense 2, OSPF interface route gets deleted (w/ workaround)

dsrepel

Anyone have some insight?
I would appreciate any and all comments!

What's happening is if I reboot PF2, the network used to negotiate OSPF stuff (192.168.1.0/28) gets deleted (the route gets deleted).
When PF2 is finished booting back up, there is an exchange of OSPF information, but it either fails or loops (pasted below).
All OSPF summary information is taken from PF1.

I created 2 PFsense systems w/ the following configuration:

|

PF1:

WAN:
Interface: em0_vlan2056
IP: 10.197.252.1/16
Gateway: 10.197.254.254
Description: Internet access, Web management

LAN:
Interface: em0
IP: 192.168.1.1/28
Description: OSPF Negotiations

OPT1:
Interface: em0_vlan11
IP: 192.168.11.1/24
Description: Test network for OSPF

|

PF2:

WAN:
Interface: em0_vlan2056
IP: 10.197.252.2/16
Gateway: 10.197.254.254
Description: Internet access, Web management

LAN:
Interface: em0
IP: 192.168.1.2/28
Description: OSPF Negotiations

OPT1:
Interface: em0_vlan12
IP: 192.168.12.1/24
Description: Test network for OSPF

|

PF1 information when PF1 and PF2 are turned on fresh.
OSPF looks good, … although if you compare the kernel routing table to what OpenBSDs looks like, you can see that OSPF is replacing/clobbering the 192.168.1.0/24 route to the physical NIC with a route to the adjacent PFsense machine (192.168.1.2 in this case).

|

OpenOSPFd Summary
Router ID: 0.0.0.1
Uptime: 00:01:14
RFC1583 compatibility flag is disabled
SPF delay is 1 sec(s), hold time between two SPFs is 5 sec(s)
Number of external LSA(s) 4
Number of areas attached to this router: 1

Area ID: 0.0.0.0
Number of interfaces in this area: 1
Number of fully adjacent neighbors in this area: 1
SPF algorithm executed 3 time(s)
Number LSA(s) 3

OpenOSPFd Neighbors
ID Pri State DeadTime Address Iface Uptime
0.0.0.2 1 FULL/DR 00:00:39 192.168.1.2 em0 00:00:30

OpenOSPFd FIB
flags: * = valid, O = OSPF, C = Connected, S = Static
Flags Prio Destination Nexthop
*S 0 0.0.0.0/0 10.197.254.254
*C 0 10.197.0.0/16 link#7
*C 0 10.197.252.1/32 link#3
*C 0 127.0.0.1/8 link#0
*C 0 127.0.0.1/32 link#3
*C 0 192.168.1.0/28 link#1
*O 16 192.168.1.0/28 192.168.1.1
*C 0 192.168.1.1/32 link#3
*C 0 192.168.12.0/24 link#9
*C 0 192.168.12.1/32 link#3
*O 16 192.168.13.0/24 192.168.1.2
*O 16 192.168.13.1/32 192.168.1.2

OpenOSPFd RIB
Destination Nexthop Path Type Type Cost Uptime
0.0.0.2 192.168.1.2 Intra-Area Router 10 00:00:22
192.168.1.0/28 192.168.1.1 Intra-Area Network 10 00:00:29
192.168.13.0/24 192.168.1.2 Type 1 ext Network 110 00:00:22
192.168.13.1/32 192.168.1.2 Type 1 ext Network 110 00:00:22

OpenOSPFd Interfaces
Interface Address State HelloTimer Linkstate Uptime nc ac
em0 192.168.1.1/28 BCKUP 00:00:09 active 00:01:14 1 1

OpenOSPFD Database
Router Link States (Area 0.0.0.0)

Link ID Adv Router Age Seq# Checksum
0.0.0.1 0.0.0.1 30 0x80000002 0x5716
0.0.0.2 0.0.0.2 31 0x80000002 0x5515

Net Link States (Area 0.0.0.0)

Link ID Adv Router Age Seq# Checksum
192.168.1.2 0.0.0.2 31 0x80000001 0x7087

Type-5 AS External Link States

Link ID Adv Router Age Seq# Checksum
192.168.12.0 0.0.0.1 74 0x80000001 0x2d4d
192.168.12.1 0.0.0.1 74 0x80000001 0x2356
192.168.13.0 0.0.0.2 71 0x80000001 0x1c5c
192.168.13.1 0.0.0.2 71 0x80000001 0x1265

|
|

$ netstat -rn
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.197.254.254 UGS 2 237 em0_vl
10.197.0.0/16 link#7 U 0 0 em0_vl
10.197.252.1 link#7 UHS 0 0 lo0
127.0.0.1 link#3 UH 0 139 lo0
192.168.1.0/28 192.168.1.1 U 0 7 em0
192.168.1.1 link#1 UHS 0 0 lo0
192.168.12.0/24 link#9 U 0 0 em0_vl
192.168.12.1 link#9 UHS 0 0 lo0
192.168.13.0/24 192.168.1.2 UG 0 0 em0
192.168.13.1/32 192.168.1.2 UG 0 0 em0

|

PF1 information after PF2 was rebooted and I waited a few minutes.
Take special note of the kernel routing table (notice how 192.168.1.0/24 is missing).

|

OpenOSPFd Summary
Router ID: 0.0.0.1
Uptime: 00:18:47
RFC1583 compatibility flag is disabled
SPF delay is 1 sec(s), hold time between two SPFs is 5 sec(s)
Number of external LSA(s) 2
Number of areas attached to this router: 1

Area ID: 0.0.0.0
Number of interfaces in this area: 1
Number of fully adjacent neighbors in this area: 0
SPF algorithm executed 1 time(s)
Number LSA(s) 1

OpenOSPFd Neighbors
ID Pri State DeadTime Address Iface Uptime
0.0.0.2 1 EXSTA/BCKUP 00:00:36 192.168.1.2 em0 -

OpenOSPFd FIB
flags: * = valid, O = OSPF, C = Connected, S = Static
Flags Prio Destination Nexthop
*S 0 0.0.0.0/0 10.197.254.254
*C 0 10.197.0.0/16 link#7
*C 0 10.197.252.1/32 link#3
*C 0 127.0.0.1/8 link#0
*C 0 127.0.0.1/32 link#3
*C 0 192.168.1.1/32 link#3
*C 0 192.168.12.0/24 link#9
*C 0 192.168.12.1/32 link#3

OpenOSPFd RIB
Destination Nexthop Path Type Type Cost Uptime

OpenOSPFd Interfaces
Interface Address State HelloTimer Linkstate Uptime nc ac
em0 192.168.1.1/28 DR 00:00:06 active 00:18:47 1 1

OpenOSPFD Database
Router Link States (Area 0.0.0.0)

Link ID Adv Router Age Seq# Checksum
0.0.0.1 0.0.0.1 1127 0x80000001 0x36b3

Type-5 AS External Link States

Link ID Adv Router Age Seq# Checksum
192.168.12.0 0.0.0.1 1127 0x80000001 0x2d4d
192.168.12.1 0.0.0.1 1127 0x80000001 0x2356

|
|

$ netstat -rn
Routing tables

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.197.254.254 UGS 3 938 em0_vl
10.197.0.0/16 link#7 U 0 0 em0_vl
10.197.252.1 link#7 UHS 0 0 lo0
127.0.0.1 link#3 UH 0 139 lo0
192.168.1.1 link#1 UHS 0 0 lo0
192.168.12.0/24 link#9 U 0 0 em0_vl
192.168.12.1 link#9 UHS 0 0 lo0

|

Here is a bit of ospfd -v -d after PF2 is up after having been rebooted (this just loops forever),
recv_db_description: dupe from ID 0.0.0.2
recv_db_description: seq num mismatch, bad flags
nbr_fsm: event SEQ_NUM_MISMATCH resulted in action RESET_DD and changing state for neighbor ID 0.0.0.2 from EXCHG to EXSTA
nbr_fsm: event NEGOTIATION_DONE resulted in action SNAPSHOT and changing state for neighbor ID 0.0.0.2 from EXSTA to SNAP
nbr_fsm: event SNAPSHOT_DONE resulted in action SNAPSHOT_DONE and changing state for neighbor ID 0.0.0.2 from SNAP to EXCHG
recv_db_description: dupe from ID 0.0.0.2

And sometimes you'll see the "failed to form adjacency" (if PF1 is starting fresh and PF2 is missing its route (related to router priority?)).

And this is the ospfd.conf file taken from PF1,

[2.0-RC3][admin@pfospf1.localdomain]/usr/local/etc(2): cat ospfd.conf

This file was created by the pfSense package manager. Do not edit!

router-id 0.0.0.1
no redistribute 10.197.0.0/16
no redistribute 192.168.1.0/28 # does this even do anything???
redistribute connected
redistribute static
area 0.0.0.0 {
interface em0
}

changed the vLAN id to 12 for PF2 (typo)

eri--

What pfSense version is this?

dsrepel

Version is:

2.0-RC3 (i386)
built on Tue Aug 16 07:25:43 EDT 2011

Thanks!

dsrepel

Just as an observation that I would like to share that shows a difference between OpenBSD and FreeBSD and their respective kernel routing table,

This is one of my OpenBSD test machines.
It's running OpenOSPF w/o any issues.
This is the routing table when everything is running,

Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 10.197.254.254 UGS 0 33 - 8 vlan0
10/8 link#4 UC 2 0 - 4 vlan0
10.197/16 192.168.11.2 UG 0 0 - 32 vlan1
10.197.254.253 00:21:9b:fc:03:3e UHLc 1 1518 - 4 vlan0
10.197.254.254 link#4 UHLc 1 0 - 4 vlan0
127/8 127.0.0.1 UGRS 0 0 33160 8 lo0
127.0.0.1 127.0.0.1 UH 1 557 33160 4 lo0
192.168.11/24 link#5 UC 3 0 - 4 vlan1
192.168.11/24 192.168.11.3 UGSP 0 0 - 8 vlan1
192.168.11/24 192.168.11.1 UG 0 0 - 32 vlan1
192.168.11.1 00:50:56:96:00:89 UHLc 1 0 - 4 lo0
192.168.11.2 00:50:56:96:00:90 UHLc 3 99241 - 4 vlan1
192.168.11.3 link#5 UHLc 1 0 - 4 vlan1
192.168.12/24 link#6 UC 0 0 - 4 vlan2
192.168.13/24 192.168.11.2 UG 0 0 - 32 vlan1
224/4 127.0.0.1 URS 0 0 33160 8 lo0

Notice how there are multiple entries for 192.168.11.0/24?
I can even add more; here's an example:

route add -net 192.168.11/24 192.168.11.5

add net 192.168.11/24: gateway 192.168.11.5

That route gets added to the routing table w/o errors.
In FreeBSD, this would error out with the message, "route already in table".

Just an observation that I wanted to share. I don't understand their respective network stacks to make any claim on how it should or should not work, but I thought it was interesting, none-the-less.

eri--

Try using no redistribute local

dsrepel

@ermal:

Try using no redistribute local

ospfd -v -d -f ospfd.conf

ospfd.conf:8: unknown redistribute type
em0 has index 1

I tried adding "no distribute local" (which isn't in any of the documentation), and that's the error message I got.
Do you mean to ask what happens if I don't distribute connected and static routes?
If so, it's the same result.

dsrepel

Workaround To This Issue

Before I created the original post, I found a way to get OpenOSPF working between PFsense machines that I will now share.
I'm not happy about having to come up with a workaround, and would love any insight from others who are using FreeBSD/PFsense w/ OpenOSPF, and what their experiences are like.

In the above example, 192.168.1.0/28 is used by PF1 and PF2 to exchange OSPF information.
Create an IP alias where the network overlaps the existing IP+network.
For example, on PF1, I created an IP alias of 192.168.1.11/24.
Ensure that 192.168.1.0/24 is not distributed (no redistribute 192.168.1.0/24).
OpenOSPF will only kill the 192.168.1.0/28 route, and the machines will still be able to negotiate correctly since they'll be able to talk over 192.168.1.0/24 when 192.168.1.0/28 is gone.

dsrepel

Because I believe this problem doesn't really have a lot to do w/ PFsense, I created a blog post about this specific issue.
I would still love for any others to share their experiences with OpenOSPF.

http://ouliakk.blogspot.com/2011/08/using-openospfd-with-freebsd-78.html