Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Kernel routing table grows and finaly crash on PFSense 2.3.2 with OSPF 0.99.24.1

    Scheduled Pinned Locked Moved pfSense Packages
    3 Posts 2 Posters 820 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • G
      greenitsolutions
      last edited by

      Hi all,

      We installed three months ago 2 PFSense (Hardware, Supermicro MB, Xeon E5, RAM 16 Go, SSD) as Core Firewalls in DC.

      In Master / Slave mode, all works fine.
      But in Master / Master mode (asymetric full BGP routing), we have routing holes in and out approx. each hour with kernel routing tables flushs and / or growing inexpectidly.

      Hope that some one in this forum can help us to diagnose and help resolving our problem :

      • an BGP / OSPF misconfiguration in Full BGP routing mode ?
      • a Quagga misconfiguration on PFSense ?
      • a bug ?

      1°) Topology****with OSPF Areas

      +–-----------+                                          +-------------+
        |            |                                          |            |
        | ISP1 ROUTER |                                          | ISP2 ROUTER |
        |            |                                          |            |
        +------+------+                                          +------+------+
              |                                                        |
              |                                                        |
              |                                                        |
              |                                                        |
      +--------+--------+                                      +--------+--------+
      |                |                                      |                |
      |  RBGP1 (CISCO)  |                                      |  RBGP2 (CISCO)  |
      |                |                                      |                |
      +-----------------+                                      +-----------------+
              |Gi0/0/2                                                |Gi0/0/2
              |                                                        |
              |                                                        |
              |                                                        |
      OSPF AREA 0.0.0.1                                        OSPF AREA 0.0.0.2
              |                                                        |
              |                                                        |
              |                                                        |
              |IGB0                                                    |IGB0
      +-----------------+                                      +-----------------+
      |                |RE1                                  |                |
      | FWN11 (PFSENSE) +----------OSPF AREA 0.0.0.0-----------+ FWN12 (PFSENSE) |
      |                |                                  RE1|                |
      +-----------------+                                      +-----------------+
              |IGB1                                                    |IGB1
              |                                                        |
              |                                                        |
              |                                                        |
      OSPF AREA 0.0.0.3                                        OSPF AREA 0.0.0.3

      2°) Topology with IP Adresses (Not real @IPs)

      +–-----------+                                          +-------------+
        |            |                                          |            |
        | ISP1 ROUTER |                                          | ISP2 ROUTER |
        |            |                                          |            |
        +------+------+                                          +------+------+
              |                                                        |
              |                                                        |
              |                                                        |
              |                                                        |
      +--------+--------+                                      +--------+--------+
      |                |                                      |                |
      |  RBGP1 (CISCO)  |                                      |  RBGP2 (CISCO)  |
      |                |                                      |                |
      +--------+--------+                                      +--------+--------+
              |                                                        |
              |Gi0/0/2                                                |Gi0/0/2
              |125.132.100.25/29                                      |125.132.100.33/29
              |                                                        |
              |                                                        |
              |                                                        |
              |IGB0                                                    |IGB0
              |125.132.100.26/29                                      |125.132.100.34/29
              |                                                        |
      +--------+--------+RE1                                  +--------+--------+
      |                |125.132.100.17/30                    |                |
      | FWN11 (PFSENSE) +--------------------------------------+ FWN12 (PFSENSE) |
      |                |                                  RE1|                |
      +--------+--------+                    125.132.100.18/30+--------+--------+
              |                                                        |
              |IGB1                                                    |IGB1
              |125.132.100.252/25                                      |125.132.100.253/25
              |                                                        |
              |          +-----------------------------------+        |
              |          | CARP IP  : 125.132.100.254/25    |        |
              +----------+ IP ALIAS : 125.132.102.254/24    +---------+
                          | IP ALIAS : 125.132.103.254/24    |
                          +-+---------------+---------------+-+
                            |              |              |
                            |              |              |
                            |              |              |
                      +-----+---+      +----+----+      +---+-----+
                      |        |      |        |      |        |
                      | HOST 01 |      | HOST 02 |      | HOST 02 |
                      |        |      |        |      |        |
                      +---------+      +---------+      +---------+

      3°) FWN11 PFSense Configuration - Quagga (version 0.99.24.1)

      
!
log file /var/log/quagga/ospfd.log informational
log syslog informational
log record-priority
!
interface enc0
 ipv6 nd suppress-ra
 no link-detect
!
interface igb0
 ip ospf hello-interval 5
 ipv6 nd suppress-ra
 ipv6 ospf6 network broadcast
 no link-detect
!
interface igb1
 ipv6 nd suppress-ra
 ipv6 ospf6 network broadcast
 ipv6 ospf6 passive
 no link-detect
!
interface lo0
 no link-detect
!
interface pflog0
 ipv6 nd suppress-ra
 no link-detect
!
interface pfsync0
 ipv6 nd suppress-ra
 no link-detect
!
interface re0
 ipv6 nd suppress-ra
 no link-detect
!
interface re1
 ip ospf hello-interval 5
 ipv6 nd suppress-ra
 ipv6 ospf6 network broadcast
 no link-detect
!
router ospf
 ospf router-id 125.132.100.17
 redistribute kernel
 network 125.132.100.16/30 area 0.0.0.0
 network 125.132.100.24/29 area 0.0.0.1
 network 125.132.100.128/25 area 0.0.0.3
 network 125.132.102.0/27 area 0.0.0.3
 network 125.132.103.0/24 area 0.0.0.3
 area 0.0.0.3 stub no-summary
!
router ospf6
 router-id 125.132.100.17
 redistribute kernel
 interface re1 area 0.0.0.0
 interface igb0 area 0.0.0.1
 interface igb1 area 0.0.0.3
!
ip forwarding
ipv6 forwarding
!
line vty
!
end

      4°) FWN12 Configuration

      
!
log file /var/log/quagga/ospfd.log informational
log syslog informational
log record-priority
!
interface enc0
 ipv6 nd suppress-ra
 no link-detect
!
interface igb0
 ip ospf hello-interval 5
 ipv6 nd suppress-ra
 no link-detect
!
interface igb1
 ipv6 nd suppress-ra
 no link-detect
!
interface lo0
 no link-detect
!
interface pflog0
 ipv6 nd suppress-ra
 no link-detect
!
interface pfsync0
 ipv6 nd suppress-ra
 no link-detect
!
interface re0
 ipv6 nd suppress-ra
 no link-detect
!
interface re1
 ip ospf hello-interval 5
 ipv6 nd suppress-ra
 no link-detect
!
router ospf
 ospf router-id 125.132.100.18
 redistribute kernel
 network 125.132.100.16/30 area 0.0.0.0
 network 125.132.100.32/29 area 0.0.0.2
 network 125.132.100.128/25 area 0.0.0.3
 network 125.132.102.0/27 area 0.0.0.3
 network 125.132.103.0/24 area 0.0.0.3
 area 0.0.0.3 stub no-summary
!
router ospf6
 router-id 125.132.100.18
 redistribute kernel
 interface re1 area 0.0.0.0
 interface igb0 area 0.0.0.2
 interface igb1 area 0.0.0.3
!
ip forwarding
ipv6 forwarding
!
line vty
!
end

```**5°) RBGP1 Configuration**

      version 15.4
      service timestamps debug datetime msec
      service timestamps log datetime msec
      service password-encryption
      no platform punt-keepalive disable-kernel-core
      !
      hostname RBGP1
      !
      boot-start-marker
      boot system flash bootflash:/asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
      boot-end-marker
      !
      !
      vrf definition Mgmt-intf
      !
      address-family ipv4
      exit-address-family
      !
      address-family ipv6
      exit-address-family
      !
      no aaa new-model
      !
      no ip domain lookup
      ip domain name xxxxxx
      !
      ipv6 unicast-routing
      !
      subscriber templating
      multilink bundle-name authenticated
      !       
      !
      hw-module subslot 0/0 ethernet vlan unlimited
      !
      redundancy
      mode none
      !
      interface Loopback0
      no ip address
      ipv6 address 2006:DC80:0:100::1/128
      !
      interface Loopback1
      ip address 125.132.100.1 255.255.255.252
      !
      interface GigabitEthernet0/0/0
      description *** Connexion vers SFR ***
      no ip address
      negotiation auto
      !
      interface GigabitEthernet0/0/0.1200
      description Connexion vers SFR
      encapsulation dot1Q 1200
      ip address 119.24.138.6 255.255.255.252
      ipv6 address 2002:8400:1:1::206/126
      !
      interface GigabitEthernet0/0/2
      description Connexion vers SWDN11
      ip address 125.132.100.25 255.255.255.248
      ip ospf dead-interval 40
      ip ospf hello-interval 5
      negotiation auto
      ipv6 address 2006:DC80:0:1100::1/56
      ipv6 ospf 1 area 0.0.0.1
      !
      interface GigabitEthernet0
      vrf forwarding Mgmt-intf
      ip address 10.19.68.171 255.255.255.0 secondary
      ip address 10.245.0.1 255.255.240.0
      negotiation auto
      !
      router ospfv3 1
      !
      address-family ipv6 unicast
        redistribute bgp 200098
        router-id 125.132.100.25
        area 0.0.0.1 normal
      exit-address-family
      !
      router ospf 1
      router-id 125.132.100.25
      redistribute bgp 200098 subnets
      network 125.132.100.24 0.0.0.7 area 0.0.0.1
      distribute-list OSPF_FILTER out
      !
      router bgp 200098
      bgp router-id 125.132.100.1
      bgp log-neighbor-changes
      no bgp default ipv4-unicast
      neighbor 2002:8400:1:1::205 remote-as 15557
      neighbor 2002:8400:1:1::205 description *** SESSION BGP 1 VERS SFR ***
      neighbor 2002:8400:1:1::205 ebgp-multihop 10
      neighbor 2002:8400:1:1::205 password 7 xxxxxxxx
      neighbor 119.24.138.5 remote-as 15557
      neighbor 119.24.138.5 description BGP-PEER1-SFR
      neighbor 119.24.138.5 ebgp-multihop 10
      neighbor 119.24.138.5 password 7 xxxxxxx
      !
      address-family ipv4
        aggregate-address 125.132.100.0 255.255.252.0 summary-only
        redistribute ospf 1
        neighbor 119.24.138.5 activate
        neighbor 119.24.138.5 advertise-map BGP_ADVERTISE exist-map BGP_CONDITION
        neighbor 119.24.138.5 soft-reconfiguration inbound
        neighbor 119.24.138.5 prefix-list DCN-IPV4->ISP out
      exit-address-family
      !
      address-family ipv6
        network 2006:DC80::/29
        neighbor 2002:8400:1:1::205 activate
        neighbor 2002:8400:1:1::205 soft-reconfiguration inbound
        neighbor 2002:8400:1:1::205 prefix-list DCN-IPV6->ISP out
      exit-address-family
      !
      ip forward-protocol nd
      !
      ip access-list standard CAP_SFR
      permit 125.132.100.54
      ip access-list standard OSPF_FILTER
      deny  125.132.100.0 0.0.3.255
      permit any
      !
      !
      ip prefix-list BGP_CONDITION seq 10 permit 125.132.100.16/30
      !
      ip prefix-list DCN-IPV4->ISP seq 5 permit 125.132.100.0/22
      access-list 1 permit any
      ipv6 route 2006:DC80::/29 GigabitEthernet0/0/2
      !
      !
      ipv6 prefix-list DCN-IPV6->ISP seq 5 permit 2006:DC80::/29
      route-map BGP_ADVERTISE permit 10
      match ip address prefix-list DCN-IPV4->ISP
      !
      route-map BGP_CONDITION permit 10
      match ip address prefix-list BGP_CONDITION
      !
      control-plane
      !
      end

      
**6°) RBGP2 Configuration**

      version 15.4
      service timestamps debug datetime msec
      service timestamps log datetime msec
      service password-encryption
      no platform punt-keepalive disable-kernel-core
      !
      hostname RBGP2
      !
      boot-start-marker
      boot system flash bootflash:/asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
      boot-end-marker
      !
      !
      vrf definition Mgmt-intf
      !
      address-family ipv4
      exit-address-family
      !
      address-family ipv6
      exit-address-family
      !
      no aaa new-model
      !
      no ip domain lookup
      ip domain name xxxxx
      !
      ipv6 unicast-routing
      !
      subscriber templating
      multilink bundle-name authenticated
      !
      redundancy
      mode none
      !
      interface Loopback0
      no ip address
      ipv6 address 2006:DC80:0:200::1/128
      !
      interface Loopback1
      ip address 125.132.100.5 255.255.255.255
      !
      interface GigabitEthernet0/0/1
      description CONNEXION VERS OBS
      ip address 222.234.112.253 255.255.255.252
      negotiation auto
      ipv6 address 2001:C915:4000:300::1/64
      !
      interface GigabitEthernet0/0/3
      description Connexion vers SWDN12
      ip address 125.132.100.33 255.255.255.248
      ip ospf dead-interval 40
      ip ospf hello-interval 5
      negotiation auto
      ipv6 address 2006:DC80:0:1200::1/56
      ipv6 ospf 1 area 0.0.0.2
      !
      interface GigabitEthernet0
      vrf forwarding Mgmt-intf
      ip address 10.245.0.2 255.255.240.0
      negotiation auto
      !
      router ospfv3 1
      !
      address-family ipv6 unicast
        redistribute bgp 200098
        router-id 125.132.100.33
        area 0.0.0.2 normal
      exit-address-family
      !
      router ospf 1
      router-id 125.132.100.33
      redistribute bgp 200098 subnets
      network 125.132.100.32 0.0.0.7 area 0.0.0.2
      distribute-list OSPF_FILTER out
      !
      router bgp 200098
      bgp router-id 125.132.100.5
      bgp log-neighbor-changes
      no bgp default ipv4-unicast
      neighbor 2001:C000:0:2010::1DF2:3 remote-as 3215
      neighbor 2001:C000:0:2010::1DF2:3 description *** SESSION BGP 1 VERS OBS ***
      neighbor 2001:C000:0:2010::1DF2:3 ebgp-multihop 64
      neighbor 2001:C000:0:2010::1DF2:3 password 7 xxxxxxxxx
      neighbor 2001:C000:0:2010::1DF2:4 remote-as 3215
      neighbor 2001:C000:0:2010::1DF2:4 description *** SESSION BGP 2 VERS OBS ***
      neighbor 2001:C000:0:2010::1DF2:4 ebgp-multihop 64
      neighbor 2001:C000:0:2010::1DF2:4 password 7 xxxxxxxxxx
      neighbor 183.253.157.241 remote-as 3215
      neighbor 183.253.157.241 description BGP-PEER1-OBS
      neighbor 183.253.157.241 ebgp-multihop 64
      neighbor 183.253.157.241 password 7 xxxxxxxxxx
      neighbor 183.253.157.242 remote-as 3215
      neighbor 183.253.157.242 description BGP-PEER2-OBS
      neighbor 183.253.157.242 ebgp-multihop 64
      neighbor 183.253.157.242 password 7 xxxxxxxxxx
      !
      address-family ipv4
        aggregate-address 125.132.100.0 255.255.252.0 summary-only
        redistribute ospf 1
        neighbor 183.253.157.241 activate
        neighbor 183.253.157.241 advertise-map BGP_ADVERTISE exist-map BGP_CONDITION
        neighbor 183.253.157.241 soft-reconfiguration inbound
        neighbor 183.253.157.241 prefix-list DCN-IPV4->ISP out
        neighbor 183.253.157.242 activate
        neighbor 183.253.157.242 advertise-map BGP_ADVERTISE exist-map BGP_CONDITION
        neighbor 183.253.157.242 soft-reconfiguration inbound
        neighbor 183.253.157.242 prefix-list DCN-IPV4->ISP out
      exit-address-family
      !
      address-family ipv6
        network 2006:DC80::/29
        neighbor 2001:C000:0:2010::1DF2:3 activate
        neighbor 2001:C000:0:2010::1DF2:3 soft-reconfiguration inbound
        neighbor 2001:C000:0:2010::1DF2:3 prefix-list DCN-IPV6->ISP out
        neighbor 2001:C000:0:2010::1DF2:4 activate
        neighbor 2001:C000:0:2010::1DF2:4 soft-reconfiguration inbound
        neighbor 2001:C000:0:2010::1DF2:4 prefix-list DCN-IPV6->ISP out
      exit-address-family
      !
      ip forward-protocol nd
      !
      ip route 183.253.157.241 255.255.255.255 222.234.112.254 name BGP-PEER1-OBS
      ip route 183.253.157.242 255.255.255.255 222.234.112.254 name BGP-PEER2-OBS
      !
      ip access-list standard OSPF_FILTER
      deny  125.132.100.0 0.0.3.255
      permit any
      !
      ip prefix-list BGP_CONDITION seq 10 permit 125.132.100.16/30
      !
      ip prefix-list DCN-IPV4->ISP seq 5 permit 125.132.100.0/22
      ipv6 route 2001:C000:0:2010::1DF2:3/128 2001:C915:4000:300::2 name BGP-PEER1-OBS
      ipv6 route 2001:C000:0:2010::1DF2:4/128 2001:C915:4000:300::2 name BGP-PEER2-OBS
      ipv6 route 2006:DC80::/29 GigabitEthernet0/0/3
      !
      !
      ipv6 prefix-list DCN-IPV6->ISP seq 5 permit 2006:DC80::/29
      route-map BGP_ADVERTISE permit 10
      match ip address prefix-list DCN-IPV4->ISP
      !
      route-map BGP_CONDITION permit 10
      match ip address prefix-list BGP_CONDITION
      !
      control-plane
      !
      end

      
**7°) Logs & Debug**

At the moment of writing this post, i have no logs at the time of the last outage (clog and pfsense logs are not so easy to manipulate than classic syslog)
We are working to have a remote syslog server to catch all logs from Cisco and PFsense routers this weekend.
I post them here as soon as possible.
      1 Reply Last reply Reply Quote 0
      • Q
        quadrinary
        last edited by

        Interesting. In my scenario, I'm essentially scrapping the use of kernel routes entirely as well as abandoning interface-based gateways (this caused some really squirrly results when trying to use learned routes). I'd suggest starting by choosing to use Quagga for all routes (static and dynamic) and not use pfSense defined routes.

        1 Reply Last reply Reply Quote 0
        • G
          greenitsolutions
          last edited by

          Hi quadrinary,

          Thanks for your reply.

          Since my post, our firewalling solution is in master/slave mode without any outage.

          Your reply seems to be a good idea. We will test in the last few nights and give you a reply.

          Bertrand

          1 Reply Last reply Reply Quote 0
          • First post
            Last post
          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.