Kernel routing table grows and finaly crash on PFSense 2.3.2 with OSPF 0.99.24.1
-
Hi all,
We installed three months ago 2 PFSense (Hardware, Supermicro MB, Xeon E5, RAM 16 Go, SSD) as Core Firewalls in DC.
In Master / Slave mode, all works fine.
But in Master / Master mode (asymetric full BGP routing), we have routing holes in and out approx. each hour with kernel routing tables flushs and / or growing inexpectidly.Hope that some one in this forum can help us to diagnose and help resolving our problem :
- an BGP / OSPF misconfiguration in Full BGP routing mode ?
- a Quagga misconfiguration on PFSense ?
- a bug ?
1°) Topology****with OSPF Areas
+–-----------+ +-------------+
| | | |
| ISP1 ROUTER | | ISP2 ROUTER |
| | | |
+------+------+ +------+------+
| |
| |
| |
| |
+--------+--------+ +--------+--------+
| | | |
| RBGP1 (CISCO) | | RBGP2 (CISCO) |
| | | |
+-----------------+ +-----------------+
|Gi0/0/2 |Gi0/0/2
| |
| |
| |
OSPF AREA 0.0.0.1 OSPF AREA 0.0.0.2
| |
| |
| |
|IGB0 |IGB0
+-----------------+ +-----------------+
| |RE1 | |
| FWN11 (PFSENSE) +----------OSPF AREA 0.0.0.0-----------+ FWN12 (PFSENSE) |
| | RE1| |
+-----------------+ +-----------------+
|IGB1 |IGB1
| |
| |
| |
OSPF AREA 0.0.0.3 OSPF AREA 0.0.0.32°) Topology with IP Adresses (Not real @IPs)
+–-----------+ +-------------+
| | | |
| ISP1 ROUTER | | ISP2 ROUTER |
| | | |
+------+------+ +------+------+
| |
| |
| |
| |
+--------+--------+ +--------+--------+
| | | |
| RBGP1 (CISCO) | | RBGP2 (CISCO) |
| | | |
+--------+--------+ +--------+--------+
| |
|Gi0/0/2 |Gi0/0/2
|125.132.100.25/29 |125.132.100.33/29
| |
| |
| |
|IGB0 |IGB0
|125.132.100.26/29 |125.132.100.34/29
| |
+--------+--------+RE1 +--------+--------+
| |125.132.100.17/30 | |
| FWN11 (PFSENSE) +--------------------------------------+ FWN12 (PFSENSE) |
| | RE1| |
+--------+--------+ 125.132.100.18/30+--------+--------+
| |
|IGB1 |IGB1
|125.132.100.252/25 |125.132.100.253/25
| |
| +-----------------------------------+ |
| | CARP IP : 125.132.100.254/25 | |
+----------+ IP ALIAS : 125.132.102.254/24 +---------+
| IP ALIAS : 125.132.103.254/24 |
+-+---------------+---------------+-+
| | |
| | |
| | |
+-----+---+ +----+----+ +---+-----+
| | | | | |
| HOST 01 | | HOST 02 | | HOST 02 |
| | | | | |
+---------+ +---------+ +---------+3°) FWN11 PFSense Configuration - Quagga (version 0.99.24.1)
! log file /var/log/quagga/ospfd.log informational log syslog informational log record-priority ! interface enc0 ipv6 nd suppress-ra no link-detect ! interface igb0 ip ospf hello-interval 5 ipv6 nd suppress-ra ipv6 ospf6 network broadcast no link-detect ! interface igb1 ipv6 nd suppress-ra ipv6 ospf6 network broadcast ipv6 ospf6 passive no link-detect ! interface lo0 no link-detect ! interface pflog0 ipv6 nd suppress-ra no link-detect ! interface pfsync0 ipv6 nd suppress-ra no link-detect ! interface re0 ipv6 nd suppress-ra no link-detect ! interface re1 ip ospf hello-interval 5 ipv6 nd suppress-ra ipv6 ospf6 network broadcast no link-detect ! router ospf ospf router-id 125.132.100.17 redistribute kernel network 125.132.100.16/30 area 0.0.0.0 network 125.132.100.24/29 area 0.0.0.1 network 125.132.100.128/25 area 0.0.0.3 network 125.132.102.0/27 area 0.0.0.3 network 125.132.103.0/24 area 0.0.0.3 area 0.0.0.3 stub no-summary ! router ospf6 router-id 125.132.100.17 redistribute kernel interface re1 area 0.0.0.0 interface igb0 area 0.0.0.1 interface igb1 area 0.0.0.3 ! ip forwarding ipv6 forwarding ! line vty ! end
4°) FWN12 Configuration
! log file /var/log/quagga/ospfd.log informational log syslog informational log record-priority ! interface enc0 ipv6 nd suppress-ra no link-detect ! interface igb0 ip ospf hello-interval 5 ipv6 nd suppress-ra no link-detect ! interface igb1 ipv6 nd suppress-ra no link-detect ! interface lo0 no link-detect ! interface pflog0 ipv6 nd suppress-ra no link-detect ! interface pfsync0 ipv6 nd suppress-ra no link-detect ! interface re0 ipv6 nd suppress-ra no link-detect ! interface re1 ip ospf hello-interval 5 ipv6 nd suppress-ra no link-detect ! router ospf ospf router-id 125.132.100.18 redistribute kernel network 125.132.100.16/30 area 0.0.0.0 network 125.132.100.32/29 area 0.0.0.2 network 125.132.100.128/25 area 0.0.0.3 network 125.132.102.0/27 area 0.0.0.3 network 125.132.103.0/24 area 0.0.0.3 area 0.0.0.3 stub no-summary ! router ospf6 router-id 125.132.100.18 redistribute kernel interface re1 area 0.0.0.0 interface igb0 area 0.0.0.2 interface igb1 area 0.0.0.3 ! ip forwarding ipv6 forwarding ! line vty ! end ```**5°) RBGP1 Configuration**
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
no platform punt-keepalive disable-kernel-core
!
hostname RBGP1
!
boot-start-marker
boot system flash bootflash:/asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
no aaa new-model
!
no ip domain lookup
ip domain name xxxxxx
!
ipv6 unicast-routing
!
subscriber templating
multilink bundle-name authenticated
!
!
hw-module subslot 0/0 ethernet vlan unlimited
!
redundancy
mode none
!
interface Loopback0
no ip address
ipv6 address 2006:DC80:0:100::1/128
!
interface Loopback1
ip address 125.132.100.1 255.255.255.252
!
interface GigabitEthernet0/0/0
description *** Connexion vers SFR ***
no ip address
negotiation auto
!
interface GigabitEthernet0/0/0.1200
description Connexion vers SFR
encapsulation dot1Q 1200
ip address 119.24.138.6 255.255.255.252
ipv6 address 2002:8400:1:1::206/126
!
interface GigabitEthernet0/0/2
description Connexion vers SWDN11
ip address 125.132.100.25 255.255.255.248
ip ospf dead-interval 40
ip ospf hello-interval 5
negotiation auto
ipv6 address 2006:DC80:0:1100::1/56
ipv6 ospf 1 area 0.0.0.1
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
ip address 10.19.68.171 255.255.255.0 secondary
ip address 10.245.0.1 255.255.240.0
negotiation auto
!
router ospfv3 1
!
address-family ipv6 unicast
redistribute bgp 200098
router-id 125.132.100.25
area 0.0.0.1 normal
exit-address-family
!
router ospf 1
router-id 125.132.100.25
redistribute bgp 200098 subnets
network 125.132.100.24 0.0.0.7 area 0.0.0.1
distribute-list OSPF_FILTER out
!
router bgp 200098
bgp router-id 125.132.100.1
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 2002:8400:1:1::205 remote-as 15557
neighbor 2002:8400:1:1::205 description *** SESSION BGP 1 VERS SFR ***
neighbor 2002:8400:1:1::205 ebgp-multihop 10
neighbor 2002:8400:1:1::205 password 7 xxxxxxxx
neighbor 119.24.138.5 remote-as 15557
neighbor 119.24.138.5 description BGP-PEER1-SFR
neighbor 119.24.138.5 ebgp-multihop 10
neighbor 119.24.138.5 password 7 xxxxxxx
!
address-family ipv4
aggregate-address 125.132.100.0 255.255.252.0 summary-only
redistribute ospf 1
neighbor 119.24.138.5 activate
neighbor 119.24.138.5 advertise-map BGP_ADVERTISE exist-map BGP_CONDITION
neighbor 119.24.138.5 soft-reconfiguration inbound
neighbor 119.24.138.5 prefix-list DCN-IPV4->ISP out
exit-address-family
!
address-family ipv6
network 2006:DC80::/29
neighbor 2002:8400:1:1::205 activate
neighbor 2002:8400:1:1::205 soft-reconfiguration inbound
neighbor 2002:8400:1:1::205 prefix-list DCN-IPV6->ISP out
exit-address-family
!
ip forward-protocol nd
!
ip access-list standard CAP_SFR
permit 125.132.100.54
ip access-list standard OSPF_FILTER
deny 125.132.100.0 0.0.3.255
permit any
!
!
ip prefix-list BGP_CONDITION seq 10 permit 125.132.100.16/30
!
ip prefix-list DCN-IPV4->ISP seq 5 permit 125.132.100.0/22
access-list 1 permit any
ipv6 route 2006:DC80::/29 GigabitEthernet0/0/2
!
!
ipv6 prefix-list DCN-IPV6->ISP seq 5 permit 2006:DC80::/29
route-map BGP_ADVERTISE permit 10
match ip address prefix-list DCN-IPV4->ISP
!
route-map BGP_CONDITION permit 10
match ip address prefix-list BGP_CONDITION
!
control-plane
!
end**6°) RBGP2 Configuration**
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
no platform punt-keepalive disable-kernel-core
!
hostname RBGP2
!
boot-start-marker
boot system flash bootflash:/asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
no aaa new-model
!
no ip domain lookup
ip domain name xxxxx
!
ipv6 unicast-routing
!
subscriber templating
multilink bundle-name authenticated
!
redundancy
mode none
!
interface Loopback0
no ip address
ipv6 address 2006:DC80:0:200::1/128
!
interface Loopback1
ip address 125.132.100.5 255.255.255.255
!
interface GigabitEthernet0/0/1
description CONNEXION VERS OBS
ip address 222.234.112.253 255.255.255.252
negotiation auto
ipv6 address 2001:C915:4000:300::1/64
!
interface GigabitEthernet0/0/3
description Connexion vers SWDN12
ip address 125.132.100.33 255.255.255.248
ip ospf dead-interval 40
ip ospf hello-interval 5
negotiation auto
ipv6 address 2006:DC80:0:1200::1/56
ipv6 ospf 1 area 0.0.0.2
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
ip address 10.245.0.2 255.255.240.0
negotiation auto
!
router ospfv3 1
!
address-family ipv6 unicast
redistribute bgp 200098
router-id 125.132.100.33
area 0.0.0.2 normal
exit-address-family
!
router ospf 1
router-id 125.132.100.33
redistribute bgp 200098 subnets
network 125.132.100.32 0.0.0.7 area 0.0.0.2
distribute-list OSPF_FILTER out
!
router bgp 200098
bgp router-id 125.132.100.5
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 2001:C000:0:2010::1DF2:3 remote-as 3215
neighbor 2001:C000:0:2010::1DF2:3 description *** SESSION BGP 1 VERS OBS ***
neighbor 2001:C000:0:2010::1DF2:3 ebgp-multihop 64
neighbor 2001:C000:0:2010::1DF2:3 password 7 xxxxxxxxx
neighbor 2001:C000:0:2010::1DF2:4 remote-as 3215
neighbor 2001:C000:0:2010::1DF2:4 description *** SESSION BGP 2 VERS OBS ***
neighbor 2001:C000:0:2010::1DF2:4 ebgp-multihop 64
neighbor 2001:C000:0:2010::1DF2:4 password 7 xxxxxxxxxx
neighbor 183.253.157.241 remote-as 3215
neighbor 183.253.157.241 description BGP-PEER1-OBS
neighbor 183.253.157.241 ebgp-multihop 64
neighbor 183.253.157.241 password 7 xxxxxxxxxx
neighbor 183.253.157.242 remote-as 3215
neighbor 183.253.157.242 description BGP-PEER2-OBS
neighbor 183.253.157.242 ebgp-multihop 64
neighbor 183.253.157.242 password 7 xxxxxxxxxx
!
address-family ipv4
aggregate-address 125.132.100.0 255.255.252.0 summary-only
redistribute ospf 1
neighbor 183.253.157.241 activate
neighbor 183.253.157.241 advertise-map BGP_ADVERTISE exist-map BGP_CONDITION
neighbor 183.253.157.241 soft-reconfiguration inbound
neighbor 183.253.157.241 prefix-list DCN-IPV4->ISP out
neighbor 183.253.157.242 activate
neighbor 183.253.157.242 advertise-map BGP_ADVERTISE exist-map BGP_CONDITION
neighbor 183.253.157.242 soft-reconfiguration inbound
neighbor 183.253.157.242 prefix-list DCN-IPV4->ISP out
exit-address-family
!
address-family ipv6
network 2006:DC80::/29
neighbor 2001:C000:0:2010::1DF2:3 activate
neighbor 2001:C000:0:2010::1DF2:3 soft-reconfiguration inbound
neighbor 2001:C000:0:2010::1DF2:3 prefix-list DCN-IPV6->ISP out
neighbor 2001:C000:0:2010::1DF2:4 activate
neighbor 2001:C000:0:2010::1DF2:4 soft-reconfiguration inbound
neighbor 2001:C000:0:2010::1DF2:4 prefix-list DCN-IPV6->ISP out
exit-address-family
!
ip forward-protocol nd
!
ip route 183.253.157.241 255.255.255.255 222.234.112.254 name BGP-PEER1-OBS
ip route 183.253.157.242 255.255.255.255 222.234.112.254 name BGP-PEER2-OBS
!
ip access-list standard OSPF_FILTER
deny 125.132.100.0 0.0.3.255
permit any
!
ip prefix-list BGP_CONDITION seq 10 permit 125.132.100.16/30
!
ip prefix-list DCN-IPV4->ISP seq 5 permit 125.132.100.0/22
ipv6 route 2001:C000:0:2010::1DF2:3/128 2001:C915:4000:300::2 name BGP-PEER1-OBS
ipv6 route 2001:C000:0:2010::1DF2:4/128 2001:C915:4000:300::2 name BGP-PEER2-OBS
ipv6 route 2006:DC80::/29 GigabitEthernet0/0/3
!
!
ipv6 prefix-list DCN-IPV6->ISP seq 5 permit 2006:DC80::/29
route-map BGP_ADVERTISE permit 10
match ip address prefix-list DCN-IPV4->ISP
!
route-map BGP_CONDITION permit 10
match ip address prefix-list BGP_CONDITION
!
control-plane
!
end**7°) Logs & Debug** At the moment of writing this post, i have no logs at the time of the last outage (clog and pfsense logs are not so easy to manipulate than classic syslog) We are working to have a remote syslog server to catch all logs from Cisco and PFsense routers this weekend. I post them here as soon as possible.
-
Interesting. In my scenario, I'm essentially scrapping the use of kernel routes entirely as well as abandoning interface-based gateways (this caused some really squirrly results when trying to use learned routes). I'd suggest starting by choosing to use Quagga for all routes (static and dynamic) and not use pfSense defined routes.
-
Hi quadrinary,
Thanks for your reply.
Since my post, our firewalling solution is in master/slave mode without any outage.
Your reply seems to be a good idea. We will test in the last few nights and give you a reply.
Bertrand