frr and 2.5.0

spearless · Feb 19, 2021, 2:34 PM

Have been upgrading out lab estate of pfsense Firewalls from 2.4.5p1 to 2.5.0, and have a problem on 2 (maybe more have stopped rollout).

I have also update a number of stanalone pFsense FW's (ie not using frr) and these have all been fine.

We have a number of Pfsense firewalls in HA pairs, (all virtual on HyperV), and upgraded 1 pair without issue.

Second pair upgrade appears to work (ie you can log in, and everything seems ok) , but frr refuses to run. frr ospf demon just stops, runs for a min or two and then stops.

Starting the ospfd service maunaly, it seems to run, but no routes are found or advertised and status querys say ospfd not running...
FRR watch frr is running ok, as is FRR Zebra

The sucsessfully upgraded pair are also running frr/ospf and seem perfectly ok.

Reverting to 2.4.5p1 and restore config and all is well again.

Have gone through the process a couple if times (following the upgrade guidelines) and same result every time.

frr on 2.4.5 pfsense = 0.6.7_5
frr on upgraded pfsense = 1.1.0_5

Is this a bug, or am i doing something wrong here?

Config and logs:

##################### DO NOT EDIT THIS FILE! ######################
###################################################################

This file was created by an automatic configuration generator.

The contents of this file will be overwritten without warning!

###################################################################
!
frr defaults traditional
hostname UCCENV-vCFW02.ucc-env.hq
password Spangl3
service integrated-vtysh-config
!
ip router-id 192.168.100.34
!
ip route 192.168.170.0/24 hn1.10
!
interface hn1.180
description "ospfd: ToExtFW"
ip ospf cost 10
ip ospf area 0.0.0.0
interface hn1.10
description "ospfd: ToExtFW - ospfd: ToCoreNetwork"
ip ospf cost 10
ip ospf area 0.0.0.0
interface hn2
description "ospfd: ToExtFW - ospfd: ToCoreNetwork - ospfd: SYNC"
ip ospf area 0.0.0.0
interface hn1.512
description "ospfd: ToExtFW - ospfd: ToCoreNetwork - ospfd: SYNC"
ip ospf area 0.0.0.0
!
router ospf
ospf router-id 192.168.100.34
redistribute connected
redistribute kernel
redistribute static
passive-interface hn2
!
ip prefix-list ACCEPTFILTER deny 192.168.8.16/30
ip prefix-list ACCEPTFILTER deny 192.168.8.18/32
ip prefix-list ACCEPTFILTER seq 10 deny 192.168.170.0/24
ip prefix-list ACCEPTFILTER seq 20 permit any
!
route-map ACCEPTFILTER permit 10
match ip address prefix-list ACCEPTFILTER
!
ip protocol bgp route-map ACCEPTFILTER
!
ip protocol ospf route-map ACCEPTFILTER
!
ipv6 protocol bgp route-map ACCEPTFILTER
!
ipv6 protocol ospf6 route-map ACCEPTFILTER
!
line vty
!
end

Feb 19 13:42:26 watchfrr 2653 watchfrr 7.5 starting: vty@0
Feb 19 13:42:26 watchfrr 2653 zebra state -> up : connect succeeded
Feb 19 13:42:26 watchfrr 2653 staticd state -> up : connect succeeded
Feb 19 13:42:26 watchfrr 2653 ospfd state -> up : connect succeeded
Feb 19 13:42:26 watchfrr 2653 all daemons up, doing startup-complete notify
Feb 19 13:43:00 watchfrr 2653 [EC 268435457] zebra state -> unresponsive : no response yet to ping sent 30 seconds ago
Feb 19 13:43:00 watchfrr 2653 Forked background command [pid 73494]: /usr/local/etc/rc.d/frr restart all
Feb 19 13:43:00 watchfrr 2653 [EC 268435457] ospfd state -> down : read returned EOF
Feb 19 13:43:02 watchfrr 2653 [EC 268435457] staticd state -> down : read returned EOF
Feb 19 13:43:21 watchfrr 2653 Warning: restart all child process 73494 still running after 20 seconds, sending signal 15
Feb 19 13:43:21 watchfrr 2653 restart all process 73494 terminated due to signal 15
Feb 19 13:43:36 watchfrr 2653 ospfd state -> up : connect succeeded
Feb 19 13:43:37 watchfrr 2653 staticd state -> up : connect succeeded
Feb 19 13:44:21 watchfrr 2653 Forked background command [pid 21479]: /usr/local/etc/rc.d/frr restart all
Feb 19 13:44:21 watchfrr 2653 [EC 268435457] ospfd state -> down : read returned EOF
Feb 19 13:44:21 watchfrr 2653 [EC 268435457] staticd state -> down : read returned EOF
Feb 19 13:44:41 watchfrr 2653 Warning: restart all child process 21479 still running after 20 seconds, sending signal 15
Feb 19 13:44:41 watchfrr 2653 restart all process 21479 terminated due to signal 15
Feb 19 13:46:42 watchfrr 2653 Forked background command [pid 97856]: /usr/local/etc/rc.d/frr restart all
Feb 19 13:47:02 watchfrr 2653 Warning: restart all child process 97856 still running after 20 seconds, sending signal 15
Feb 19 13:47:02 watchfrr 2653 restart all process 97856 terminated due to signal 15
Feb 19 13:51:04 watchfrr 2653 Forked background command [pid 85042]: /usr/local/etc/rc.d/frr restart all
Feb 19 13:51:24 watchfrr 2653 Warning: restart all child process 85042 still running after 20 seconds, sending signal 15
Feb 19 13:51:24 watchfrr 2653 restart all process 85042 terminated due to signal 15
Feb 19 13:54:01 watchfrr 2653 ospfd state -> up : connect succeeded
Feb 19 13:54:01 watchfrr 2653 staticd state -> up : connect succeeded
Feb 19 13:59:24 watchfrr 2653 Forked background command [pid 22814]: /usr/local/etc/rc.d/frr restart all
Feb 19 13:59:24 watchfrr 2653 [EC 268435457] ospfd state -> down : read returned EOF
Feb 19 13:59:25 watchfrr 2653 [EC 268435457] staticd state -> down : read returned EOF
Feb 19 13:59:45 watchfrr 2653 Warning: restart all child process 22814 still running after 20 seconds, sending signal 15
Feb 19 13:59:45 watchfrr 2653 restart all process 22814 terminated due to signal 15

Feb 19 13:52:36 check_reload_status 370 Syncing firewall
Feb 19 13:52:36 php-fpm 8559 FRR Package: FRR BGPd: No config data found.
Feb 19 13:52:36 php-fpm 8559 FRR Package: FRR OSPF6d: No config data found.
Feb 19 13:52:36 php-fpm 8559 FRR Package: FRR BFDd: No config data found.
Feb 19 13:52:36 php-fpm 8559 FRR Package: FRR: Rebuild configuration.
Feb 19 13:52:36 php-fpm 8559 FRR Package: FRR: Restarting services.
Feb 19 13:54:00 kernel sonewconn: pcb 0xfffff8001c39d500: Listen queue overflow: 8 already in queue awaiting acceptance (4 occurrences)
Feb 19 13:55:18 kernel sonewconn: pcb 0xfffff8001c39d500: Listen queue overflow: 8 already in queue awaiting acceptance (1 occurrences)
Feb 19 13:56:29 check_reload_status 370 Syncing firewall
Feb 19 13:56:29 php-fpm 8559 FRR Package: FRR BGPd: No config data found.
Feb 19 13:56:29 php-fpm 8559 FRR Package: FRR OSPF6d: No config data found.
Feb 19 13:56:29 php-fpm 8559 FRR Package: FRR BFDd: No config data found.
Feb 19 13:56:29 php-fpm 8559 FRR Package: FRR: Rebuild configuration.
Feb 19 13:56:29 php-fpm 8559 FRR Package: FRR: Daemon state: zebra: running | staticd: running | ospfd: running
Feb 19 13:56:29 php-fpm 8559 FRR Package: FRR: Reloading configuration.
Feb 19 13:56:30 kernel sonewconn: pcb 0xfffff8001c39d500: Listen queue overflow: 8 already in queue awaiting acceptance (8 occurrences)
Feb 19 13:57:54 kernel sonewconn: pcb 0xfffff8001c39d500: Listen queue overflow: 8 already in queue awaiting acceptance (5 occurrences)
Feb 19 13:59:17 kernel sonewconn: pcb 0xfffff8001c39d500: Listen queue overflow: 8 already in queue awaiting acceptance (16 occurrences)
Feb 19 14:02:05 check_reload_status 370 Syncing firewall

spearless · Feb 20, 2021, 2:48 PM

@spearless.... So things have moved on.

Things I have tried:

1, Deleting all frr related config from config.xml, rebooting and the reconfiguring all frr bits.... made no difference.

2, Removed some wierd deny entries in the frr.conf file that seemed to have come from nowhere.... no difference.

Lastly.. after much digging, deleted /tmp/config.cache, rebooted and everthing started to work.

Having gone through all this on one firewall where frr would not run, I deleted just the config.cache on another that would not run (frr) rebooted and that now works too.

Wether its the 2.5.0 upgrade or the frr upgrade you have to do too, i have no idea... but it all is now working!

scourtney2000 · Feb 20, 2021, 8:01 PM

have a similar issue, but i'm using BGP and OSPF. FRR services will not start. tried your solution of deleting /tmp/config.cache but it not work. i'm still searching.

viktor_g · Feb 20, 2021, 8:55 PM

Please show the /var/log/frr/frr-reload.log to check the frr parser

spearless · Feb 22, 2021, 4:26 PM

@viktor_g

Here is is as of now. However as it is now working (following the config.cache deletion) not sure if it is any help.

2021-02-20 15:24:52,205 INFO: Called via "Namespace(bindir='/usr/local/bin', confdir='/var/etc/frr', daemon='', debug=False, filename='/var/etc/frr/frr.conf', input=None, log_level='info', overwrite=False, pathspace=None, reload=True, rundir='/var/run/frr', stdout=False, test=False, vty_socket=None)"
2021-02-20 15:24:52,206 INFO: Loading Config object from file /var/etc/frr/frr.conf
2021-02-20 15:24:52,472 INFO: Loading Config object from vtysh show running
2021-02-20 15:24:52,595 INFO: "frr version 7.5" cannot be removed
2021-02-20 15:24:52,595 INFO: Loading Config object from vtysh show running
2021-02-20 15:24:52,707 INFO: "frr version 7.5" cannot be removed
2021-02-20 15:33:49,745 INFO: Called via "Namespace(bindir='/usr/local/bin', confdir='/var/etc/frr', daemon='', debug=False, filename='/var/etc/frr/frr.conf', input=None, log_level='info', overwrite=False, pathspace=None, reload=True, rundir='/var/run/frr', stdout=False, test=False, vty_socket=None)"
2021-02-20 15:33:49,745 INFO: Loading Config object from file /var/etc/frr/frr.conf
2021-02-20 15:33:49,992 INFO: Loading Config object from vtysh show running
2021-02-20 15:33:50,124 INFO: "frr version 7.5" cannot be removed
2021-02-20 15:33:50,124 INFO: Loading Config object from vtysh show running
2021-02-20 15:33:50,245 INFO: "frr version 7.5" cannot be removed

scourtney2000 · Feb 22, 2021, 5:04 PM

@viktor_g

Hello,

I go my config to work by deleting all the route maps, acls, and prefix lists.

I have a bunch of pfsense firewalls that i'm upgrading and will be sending logs.

Ty,
Sean