Quagga OSPF installation hanging…

[SkyNET]

While the install was still running, and hanging on the "Executing custom_php_resync_config_command()…" message, I logged in to the WUI.
There under Services I pressed Start on Quagga_OSPFd to see what would happen, get a message like "nothing to do" on the screen attached to the pfSense machine, and the install process continued to the next package install..

Only output I can read on the screen at this moment is

defaultrouter="NO"
quagga_enable="YES"

"Also, you may wish to set the following options:
Quagga_deamons="zebra bgpd etc"
quagga_flags="...."
quagga_extralibs_path="... ..."
router_enable="NO"
watchquagga_enable"YES"

Noting to do..
Starting syslog...done
Starting CRON... done
and so on..

[BBcan177]

@SkyNET:

On a different system i run a i386 install, and there the upgrade went smooth.
That system suffered from "forgetting" every SNORT setting.. but that's a different story, and an easy fix..  :-X

For Snort/Suricata, there is a "Checkbox" in Global Settings, to "Keep Snort Settings After Deinstall"… That needs to be enabled, or it will clear out your settings...

[jimp]

Anyone hitting this, can you share the quagga settings section from your config.xml? You can send it via PM if you wish to do so privately, or mask out any password/IP addresses/etc.

[SkyNET]

Successfully upgraded system 3 (amd64 - full install).

I will do a compare from all of the system-config files and post the differences, if I spot any.
On the system1, I successfully removed the Quagga package. Installing it again did not help. Lets see if there are any "left-overs" of quagga in the config..

Will post my results shortly.

[mph]

I have the same problem on all of my systems (8 routers)
Also, qugga ospf is working incorrectly. Route table doesn't update with actions on  ospf, only on start. For example, i had stopped the ospf demon, but route tables hasn't changed.

I think, the problems has the same root.

PS. sorry for my english

[mph]

@jimp:

Anyone hitting this, can you share the quagga settings section from your config.xml? You can send it via PM if you wish to do so privately, or mask out any password/IP addresses/etc.

	 <installedpackages><package><name>Open-VM-Tools</name>

			<website>http://open-vm-tools.sourceforge.net/</website>
			<version>1280544.13_2</version>
			<pkginfolink>https://doc.pfsense.org/index.php/Open_VM_Tools_package</pkginfolink>
			<configurationfile>open-vm-tools.xml</configurationfile></package> 
		 <package><name>Quagga OSPF</name>
			<internal_name>Quagga_OSPF</internal_name>
			 <descr><strong>WARNING! Installs files to the same place as OpenBGPD. Installing both will break things.</strong>]]></descr>
			<version>0.6.12.1_4</version>
			<configurationfile>quagga_ospfd.xml</configurationfile></package> 
		 <service><name>vmware-guestd</name>
			<rcfile>vmware-guestd.sh</rcfile>
			<executable>vmtoolsd</executable></service> 
		 <tab><text>Global Settings</text>
			<url>pkg_edit.php?xml=quagga_ospfd.xml&id=0</url>
			 <active></active></tab> 
		 <quaggaospfdinterfaces><config><interface>wan</interface>
				 <networktype><metric><interfacearea><descr><passive>on</passive>
				 <acceptfilter><md5password><password><routerpriorityelections><hellointervalinseconds><retransmitinterval><deadtimer></deadtimer></retransmitinterval></hellointervalinseconds></routerpriorityelections></password></md5password></acceptfilter></descr></interfacearea></metric></networktype></config> 
			 <config><interface>ovpns1</interface>
				 <networktype><metric><interfacearea><descr><passive><acceptfilter><md5password><password><routerpriorityelections><hellointervalinseconds><retransmitinterval><deadtimer></deadtimer></retransmitinterval></hellointervalinseconds></routerpriorityelections></password></md5password></acceptfilter></passive></descr></interfacearea></metric></networktype></config> 
			 <config><interface>ovpns2</interface>
				 <networktype><metric><interfacearea><descr><passive><acceptfilter><md5password><password><routerpriorityelections><hellointervalinseconds><retransmitinterval><deadtimer></deadtimer></retransmitinterval></hellointervalinseconds></routerpriorityelections></password></md5password></acceptfilter></passive></descr></interfacearea></metric></networktype></config> 
			 <config><interface>ovpns3</interface>
				 <networktype><metric><interfacearea><descr><passive><acceptfilter><md5password><password><routerpriorityelections><hellointervalinseconds><retransmitinterval><deadtimer></deadtimer></retransmitinterval></hellointervalinseconds></routerpriorityelections></password></md5password></acceptfilter></passive></descr></interfacearea></metric></networktype></config> 
			 <config><interface>ovpns4</interface>
				 <networktype><metric><interfacearea><descr><passive><acceptfilter><md5password><password><routerpriorityelections><hellointervalinseconds><retransmitinterval><deadtimer></deadtimer></retransmitinterval></hellointervalinseconds></routerpriorityelections></password></md5password></acceptfilter></passive></descr></interfacearea></metric></networktype></config> 
			 <config><interface>ovpns5</interface>
				 <networktype><metric><interfacearea><descr><passive><acceptfilter><md5password><password><routerpriorityelections><hellointervalinseconds><retransmitinterval><deadtimer></deadtimer></retransmitinterval></hellointervalinseconds></routerpriorityelections></password></md5password></acceptfilter></passive></descr></interfacearea></metric></networktype></config> 
			 <config><interface>ovpns6</interface>
				 <networktype><metric><interfacearea><descr><passive><acceptfilter><md5password><password><routerpriorityelections><hellointervalinseconds><retransmitinterval><deadtimer></deadtimer></retransmitinterval></hellointervalinseconds></routerpriorityelections></password></md5password></acceptfilter></passive></descr></interfacearea></metric></networktype></config> 
			 <config><interface>ovpns7</interface>
				 <networktype><metric><interfacearea><descr><passive><acceptfilter><md5password><password><routerpriorityelections><hellointervalinseconds><retransmitinterval><deadtimer></deadtimer></retransmitinterval></hellointervalinseconds></routerpriorityelections></password></md5password></acceptfilter></passive></descr></interfacearea></metric></networktype></config></quaggaospfdinterfaces> 
		 <quaggaospfd><config><password>zebra</password>
				 <logging><adjacencylog><routerid><area>0.0.0.0
				 <updatefib><redistributeconnectedsubnets><redistributedefaultroute><redistributestatic><redistributekernel><spfholdtime><spfdelay><rfc1583><row><routevalue><routearea></routearea></routevalue></row> 
				<carpstatusvid>none</carpstatusvid></rfc1583></spfdelay></spfholdtime></redistributekernel></redistributestatic></redistributedefaultroute></redistributeconnectedsubnets></updatefib></routerid></adjacencylog></logging></config></quaggaospfd></installedpackages> 

[SkyNET]

Got OSPFd running again..

Removed Quagga_OSPF from packages.. hangs.. rebooted..
Package shows up under Services.. At status it shows "OSPF appears not to be running".

Checked if all interfaces are correct. Set "Network Type" for each interface. Broadcast/point-to-point.
Somehow some interface got binded to the WAN interface.
Hit Save. Then I got an "504 Gateway Time-out" message of nginx.

In a second browser tab I noticed that OSPFd is now running. Under status I see all the OSPF neighbors I should see.
Restarted OSPFd via Services and noticed that the SPF algorithm was newly executed.

So conclusion.. check interfaces, or remove all interfaces from OSPF and configure them again.

System log regarding nginx

nginx: 2016/04/18 14:10:41 [error] 46201#0: *103 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.x.y.z, server: , request: "POST /pkg_edit.php?xml=quagga_ospfd_interfaces.xml&act=edit&id=3 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.x.x.x", referrer: "https://10.x.x.x/pkg_edit.php?xml=quagga_ospfd_interfaces.xml&act=edit&id=3"

[jimp]

OK I was able to reproduce this using that config section above. I'm preparing a fix, but it might need a local fix in your quagga settings if it seems to be stuck there.

If you run this command you'll see the actual problem:

/usr/local/etc/rc.d/quagga.sh restart

Run that from a shell prompt, NOT the GUI Diagnostics > Command prompt.

In the above case, at least on my VM, it was a blank/empty interface. Fixing the interfaces allowed it to function.

I'm adding two things to prevent it from being a problem in the future:
1. A check to ensure blank interfaces don't make it into the Quagga config
2. Using mwexec_bg rather than mwexec on the calls to (re)start the service so that it can't hang up the PHP process in the GUI.

[jimp]

New version pushed, it should be up within ~30m.

[SkyNET]

Just performed the updated.. OSPF still working great.
Thanks for the effort jimp!