Hello, I just set up the same config and had it working. The issue I ran into is that I needed some layer2 stuff to cross the network and pf was placing in layer 3, thus breaking my config.
Anyhow, I have Cisco switches that were connecting to my pf setup.
I have three Nics in my pf boxes (1 for LAN, 1 for WLAN and 1 for Internet)
I created openvpn tun sharedkey tunnels between my pf boxes, and assigned the openvpn clients to interfaces. In QuagaOSPF add the three interfaces to area 0.0.0.0.
On the switch side I added the pf LAN network to area 0 and my failover was good to go. Just play with the interface cost in quagga to determine when a failover should occur. I think my fail over was sub 2 minutes.
In pfsense you will want to set up some rules to handle traffic that ospf doesn't know about. I used the gateway groups to handle this so that in a failover my internet traffic would still go out. However, I route all my outgoing internet traffic through my data center so YMMV.
BTW if you need to trunk (802.1q) between your switches and they support ospf you can connect the wlan to the switch use pfsense to create a vpn backup there. At least that's what I am trying now….
Fred