The problem went away after I first delete the GW group, changed the WAN interfaces to an outside monitoring address and re-created the GW group. Thanks for your respons.

@heper: i have no clue what you want to do … could you make a network diagram and try to explain 'the plan' with a lot more detail? Agreed. I was trying to figure that out too. Thats why I ask for short descriptions and config files (obscured IPs, etc, better than pictures)

I was wondering, is there a way to manage 100 pfSense VPS One by one. Is there a way to have a master for all these pfsense VPS? Not at the moment, nor any time in the near future.

My experience is that after a period of real internet problems (ISP is down but not at the first hop, or real latency is high), when the real problem is fixed, then apinger get too optimistic, reporting latency that is too good to be true. I guess something in its rolling averaging algorithm goes wrong after it has had big values and then gets much smaller ones come in. Anyway, for me it usually stabilizes back to a believable value after 5 minutes. Yes, the code has got too complicated trying to patch it to handle all sorts of things that can happen on-the-fly as interface events happen and outstanding IOs die and… Maybe it can be fixed by just 1 more patch?! Realistically, as others have commented elsewhere, it is time to make some requirements and write a multi-WAN multi-target monitoring program from scratch.

I posted this while i diagnosed this and ultimately determined this to be the fault of the modem. Replacing it was the key. All quiet all the time now.

I think they will come up as backup until they don't see hellos for a multiple of the advertising frequency so your settings might be a little excessive.  Just going to 2 seconds cuts the hello traffic in half.  Going to 10 cuts it by 90%.  in general, as I understand it, YMMV etc.

I have an inbound NAT rule on each interface for port 80/443 that routes into the main web server.  Response from the web request returns out the link it came in on.  Not sure exactly where RRD gets its data from, but it is obviously different than where SNMP gets its numbers from as the SNMP data is correct, validated by looking at my usage graphs from the upstream ISP links. Thanks,

This is a bug. I've filed a report: https://redmine.pfsense.org/issues/4479

Hmm.  Seems a default route to pfSense would have been sufficient.

If you can match the traffic using firewall rules you can policy route / NAT it however you like.

Hey! So, my 2 dyndns entry is different. I have solved this problem setup the trigger level of gateway group FailOver to Member Down. It Works! Thanks! I have another problem with Squid + FailOver, when one of WAN is down, squid returns 500 error int he clients…..but i will open another topic! Thanks again!

@Derelict: You need to reject LAN net to OPT1 net followed by your pass any any. What you have there won't block anything from LAN to OPT1. Assuming you are trying to limit connectivity from LAN to OPT1 to all but Servers. ETA: Not true - I didn't notice that last rule was IPv6. I'd still rather see and explicit reject rule for the traffic you want to block followed by a pass rule instead of that exclusion. Good Idea. I have added such rules now. Thanks.

Put 8.8.8.8 on monitor ip in the gateway. It works with me. Like @ristosu said.

pfsense 2.2 one end <–> 2.1.4 other end of openvpn tunnel. quagga: Installed: 0.99.22.3.1_2 v0.6.2 <-- not all that much has changed in functionality lately (i think). what i did notice was some weirdness on the quagga 'status' page on the pfsense that was injecting the routes ( Quagga Zebra Routes = empty) ... but the receiving end added the route automagically, and everything keeps working ;) injector quagga config: Quagga ospfd.conf # This file was created by the pfSense package manager.  Do not edit! password xxxxx log syslog interface ovpnc1   ip ospf cost 100 router ospf   ospf router-id 10.0.0.1   redistribute static   network 192.168.222.0/30 area 0.0.0.1   network 192.168.226.0/24 area 0.0.0.1   network 10.0.0.0/24 area 0.0.0.1 Quagga zebra.conf ip route 192.168.213.0/24 lo0 Quagga OSPF Database on inject side       OSPF Router with ID (10.0.0.1)                 Router Link States (Area 0.0.0.1) Link ID        ADV Router      Age  Seq#      CkSum  Link count 10.0.0.1        10.0.0.1          3 0x800013d7 0xd3cf 3 10.10.10.1      10.10.10.1        98 0x8000be71 0x1f18 11 10.20.10.1      10.20.10.1      776 0x80007af9 0x447c 10 10.30.10.1      10.30.10.1      1246 0x8000049f 0xd60d 5                 AS External Link States Link ID        ADV Router      Age  Seq#      CkSum  Route 192.168.213.0  10.0.0.1          3 0x80000002 0x670c E2 192.168.213.0/24 [0x0] Quagga OSPF Database on receiving end       OSPF Router with ID (10.10.10.1)                 Router Link States (Area 0.0.0.1) Link ID        ADV Router      Age  Seq#      CkSum  Link count 10.0.0.1        10.0.0.1        233 0x800013d8 0xd1d0 3 10.10.10.1      10.10.10.1      268 0x8000be72 0xdbed 12 10.20.10.1      10.20.10.1      1041 0x80007af9 0x447c 10 10.30.10.1      10.30.10.1      1511 0x8000049f 0xd60d 5                 AS External Link States Link ID        ADV Router      Age  Seq#      CkSum  Route 192.168.213.0  10.0.0.1        268 0x80000002 0x670c E2 192.168.213.0/24 [0x0]

No.  You have to policy route to the gateway group. If you think about it a minute, a gateway group being the "default gateway" is nonsensical.

We sorted this out, it was a config within the HP switch for VLAN trunking that was causing issues with packet routing. Derelict, thanks again dude and sorry to waste your time. 3 days, new infant, new work project, 6 hours total sleep; Kills the brain.

@Derelict: You have firewall rule and/or NAT issues.  See the other thread. Note that rules permitting DHCP traffic on a pfSense interface are automatically added to interfaces with a DHCP server defined.  That's probably why DHCP is working but nothing else is. You're right. Turns out my firewall rule set only to allow any TCP so I was actually able to access the pfsense web configurator but nothing else.