WAN interfaces flapping with multiWAN
-
Thank you for the generous offer to help. I'll gather as much info as I can. The thing that really has me scratching my head is that OPT1 seems to be the "problem child," though the WAN interface periodically flaps too. I've changed the monitor IP for OPT1 to be OPt1's own IP address and it STILL happens. That is with both a dc ethernet card AND with the integrated Intel nic on the motherboard that uses the fxp driver. So if the pings are failing even when it is monitoring the OPT1 interface itself, I'm just plain confused.
One more thing I'm going to try is to use a more current snapshot and see if that helps at all.
Cheers,
-
to be very clear about this.
We replaced ping explicitly by fping because ping was flapping so much. When we used ping it sent a single packet with a 2 second timeout. This failed too often.
When using fping the default timeout is 400 ms, the backoff factor is 1.5 and the number of retries is 3.
This means that we wait 400ms, 600ms, 900ms and 1.35 seconds for a ping response.fping should only return failure after all retries have failed which is takes at most 3.5 seconds before failure is detected.
If 1.2 does not exhibit this problem there might be a different issue at play totally.
But as said before, this requires a tcpdump of the icmp traffic to be able to debug this.
-
We found a issue with FreeBSD7 which might be causing us grief.
We have filed a PR and hope to get any response.
-
We found a issue with FreeBSD7 which might be causing us grief.
We have filed a PR and hope to get any response.
For those of us following this sisue can you give some more information, or at least a reference to the FreeBSD PR?
-
We found a issue with FreeBSD7 which might be causing us grief.
We have filed a PR and hope to get any response.
For those of us following this sisue can you give some more information, or at least a reference to the FreeBSD PR?
Yes, I would also find that interesting (but perhaps not directly helpful since I'm not much of a programmer).
Best,
-
The PR is http://www.freebsd.org/cgi/query-pr.cgi?pr=127528
-
@cmb:
The PR is http://www.freebsd.org/cgi/query-pr.cgi?pr=127528
Doesn't appear that they intend to "fix" this and they're saying it's an application level issue. Where does that leave those of us that are experiencing the problem? Go back to pre-FreeBSD7 distro?
Best,
-
Can you all see at what hz are you running?
it should come out of sysctl kern.hz if greater than 1000 try setting it to 500 and retry.
Interesting would be hz 2000 but we will see. -
@ermal:
Can you all see at what hz are you running?
it should come out of sysctl kern.hz if greater than 1000 try setting it to 500 and retry.
Interesting would be hz 2000 but we will see.Huh? I don't understand what you just said. What are you suggesting we change and why?
Best,
-
I'm sure they're all at default hz. Changing that isn't a solution regardless.
We'll get a resolution to this eventually, if it's an immediate problem for you, you'll have to downgrade to 1.2. This isn't going to be easy or quick to resolve.
-
@cmb:
I'm sure they're all at default hz. Changing that isn't a solution regardless.
We'll get a resolution to this eventually, if it's an immediate problem for you, you'll have to downgrade to 1.2. This isn't going to be easy or quick to resolve.
OK. I think downgrading looks like the path of least resistance. The complaining from folks with frequently dropped connections at the office is getting rather shrill. Looking forward to an eventual fix.
Best,
-
For what its worth I was seeing this too and have also downgraded to 1.2-Release. Its a shame, as I hate going backwards. You need a firewall to be reliable and stable and its hard to test a new beta without putting it in 'service'.
-
The latest snapshots have a fix for this can you, if possible, test and report if it behaves correctly now.
-
@ermal:
The latest snapshots have a fix for this can you, if possible, test and report if it behaves correctly now.
I'll give it a try next time I'm on site. What was the nature of the fix?
Best,
-
slbd used to use fping to determine if a WAN was online. There is some kernel change in FreeBSD 7.0 that causes problems because fping sees replies from pings initiated by other processes. Usually RRD for quality graph and slbd for monitor IP are both pinging the gateway IPs on your WAN (the fact that two processes are pinging the same thing is something we're eliminating in 1.3, but is too significant a change to pull into a maintenance release).
Now, slbd runs a shell script (for easy changing and testing, because the process being run is hard coded into the slbd binary) which runs FreeBSD's ping. It knows which replies are supposed to go where, and should behave properly unlike fping. The ping in FreeBSD 7.0 supports everything we were doing with fping. This should hopefully be resolved now.
-
Confirm flapping stopped. Thanks for the fix.
Nick.
-
Can you test that it behaves propperly if you disconnect one of the wans even in failover or loadbalance?
This would help pushing the 1.2.1 release. -
I'm running multiple failover (not balance) multi-WAN on a CARP cluster.
Watching syslog messages as they come through I unplugged the phone line so the ping would fail but leave interfaces up.It took 30s for a the message to come through:
"ICMP poll failed…marking service DOWN"Plugged back in and "marking service as UP"
I don't know how long it took before but I think it was a little more responsive than this. If you think I'm just seeing a delay in the syslog pathway I can time it a more carefully using the logs.
Nick.
-
Nothing has changed in that way apart the good thing that it is working now.
If you wish /usr/local/sbin/slbd.sh has the command to check the status. As far as i am conerned you may replace it with anything you please just return the status.Maybe worth confirming that it is the syslog latency though 30sec are not that bad too :P.
-
I'm discovering that on the latest nightly of 1.2.1rc1, the interface does get marked down after a few seconds. But unfortunately, the failover never actually takes places. If the interface goes down, it's simply down until you bring it back up. The downed network never falls back to the other gateway.
