Multi-WAN gateway failover not switching back to tier 1 gw after back online
-
hello,
I have a problem with pfSense 2.2.2 with Multi-WAN and failover not returning routing to tier 1 gateway after it failed and is back online.
More details:
Two WAN connections, WAN1 and WAN2. WAN1 is a reliable but slow DSL connection and also the default gateway. WAN1 has static IP assignment and basically almost never goes down.
Now I've added a 4G/LTE connection as WAN2. WAN2 is fast, has dynamic IP assignment, disconnects once every 24 hours for 1-2 minutes, to come back up with a new IP. WAN2 also sometimes has longer, unplanned outages. WAN2 connects to LTE in bridge mode, so the dynamic public IP is assigned directly to the WAN2 interface via DHCP.
I've setup a gateway group with WAN2 as tier 1, and WAN1 as tier 2, and assigned an appropriate rule to use this gateway group. Trigger level for gateway group is set to "Member down".
So far so good, this basically works until WAN2 goes down a couple times (usually after 2 or 3 times): traffic fails over to WAN1 automatically, but then after WAN2 returns back online, all traffic is still routed via WAN1 and stays there until I manually change some interface, gateway or other setting on pfSense – this apparently somehow solves the stuck routing via WAN1. This happens after a couple failovers, no matter if WAN2 was down for just one minute (for the new IP every 24h), or one hour or more.
I can see that both gateways are online again in the status page, and here are the apinger log entries for when WAN2 went down and back online today (GW_OPT1 is WAN2):
May 16 15:04:49 apinger: alarm canceled (config reload): GW_OPT1(8.8.8.8) *** down *** May 16 15:04:49 apinger: SIGHUP received, reloading configuration. May 16 15:03:23 apinger: ALARM: GW_OPT1(8.8.8.8) *** down ***
Just after this cycle the connections were stuck on WAN1 again.
Does anyone have some suggestions what I could do to troubleshoot this or how to fix this?
Are there any other settings that need to be made for failover to work properly?
Is the SIGHUP log entry normal for apinger? I did not do anything on pfSense at this time.
thanks.
-
Hi,
Working fine on my pfsense2.2.2, but I set up my main WAN (WAN2 also) as default gateway (system/routing). But I do have several cut down everyday..You also have some parameters hidden in System/advanced/mescellanous ==> Load Balancing
/Klona
-
As the issue happens almost daily now, I was trying different things in the webinterface to figure out workarounds – one method that always seems to help is the filter reload page (status_filter_reload.php) and then clicking the reload filter button.
Is there some way to do this programmatically in pfsense (cron or such)?
of course, a real solution would be the proper way, but a working workaround is better than having to manually reload via the web interface.
-
Same problem with 2.2.4. In fact, I have to different problems related to this. Similar to your case, I have two wan connections configures with failover (tier 1 and tier 2).
-
When both are ok, it uses the wan with tier 1 (wan1), which is ok. If I disconnect wan1, it starts using wan with tier 2 (wan2), wich is also ok. After 2-3 minutes, I reconnect wan1, but the "Status Gateway" page doens't recover wan1, and keeps appearing as offline. If I restart apinger service, wan1 appears as online again.
-
Besides that, like in your case, although wan1 is back online, pfsense keeps using wan2 until I do some change or manually make wan2 fail.
To sum up, commutation is done ok in case of fail, but when you recover the failed gateway, nothing comes bak automatically. I've tested with two different machines, and always happens the same.
Any ideas??
Regards
-
-
Hi. I have the same problem as you guys (never switch to main unless I change something in config) but it seems we also have another thread about this:
https://forum.pfsense.org/index.php?topic=88723.0should we all post in a single thread?
Thanks
-
Update: I read somewhere on the forum that a broken install might be the problem.
I reinstalled from scratch and it's working, a simple failover only. I anyone interested, I will add the full setup later
-
Hi yanakis
And in your case, when it was failing, did pfsense detect that the gateway was online again or did you have to restart apinger like in my case? Apart from the fact that even after restarting apinger and main gateway appears as online again, it doesn't goes back…
After the reinstall, have you installed a backup with previous configuration or have you reconfigured everything manually??
Thanks in advance
-
Hi yanakis
And in your case, when it was failing, did pfsense detect that the gateway was online again or did you have to restart apinger like in my case? Apart from the fact that even after restarting apinger and main gateway appears as online again, it doesn't goes back…
After the reinstall, have you installed a backup with previous configuration or have you reconfigured everything manually??
Thanks in advance
Hi.
Yes, the gateway appeared back online (green), no need to restart apinger. I suggest you to add in GW settings, IP MONITOR google dns servers (8.8.8.8 and 8.8.4.4)After reinstall I reconfigured everything manually to avoid any bad setting in the previous config.
regards,
-
Hi yanakis
And in your case, when it was failing, did pfsense detect that the gateway was online again or did you have to restart apinger like in my case? Apart from the fact that even after restarting apinger and main gateway appears as online again, it doesn't goes back…
After the reinstall, have you installed a backup with previous configuration or have you reconfigured everything manually??
Thanks in advance
Hi.
Yes, the gateway appeared back online (green), no need to restart apinger. I suggest you to add in GW settings, IP MONITOR google dns servers (8.8.8.8 and 8.8.4.4)After reinstall I reconfigured everything manually to avoid any bad setting in the previous config.
regards,
Well, it seems that after more configs (just added dynamic dns & NAT) it stopped working again :(. No idea what's going on but I feel let down by pfsense.
Here are some logs, maybe some is figuring it out:
Sep 3 23:44:42 kernel: em0: link state changed to DOWN
Sep 3 23:45:01 check_reload_status: updating dyndns WAN_PPPOE
Sep 3 23:45:01 check_reload_status: Restarting ipsec tunnels
Sep 3 23:45:01 check_reload_status: Restarting OpenVPN tunnels/interfaces
Sep 3 23:45:01 check_reload_status: Reloading filter
Sep 3 23:45:02 php-fpm[42490]: /rc.dyndns.update: MONITOR: WAN_PPPOE is down, omitting from routing group WANFailover
Sep 3 23:45:02 php-fpm[42490]: /rc.dyndns.update: phpDynDNS (xxxxxxxxxxxxxx): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 3 23:45:03 php-fpm[52556]: /rc.filter_configure_sync: MONITOR: WAN_PPPOE is down, omitting from routing group WANFailover
Sep 3 23:45:46 check_reload_status: Rewriting resolv.conf
Sep 3 23:45:46 check_reload_status: Rewriting resolv.conf
______________________________________________________________________interface upheck_reload_status: Linkup starting em0
Sep 3 23:47:30 kernel: em0: link state changed to UP
Sep 3 23:47:37 php-fpm[18877]: /rc.newwanipv6: rc.newwanipv6: Failed to update WAN[wan] IPv6, restarting…
Sep 3 23:47:37 php-fpm[18877]: /rc.newwanip: IP has changed, killing states on former IP 86.xxx.xxx.xxx.
Sep 3 23:47:37 php-fpm[18877]: /rc.newwanip: MONITOR: WAN_PPPOE is down, omitting from routing group WANFailover
Sep 3 23:47:37 php-fpm[18877]: /rc.newwanip: ROUTING: setting default route to 10.0.0.1
Sep 3 23:47:37 php-fpm[18877]: /rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through 192.168.0.1
Sep 3 23:47:37 php-fpm[18877]: /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 10.0.0.1
Sep 3 23:47:40 php-fpm[18877]: /rc.newwanip: phpDynDNS: updating cache file /conf/dyndns_wannoip'yanakis.xxxxxxxxxx'0.cache: 188.xxx.xxx.xxx
Sep 3 23:47:40 php-fpm[18877]: /rc.newwanip: phpDynDNS (yanakis.xxxxxxxxxxx): (Success) DNS hostname update successful.
Sep 3 23:47:41 php-fpm[18877]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Sep 3 23:47:41 php-fpm[18877]: /rc.newwanip: Creating rrd update script
Sep 3 23:47:41 php-fpm[73772]: /rc.linkup: Accept router advertisements on interface em0
Sep 3 23:47:41 php-fpm[73772]: /rc.linkup: ROUTING: setting default route to 10.0.0.1
Sep 3 23:47:41 check_reload_status: Restarting ipsec tunnels
Sep 3 23:47:42 rtsold[8762]: <sendpacket>sendmsg on em0: Operation not permitted
Sep 3 23:47:43 php-fpm[18877]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 86.xxx.xxx.xxx -> 188.26.229.135 - Restarting packages.
Sep 3 23:47:43 check_reload_status: Starting packages
Sep 3 23:47:44 php-fpm[6347]: /rc.start_packages: Restarting/Starting all packages.
Sep 3 23:47:44 check_reload_status: updating dyndns wan
Sep 3 23:47:45 php-fpm[6347]: /rc.dyndns.update: phpDynDNS (yanakis.xxxxxxxxxxx): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 3 23:47:46 rtsold[8762]: <sendpacket>sendmsg on em0: Operation not permitted
Sep 3 23:47:50 rtsold[8762]: <sendpacket>sendmsg on em0: Operation not permittedAnd that's all in system logs.</sendpacket></sendpacket></sendpacket>
-
:( :(
That's bad news. Yes, I already use Google DNS servers 8.8.8.8 and 8.8.4.4 as monitor IP's, and DNS association selected in the General Section. I'm pretty sure configuration is OK (at this point, I've reviewed it a hundred times :) ) and this looks like a pfsense problem.
I've openned a bug (https://redmine.pfsense.org/issues/5090#change-20401). Would be good if you add your comments there, so they can investigate the problem.
-
:( :(
That's bad news. Yes, I already use Google DNS servers 8.8.8.8 and 8.8.4.4 as monitor IP's, and DNS association selected in the General Section. I'm pretty sure configuration is OK (at this point, I've reviewed it a hundred times :) ) and this looks like a pfsense problem.
I also tend to believe the same, pfsense has an issue. I spent a week trying to figure this out, I will try one more setup from scratch and make snapshots in vmware.
I've openned a bug (https://redmine.pfsense.org/issues/5090#change-20401). Would be good if you add your comments there, so they can investigate the problem.
-
…and this looks like a pfsense problem...
I cannot second that!
I have this working for quite some time now with WAN1 (100Mb cable) and a rather old WAN2 (6Mb DSL).
I have failover to W2 if W1 is down and immediately W1 again when available.Show us your System | Routing | Gateway Groups page.
-
…and this looks like a pfsense problem...
I cannot second that!
I have this working for quite some time now with WAN1 (100Mb cable) and a rather old WAN2 (6Mb DSL).
I have failover to W2 if W1 is down and immediately W1 again when available.Show us your System | Routing | Gateway Groups page.
Hi. Thanks for reply. Please see the screenshots
-
You have one or two Gateway Groups defined? The one with time stamp 02-25-44.
What you call "WANGROUP" is easier to handle when called " PPPoE 2 UPC"
Now you need an additional "UPC 2 PPPoE" group with reversed tiers.
Add another firewall rule for that one as well and it should work.And start with setting both "Trigger levels" to "Member Down".
-
Hi again
But this is a temporal solution you've found to make it work or is the normal configuration for failover? I can't understand why we have to create two Gateways Groups, cause we only want one failover direction, not the oposite. I've been reviewing documentation and online info, and you should only need to create one gateway group.
When you create the second firewall rule with the inverted Tier numbers, if that rule goes after the normal rule, the firewall should never reach the second one, because it looks at the rules sequentially, so when it reaches the first one, it directs the traffic through the main group (the group is online, because wan2 is online). It should never reach the second rule, so if it's doing it in your case, I think something strange happens.
I'm probably wrong or I'm missing something, but I just want to clarify if with your configuration it's working because that's the normal config or is some other problem that makes it work although is not the right configuration.
Now I can't test it in my client, because is a production system, but I'll try to make a demo in our office to see if I can verify your configuration. If it works, it could be a good temporal patch to solve the problem, but I still think something is not going well. The group should recover gateways and order preference automatically (that's what Tier is for).
Thanks for your help
-
Hi again
But this is a temporal solution you've found to make it work or is the normal configuration for failover? I can't understand why we have to create two Gateways Groups, cause we only want one failover direction, not the oposite. I've been reviewing documentation and online info, and you should only need to create one gateway group.
When you create the second firewall rule with the inverted Tier numbers, if that rule goes after the normal rule, the firewall should never reach the second one, because it looks at the rules sequentially, so when it reaches the first one, it directs the traffic through the main group (the group is online, because wan2 is online). It should never reach the second rule, so if it's doing it in your case, I think something strange happens.
I'm probably wrong or I'm missing something, but I just want to clarify if with your configuration it's working because that's the normal config or is some other problem that makes it work although is not the right configuration.
Now I can't test it in my client, because is a production system, but I'll try to make a demo in our office to see if I can verify your configuration. If it works, it could be a good temporal patch to solve the problem, but I still think something is not going well. The group should recover gateways and order preference automatically (that's what Tier is for).
Thanks for your help
Hi. Indeed, a second rule makes no sense to me also but I will test it as advised, test it for the second time actuallly. I've tried even with 3 rules, same results.
In the mean time I've done more testing and got to a conclusion but first please let me know how did you simulate the main wan failure?
-
Second rule and gateway group is not necessary unless you want some traffic to prefer the second route and fail over the other way.
You only need the one Tier 1 to Tier 2 to fail all traffic over is that direction.
It certainly should recover and "fail back" when the Tier 1 route comes back up.
-
Second rule and gateway group is not necessary unless you want some traffic to prefer the second route and fail over the other way.
You only need the one Tier 1 to Tier 2 to fail all traffic over is that direction.
It certainly should recover and "fail back" when the Tier 1 route comes back up.
Well, in my case it doesn't. WAN1 is a PPPOE connection, and after I re-plug or the Ethernet cable in WAN 1 all connections still go through OPT1.
For the testing purpose, I added another router in front of pfsense so it won't have to use a PPPOE connection, I assigned to WAN1 a static IP like OPT1 has. In this case to some extent it works if I unplug/re-plug the connection on the first router (take down the ISP Interface, the fiber media converter) so both WAN and OPT1 stay up in pfsense. Still, some sites refuses to load in Chrome with the following error: DNS_PROBE_FINISHED_NXDOMAIN
So, for now my only conclusion is that there is a problem with pfsense when you unplug and re-plug the cable on the interface using a PPPOE connection. The dns error is still a mystery to me, I still need to figure it out.
-
Well you need to fix your DNS. Sounds like it might not be working right on one or both WANs. Are you using the forwarder or the resolver?
It shouldn't matter which WAN the resolver uses because it should only be trying to talk to authoritative name servers that should accept queries from everywhere.
The problem lies in forwarders because you usually point the forwarder at ISP caching servers and they might only accept connections from their network so it matters which DNS servers are used out which interface.
-
Well you need to fix your DNS. Sounds like it might not be working right on one or both WANs. Are you using the forwarder or the resolver?
It shouldn't matter which WAN the resolver uses because it should only be trying to talk to authoritative name servers that should accept queries from everywhere.
The problem lies in forwarders because you usually point the forwarder at ISP caching servers and they might only accept connections from their network so it matters which DNS servers are used out which interface.
I tried both the resolver and the forwarder, some sites are just not resolved. Unfortunately I don't think I can use pfsense in a production environment, for me at least failover it's not working with pppoe :(.
should "State Killing on Gateway Failure" should be on?
Thanks
-
Depends on whether or not you want states killed on a gateway failure.
-
Depends on whether or not you want states killed on a gateway failure.
Well, isn't better to have them reset on a gw failure? The definition is a bit tricky for this option
-
I only skimmed through this thread so I apologize if this was already suggested but – are you certain your clients are set to use the pfSense IP as their DNS resolver? If e.g. you have a gateway defined with a custom monitor IP of 8.8.8.8 or the DNS servers on your General settings page are locked to a specific gateway, then static routes are built which will force traffic out that specific gateway, even if it's down. So this could result in DNS being "dead" when one of the gateways goes down. Is this possibly what's happening?
-
I only skimmed through this thread so I apologize if this was already suggested but – are you certain your clients are set to use the pfSense IP as their DNS resolver? If e.g. you have a gateway defined with a custom monitor IP of 8.8.8.8 or the DNS servers on your General settings page are locked to a specific gateway, then static routes are built which will force traffic out that specific gateway, even if it's down. So this could result in DNS being "dead" when one of the gateways goes down. Is this possibly what's happening?
Monitor IPs are currently set to one of each ISP, in General I have a pair of DNSes set for each gateway (four servers in total). Clients DNS is manullay set 192.168.1.1 (pfsense)
-
I tried both the resolver and the forwarder, some sites are just not resolved.
If you do not know how to get more information than that about what's actually happening, you are probably in over your head.
-
I tried both the resolver and the forwarder, some sites are just not resolved.
If you do not know how to get more information than that about what's actually happening, you are probably in over your head.
Oh, nice. What can I say?Thanks? :)…..Thanks.
-
Hi
yanakis, in my case we have the fiber media converter and the router (not PPPoE), and happens the same. I switched off or disconnected the router, but never tried switching off media converter (good idea).
In my last installations I don't usually use DNS forwarder/resolver for localhost, but in this case I do (I configured it in the past and never change it). Have you tried deactivating that option in General Settings? Just to see if something changes.
I understand luckman212 concerns about DNS and static routes created by pfsense for each DNS associated to a wan, but in my case we had two different DNS configured and working, and failed. And in any case, once wan is recovered again, DNS works again and everything should work again.
By the way, I tried with "State Killing on Gateway Failure" on and off, and recover fails in both cases. I keep it unchecked, because with external sip connections is mandatory to make failover work (at least in my case). And I personally prefer to reset states if a gateway fails, to avoid problems.
Regards
PD: I don't think you are in over your head… Thanks for all
-
Hi
yanakis, in my case we have the fiber media converter and the router (not PPPoE), and happens the same. I switched off or disconnected the router, but never tried switching off media converter (good idea).
In my last installations I don't usually use DNS forwarder/resolver for localhost, but in this case I do (I configured it in the past and never change it). Have you tried deactivating that option in General Settings? Just to see if something changes.
I understand luckman212 concerns about DNS and static routes created by pfsense for each DNS associated to a wan, but in my case we had two different DNS configured and working, and failed. And in any case, once wan is recovered again, DNS works again and everything should work again.
By the way, I tried with "State Killing on Gateway Failure" on and off, and recover fails in both cases. I keep it unchecked, because with external sip connections is mandatory to make failover work (at least in my case). And I personally prefer to reset states if a gateway fails, to avoid problems.
Regards
PD: I don't think you are in over your head… Thanks for all
Well, I left empty the DNS fields in General but failback to WAN still not working after WAN recovery unless I change something in Firewall or Routing and apply changes :(
-
…and this looks like a pfsense problem...
I cannot second that!
I have this working for quite some time now with WAN1 (100Mb cable) and a rather old WAN2 (6Mb DSL).
I have failover to W2 if W1 is down and immediately W1 again when available.Show us your System | Routing | Gateway Groups page.
Hi Cris. Can you please post your setup? Thanks
-
Well, Derelict wrote that my config seems to be a bit more complicated than necessary.
Since I trust him I usually would test his suggestion first and post afterwards. I just don't have the time for that in the foreseeable future…I only use DSL as failover (it's 6Mbit) and rely on cable which is 100Mbit.
You will know why if you have teen kids...
Just checked and failover to DSL is working as well as fallback to cable when available again.I set this up about a year ago and used the pfsense docs for that.
-
I have mine set up exactly like you do (A group with Tier 1 Cable, Tier 2 DSL and a group with Tier 1 DSL, Tier 2 Cable).
I was just commenting it's only necessary if you want to have rules that prefer the other circuit while maintaining the ability for those rules to fail over too.
I tested my failover yesterday since I was putting new splitters on my cable in anticipation of MoCA 2.0. It all worked exactly as configured and when I was done it brought my Tier 1 back online just as it has many times before.
-
i didn't criticize! I only mentioned that I might be done with half the work. ;)
-
Hi again
Past week we've installed a new machine with 2.2.4 and two WAN with failover, and same problem. In this case we have to different LANs, and each one has one failover group with different order (one with wan1->wan2 and the other with wan2->wan1), and none of them redirect traffic to the main one when it's recovered (we have to do some change and Save, as yanakis says).
It's a new installation without anything strange. We run several tests in both directions, and I can confirm the problem exists. Never went back automatically to the recovered main wan.
We didn't find nothing new or more clues, it just doesn't work.
Regards
-
+1 with arcanos.
In fact, instead of attempting to troubleshoot this and failing, it would be better if someone that has this working, would post a complete series of screenshots showing their setup. Then we can all learn from a working environment.
-
Is PPPoE a common factor for those that don't work? Both my WANs are DHCP.
There's really nothing to it. Create a gateway group with a Tier 1 and Tier 2 with member down as the trigger level and policy route to it.
-
Correct, PPPoE is the default and DHCP is the failover.
-
Not PPPoE in my case. This last case are two cable connections with routers with NAT and DMZ pointing to the wan interface of pfsense. But I've seen the problem with DSL and cable in bridge mode.
-
On the first view I have the same issue but looking deeper I can see that my Gateway keeps really offline until I reapply the interface config page or reboot.
Short decription of config and behavior:
I have two gateways (1. fiber / 2. cable modem) with a routing group for balancing (tier 1 / tier 1). Both gateways are monitored against external DNS servers. The routing group is defined as gateway in FW rule. "Use sticky connection" on System-advanced-misc. is on.
At the beginning after reboot all works fine an traffic is distributed to both gateways with weight 1:4.
But after some minutes / hours always second gateway goes offline (100% package loss) and keeps this status until I reapply the interface config or reboot. It's not an apinger problem. The gateway is really broken. A ping from Diagnostic - ping with source of gateway doesn't work (100% loss). The cable modem isn't disconnect and it works if a plugin a notebook there. So Pfsense stops the gateway really and keep it broken. Even if I disconnect the lan wire and reconnect no reaction.
Same happens on my backup Pfsense which is running in CARP mode. There is no traffic load but GW stops also.
If I set routing group in redundant mode (GW 1 tier 1 / GW 2 tier 2 OR GW 2 tier 1 / GW 1 tier 2) then all work OK. The gateways keep online. Also after reconnection of wire the interface comes online again.
My estimation is that there must be something wrong with balancing gateways. But I need the capacity of both gateways.
-
you should put a working monitor ip for each interfaces like dns ip
-
Hello,
I'm facing the same problem. I've read those (with no solution)
I'm runing 2.3.1 wit 2 wans (1 cable/main and 1dsl-pppoe/secondary), 2 groups. Failover is working (trigger ok) but not switching back after weak connection is back at 100%.
Ready to send screenshots. Ask
logs:
Jun 16 13:49:20 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 9274us stddev 5829us loss 21%
Jun 16 13:49:42 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 7586us stddev 4056us loss 15%
Jun 16 13:53:58 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 7362us stddev 4941us loss 21%
Jun 16 13:54:15 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 6543us stddev 3719us loss 19%
Jun 16 13:54:39 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 6692us stddev 3840us loss 21%
Jun 16 13:54:57 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 6644us stddev 3338us loss 15%
Jun 16 13:56:03 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 8839us stddev 5402us loss 21%
Jun 16 13:56:19 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 8292us stddev 4864us loss 19%
Jun 16 13:56:43 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 8431us stddev 5556us loss 22%
Jun 16 13:57:02 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 7940us stddev 5158us loss 15%
Jun 16 13:58:35 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 12630us stddev 12111us loss 21%
Jun 16 13:58:53 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 8282us stddev 4592us loss 15%
Jun 16 13:59:21 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 8983us stddev 5856us loss 21%
Jun 16 13:59:32 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 8447us stddev 5473us loss 16%
Jun 16 13:59:58 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 8206us stddev 5630us loss 21%
Jun 16 14:00:11 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 7373us stddev 4132us loss 14%
Jun 16 14:01:14 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 8049us stddev 4691us loss 21%
Jun 16 14:01:44 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 7842us stddev 3865us loss 18%
Jun 16 14:01:47 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 7944us stddev 3892us loss 21%
Jun 16 14:02:18 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 7717us stddev 3673us loss 12%
Jun 16 14:03:51 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 7952us stddev 4608us loss 21%
Jun 16 14:04:16 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 7030us stddev 3415us loss 12%
Jun 16 14:04:28 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 7711us stddev 4555us loss 21%
Jun 16 14:04:56 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 7538us stddev 4081us loss 14%
Jun 16 14:05:10 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 8504us stddev 5216us loss 21%
Jun 16 14:05:32 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 8245us stddev 4794us loss 13%
Jun 16 14:05:51 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 8544us stddev 5200us loss 21%
Jun 16 14:06:14 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 7369us stddev 3934us loss 16%
Jun 16 14:06:26 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 7862us stddev 4613us loss 21%
Jun 16 14:06:56 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 7151us stddev 3861us loss 13%
Jun 16 14:11:12 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 7910us stddev 4976us loss 21%
Jun 16 14:11:26 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 6648us stddev 3553us loss 15%
Jun 16 14:11:49 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 6910us stddev 4027us loss 21%
Jun 16 14:12:11 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 6271us stddev 2901us loss 15%
Jun 16 14:12:28 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 6705us stddev 3698us loss 21%
Jun 16 14:12:52 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 6371us stddev 2763us loss 11%
Jun 16 14:13:45 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Alarm latency 8346us stddev 5486us loss 21%
Jun 16 14:14:57 dpinger WAN2CABLEGW xxx.xxx.xxx.xxx: Clear latency 8065us stddev 5624us loss 16%