Apinger only working on wan 8/6/13 64bit snapshot

phil.davis

WAN - DHCP, attached to a WiMax device that has its own private IP and NATs out to internet. (Gets an address 10.1.1.x from the WiMax DHCP server)
OPT1 - static private IP to a TP-Link ADSL router, which again NATs out to the real internet.

WANGW - Monitor IP 8.8.8.8 - latency thresholds 4000 to 5000ms - packet loss thresholds 40 to 50% - probe interval 2 sec - down 30 sec.

OPT1GW - Monnitor IP 8.8.4.4 - latency thresholds 4000 to 5000ms - packet loss thresholds 40 to 50% - probe interval 2 sec - down 30 sec.

These connections have reasonably high latency normally, and when saturating the links with downloads the latency would normally go high, hence the wacky high gateway monitoring parameters to prevent gateways from being declared down when they are in fact "working".

Unfortunately I can't tell the exact symptoms, since it was a phone call and instructions about how to go back. The CF card multi-slice thing is very useful. As per previous post, I do know that links were coming and going, as I observed OpenVPN site-to-site links establishing for a minute or so, then dropping out.

I am at another site with multi-WAN at the moment. If I can gain a little confidence that apinger in the latest build is working OK and seems to be controlling failover OK, then I can upgrade here this evening and will be around to monitor it the next few days. This site is on a 31 Jul snap, which was before the recent apinger changes. So I will easily be able to switch back slices if needed. (I am not at home with a real test box)

jimp

I pulled up another VM that has a better multi-WAN config and it was still OK there.

Though when I was experiencing problems before the latest round of fixes, it was worse with high-latency gateways, so it's possible that the issue is compounded by the actual latency there. To reproduce it you may have to artificially induce the same level of latency.

vielfede

@jimp:

I pulled up another VM that has a better multi-WAN config and it was still OK there.

Though when I was experiencing problems before the latest round of fixes, it was worse with high-latency gateways, so it's possible that the issue is compounded by the actual latency there. To reproduce it you may have to artificially induce the same level of latency.

Did you try to test failover?
As I state on this thread http://forum.pfsense.org/index.php/topic,65455.0.html, on RC1-20130812 failover does not work anymore (in my case).
Thanks
FV

ggzengel

I have 2 pfsense with this.

1. pfsense:
2.1-RC1 (amd64)
built on Thu Aug 8 14:25:22 EDT 2013
FreeBSD 8.3-RELEASE-p9
1 WAN (0.4ms) (always green). The apinger shows 0ms which is wrong since update (pfsense1_WAN.png).
2 OpenVPN Server (23ms + 16ms) which have growing latencies. The corresponding clients at the other sides are green.

2. pfsense:
2.1-RC1 (amd64)
built on Wed Aug 7 20:59:21 EDT 2013
FreeBSD 8.3-RELEASE-p9
2 WANs: static WAN (1.4ms) + DSL (22ms). The DSL has growing latency. WAN shows less latency (pfsense2_WAN.png).
2 OpenVPN Server. Both have growing latency.
1 OpenVPN Client which has growing latency, too.

pfsense1_WAN.png_thumb

pfsense2_DSL.png_thumb

pfsense2_LAN.png_thumb

pfsense2_OVPN1.png_thumb

pfsense2_OVPN2.png_thumb

pfsense2_OVPN3.png_thumb

pfsense2_WAN.png_thumb

jimp

Those snapshots are known to have apinger issues, upgrade to a current snapshot.

ggzengel

I forgot to write:
The LAN shows strange values, too.

jimp

@vielfede:

@jimp:

I pulled up another VM that has a better multi-WAN config and it was still OK there.

Though when I was experiencing problems before the latest round of fixes, it was worse with high-latency gateways, so it's possible that the issue is compounded by the actual latency there. To reproduce it you may have to artificially induce the same level of latency.

Did you try to test failover?
As I state on this thread http://forum.pfsense.org/index.php/topic,65455.0.html, on RC1-20130812 failover does not work anymore (in my case).
Thanks
FV

It does appear as though the filter reload at the end of the apinger event isn't doing what it should there. I'll need to run some more tests to narrow it down though.

ggzengel

I updated pfsense1.
While the first minutes a didn't see growing latencies.
But WAN still has 0ms in RRD and is less than real 0.4ms.

jimp

The lack of failover working seems to be this:
http://redmine.pfsense.org/issues/3146

phil.davis

2.1-RC1 (i386)
built on Wed Aug 14 14:47:24 EDT 2013
FreeBSD 8.3-RELEASE-p9

Looking good so far. Someone was downloading on our 1Mbps link speed for an hour or so. Latency went up to around 930ms. When the download finished the latency dropped back to under 200ms. The backup link latency is hovering around 300ms. During all this time there was no "panic" from apinger, check_reload_status or anything else to failover links.

At another site, latency on one link is changing in a range from 400 to 1100ms (people working it hard) and another 120ms (less used). apinger is coping fine.

ggzengel

After updating to the latest release
2.1-RC1 (amd64)
built on Thu Aug 15 03:12:29 EDT 2013
FreeBSD 8.3-RELEASE-p9

the fast interfaces still shows 0ms instead of 0.400 on dashboard and RRD.

Raul Ramos

Hi

To ggzengel: have you "Disable Gateway Monitoring"?

mastahfr

@ggzengel:

After updating to the latest release
2.1-RC1 (amd64)
built on Thu Aug 15 03:12:29 EDT 2013
FreeBSD 8.3-RELEASE-p9

the fast interfaces still shows 0ms instead of 0.400 on dashboard and RRD.

Same here, 2 of my 3 gateways show 0ms but they should show respectivly arround 14ms and 1ms.
My main gateway (Wan) is showing 1ms, which is correct.

ggzengel

It's not exactly 0ms and it goes up on some interfaces.

1.png_thumb

2.png_thumb

3.png_thumb

4.png_thumb

5.png_thumb

6.png_thumb

phil.davis

2.1-RC1 (i386)
built on Thu Aug 15 16:30:19 EDT 2013
FreeBSD 8.3-RELEASE-p9

2 multi-WAN systems are on this snap now. Gateway status is reporting reasonable latency numbers, and RRD quality numbers also look OK. I only have IPv4 and 2 gateways on each, so I can't speak for IPv6 or more complex systems with more gateways.

vielfede

@jimp:

The lack of failover working seems to be this:
http://redmine.pfsense.org/issues/3146

2.1-RC1 (amd64)
built on Thu Aug 15 16:30:12 EDT 2013
FreeBSD 8.3-RELEASE-p9

Failover (internet) is working again!

Unfortunately squid proxy failover (http://forum.pfsense.org/index.php/topic,60977.0.html) does not.
Maybe this is off topic (it's not due to apinger issues) but I cant find any help to get it works on 2.1.

jimp

It's not related. Keep it out of this thread.

NOYB

@ggzengel:

After updating to the latest release
2.1-RC1 (amd64)
built on Thu Aug 15 03:12:29 EDT 2013
FreeBSD 8.3-RELEASE-p9

the fast interfaces still shows 0ms instead of 0.400 on dashboard and RRD.

Same here. But on today's snapshot.
2.1-RC1 (i386)
built on Fri Aug 16 16:28:22 EDT 2013

Up-until now it had been working okay.
It works with a manually entered alternate monitoring address. But not with the interfaces' gateway. The WAN interface is working okay. But the optional interface is reporting 0ms. Reality is about 0.35ms.

Both interfaces are VLAN's on the same physical interface (bfe0).

doktornotor

Still showing 0ms everywhere with the Aug 17 snapshot…

jimp

After Ermal's last changes it should have been back to normal there, it was on my test VM. I was seeing 0.3-0.8ms reported. I rebuilt apinger again on the snapshot builders to see if maybe I missed rebuilding it one of them yesterday. Try it again later tonight/tomorrow when the next snap shows up.