Pfsense 2.1-release - Gateway aPinger broken?

miles267

For those who were encountering this issue, it appears to have been corrected in the pfsense 2.1.1 PRERELEASE. Can any of the ops confirm?

krylou

Issue reproduced at my env too. I have 2 installations, each with 2 WAN. Sometimes i have to restart apinger manually to get back WAN online.

Maybe someone could share cron script to do this?

krylou

I`ve found temporary solution - restarting apinger every 5 minutes.

Open shell via putty

Go to /usr/local/www
and create file apinger.php with the following content:
require_once("/etc/inc/service-utils.inc");
require_once("/etc/inc/globals.inc");
require_once("/etc/inc/vslb.inc");
require_once("/etc/inc/gwlb.inc");

service_control_restart(apinger,restartservice)
?>

chmod 755 apinger.php

To run it manually you can just open http://pfsense/apinger.php

Then install cron package via pfsense UI and add new job:
*/5 * * * * /usr/local/bin/php /usr/local/www/apinger.php

Jabroni

Same issue as everyone on this forum..

Different plataforms (2 running as VM under ESXi, and other on an alix) on different type of interfaces (some ethernets, and some openvpn interfaces). And ive been restarting the apinger in order to 'clear' the Packet loss % so the gw can be detected up again…

Hope pfsense devs fix this on the next release since its a bummer for pfsenses users who rely on multiwan

coolspot

@miles267:

For those who were encountering this issue, it appears to have been corrected in the pfsense 2.1.1 PRERELEASE. Can any of the ops confirm?

I'm still having the issue with 2.1.1 release; pfsense reports 2% packet loss, but when I ping from the command line, it's all clean?

eSpezi

Similar issues here.

How do you connect to your WAN(s)?
Do you use a modem or a router in front of pfSense?

It seems that a (DSL/Cable) router makes more problems at my installations.
Unfortunately it is not always possible to exchange router with a modem.

Are there any upcomming solutions to the issues, without hacking around in pfS?

Cheers,
Harry

camlin

I am using 2.1.2, and I can confirm there is definitively a problem. Now, in my case, it does not look like a pure aPinger problem

The issue is that I am using PPoE for my WAN connection and the script responsible for configuring aPinger (gwlb.inc) gets it wrong.

Basically PPoE marks the gateway as "dynamic" but the script tries to check if that's an IP address (which is clearly isn't!).

This then leads aPinger to fail and the gateway report page stays stuck on "Pending"…

I am going to try and see if I can hack the thing on my test machine for the time being but I would welcome any recommendations on how I should report this.

Thanks.

camlin

Yes, I can confirm I got it to work by changing the gwlb.inc script.

Part of the problem is that GUI checks (the story about providing only an IP address) are present here. So when PPoE sets the IP to "dynamic" it all goes wrong (the script says: Not an IPv4 or an IPv6 address => Do not configure apinger => apinger then fails).

Ideally the PPoE system should be improved to provide something more useful than just "dynamic" but this is probably a much bigger scope of work.

In the meantime, changing the block (in the setup_gateways_monitor method):


if (is_ipaddrv4($gateway['gateway'])) {
			...
} else if (is_ipaddrv6($gateway['gateway'])) {
			...
} else
			continue;

by

		if ($gateway['ipprotocol'] == "inet") {
			$gwifip = find_interface_ip($gateway['interface'], true);
			if (!is_ipaddrv4($gwifip))
				continue; //Skip this target

			/*
			 * If the gateway is the same as the monitor we do not add a
			 * route as this will break the routing table.
			 * Add static routes for each gateway with their monitor IP
			 * not strictly necessary but is a added level of protection.
			 */
			if (is_ipaddrv4($gateway['gateway']) && $gateway['monitor'] != $gateway['gateway']) {
				log_error("Removing static route for monitor {$gateway['monitor']} and adding a new route through {$gateway['gateway']}");
				mwexec("/sbin/route change -host " . escapeshellarg($gateway['monitor']) .
					" " . escapeshellarg($gateway['gateway']), true);
			}
		} else if ($gateway['ipprotocol'] == "inet6") {
			if ($gateway['monitor'] == $gateway['gateway']) {
				/* link locals really need a different src ip */
				if (is_linklocal($gateway['gateway'])) {
					$gwifip = find_interface_ipv6_ll($gateway['interface'], true);
				} else {
					$gwifip = find_interface_ipv6($gateway['interface'], true);
				}
			} else {
				$gwifip = find_interface_ipv6($gateway['interface'], true);
				if (is_linklocal($gateway['monitor'])) {
					if (!strstr($gateway['monitor'], '%')) {
						$gateway['monitor'] .= "%{$gateway['interface']}";
					}
				} else {
					// Monitor is a routable address, so use a routable address for the "src" part
					$gwifip = find_interface_ipv6($gateway['interface'], true);
				}
			}
			if (!is_ipaddrv6($gwifip))
				continue; //Skip this target

			/*
			 * If the gateway is the same as the monitor we do not add a
			 * route as this will break the routing table.
			 * Add static routes for each gateway with their monitor IP
			 * not strictly necessary but is a added level of protection.
			 */
			if (is_ipaddrv6($gateway['gateway']) && $gateway['monitor'] != $gateway['gateway']) {
				log_error("Removing static route for monitor {$gateway['monitor']} and adding a new route through {$gateway['gateway']}");
				mwexec("/sbin/route change -host -inet6 " . escapeshellarg($gateway['monitor']) .
					" " . escapeshellarg($gateway['gateway']), true);
			}
		} else { 
			continue;
		}

Please note that in my case I also had to change the logic of how it determines the source address: I was trying to force the monitoring of a routable IPv6 address but the script was forcing a local link address as the source address because the default monitor was a local link address (so I change the order so that it considers first if the monitor gateway address had been manually specified in the config).

HTH

mukidu

Hi Guys,
I am experiencing exactly the same issue, got 2 WANs configured for loadbalancing, but since the update I noticed that loadbalancing is not working anymore besides the fact that the 2nd WAN in marked as offline.
I have also a tier 2 H3g connection set up on the loadbalancing, I noticed that pfsense is keeping it up even though both WANs are up and running….
My broadband connections are both Pppoe.

I'am planning to roolback to 2.0.3

Bye

camlin

Yes, that does not surprise me (quite clearly part of the load balancing work depends on properly determining which gateway is working fine).

For info, I have submitted my changes to GitHub. They improve things a bit but I still see issues after link loss where the monitoring of my IPv6 gateway is not properly reset. I suspect this is caused by timing issues where pfSense is trying to setup apinger whilst not everything is ready yet…

krylou

@mukidu:

Hi Guys,
I am experiencing exactly the same issue, got 2 WANs configured for loadbalancing, but since the update I noticed that loadbalancing is not working anymore besides the fact that the 2nd WAN in marked as offline.
I have also a tier 2 H3g connection set up on the loadbalancing, I noticed that pfsense is keeping it up even though both WANs are up and running….
My broadband connections are both Pppoe.

I'am planning to roolback to 2.0.3

Bye

What version you are using? Does anyone tried 2.1.3 version?

camlin

The fix I submitted was accepted 13 days ago, however it was too late for it to make it in 2.1.3…

So I suppose we will only see it on 2.1.4 and later.

My fix improves the situation as before link monitoring would just not work at all with PPoE (and probably any kind of dynamically established links, so that would include VPNs). This means you can now see packet loss and latency on PPoE links (and I suspect VPN links too, although I haven't tried).

Now, my fix still needs some work admittedly. I am still seeing some issues sometimes after a complete link loss. When the link comes back up it looks like pfSense calls the script way too early (at a stage where we don't even know the IP to monitor). It seems to be a timing issue as this only happens from time to time.

It is easy to "fix" as once the link is established you just wait a few seconds and then have to stop/start the "apinger" service and it will then get the proper addresses.

But it still means the solution won't be good enough to allow proper automated link failover...

I am probably going to try and have a look at some point, but I suspect it is an issue that will be caused by something deep in the internals of pfSense. So could well be out of my league...

yaboc

i have mutli wan set up with cable as WAN1 and PPPoE DSL as WAN2. my cable gateway is up and running but PPPoE is showing as Pending and restarting the apinger doesn't bring it up online. but im able to ping using DSL gateway. is this the problem that apinger is causing ?

im on the 2.1.3

pfSensible

I ran into this problem for the first time. I have WAN failover configured but not load balancing. I rebooted one WAN modem and now the GW monitor just died and will not come back.

It is configured to ping every 2 seconds

pfSense_GW_group.PNG_thumb

mukidu

@krylou:

@mukidu:

Hi Guys,
I am experiencing exactly the same issue, got 2 WANs configured for loadbalancing, but since the update I noticed that loadbalancing is not working anymore besides the fact that the 2nd WAN in marked as offline.
I have also a tier 2 H3g connection set up on the loadbalancing, I noticed that pfsense is keeping it up even though both WANs are up and running….
My broadband connections are both Pppoe.

I'am planning to roolback to 2.0.3

Bye

What version you are using? Does anyone tried 2.1.3 version?

I upgraded from 2.0.1 to 2.1.2, I also tried 2.1.3 but the issue is still there…

pubmsu

I am on 2.1.3 and I can confirm huge issue with apinger. My otherwise stable WANs are shown as "Pending" no matter what monitor IP I set. Even setting the gateway IP address one hop away causes the apinger to show the WAN as Pending and therefore exclude it from the gateway group, causing interruptions all the time.

BTW, all of you who are experiencing this issue, is it the case that this is due to having traffic shaper configured? Or does it happen even without traffic shaper?

This is a very core, basic function and must be fixed asap.

BobDarnit

Can anyone specify if apinger has been fixed yet… or if there is a good work around? I've got two WANs, both static, and since upgrading to 2.1p1 it's needed the apinger service reset frequently. I've got multiple other devices at other locations, DHCP and Static that have had to reset the apinger since the 2.1 upgrades started.

Thanks for any help.

Bob

pubmsu

Hi Bob,

Please try the work around we used and got good result:

https://forum.pfsense.org/index.php?topic=78502.0

Thanks,
msu

saxd40

There's definitely still a problem in 2.1.4 and since 2.1.5 just released and I know that they are still trying to figure out the problem, I'm assuming it is still an issue in 2.1.5.

I have this problem on ONE pfSense box with a flakey T1 (yes, they DO exist, but apparently only in Detroit). I have never seen this on any other of my many pfSense installations.

They are aware of the issue and trying to figure it out. I'm in communication with them and have given them access to the ONE firewall where I have this problem.

In the meantime, I'm going to create a CRON job to restart apinger service at given intervals. BTW, this command will restart it:

/usr/local/sbin/pfSsh.php playback svc restart apinger

naras

Yes, it's still there, hope it will disappear in 2.2.