Pfsense 2.1-release - Gateway aPinger broken?

miles267

Brutal, appreciate your insights. Fortunately or not, I'm glad to hear I'm not alone. Figured this thread would be lighting up w/ similar experiences unless others just haven't realized their experiencing the same issue. I've since rebooted and disabled the Gateway Monitor completely. Since then, everything appears stable. Despite not seeing the ping response, it doesn't appear to impact my connection as it's always been stable and responsive. Considering the apinger service bugs, I'm not entirely convinced what I was seeing (green, 0% loss, etc.) was even accurate.

zillo

I can confirm this problem.
I run a system with 2 WAN. Apinger degrade my second interface WAN2 in the first place. After enough time it will affect the first WAN interface too. I need to restart apinger service daily for the moment.
If there will be necessary I will provide required information.

Yours
VV

miles267

Thanks Zillo. My experience is similar to yours only w/ a single WAN interface. I hope that this issue will be resolved soon.

BTW, are there still daily snapshots available with 2.1-release? If so, what is the URL for those to enter into the firmware update interface?

sibsib

Hi,

New member of the forum, but using pfsense since half a year, I have similar issues.

I have two Internet access, the whole was working fine, and since one week (a little more, probably), I had a lot of loos on my 'primary link'. One good part of that trouble is that actually, my backup didn't work, now it does :D

But still, I swapped my 'primary' and my secondary access, and for sure, I started to see my line going offline. The stangest thing is to see in the monitor dashboard, message like : RTT 5.5 ms, loss 100 % ???

I'll see if restarting the apinger process is also a solution here.

BTW, can I easily go back to pfsense 2.0.3 ?

Thanks,
Pascal

Reiner030

@miles267:

Have even created floating rules in traffic shaper (HFSC) to prioritize ICMP packets during heavy traffic times. Frustrating.

Floating rules are broken so try to avoid them…. perhaps then your problem is temporary fixed ?

sibsib

Hi,

Even after restarting apinger, no luck I did lost my gateway, (at least pfsense 'sense' it !)

As my pfsense is actually a VM in my little ESXi, I just installed a PFsense 2.0.3 in a brand new VM, restored the configuration from the 2.1 (with a big warning about my silly action), and so far, it seems that everything is OK.

The plan is to wait a week, and if everything is OK, I'll try again an 2.1 update (but with an active Snap, to be able to go bak a little faster !).

I'll let you know.

For the record : pfsense is barely from scratch, with only three additional packages : ntop, Open VM Tools and tftp (to let my SIP phone boot).

Thanks,
Pascal

ronnysa

I have the exact same problem, apinger is broken in 2.1 I had to roll back 4 installs to 2.0.3

miles267

For those who were encountering this issue, it appears to have been corrected in the pfsense 2.1.1 PRERELEASE. Can any of the ops confirm?

krylou

Issue reproduced at my env too. I have 2 installations, each with 2 WAN. Sometimes i have to restart apinger manually to get back WAN online.

Maybe someone could share cron script to do this?

krylou

I`ve found temporary solution - restarting apinger every 5 minutes.

Open shell via putty

Go to /usr/local/www
and create file apinger.php with the following content:
require_once("/etc/inc/service-utils.inc");
require_once("/etc/inc/globals.inc");
require_once("/etc/inc/vslb.inc");
require_once("/etc/inc/gwlb.inc");

service_control_restart(apinger,restartservice)
?>

chmod 755 apinger.php

To run it manually you can just open http://pfsense/apinger.php

Then install cron package via pfsense UI and add new job:
*/5 * * * * /usr/local/bin/php /usr/local/www/apinger.php

Jabroni

Same issue as everyone on this forum..

Different plataforms (2 running as VM under ESXi, and other on an alix) on different type of interfaces (some ethernets, and some openvpn interfaces). And ive been restarting the apinger in order to 'clear' the Packet loss % so the gw can be detected up again…

Hope pfsense devs fix this on the next release since its a bummer for pfsenses users who rely on multiwan

coolspot

@miles267:

For those who were encountering this issue, it appears to have been corrected in the pfsense 2.1.1 PRERELEASE. Can any of the ops confirm?

I'm still having the issue with 2.1.1 release; pfsense reports 2% packet loss, but when I ping from the command line, it's all clean?

eSpezi

Similar issues here.

How do you connect to your WAN(s)?
Do you use a modem or a router in front of pfSense?

It seems that a (DSL/Cable) router makes more problems at my installations.
Unfortunately it is not always possible to exchange router with a modem.

Are there any upcomming solutions to the issues, without hacking around in pfS?

Cheers,
Harry

camlin

I am using 2.1.2, and I can confirm there is definitively a problem. Now, in my case, it does not look like a pure aPinger problem

The issue is that I am using PPoE for my WAN connection and the script responsible for configuring aPinger (gwlb.inc) gets it wrong.

Basically PPoE marks the gateway as "dynamic" but the script tries to check if that's an IP address (which is clearly isn't!).

This then leads aPinger to fail and the gateway report page stays stuck on "Pending"…

I am going to try and see if I can hack the thing on my test machine for the time being but I would welcome any recommendations on how I should report this.

Thanks.

camlin

Yes, I can confirm I got it to work by changing the gwlb.inc script.

Part of the problem is that GUI checks (the story about providing only an IP address) are present here. So when PPoE sets the IP to "dynamic" it all goes wrong (the script says: Not an IPv4 or an IPv6 address => Do not configure apinger => apinger then fails).

Ideally the PPoE system should be improved to provide something more useful than just "dynamic" but this is probably a much bigger scope of work.

In the meantime, changing the block (in the setup_gateways_monitor method):


if (is_ipaddrv4($gateway['gateway'])) {
			...
} else if (is_ipaddrv6($gateway['gateway'])) {
			...
} else
			continue;

by

		if ($gateway['ipprotocol'] == "inet") {
			$gwifip = find_interface_ip($gateway['interface'], true);
			if (!is_ipaddrv4($gwifip))
				continue; //Skip this target

			/*
			 * If the gateway is the same as the monitor we do not add a
			 * route as this will break the routing table.
			 * Add static routes for each gateway with their monitor IP
			 * not strictly necessary but is a added level of protection.
			 */
			if (is_ipaddrv4($gateway['gateway']) && $gateway['monitor'] != $gateway['gateway']) {
				log_error("Removing static route for monitor {$gateway['monitor']} and adding a new route through {$gateway['gateway']}");
				mwexec("/sbin/route change -host " . escapeshellarg($gateway['monitor']) .
					" " . escapeshellarg($gateway['gateway']), true);
			}
		} else if ($gateway['ipprotocol'] == "inet6") {
			if ($gateway['monitor'] == $gateway['gateway']) {
				/* link locals really need a different src ip */
				if (is_linklocal($gateway['gateway'])) {
					$gwifip = find_interface_ipv6_ll($gateway['interface'], true);
				} else {
					$gwifip = find_interface_ipv6($gateway['interface'], true);
				}
			} else {
				$gwifip = find_interface_ipv6($gateway['interface'], true);
				if (is_linklocal($gateway['monitor'])) {
					if (!strstr($gateway['monitor'], '%')) {
						$gateway['monitor'] .= "%{$gateway['interface']}";
					}
				} else {
					// Monitor is a routable address, so use a routable address for the "src" part
					$gwifip = find_interface_ipv6($gateway['interface'], true);
				}
			}
			if (!is_ipaddrv6($gwifip))
				continue; //Skip this target

			/*
			 * If the gateway is the same as the monitor we do not add a
			 * route as this will break the routing table.
			 * Add static routes for each gateway with their monitor IP
			 * not strictly necessary but is a added level of protection.
			 */
			if (is_ipaddrv6($gateway['gateway']) && $gateway['monitor'] != $gateway['gateway']) {
				log_error("Removing static route for monitor {$gateway['monitor']} and adding a new route through {$gateway['gateway']}");
				mwexec("/sbin/route change -host -inet6 " . escapeshellarg($gateway['monitor']) .
					" " . escapeshellarg($gateway['gateway']), true);
			}
		} else { 
			continue;
		}

Please note that in my case I also had to change the logic of how it determines the source address: I was trying to force the monitoring of a routable IPv6 address but the script was forcing a local link address as the source address because the default monitor was a local link address (so I change the order so that it considers first if the monitor gateway address had been manually specified in the config).

HTH

mukidu

Hi Guys,
I am experiencing exactly the same issue, got 2 WANs configured for loadbalancing, but since the update I noticed that loadbalancing is not working anymore besides the fact that the 2nd WAN in marked as offline.
I have also a tier 2 H3g connection set up on the loadbalancing, I noticed that pfsense is keeping it up even though both WANs are up and running….
My broadband connections are both Pppoe.

I'am planning to roolback to 2.0.3

Bye

camlin

Yes, that does not surprise me (quite clearly part of the load balancing work depends on properly determining which gateway is working fine).

For info, I have submitted my changes to GitHub. They improve things a bit but I still see issues after link loss where the monitoring of my IPv6 gateway is not properly reset. I suspect this is caused by timing issues where pfSense is trying to setup apinger whilst not everything is ready yet…

krylou

@mukidu:

Hi Guys,
I am experiencing exactly the same issue, got 2 WANs configured for loadbalancing, but since the update I noticed that loadbalancing is not working anymore besides the fact that the 2nd WAN in marked as offline.
I have also a tier 2 H3g connection set up on the loadbalancing, I noticed that pfsense is keeping it up even though both WANs are up and running….
My broadband connections are both Pppoe.

I'am planning to roolback to 2.0.3

Bye

What version you are using? Does anyone tried 2.1.3 version?

camlin

The fix I submitted was accepted 13 days ago, however it was too late for it to make it in 2.1.3…

So I suppose we will only see it on 2.1.4 and later.

My fix improves the situation as before link monitoring would just not work at all with PPoE (and probably any kind of dynamically established links, so that would include VPNs). This means you can now see packet loss and latency on PPoE links (and I suspect VPN links too, although I haven't tried).

Now, my fix still needs some work admittedly. I am still seeing some issues sometimes after a complete link loss. When the link comes back up it looks like pfSense calls the script way too early (at a stage where we don't even know the IP to monitor). It seems to be a timing issue as this only happens from time to time.

It is easy to "fix" as once the link is established you just wait a few seconds and then have to stop/start the "apinger" service and it will then get the proper addresses.

But it still means the solution won't be good enough to allow proper automated link failover...

I am probably going to try and have a look at some point, but I suspect it is an issue that will be caused by something deep in the internals of pfSense. So could well be out of my league...

yaboc

i have mutli wan set up with cable as WAN1 and PPPoE DSL as WAN2. my cable gateway is up and running but PPPoE is showing as Pending and restarting the apinger doesn't bring it up online. but im able to ping using DSL gateway. is this the problem that apinger is causing ?

im on the 2.1.3