Crash Report every time interface goes down

[megapearl]

Hi,

I'm Running PFSense latest 2.3.3 branche.

Everytime an interface goes down pfsense generates a crash report as below:

Crash report begins.  Anonymous machine information:

amd64
10.3-RELEASE-p9
FreeBSD 10.3-RELEASE-p9 #303 7a802f5(RELENG_2_3): Tue Oct 11 04:27:29 CDT 2016    root@ce23-amd64-builder:/builder/pfsense/tmp/obj/builder/pfsense/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP Warning:  unlink(/tmp/.down.MOBILE_DHCP): No such file or directory in /etc/inc/gwlb.inc on line 977
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP Stack trace:
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  1. {main}() /etc/rc.filter_configure_sync:0
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  2. filter_configure_sync() /etc/rc.filter_configure_sync:61
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  3. filter_generate_aliases() /etc/inc/filter.inc:287
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  4. filter_get_vpns_list() /etc/inc/filter.inc:779
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  5. ipsec_idinfo_to_cidr() /etc/inc/filter.inc:983
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  6. get_interface_ip() /etc/inc/ipsec.inc:376
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  7. get_failover_interface() /etc/inc/interfaces.inc:5421
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  8. return_gateway_groups_array() /etc/inc/interfaces.inc:5972
[12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  9. unlink() /etc/inc/gwlb.inc:977

Filename: /var/crash/minfree
2048

What could be wrong here and how to fix it?

Best Regards,
Donald.

[JorgeOliveira]

Similar issue reported at:
https://forum.pfsense.org/index.php?topic=118959.0

You can try the following change and let us know if there is any improvement.

Find at /etc/inc/gwlb.inc:

						if (file_exists("/tmp/.down.$gwname")) {
							$msg = "MONITOR: {$gwname} is available now, adding to routing group";
							$msg .= "\n".implode("|", $status);
							log_error($msg);
							notify_via_growl($msg);
							notify_via_smtp($msg);
							unlink("/tmp/.down.$gwname");
						}

Replace with:

						if (file_exists("/tmp/.down.$gwname")) {
							$msg = "MONITOR: {$gwname} is available now, adding to routing group";
							$msg .= "\n".implode("|", $status);
							unlink("/tmp/.down.$gwname");
							log_error($msg);
							notify_via_growl($msg);
							notify_via_smtp($msg);
						}

This issue is most likely caused by simultaneous events. A latent problem in pfSense is that it triggers the same events multiple times in certain corner cases. I am hoping that unlinking a little earlier would prevent the other event from hitting the same block of code.

Regards,
Jorge M. Oliveira

[JorgeOliveira]

Submitted a PR with this and more changes in the same code area:
https://github.com/pfsense/pfsense/pull/3184

Tested on my environment and worked. Please confirm it also fixes your issues ;)

[megapearl]

Thank you,

Changed the code, and testing out.
Will report back later.

UPDATE: Problem is solved using patch above.

[phil.davis]

IMHO this type of thing could be improved even more, to avoid the race condition between the file_exists() test and the unlink() by enhancing the unlink_if_exists() function so that it returns a true/false status to indicate if the file existed and was unlinked OK.

Pull request: https://github.com/pfsense/pfsense/pull/3186

Discussion welcome!

Such a thing could be used in other places that currently have a similar sequence of test-if-exists-then-unlink.