Crash Report every time interface goes down



  • Hi,

    I'm Running PFSense latest 2.3.3 branche.

    Everytime an interface goes down pfsense generates a crash report as below:

    Crash report begins.  Anonymous machine information:

    amd64
    10.3-RELEASE-p9
    FreeBSD 10.3-RELEASE-p9 #303 7a802f5(RELENG_2_3): Tue Oct 11 04:27:29 CDT 2016    root@ce23-amd64-builder:/builder/pfsense/tmp/obj/builder/pfsense/tmp/FreeBSD-src/sys/pfSense

    Crash report details:

    PHP Errors:
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP Warning:  unlink(/tmp/.down.MOBILE_DHCP): No such file or directory in /etc/inc/gwlb.inc on line 977
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP Stack trace:
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  1. {main}() /etc/rc.filter_configure_sync:0
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  2. filter_configure_sync() /etc/rc.filter_configure_sync:61
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  3. filter_generate_aliases() /etc/inc/filter.inc:287
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  4. filter_get_vpns_list() /etc/inc/filter.inc:779
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  5. ipsec_idinfo_to_cidr() /etc/inc/filter.inc:983
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  6. get_interface_ip() /etc/inc/ipsec.inc:376
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  7. get_failover_interface() /etc/inc/interfaces.inc:5421
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  8. return_gateway_groups_array() /etc/inc/interfaces.inc:5972
    [12-Oct-2016 02:40:34 Europe/Amsterdam] PHP  9. unlink() /etc/inc/gwlb.inc:977

    Filename: /var/crash/minfree
    2048

    What could be wrong here and how to fix it?

    Best Regards,
    Donald.



  • Similar issue reported at:
    https://forum.pfsense.org/index.php?topic=118959.0

    You can try the following change and let us know if there is any improvement.

    Find at /etc/inc/gwlb.inc:

    
    						if (file_exists("/tmp/.down.$gwname")) {
    							$msg = "MONITOR: {$gwname} is available now, adding to routing group";
    							$msg .= "\n".implode("|", $status);
    							log_error($msg);
    							notify_via_growl($msg);
    							notify_via_smtp($msg);
    							unlink("/tmp/.down.$gwname");
    						}
    
    

    Replace with:

    
    						if (file_exists("/tmp/.down.$gwname")) {
    							$msg = "MONITOR: {$gwname} is available now, adding to routing group";
    							$msg .= "\n".implode("|", $status);
    							unlink("/tmp/.down.$gwname");
    							log_error($msg);
    							notify_via_growl($msg);
    							notify_via_smtp($msg);
    						}
    
    

    This issue is most likely caused by simultaneous events. A latent problem in pfSense is that it triggers the same events multiple times in certain corner cases. I am hoping that unlinking a little earlier would prevent the other event from hitting the same block of code.

    Regards,
    Jorge M. Oliveira



  • Submitted a PR with this and more changes in the same code area:
    https://github.com/pfsense/pfsense/pull/3184

    Tested on my environment and worked. Please confirm it also fixes your issues ;)



  • Thank you,

    Changed the code, and testing out.
    Will report back later.

    UPDATE: Problem is solved using patch above.



  • IMHO this type of thing could be improved even more, to avoid the race condition between the file_exists() test and the unlink() by enhancing the unlink_if_exists() function so that it returns a true/false status to indicate if the file existed and was unlinked OK.

    Pull request: https://github.com/pfsense/pfsense/pull/3186

    Discussion welcome!

    Such a thing could be used in other places that currently have a similar sequence of test-if-exists-then-unlink.