@pfpv I don't know. I did see that emails have a timeout, default is 20 seconds. When I pull the wan1 link, the failover happens in <5 seconds. I've run multi wan for decades... never had problems w/ this feature even when I used linksys/dlink.

The second part worked on second attempt. Below is the third last part, hopefully. It looks like I need to write more not to be flagged as spam. Aug 21 05:23:24 php-fpm 453 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 15 days has not passed. Not updating dynamic DNS entry. Aug 21 05:23:24 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:24 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:24 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:23 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:23 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:23 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:23 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:23 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:22 php-fpm 453 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 15 days has not passed. Not updating dynamic DNS entry. Aug 21 05:23:22 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:22 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:22 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:22 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 php-fpm 453 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 15 days has not passed. Not updating dynamic DNS entry. Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:21 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:20 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:20 php-fpm 453 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 15 days has not passed. Not updating dynamic DNS entry. Aug 21 05:23:20 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:20 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 php-fpm 453 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 15 days has not passed. Not updating dynamic DNS entry. Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:19 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:18 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:18 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:18 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:18 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:18 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:18 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:18 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:18 php 33487 notify_monitor.php: Message sent to Email OK Aug 21 05:23:18 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:18 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:17 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:17 php-fpm 453 /rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 15 days has not passed. Not updating dynamic DNS entry. Aug 21 05:23:17 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:17 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 php-fpm 453 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 php-fpm 453 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_PreferWANcustom''5.cache: q.q.q.q(WAN2_IP) Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:16 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:15 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:15 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:15 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:15 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:15 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:15 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:15 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:15 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 php-fpm 453 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 21 05:23:14 php-fpm 453 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_PreferWANcustom''4.cache: q.q.q.q(WAN2_IP) Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:14 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:13 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:13 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:13 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:13 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:13 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:13 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:13 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:13 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:13 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:12 php-fpm 453 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 21 05:23:12 php-fpm 453 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_PreferWANcustom''3.cache: q.q.q.q(WAN2_IP) Aug 21 05:23:12 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:12 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:12 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:12 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:12 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:12 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:12 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:11 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 php-fpm 453 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 21 05:23:10 php-fpm 453 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_PreferWANcustom''2.cache: q.q.q.q(WAN2_IP) Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 php-fpm 40773 /rc.filter_configure_sync: GW States: Gateway is down but its IP address cannot be located. Skipping state kill.: WAN_DHCP Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:10 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:09 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:08 php-fpm 453 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 21 05:23:08 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:08 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:08 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:08 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:08 php-fpm 453 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_PreferWANcustom''1.cache: q.q.q.q(WAN2_IP) Aug 21 05:23:08 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:08 php-fpm 64592 /rc.filter_configure_sync: GW States: Gateway is down but its IP address cannot be located. Skipping state kill.: WAN_DHCP Aug 21 05:23:08 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:08 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:08 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:07 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:07 php 33487 notify_monitor.php: Message sent to Email OK Aug 21 05:23:07 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:07 php-fpm 453 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 21 05:23:07 php-fpm 453 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_PreferWANcustom''0.cache: q.q.q.q(WAN2_IP) Aug 21 05:23:07 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:07 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 kernel arpresolve: can't allocate llinfo for y.y.y.y(WAN1_GW_IP) on igb0 Aug 21 05:23:06 php-fpm 454 /rc.openvpn: Default gateway setting Interface WAN2_PPPOE Gateway as default. Aug 21 05:23:06 php-fpm 454 /rc.openvpn: Gateway, switch to: WAN2_PPPOE Aug 21 05:23:05 php-fpm 64592 z.z.z.z(WAN1_monitoring_IP)|192.168.100.2|WAN_DHCP|0ms|0ms|100%|down|highloss Aug 21 05:23:05 php-fpm 64592 /rc.filter_configure_sync: MONITOR: WAN_DHCP has packet loss, omitting from routing group PreferWAN Aug 21 05:23:05 check_reload_status 483 Reloading filter Aug 21 05:23:05 check_reload_status 483 Restarting OpenVPN tunnels/interfaces Aug 21 05:23:05 check_reload_status 483 Restarting IPsec tunnels Aug 21 05:23:05 check_reload_status 483 updating dyndns WAN_DHCP Aug 21 05:23:05 rc.gateway_alarm 25620 >>> Gateway alarm: WAN_DHCP (Addr:z.z.z.z(WAN1_monitoring_IP) Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%) Aug 21 05:23:03 check_reload_status 483 Reloading filter Aug 21 05:23:03 php-fpm 40773 /rc.linkup: Removing static route for monitor p.p.p.p(WAN3_monitoring_IP) and adding a new route through k.k.k.k(WAN3_IP) Aug 21 05:23:01 php-fpm 40773 /rc.linkup: DEVD Ethernet detached event for wan Aug 21 05:23:00 check_reload_status 483 Linkup starting igb0 Aug 21 05:23:00 kernel igb0: link state changed to DOWN

[image: 1660745076451-overview.png] I made a sketch to clarify

@alvescaio can I help me?

@indiankid Have you looked at tnsr? I suspect between tnsr and multiple pf installs you'd be best situated.

I have even tried replacing hardware such as Hdds to 2 brand new SSD WD RED in Zfs "raid 1" and replaced Ram. Same thing happens when enabling the option........

I would verify with a packet capture that the traffic is crossing into the pfSense side properly. What's the LAN interface VLAN in the that list? 4084? 4083? 4082?

In order for replies from those servers to go back over the tunnel you will need the states to be tagged reply-to unless the tunnel is the default route. That usually means the tunnel must be a routed type and the states must be opened by firewall rules on the assigned interface. So I'd start out by using OpenVPN and assigning the house side interface. Steve

@ansarmehmood said in wan settings: when i select staic on wan side it only ask for ip address and subnet mask The gateway is requested as well, either in the initial setup in the console or when you configure the interface in the GUI. In the GUI click "Add new gateway", then you're able to enter it. The routed host IPs can be directly used in port forwarding rules as destination addresses.

@aziz-1 said in Static route over OpenVPN VPN: The other way only requests So that's the point where you should check the firewall on the destination device. Do you get responses if you ping a LAN2 device from LAN1 pfSense? To be sure, are both VPN endpoints the default gateway in their local networks?

@akuma1x simply blocking ip spoofing :)

I encountered another issue with dpinger. I changed one unimportant parameter in the WAN interface (CIDR length for the alias IPv4 address). Pressed save and apply and noticed that my IP switched over to the failover WAN2, then back to WAN1. I looked at the logs and found this: php-fpm 89297 /interfaces.php: The command '/usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B x.x.x.x -p /var/run/dpinger_WAN_DHCP~x.x.x.x~y.y.y.y.pid -u /var/run/dpinger_WAN_DHCP~x.x.x.x~y.y.y.y.sock -C "/etc/rc.gateway_alarm" -d 1 -s 500 -l 2000 -t 30000 -A 1000 -D 500 -L 20 y.y.y.y.>/dev/null' returned exit code '1', the output was '' It's weird that I saw changes for WAN3 that I haven't touched. I think the developers should have a look at the whole multi-WAN system. More logs: Aug 4 19:02:27 php-fpm 73902 /rc.filter_configure_sync: Default gateway setting Interface WAN_DHCP Gateway as default. Aug 4 19:02:27 php-fpm 73902 /rc.filter_configure_sync: Gateway, switch to: WAN_DHCP Aug 4 19:02:25 upsmon 11497 Startup successful Aug 4 19:02:25 php-fpm 48340 /rc.start_packages: Starting service nut Aug 4 19:02:25 upsmon 85376 Signal 15: exiting Aug 4 19:02:25 php-fpm 48340 /rc.start_packages: Stopping service nut Aug 4 19:02:25 php-fpm 48340 /rc.start_packages: Restarting/Starting all packages. Aug 4 19:02:24 check_reload_status 483 Starting packages Aug 4 19:02:24 php-fpm 89297 /interfaces.php: Creating rrd update script Aug 4 19:02:24 check_reload_status 483 Reloading filter Aug 4 19:02:24 php-fpm 89297 /interfaces.php: Removing static route for monitor m.m.m.m and adding a new route through n.n.n.n [this is untouched WAN3] Aug 4 19:02:24 php-fpm 89297 /interfaces.php: Removing static route for monitor y.y.y.y and adding a new route through x1.x1.x1.x1 [this is WAN1] Aug 4 19:02:24 php-fpm 89297 /interfaces.php: Resyncing OpenVPN instances for interface WAN. Aug 4 19:02:23 php 77068 notify_monitor.php: Message sent to X OK Aug 4 19:02:22 php-fpm 202 /rc.dyndns.update: phpDynDNS (): (Success) IP Address Updated Successfully! Aug 4 19:02:22 php-fpm 202 /rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_PreferWANcustom''0.cache: k.k.k.k [this is WAN2 IP] Aug 4 19:02:21 check_reload_status 483 updating dyndns wan Aug 4 19:02:19 dhcpleases 50889 Could not deliver signal HUP to process 71960: No such process. Aug 4 19:02:18 check_reload_status 483 Restarting IPsec tunnels Aug 4 19:02:18 php-fpm 775 /rc.filter_configure_sync: Gateway, switch to: WAN2_PPPOE Aug 4 19:02:18 php-fpm 89297 /interfaces.php: Default gateway setting Interface WAN2_PPPOE Gateway as default. Aug 4 19:02:18 php-fpm 89297 /interfaces.php: Gateway, switch to: WAN2_PPPOE Aug 4 19:02:17 check_reload_status 483 Reloading filter Aug 4 19:02:17 php-fpm 98601 /rc.newwanip: rc.newwanip: on (IP address: x.x.x.x) (interface: WAN[wan]) (real interface: igb0). Aug 4 19:02:17 php-fpm 98601 /rc.newwanip: rc.newwanip: Info: starting on igb0. Aug 4 19:02:16 check_reload_status 483 rc.newwanip starting igb0 Aug 4 19:02:16 php-fpm 89297 /interfaces.php: Error starting gateway monitor for WAN_DHCP Aug 4 19:02:16 php-fpm 89297 /interfaces.php: The command '/usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B x.x.x.x -p /var/run/dpinger_WAN_DHCP~x.x.x.x~y.y.y.y.pid -u /var/run/dpinger_WAN_DHCP~x.x.x.x~y.y.y.y.sock -C "/etc/rc.gateway_alarm" -d 1 -s 500 -l 2000 -t 30000 -A 1000 -D 500 -L 20 y.y.y.y.>/dev/null' returned exit code '1', the output was '' Aug 4 19:02:16 php-fpm 89297 /interfaces.php: Removing static route for monitor m.m.m.m and adding a new route through n.n.n.n [this is really weird - this is untouched WAN3]

@netblues said in Sticky connections not working with dual WAN: @bijore8887 Yes, you are right about the documentation, my bad for that. However working with smaller values of stickiness (or the default 0) the gateway is switced everytime all connections complete. It would also be a bug if the gateway was switched every time all connections complete if stickiness was set to more than 0. Because it should be sticky for the number of seconds specified. Do you have a high value in stickiness? In my first comment I mentioned 1200. Try it out. No thanks. You have not bothered to read the pfSense documentation or the comments properly. But at least others can now know that there could be a bug and use workarounds instead and not waste further time.

@justconfused And to reply to my own post Oh Shit - I am telnetting on the wrong bloody port. Note that this is a demonstration of how writing things down can help think things through - even when you are being an utter moron. I am certainly not going to admit just how long I have spent struggling with this (and a very similar related issue which is the reason for this test setup) Sigh

@jeryd did you read the link provided by @viragomann It even gives an example prefer wan over load balance https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html [image: 1659277981165-example.jpg]