"tun_wg0: loop detected" following WireGuard gateway 100% loss and WAN interface rc.newwanip

Jarhead

@mixka Right, it will. But the peer is on the other end of the tunnel, correct? It should use the WG gateway, then you have the WG gateway set as the upstream gateway in the interface or maybe I'm missing something. You have a lot redacted that doesn't need to be.

Mixka

@jarhead

I'm new with WG (this abbreviation always got me, because WG is WatchGuard for me ) so this setup might be wrong in many places. I applied this guide for my use https://youtu.be/wYe7FzZ_0X8 (not using Mullvad).

What might be right way to use WG gateway as default gateway? Peer endpoint is in public domain and interface+gateway private.

stephenw10

Yes, the static route for subnets at the remote end of the tunnel must be the Wireguard gateway.

The public peer end point IP would not normally need a static route. The only reason you might need that is if you change the system default route to be the Wireguard gateway. I would not recommend doing that though.

Jarhead

@mixka Use this:
https://youtu.be/2oe7rTMFmqc

Mixka

@jarhead

Golden video as Christians videos always are. But I can't see relations to my setup there (other than that WG is used to make tunnel). Video anyway explains routes and NAT's nicely.

@stephenw10

So, setup is somewhat right, but not recommended. I see, but goal is to use WG gateway as default route, so recommended or not this is the way.

Anything else to test out?

I also found when started to look other logs, that ntpd have log lines just after first "loop detected" message is timestamped. Is it root cause or just following system melt down?

Log:

Aug 8 20:16:14 	check_reload_status 	487 	Reloading filter 
Aug 8 20:14:26 	kernel 		tun_wg0: loop detected 
Aug 8 20:14:05 	ntpd 	1691 	Listen normally on 25 igb0 80.221.147.161:123
Aug 8 20:14:02 	ntpd 	1691 	Deleting interface #24 igb0, 80.221.147.161#123, interface stats: received=0, sent=0, dropped=0, active_time=14804 secs
Aug 8 20:14:01 	kernel 		tun_wg0: loop detected 

Aug 8 16:09:28 	check_reload_status 	487 	Reloading filter
Aug 8 16:07:42 	kernel 		tun_wg0: loop detected 
Aug 8 16:07:18 	ntpd 	1691 	Listen normally on 24 igb0 80.221.147.161:123
Aug 8 16:07:16 	ntpd 	1691 	Deleting interface #23 igb0, 80.221.147.161#123, interface stats: received=0, sent=0, dropped=0, active_time=14756 secs
Aug 8 16:07:15 	kernel 		tun_wg0: loop detected

Aug 8 12:03:30 	check_reload_status 	487 	Reloading filter
Aug 8 12:01:44 	kernel 		tun_wg0: loop detected
Aug 8 12:01:20 	ntpd 	1691 	Listen normally on 23 igb0 80.221.147.161:123
Aug 8 12:01:18 	ntpd 	1691 	Deleting interface #22 igb0, 80.221.147.161#123, interface stats: received=0, sent=0, dropped=0, active_time=1782 secs 
Aug 8 12:01:17 	kernel 		tun_wg0: loop detected

stephenw10

Mmm, it's unclear where the loop is being created there but I'd have to guess it's because the system default route is switched back to the WAN when the WG gateway goes down and you have something else routing traffic back.
Do you have the default gateway set specifically to the WG gateway or to a group?

Jarhead

@mixka Did you fix the WG config? That's what the video was for.
From what I can see, you need to remove the upstream gateway on the interface and correct the static route.

Mixka

@stephenw10

Specifically to WG gateway.

@Jarhead

I found out that video really helpful, but not on my case. I do not want to remove upstream gateway, because traffic out of tunnel needs to be NAT'd. Or does NAT rules still operate even upstream gateway is disabled? What does it help in that case? That video did not mention. And what comes to static route, I think there is no things to correct. In video WAN gateway was used and I need to use WG as my gateway.

stephenw10

If you have a gateway on the interface automatic outbound NAT will be applied but you can still add manual rules to an interface that has no gateway.
You need a gateway there to use as the default route or to add policy routing for but it doesn't have to be on the interface itself. However I wouldn't expect it to make any difference.

Steve

Mixka

It's been long days trying to solve this. I even deleted all firewall rules and redo only basic setup.

I have for example tried these:

Disable NTP: No affect
Disable WG interface upstream gateway: Interface errors out statistics goes down by ~90%
Disable Negate rules: No affect
Disable gateways monitoring: After that it did not recover from loop
Started firewall rules from scrach with basic setup: Maybe loop occurances are lower, or not...
Put "supersede interface-mtu 0" to WAN interface DHCP option modifiers: Definately made loop occurance lower, or maybe just coincidence

Now I did get 15 hours before problem occured after reboot.

Should I try to play with WAN interface DHCP protocol timing or other options? Connection is optical fiber 100/100M.

I also found this Reddit post https://reddit.com/r/PFSENSE/comments/ln14gu/arpresolve_cant_allocate_llinfo_ip_crippling_wan
Should I try dumb switch before WAN interface and what difference it might doing? I do not think this is tagged traffic anyway.

I just did speedtest with different services and my connection speed is low like 8/30M. I have run speedtest often and haven't got problems before. I think only thing what I have changed after last time is that DHCP option modifier.

During speedtest:

I have had traffic shaper, but disabling it won't help any. I do not know if this is related in anyway, but is this right way to reduce buffer bloating? It is (was) working really well for me. https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/775

And I have this spam for 1000 lines per minute:
(51820 is WG peer endpoint port and 43279 is WG tunnel listening port)

Aug 14 14:06:00 	WAN1 	Default deny rule IPv4 (1000000103) 	*WG_Endpoint_IP*:51820		*WAN_Interface_IP*:43279		UDP
Aug 14 14:06:00 	WAN1 	Default deny rule IPv4 (1000000103) 	*WG_Endpoint_IP*:51820		*WAN_Interface_IP*:43279		UDP
Aug 14 14:06:00 	WAN1 	Default deny rule IPv4 (1000000103) 	*WG_Endpoint_IP*:51820		*WAN_Interface_IP*:43279		UDP

Should I allow this traffic or is there misconfiguration?

I have now used way over 50 hours to trying to solve this. I am even thinking to clear install whole box. I do not know what to do. I have tried everything and problems just pile up.

Loop occurance:

Aug 14 10:18:34 	check_reload_status 	484 	Reloading filter
Aug 14 10:18:34 	check_reload_status 	484 	Restarting OpenVPN tunnels/interfaces
Aug 14 10:18:34 	check_reload_status 	484 	Restarting IPsec tunnels
Aug 14 10:18:34 	check_reload_status 	484 	updating dyndns *WG_GW*
Aug 14 10:18:34 	rc.gateway_alarm 	17746 	>>> Gateway alarm: *WG_GW* (Addr:8.8.8.8 Alarm:0 RTT:7.390ms RTTsd:36.380ms Loss:11%)
Aug 14 10:18:25 	php 	14097 	[pfBlockerNG] filterlog daemon started
Aug 14 10:18:25 	tail_pfb 	13938 	[pfBlockerNG] Firewall Filter Service started
Aug 14 10:18:25 	lighttpd_pfb 	12699 	[pfBlockerNG] DNSBL Webserver started
Aug 14 10:18:25 	php_pfb 	10645 	[pfBlockerNG] filterlog daemon stopped
Aug 14 10:18:25 	lighttpd_pfb 	10375 	[pfBlockerNG] DNSBL Webserver stopped
Aug 14 10:18:25 	tail_pfb 	10461 	[pfBlockerNG] Firewall Filter Service stopped
Aug 14 10:18:24 	php-fpm 	30665 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
Aug 14 10:18:24 	php-fpm 	30665 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
Aug 14 10:18:24 	php-fpm 	30665 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
Aug 14 10:18:24 	php-fpm 	30665 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
Aug 14 10:18:24 	php-fpm 	30665 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
Aug 14 10:18:24 	check_reload_status 	484 	Syncing firewall
Aug 14 10:18:24 	php-fpm 	30665 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
Aug 14 10:18:24 	check_reload_status 	484 	Syncing firewall
Aug 14 10:18:24 	php-fpm 	30665 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
Aug 14 10:18:24 	php 	83866 	haproxy: started new pid:87304
Aug 14 10:18:24 	php 	83866 	haproxy: reload old pid:42610
Aug 14 10:18:23 	php-fpm 	30665 	/rc.start_packages: Restarting/Starting all packages.
Aug 14 10:18:22 	check_reload_status 	484 	Starting packages
Aug 14 10:18:22 	php-fpm 	10714 	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> *WAN_Interface_IP* - Restarting packages.
Aug 14 10:18:20 	php-fpm 	10714 	/rc.newwanip: Creating rrd update script
Aug 14 10:18:20 	php-fpm 	10714 	/rc.newwanip: Resyncing OpenVPN instances for interface WAN1.
Aug 14 10:18:15 	php-fpm 	10714 	/rc.newwanip: IP Address has changed, killing all states (ip_change_kill_states is set).
Aug 14 10:18:15 	check_reload_status 	484 	Reloading filter
Aug 14 10:18:15 	check_reload_status 	484 	Restarting OpenVPN tunnels/interfaces
Aug 14 10:18:15 	check_reload_status 	484 	Restarting IPsec tunnels
Aug 14 10:18:15 	check_reload_status 	484 	updating dyndns *WG_GW*
Aug 14 10:18:15 	rc.gateway_alarm 	70659 	>>> Gateway alarm: *WG_GW* (Addr:8.8.8.8 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%)
Aug 14 10:18:13 	php-fpm 	10714 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through *WG_GW_IP*
Aug 14 10:18:13 	php-fpm 	10714 	/rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through *WAN_GW_IP*
Aug 14 10:18:11 	php-fpm 	10714 	/rc.newwanip: rc.newwanip: on (IP address: *WAN_Interface_IP*) (interface: WAN1[wan]) (real interface: igb0).
Aug 14 10:18:11 	php-fpm 	10714 	/rc.newwanip: rc.newwanip: Info: starting on igb0.
Aug 14 10:18:10 	check_reload_status 	484 	rc.newwanip starting igb0
Aug 14 10:18:10 	kernel 		tun_wg0: loop detected
Aug 14 10:18:10 	kernel 		tun_wg0: loop detected
Aug 14 10:18:10 	kernel 		tun_wg0: loop detected
Aug 14 10:18:10 	kernel 		arpresolve: can't allocate llinfo for *WAN_GW_IP* on igb0
Aug 14 10:18:09 	kernel 		tun_wg0: loop detected
Aug 14 10:18:09 	kernel 		tun_wg0: loop detected
Aug 14 10:18:09 	kernel 		arpresolve: can't allocate llinfo for *WAN_GW_IP* on igb0
Aug 14 10:18:09 	kernel 		tun_wg0: loop detected
Aug 14 10:18:09 	kernel 		tun_wg0: loop detected
Aug 14 10:18:09 	kernel 		tun_wg0: loop detected
Aug 14 10:18:09 	kernel 		tun_wg0: loop detected
Aug 14 10:18:09 	kernel 		arpresolve: can't allocate llinfo for *WAN_GW_IP* on igb0
Aug 14 10:18:08 	kernel 		tun_wg0: loop detected
Aug 14 10:18:08 	kernel 		arpresolve: can't allocate llinfo for *WAN_GW_IP* on igb0

Aug 14 10:18:34 	dpinger 	65033 	*WG_GW* 8.8.8.8: Clear latency 7390us stddev 36380us loss 11%
Aug 14 10:18:15 	dpinger 	65033 	*WG_GW* 8.8.8.8: Alarm latency 0us stddev 0us loss 100%
Aug 14 10:18:13 	dpinger 	65033 	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.8.8 bind_addr *WG_Interface_IP* identifier "*WG_GW*"
Aug 14 10:18:13 	dpinger 	64735 	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr *WAN_Interface_IP* identifier "WAN_DHCP"

stephenw10

@mixka said in "tun_wg0: loop detected" following WireGuard gateway 100% loss and WAN interface rc.newwanip:

Should I try dumb switch before WAN interface and what difference it might doing?

That's unlikely to help here because, AFAIK, you are not seeing igb0 actually lose link which is what that would prevent.
It would still hit whatever is causing it to see a new WAN IP triggering the reload of the interface and everything on it.

Reviewing the thread though, this seems odd:
@mixka said in "tun_wg0: loop detected" following WireGuard gateway 100% loss and WAN interface rc.newwanip:

I pin down problem that they all was behind same dummy switch. I restarted that switch and everything worked like 2 minutes and then goes down. I removed that switch and everything back to normal.

Could it be that switch has some loop detection and was shutting down one or more ports to prevent it?
If so that implies whatever is creating the loop ends up sending traffic to the LAN side incorrectly.

Try checking the routing tables when it's working and compare that to the failed state.

Steve

Mixka

This post is deleted!

Mixka

This post is deleted!

Mixka

This post is deleted!

Mixka

@stephenw10

I have now connected one host directly to pfSense interface and deleted every VLAN and unnecessary interface. So every switch is out. Problem still exist.

How I can check routing tables when problem occur? It last under minute. Maybe I remove interface monitoring, so it won't recover from loop.

Also do you know if these settings are ok for eliminating buffer bloat? https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/775

Aug 15 11:19:08 	dhclient 	20300 	Creating resolv.conf
Aug 15 11:19:08 	dhclient 	20119 	Adding new routes to interface: igb0
Aug 15 11:19:08 	dhclient 	19776 	New Routers (igb0): WAN_GW_IP
Aug 15 11:19:08 	dhclient 	19433 	New Broadcast Address (igb0): WAN_255_IP
Aug 15 11:19:08 	dhclient 	19284 	New Subnet Mask (igb0): 255.255.240.0
Aug 15 11:19:08 	dhclient 	19165 	New IP Address (igb0): WAN_IP
Aug 15 11:19:08 	dhclient 	18956 	ifconfig igb0 inet WAN_IP netmask 255.255.240.0 broadcast WAN_255_IP
Aug 15 11:19:08 	dhclient 	18914 	Starting add_new_address()
Aug 15 11:19:08 	dhclient 	18658 	BOUND
Aug 15 11:19:08 	dhclient 	18543 	ARPCHECK
Aug 15 11:19:05 	dhclient 	17989 	ARPSEND
Aug 15 11:19:05 	dhclient 	17554 	PREINIT
Aug 15 11:19:05 	dhclient 	16730 	Deleting old routes
Aug 15 11:19:05 	dhclient 	16311 	EXPIRE

Aug 15 11:19:09 	check_reload_status 	487 	Reloading filter
Aug 15 11:19:09 	php-fpm 	1307 	/rc.newwanip: rc.newwanip: on (IP address: WAN_IP) (interface: WAN1[wan]) (real interface: igb0).
Aug 15 11:19:09 	php-fpm 	1307 	/rc.newwanip: rc.newwanip: Info: starting on igb0.
Aug 15 11:19:08 	check_reload_status 	487 	rc.newwanip starting igb0
Aug 15 11:19:08 	kernel 		tun_wg1: loop detected
Aug 15 11:19:07 	kernel 		tun_wg1: loop detected
Aug 15 11:19:07 	kernel 		arpresolve: can't allocate llinfo for WAN_GW_IP on igb0
Aug 15 11:19:07 	kernel 		tun_wg1: loop detected
Aug 15 11:19:07 	kernel 		tun_wg1: loop detected
Aug 15 11:19:07 	kernel 		arpresolve: can't allocate llinfo for WAN_GW_IP on igb0
Aug 15 11:19:06 	kernel 		tun_wg1: loop detected
Aug 15 11:19:06 	kernel 		arpresolve: can't allocate llinfo for WAN_GW_IP on igb0
Aug 15 11:19:06 	kernel 		tun_wg1: loop detected
Aug 15 11:19:06 	kernel 		arpresolve: can't allocate llinfo for WAN_GW_IP on igb0

stephenw10

Ah, I was just reviewing. Did something change?

Mixka

@stephenw10

I did not find anything change in behaviour, if that is what you mean.

Now I even tried with different connection. I have 5G router in bridged mode with external IP connected to WAN port. Still seeing loop messages in log:

Aug 15 21:16:59 	php 	5817 	[pfBlockerNG] filterlog daemon started
Aug 15 21:16:59 	tail_pfb 	5432 	[pfBlockerNG] Firewall Filter Service started
Aug 15 21:16:59 	lighttpd_pfb 	4636 	[pfBlockerNG] DNSBL Webserver started
Aug 15 21:16:59 	php_pfb 	1738 	[pfBlockerNG] filterlog daemon stopped
Aug 15 21:16:59 	tail_pfb 	1131 	[pfBlockerNG] Firewall Filter Service stopped
Aug 15 21:16:59 	lighttpd_pfb 	872 	[pfBlockerNG] DNSBL Webserver stopped
Aug 15 21:16:59 	php-fpm 	445 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
Aug 15 21:16:58 	php-fpm 	445 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
Aug 15 21:16:58 	php-fpm 	445 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
Aug 15 21:16:58 	php-fpm 	445 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
Aug 15 21:16:58 	php-fpm 	445 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
Aug 15 21:16:58 	php-fpm 	24315 	/rc.start_packages: Skipping STARTing packages process because previous/another instance is already running
Aug 15 21:16:58 	check_reload_status 	484 	Syncing firewall
Aug 15 21:16:58 	php-fpm 	445 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
Aug 15 21:16:58 	check_reload_status 	484 	Syncing firewall
Aug 15 21:16:58 	php-fpm 	445 	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
Aug 15 21:16:58 	php 	73522 	haproxy: started new pid:76645
Aug 15 21:16:58 	php 	73522 	haproxy: reload old pid:60858
Aug 15 21:16:58 	php-fpm 	445 	/rc.start_packages: Restarting/Starting all packages.
Aug 15 21:16:57 	check_reload_status 	484 	Starting packages
Aug 15 21:16:57 	php-fpm 	40603 	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - OLD_WAN_GW_IP -> WAN_IP - Restarting packages.
Aug 15 21:16:57 	check_reload_status 	484 	Starting packages
Aug 15 21:16:57 	php-fpm 	446 	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - OLD_WAN_GW_IP -> WAN_IP - Restarting packages.
Aug 15 21:16:55 	php-fpm 	40603 	/rc.newwanip: Creating rrd update script
Aug 15 21:16:55 	php-fpm 	40603 	/rc.newwanip: Resyncing OpenVPN instances for interface WAN1.
Aug 15 21:16:55 	php-fpm 	446 	/rc.newwanip: Creating rrd update script
Aug 15 21:16:55 	php-fpm 	446 	/rc.newwanip: Resyncing OpenVPN instances for interface WAN1.
Aug 15 21:16:54 	php-fpm 	40603 	/rc.newwanip: phpDynDNS (pfsense.example.fi): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 15 21:16:54 	php-fpm 	90936 	/rc.dyndns.update: phpDynDNS (pfsense.example.fi): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Aug 15 21:16:54 	php-fpm 	446 	/rc.newwanip: phpDynDNS (pfsense): (Success) pfsense updated to WAN_IP
Aug 15 21:16:54 	php-fpm 	446 	/rc.newwanip: phpDynDNS: updating cache file /conf/dyndns_wancloudflare'pfsense.example.fi'0.cache: WAN_IP
Aug 15 21:16:46 	php-fpm 	40603 	/rc.newwanip: IP Address has changed, killing states on former IP Address OLD_WAN_GW_IP.
Aug 15 21:16:46 	check_reload_status 	484 	Reloading filter
Aug 15 21:16:46 	check_reload_status 	484 	updating dyndns wan
Aug 15 21:16:46 	php-fpm 	40603 	/rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through WG_VPN_GW_IP
Aug 15 21:16:46 	php-fpm 	40603 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through WG_VPN_GW_IP
Aug 15 21:16:46 	php-fpm 	40603 	/rc.newwanip: Removing static route for monitor 1.1.1.1 and adding a new route through WAN_GW_IP
Aug 15 21:16:45 	php-fpm 	79596 	/rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1660587405] unbound[93610:0] error: bind: address already in use [1660587405] unbound[93610:0] fatal error: could not open ports'
Aug 15 21:16:43 	check_reload_status 	484 	Restarting IPsec tunnels
Aug 15 21:16:43 	check_reload_status 	484 	Restarting IPsec tunnels
Aug 15 21:16:43 	check_reload_status 	484 	updating dyndns GW_VPN3_gw
Aug 15 21:16:43 	check_reload_status 	484 	Restarting OpenVPN tunnels/interfaces
Aug 15 21:16:43 	check_reload_status 	484 	Restarting IPsec tunnels
Aug 15 21:16:43 	check_reload_status 	484 	updating dyndns GW_VPN2_gw
Aug 15 21:16:43 	rc.gateway_alarm 	39811 	>>> Gateway alarm: GW_VPN3_gw (Addr:8.8.8.8 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%)
Aug 15 21:16:43 	rc.gateway_alarm 	41358 	>>> Gateway alarm: GW_VPN2_gw (Addr:8.8.4.4 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%)
Aug 15 21:16:43 	kernel 		tun_wg0: loop detected
Aug 15 21:16:43 	kernel 		tun_wg0: loop detected
Aug 15 21:16:42 	kernel 		tun_wg0: loop detected
Aug 15 21:16:42 	kernel 		tun_wg0: loop detected
Aug 15 21:16:42 	kernel 		tun_wg0: loop detected
Aug 15 21:16:42 	php-fpm 	40603 	/rc.newwanip: rc.newwanip: on (IP address: WAN_IP) (interface: WAN1[wan]) (real interface: igb0).
Aug 15 21:16:42 	kernel 		tun_wg0: loop detected
Aug 15 21:16:42 	php-fpm 	40603 	/rc.newwanip: rc.newwanip: Info: starting on igb0.
Aug 15 21:16:41 	kernel 		tun_wg0: loop detected
Aug 15 21:16:41 	kernel 		tun_wg0: loop detected
Aug 15 21:16:41 	check_reload_status 	484 	rc.newwanip starting igb0
Aug 15 21:16:41 	kernel 		tun_wg0: loop detected
Aug 15 21:16:41 	php-fpm 	445 	/rc.dyndns.update: Dynamic DNS (pfsense.example.fi) There was an error trying to determine the public IP for interface - wan (igb0 ).
Aug 15 21:16:41 	php-fpm 	445 	/rc.dyndns.update: Dynamic DNS cloudflare (pfsense.example.fi): IP address could not be extracted from Check IP Service
Aug 15 21:16:41 	kernel 		tun_wg0: loop detected
Aug 15 21:16:41 	php-fpm 	79596 	/rc.linkup: HOTPLUG: Configuring interface wan
Aug 15 21:16:41 	php-fpm 	79596 	/rc.linkup: DEVD Ethernet attached event for wan
Aug 15 21:16:41 	check_reload_status 	484 	Reloading filter
Aug 15 21:16:41 	kernel 		tun_wg0: loop detected
Aug 15 21:16:41 	php-fpm 	445 	/rc.dyndns.update: dpinger: No dpinger session running for gateway GW_VPN3_gw
Aug 15 21:16:41 	kernel 		tun_wg0: loop detected
Aug 15 21:16:41 	php-fpm 	445 	/rc.dyndns.update: dpinger: No dpinger session running for gateway GW_VPN2_gw
Aug 15 21:16:41 	php-fpm 	90936 	/rc.linkup: Removing static route for monitor 8.8.4.4 and adding a new route through WG_VPN_GW_IP
Aug 15 21:16:41 	php-fpm 	90936 	/rc.linkup: Removing static route for monitor 8.8.8.8 and adding a new route through WG_VPN_GW_IP
Aug 15 21:16:40 	php-fpm 	446 	/rc.newwanip: IP Address has changed, killing states on former IP Address OLD_WAN_GW_IP.
Aug 15 21:16:40 	check_reload_status 	484 	Restarting OpenVPN tunnels/interfaces
Aug 15 21:16:40 	check_reload_status 	484 	Restarting IPsec tunnels
Aug 15 21:16:40 	rc.gateway_alarm 	28598 	>>> Gateway alarm: GW_VPN3_gw (Addr:8.8.8.8 Alarm:1 RTT:1858.240ms RTTsd:0.000ms Loss:0%)
Aug 15 21:16:40 	php-fpm 	90936 	/rc.linkup: DEVD Ethernet detached event for wan
Aug 15 21:16:40 	check_reload_status 	484 	Reloading filter
Aug 15 21:16:40 	check_reload_status 	484 	updating dyndns wan
Aug 15 21:16:38 	php-fpm 	446 	/rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through WG_VPN_GW_IP
Aug 15 21:16:38 	php-fpm 	446 	/rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through WG_VPN_GW_IP
Aug 15 21:16:38 	php-fpm 	446 	/rc.newwanip: Removing static route for monitor 1.1.1.1 and adding a new route through WAN_GW_IP
Aug 15 21:16:37 	check_reload_status 	484 	Reloading filter
Aug 15 21:16:37 	check_reload_status 	484 	Restarting OpenVPN tunnels/interfaces
Aug 15 21:16:37 	check_reload_status 	484 	Restarting IPsec tunnels
Aug 15 21:16:37 	check_reload_status 	484 	updating dyndns GW_VPN2_gw
Aug 15 21:16:37 	rc.gateway_alarm 	80506 	>>> Gateway alarm: GW_VPN2_gw (Addr:8.8.4.4 Alarm:1 RTT:14837.011ms RTTsd:12222.434ms Loss:58%)
Aug 15 21:16:37 	php-fpm 	446 	/rc.newwanip: rc.newwanip: on (IP address: WAN_IP) (interface: WAN1[wan]) (real interface: igb0).
Aug 15 21:16:37 	php-fpm 	446 	/rc.newwanip: rc.newwanip: Info: starting on igb0.
Aug 15 21:16:36 	check_reload_status 	484 	Restarting IPsec tunnels
Aug 15 21:16:36 	check_reload_status 	484 	rc.newwanip starting igb0
Aug 15 21:16:27 	check_reload_status 	484 	Linkup starting igb0
Aug 15 21:16:27 	kernel 		igb0: link state changed to UP
Aug 15 21:16:23 	check_reload_status 	484 	Linkup starting igb0
Aug 15 21:16:23 	kernel 		igb0: link state changed to DOWN
Aug 15 21:16:18 	php-fpm 	445 	/rc.linkup: HOTPLUG: Configuring interface wan
Aug 15 21:16:18 	php-fpm 	445 	/rc.linkup: DEVD Ethernet attached event for wan
Aug 15 21:16:17 	kernel 		igb0: link state changed to UP
Aug 15 21:16:17 	check_reload_status 	484 	Linkup starting igb0

stephenw10

That log message is created by interface recursions such as a tunnel trying to open connections inside itself. The last time I saw this it was because of peer IPs that were defined as IPs inside the tunnel subnet:
https://forum.netgate.com/topic/168357/system-log-tun_wg0-loop-detected

However, since you're not seeing this continually it seems very likely that the default gateway setting being via the WG tunnel is causing problems when it fails or perhaps when it recovers.

If you don't configure the system default gateway to be the WG gateway and instead use policy based routing does it stop those logs?

Steve

Mixka

I did lot of testing. Really pulled out my hairs out.

TL;DR: After I let my WAN interface DHCP lease to expire I changed interface MAC address to make sure that I got another IP. With new IP everything is working as should.

Really nice :)