WAN only connects on unplug and plug back in

[WN1X]

@randombits You should post your system log after a reboot.

[randombits]

Will do, the one under Status/System Logs/System/General in the gui ?

[Gertjan]

@randombits said in WAN only connects on unplug and plug back in:

and gets a WAN IP address

How's your WAN setup ?
DHCP for IPv4, or something else ?
Did you set up the IPv6 part, and if so, how ?

Typically, you should have this - the most common setup :

(disregard the VPNS line)

Getting WAN IP doesn't make the connection work. It's just a part of what is needed.
A remote (ISP) gateway is as important as the IPv4 WAN.
Because you use pfSense, the DNS IPs - if you got any from your ISP - can be thrown in the waste-bucket or discarded. pfSense uses a resolver.

[WN1X]

@randombits Correct.

[randombits]

This post is deleted!

[randombits]

Ok Logs - now redacted

The first is after pfsense reboot then toggled WAN UP/DOWN WAN in Proxmox is the second part.

Aug 19 14:27:56	php-fpm	397	/index.php: The command '/usr/local/etc/rc.d/vmware-kmod status' returned exit code '1', the output was 'Checking vmmemctl kernel module: not loaded Checking vmblock kernel module: not loaded'
Aug 19 14:27:56	php-fpm	397	/index.php: The command '/usr/local/etc/rc.d/vmware-kmod status' returned exit code '1', the output was 'Checking vmmemctl kernel module: not loaded Checking vmblock kernel module: not loaded'
Aug 19 14:27:56	php-fpm	397	/index.php: The command '/usr/local/etc/rc.d/vmware-kmod status' returned exit code '1', the output was 'Checking vmmemctl kernel module: not loaded Checking vmblock kernel module: not loaded'
Aug 19 14:27:18	arpwatch	35925	bogon 0.0.0.0 fc:b4:67:d4:db:b0
Aug 19 14:27:18	arpwatch	35925	bogon 0.0.0.0 fc:b4:67:d4:db:b0
Aug 19 14:27:18	arpwatch	35925	bogon 0.0.0.0 fc:b4:67:d4:db:b0
Aug 19 14:27:15	check_reload_status	428	Reloading filter
Aug 19 14:27:15	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Aug 19 14:27:15	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Default gateway setting Interface WAN_DHCP Gateway as default.
Aug 19 14:27:15	vnstatd	48634	Interface "tun_wg1" enabled.
Aug 19 14:27:14	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
Aug 19 14:27:14	check_reload_status	428	Syncing firewall
Aug 19 14:27:14	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
Aug 19 14:27:14	check_reload_status	428	Syncing firewall
Aug 19 14:27:13	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
Aug 19 14:27:13	check_reload_status	428	Syncing firewall
Aug 19 14:27:13	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
Aug 19 14:27:13	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
Aug 19 14:27:12	check_reload_status	428	Syncing firewall
Aug 19 14:27:12	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
Aug 19 14:27:11	check_reload_status	428	Syncing firewall
Aug 19 14:27:11	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
Aug 19 14:27:10	kernel		tun_wg1: link state changed to UP
Aug 19 14:27:10	kernel		wg1: changing name to 'tun_wg1'
Aug 19 14:27:10	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
Aug 19 14:27:10	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
Aug 19 14:27:10	vnstatd	48634	Interface "tun_wg0" enabled.
Aug 19 14:27:10	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
Aug 19 14:27:09	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
Aug 19 14:27:09	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
Aug 19 14:27:09	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
Aug 19 14:27:09	check_reload_status	428	Syncing firewall
Aug 19 14:27:09	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
Aug 19 14:27:09	kernel		tun_wg0: link state changed to UP
Aug 19 14:27:09	kernel		wg0: changing name to 'tun_wg0'
Aug 19 14:27:08	check_reload_status	428	Syncing firewall
Aug 19 14:27:08	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
Aug 19 14:27:07	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
Aug 19 14:27:07	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
Aug 19 14:27:07	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
Aug 19 14:27:07	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
Aug 19 14:27:07	login	85707	login on ttyv0 as root
Aug 19 14:27:07	check_reload_status	428	Syncing firewall
Aug 19 14:27:06	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
Aug 19 14:27:06	check_reload_status	428	Syncing firewall
Aug 19 14:27:06	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
Aug 19 14:27:06	php	77093	[pfBlockerNG] DNSBL parser daemon started
Aug 19 14:27:05	php_pfb	70160	[pfBlockerNG] filterlog daemon started
Aug 19 14:27:05	root	72945	Bootup complete
Aug 19 14:27:05	tail_pfb	69910	[pfBlockerNG] Firewall Filter Service started
Aug 19 14:27:05	lighttpd_pfb	67801	[pfBlockerNG] DNSBL Webserver started
Aug 19 14:27:05	vnstatd	48634	Interface "tun_wg0" disabled.
Aug 19 14:27:05	vnstatd	48634	Interface "tun_wg1" disabled.
Aug 19 14:27:05	php_pfb	65170	[pfBlockerNG] filterlog daemon stopped
Aug 19 14:27:05	lighttpd_pfb	64162	[pfBlockerNG] DNSBL Webserver stopped
Aug 19 14:27:05	tail_pfb	63685	[pfBlockerNG] Firewall Filter Service stopped
Aug 19 14:27:04	vnstatd	62273	Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.
Aug 19 14:27:04	kernel		tun_wg1: link state changed to DOWN
Aug 19 14:27:04	kernel		tun_wg0: link state changed to DOWN
Aug 19 14:27:04	php-fpm	397	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
Aug 19 14:27:04	php-fpm	397	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
Aug 19 14:27:04	php-fpm	397	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
Aug 19 14:27:04	php-fpm	397	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
Aug 19 14:27:03	php-fpm	397	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
Aug 19 14:27:03	check_reload_status	428	Syncing firewall
Aug 19 14:27:03	check_reload_status	428	Syncing firewall
Aug 19 14:27:03	php-fpm	397	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
Aug 19 14:26:58	php-fpm	397	/rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
Aug 19 14:26:58	vnstatd	48634	Interface "em0" disabled.
Aug 19 14:26:58	vnstatd	48634	Interface "em0.10" disabled.
Aug 19 14:26:58	vnstatd	48634	Interface "em0.20" disabled.
Aug 19 14:26:58	vnstatd	48634	Interface "em0.30" disabled.
Aug 19 14:26:58	vnstatd	48634	Interface "em0.40" disabled.
Aug 19 14:26:58	vnstatd	48634	Interface "em0.999" disabled.
Aug 19 14:26:58	vnstatd	48634	Interface "em1" disabled.
Aug 19 14:26:58	vnstatd	48634	Interface "pppoe0" disabled.
Aug 19 14:26:58	vnstatd	48634	Monitoring (22): vtnet1 (10000 Mbit) vtnet0.999 (10000 Mbit) vtnet0.40 (10000 Mbit) vtnet0.30 (10000 Mbit) vtnet0.20 (10000 Mbit) vtnet0.10 (10000 Mbit) vtnet0 (10000 Mbit) tun_wg1 (1000 Mbit) tun_wg0 (1000 Mbit) pppoe0 (1000 Mbit) pfsync0 (1000 Mbit) pflog0 (1000 Mbit) ovpns1 (1000 Mbit) ovpnc2 (1000 Mbit) enc0 (1000 Mbit) em1 (1000 Mbit) em0.999 (1000 Mbit) em0.40 (1000 Mbit) em0.30 (1000 Mbit) em0.20 (1000 Mbit) em0.10 (1000 Mbit) em0 (1000 Mbit)
Aug 19 14:26:58	vnstatd	48634	Data retention: 48 5MinuteHours, 4 HourlyDays, 62 DailyDays, 25 MonthlyMonths, -1 YearlyYears, 20 TopDayEntries
Aug 19 14:26:58	vnstatd	48634	vnStat daemon 2.11 (pid:48634 uid:0 gid:0, SQLite 3.43.1)
Aug 19 14:26:57	vnstatd	34674	SIGTERM received, exiting.
Aug 19 14:26:53	php-fpm	56866	/index.php: The command '/usr/local/etc/rc.d/vmware-kmod status' returned exit code '1', the output was 'Checking vmmemctl kernel module: not loaded Checking vmblock kernel module: not loaded'
Aug 19 14:26:52	php-fpm	56866	/index.php: The command '/usr/local/etc/rc.d/vmware-kmod status' returned exit code '1', the output was 'Checking vmmemctl kernel module: not loaded Checking vmblock kernel module: not loaded'
Aug 19 14:26:52	php-fpm	56866	/index.php: The command '/usr/local/etc/rc.d/vmware-kmod status' returned exit code '1', the output was 'Checking vmmemctl kernel module: not loaded Checking vmblock kernel module: not loaded'
Aug 19 14:26:51	sshguard	39798	Now monitoring attacks.
Aug 19 14:26:51	php-fpm	56866	/index.php: Successful login for user 'admin' from: 192.168.2.21 (Local Database)
Aug 19 14:26:50	kernel		ovpnc2: promiscuous mode disabled
Aug 19 14:26:50	kernel		ovpnc2: promiscuous mode enabled
Aug 19 14:26:50	arpwatch	37518	Link layer type 0 not ethernet or fddi
Aug 19 14:26:50	kernel		vtnet1: promiscuous mode enabled
Aug 19 14:26:50	arpwatch	38684	listening on vtnet1
Aug 19 14:26:50	arpwatch	38300	pcap open : : No such device exists (No such device exists)
Aug 19 14:26:50	kernel		vtnet0.999: promiscuous mode enabled
Aug 19 14:26:50	arpwatch	37953	listening on vtnet0.999
Aug 19 14:26:50	arpwatch	37332	listening on vtnet0.30
Aug 19 14:26:50	kernel		vtnet0.30: promiscuous mode enabled
Aug 19 14:26:50	kernel		vtnet0.20: promiscuous mode enabled
Aug 19 14:26:50	arpwatch	36727	listening on vtnet0.20
Aug 19 14:26:50	kernel		vtnet0.10: promiscuous mode enabled
Aug 19 14:26:50	arpwatch	36085	listening on vtnet0.10
Aug 19 14:26:50	arpwatch	35925	listening on vtnet0





Aug 19 14:31:05	php-fpm	58327	/rc.newwanip: rc.newwanip: on (IP address: xx.xx.xx.xx) (interface: PROTONVPN_NL[opt4]) (real interface: ovpnc2).
Aug 19 14:31:05	php-fpm	58327	/rc.newwanip: rc.newwanip: Info: starting on ovpnc2.
Aug 19 14:31:04	check_reload_status	428	rc.newwanip starting ovpnc2
Aug 19 14:31:03	kernel		ovpnc2: link state changed to UP
Aug 19 14:31:00	php-cgi	52317	servicewatchdog_cron.php: Service Watchdog detected service dpinger stopped. Restarting dpinger (Gateway Monitoring Daemon)
Aug 19 14:30:48	php-fpm	56866	/rc.newwanip: rc.newwanip: on (IP address: xxx.xxx.xxx.xxx) (interface: WAN[wan]) (real interface: vtnet1).
Aug 19 14:30:48	php-fpm	56866	/rc.newwanip: rc.newwanip: Info: starting on vtnet1.
Aug 19 14:30:48	check_reload_status	428	Restarting IPsec tunnels
Aug 19 14:30:48	php-fpm	396	/rc.linkup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Aug 19 14:30:48	arpwatch	38684	bogon xxx.xxx.xxx.xxx d4:5a:3f:3c:db:09
Aug 19 14:30:47	check_reload_status	428	rc.newwanip starting vtnet1
Aug 19 14:30:47	php-fpm	396	/rc.linkup: The command '/sbin/ifconfig 'vtnet1' media 'autoselect'' returned exit code '1', the output was 'ifconfig: SIOCSIFMEDIA (media): Operation not supported'
Aug 19 14:30:47	php-fpm	396	/rc.linkup: HOTPLUG: Configuring interface wan
Aug 19 14:30:47	php-fpm	396	/rc.linkup: DEVD Ethernet attached event for wan
Aug 19 14:30:47	php-fpm	396	/rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
Aug 19 14:30:47	check_reload_status	428	Reloading filter
Aug 19 14:30:41	kernel		vtnet1: link state changed to UP
Aug 19 14:30:41	check_reload_status	428	Linkup starting vtnet1
Aug 19 14:30:39	php-fpm	58327	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_DHCP.
Aug 19 14:30:39	php-fpm	58327	/rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Aug 19 14:30:38	check_reload_status	428	Reloading filter
Aug 19 14:30:38	check_reload_status	428	Restarting OpenVPN tunnels/interfaces
Aug 19 14:30:38	check_reload_status	428	Restarting IPsec tunnels
Aug 19 14:30:38	check_reload_status	428	updating dyndns WAN_DHCP
Aug 19 14:30:38	rc.gateway_alarm	81356	>>> Gateway alarm: WAN_DHCP (Addr:xxx.xxx.xxx.xxx Alarm:down RTT:0ms RTTsd:0ms Loss:100%)
Aug 19 14:30:37	php-fpm	397	/rc.linkup: DEVD Ethernet detached event for wan
Aug 19 14:30:37	php-fpm	397	/rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp)
Aug 19 14:30:35	check_reload_status	428	Linkup starting vtnet1
Aug 19 14:30:35	kernel		vtnet1: link state changed to DOWN

The WAN is set like this

My ISP doesn't supply any DNS and I have a fixed IP

It's all very standard no IPv6 although my ISP does supply it - a /48 block I think.

[Gertjan]

@randombits said in WAN only connects on unplug and plug back in:

It's all very standard no IPv6 although my ISP does supply it - a /48 block I think.

As soon as your get your connection issues sorted out, you know what will be next on the list

[stephenw10]

Before you replug the WAN is there a valid default route present in Diag > Routes?

[randombits]

@Gertjan I know, I'm scared of IPv6 ...

[randombits]

@stephenw10 No, vtnet1 doesn't appear to after I toggle the WAN. Scrub that I see my IP listed further down with the lo0 interface

After reboot

And after toggle

And the Gateway tab

[randombits]

This post is deleted!

[stephenw10]

Hmm, OK well no default gateway is what's causing the problem. To confirm that try resaving the gateways page rather then reconnecting the WAN and make sure that also fixes it.

But what is causing that missing gateway... Any errors in the gateway or routing log after reboot?

[randombits]

@stephenw10 Nothing in the routing log and repeated below in the gateway log

Aug 19 17:42:31 dpinger 64161 WAN_DHCP XXX.XXX.XXX.XXX: sendto error: 65

until I toggle the WAN.

(I wonder if anything in Proxmox is the issue ? I can't see why. Although I do split the LAN/WAN in the switch as the VM only has one ethernet port, this has never been an issue before)

[stephenw10]

I would first try disabling some of those packages as a test. You seem to have a bunch of references to old interfaces that are no longer present. And it's also trying to load vmware tools in Proxmox.

This looks to be the issue but it's not clear what the cause is:

Aug 19 14:27:15	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was ''
Aug 19 14:27:15	php_wg	59854	/usr/local/pkg/wireguard/includes/wg_service.inc: Default gateway setting Interface WAN_DHCP Gateway as default.

I'd guess it's a timing issue due to one of those previous errors. Disabling some packages to prevent that may allow it to comes up with a default route and if so we can dig into that.

[randombits]

@stephenw10 Morning, I've removed Vmtools and removed Wireguard that made no difference. Where do I remove the those orphaned interfaces em0, pppoe, em0.xx etc ?

I've just noticed watching the boot screen I see

Configuring WAN interface...[fib_algo] Inet.0 (bsearch4#28) rebuild_fb_flm: switching algo to radix4_lockless

[stephenw10]

That message at boot is fine, not an issue.

The old interfaces look to be in vmstat.

[randombits]

@stephenw10 What I meant was where are they located in the cli (putty) so I remove them if that's the right thing to do. I did use pppoe with my old ADSL ISP, I just swapped from pppoe over to DHCP for the WAN

[stephenw10]

They are in the config file (/conf/config.xml) but you should be able to remove them in the gui by re-configuring vmstat.

[randombits]

@stephenw10 I was the right place I assumed, but a bit daunted pruning the xml. What do you mean by re-configuring the xmstat ?

The interfaces look like this

Does pfSense assume the WAN is vtnet0 or vtnet1 ? I vaguely remember em0 and em1 would get mixed up

[stephenw10]

@randombits said in WAN only connects on unplug and plug back in:

Does pfSense assume the WAN is vtnet0 or vtnet1 ?

It has no preset config for vtnet, the user has to assign those interfaces manually.

If you have em NICs it will initially use em0 as WAN unless you set it differently.

vmstat is used by a package you have installed so probably darkstat or Banwidthd. If you resave the config in that package it should remove those old NIC references.