pfSense randomly dropping WAN

phobes

Apologies if this isn't the correct section of the forum, I can relocate if necessary.

pfSense worked perfectly for about 2 years but I started having WAN issues about a week ago. Setup is ethernet from Verizon FiOS ONT directly to WAN port. The connection drops at random, not any specific interval, and refuses to retrieve an IP for a random amount of time. I've tried 2 different physical machines, currently running it in a VM on Proxmox. It's a 2013 Mac Pro that has dual Intel 1g NICs (they show as 10Gbase-T <full-duplex> which cannot be changed in pfSense).

I swapped out ethernet cables so I believe this is either a problem with pfSense or Verizon. I have aliases setup to force certain IP ranges through my VPN, but no other complicated setups regarding firewall etc. The only installed package is Service_Watchdog to monitor the VPN.

I followed this guide as well, using the MAC address and other information from the G3100 router Verizon provided. Screenshots of my configuration are attached.

Verizon ran remote tests and said everything looks fine on their end, but I'm starting to wonder if there isn't something wrong with the ONT or cabling going to/from the ONT. If they come out and confirm there's nothing wrong on their end they'll likely charge me, so I'd like to avoid that.

The directory /var/run/dhclient doesn't exist and seen other posts about this, so I just created it and manually ran dhclient vtnet1, it did not successfully receive an IP when I did this, but it did on its own a few minutes later.

I would appreciate any help! Info is below, if I left out important details please let me know and I'll edit this post. I truncated the Gateway log as after the WAN went down it repeatedly spammed sendto error: 64.

General:

Jun 21 13:53:30	check_reload_status	411	Syncing firewall
Jun 21 13:53:30	php-fpm	380	/widgets/widgets/speedtest.widget.php: Configuration Change: admin@192.168.1.140 (Local Database): Save speedtest results
Jun 21 13:51:00	sshguard	37006	Now monitoring attacks.
Jun 21 13:51:00	sshguard	12501	Exiting on signal.
Jun 21 13:50:40	sshguard	12501	Now monitoring attacks.
Jun 21 13:50:40	login	8874	login on ttyv0 as root
Jun 21 13:50:40	xinetd	8522	Reconfigured: new=0 old=2 dropped=0 (services)
Jun 21 13:50:40	xinetd	8522	readjusting service 19000-udp
Jun 21 13:50:40	xinetd	8522	readjusting service 6969-udp
Jun 21 13:50:40	xinetd	8522	Swapping defaults
Jun 21 13:50:40	xinetd	8522	Starting reconfiguration
Jun 21 13:50:39	root	82159	Bootup complete
Jun 21 13:50:39	php-fpm	380	/rc.start_packages: Restarting/Starting all packages.

Routing:

Jun 21 13:50:38	miniupnpd	21840	Listening for NAT-PMP/PCP traffic on port 5351
Jun 21 13:50:38	miniupnpd	21840	no HTTP IPv6 address, disabling IPv6
Jun 21 13:50:38	miniupnpd	21840	HTTP listening on port 2189
Jun 21 13:45:58	miniupnpd	55239	PCPSendUnsolicitedAnnounce(sockets[0]) sendto(): No route to host
Jun 21 13:45:58	miniupnpd	55239	SendNATPMPPublicAddressChangeNotification: cannot get public IP address, stopping
Jun 21 13:45:58	miniupnpd	55239	Failed to get IP for interface vtnet1
Jun 21 13:45:58	miniupnpd	55239	ioctl(s, SIOCGIFADDR, ...): Can't assign requested address
Jun 21 13:45:58	miniupnpd	55239	Cannot get IP address for ext interface vtnet1. Network is down
Jun 21 13:45:58	miniupnpd	55239	ioctl(s, SIOCGIFADDR, ...): Can't assign requested address
Jun 21 13:45:56	miniupnpd	55239	PCPSendUnsolicitedAnnounce(sockets[0]) sendto(): No route to host
Jun 21 13:45:56	miniupnpd	55239	SendNATPMPPublicAddressChangeNotification: sendto(s_udp=11, port=5351): No route to host
Jun 21 13:39:56	miniupnpd	55239	PCPSendUnsolicitedAnnounce(sockets[0]) sendto(): No route to host
Jun 21 13:39:56	miniupnpd	55239	SendNATPMPPublicAddressChangeNotification: cannot get public IP address, stopping
Jun 21 13:39:56	miniupnpd	55239	Failed to get IP for interface vtnet1
Jun 21 13:39:56	miniupnpd	55239	ioctl(s, SIOCGIFADDR, ...): Can't assign requested address
Jun 21 13:39:56	miniupnpd	55239	Cannot get IP address for ext interface vtnet1. Network is down
Jun 21 13:39:56	miniupnpd	55239	ioctl(s, SIOCGIFADDR, ...): Can't assign requested address
Jun 21 13:39:54	miniupnpd	55239	PCPSendUnsolicitedAnnounce(sockets[0]) sendto(): No route to host
Jun 21 13:39:54	miniupnpd	55239	SendNATPMPPublicAddressChangeNotification: sendto(s_udp=11, port=5351): No route to host
Jun 21 13:36:09	miniupnpd	55239	PCPSendUnsolicitedAnnounce(sockets[0]) sendto(): No route to host
Jun 21 13:36:09	miniupnpd	55239	Listening for NAT-PMP/PCP traffic on port 5351
Jun 21 13:36:09	miniupnpd	55239	no HTTP IPv6 address, disabling IPv6
Jun 21 13:36:09	miniupnpd	55239	HTTP listening on port 2189
Jun 21 13:36:09	miniupnpd	55239	Cannot get IP address for ext interface vtnet1. Network is down
Jun 21 13:36:09	miniupnpd	55239	ioctl(s, SIOCGIFADDR, ...): Can't assign requested address

Gateways:

Jun 21 15:12:39	dpinger	79961	WAN_DHCP [WAN_IP]: sendto error: 64
Jun 21 15:12:38	dpinger	79961	WAN_DHCP [WAN_IP]: sendto error: 64
Jun 21 15:12:37	dpinger	79961	WAN_DHCP [WAN_IP]: sendto error: 64
Jun 21 15:04:59	dpinger	79961	WAN_DHCP [WAN_IP]: Alarm latency 3723us stddev 2552us loss 21%
Jun 21 13:50:29	dpinger	80240	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 10.18.0.2 bind_addr 10.18.0.2 identifier "PROTONVPN_VPNV4 "
Jun 21 13:50:29	dpinger	79961	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr [WAN_IP] bind_addr [WAN_IP]79 identifier "WAN_DHCP "
Jun 21 13:50:29	dpinger	18564	exiting on signal 15
Jun 21 13:50:22	dpinger	18564	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr [WAN_IP] bind_addr [WAN_IP]79 identifier "WAN_DHCP "
Jun 21 13:50:22	dpinger	92937	exiting on signal 15
Jun 21 13:50:21	dpinger	92937	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr [WAN_IP] bind_addr [WAN_IP]79 identifier "WAN_DHCP "

config-verizon-20240621153453.xml

phobes

I am at a loss here. I made no changes to my configuration when the issue started. The cable between the ONT and my machine has been replaced and the ONT is confirmed working properly.

I just released WAN and relinquished lease, manually started dhclient for WAN, internet works on router but no other device on network - until I reroot/reboot pfSense.

After doing the above, within minutes I begin getting latency errors. This happens on the VM as well as 2 separate physical devices.

Jun 23 22:52:36	dpinger	22789	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 10.18.0.2 bind_addr 10.18.0.2 identifier "PROTONVPN_VPNV4 "
Jun 23 22:52:36	dpinger	22284	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 1.1.1.1 bind_addr 100.4.58.136 identifier "WAN_DHCP "
Jun 23 22:52:36	dpinger	63877	exiting on signal 15
Jun 23 22:52:32	dpinger	63877	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 1.1.1.1 bind_addr 100.4.58.136 identifier "WAN_DHCP "
Jun 23 22:52:32	dpinger	60770	exiting on signal 15
Jun 23 22:52:32	dpinger	60770	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 1.1.1.1 bind_addr 100.4.58.136 identifier "WAN_DHCP "
Jun 23 22:45:16	dpinger	9914	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 10.25.0.3 bind_addr 10.25.0.3 identifier "PROTONVPN_VPNV4 "
Jun 23 22:45:16	dpinger	9411	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 1.1.1.1 bind_addr 100.4.58.136 identifier "WAN_DHCP "

Gertjan

@phobes said in pfSense randomly dropping WAN:

I am at a loss here.

Not only you, everybody is.
Look again at :

AFAIK, if your PROTONVPN wants to connect, it needs a working WAN connection.
Without your conenction to the ISP no VPN nothing.
Or, your PROTONVPN connection says : all is well : Online. (I presume 10.18.0.2 is the remote, server side VPN IP).

Maybe your WAN_DHCP gateway IP isn't answering to ping ? Pick another one ?!

phobes

@Gertjan Appreciate the reply.

I'm trying to figure out what's causing WAN_DHCP to drop in the first place. I've thrown the logs and configuration at ChatGPT 4o with no positive results. I'm not sure why the VPN says it's up, it doesn't actually function, but regardless the VPN is only relevant for a specific IP range I have specified in Aliases. My PC is exempt from said aliases as I use a VPN client application, but when the WAN is dropped it also has no internet access.

Using my Archer A7 router works fine, the Verizon G3100 router works fine, plugging the ethernet from the ONT directly into my PC works fine. I'm currently trying to see whether I can use pfSense behind one of those routers, but search results aren't promising thus far.

Everything worked for ~2 years and now all the sudden, with absolutely no changes to pfSense or my configuration, I have connectivity about 20% of the day. My homelab is completely crippled, I have websites and services that are no longer accessible outside of my network. Insanely frustrating.

If I can't get it resolved I'll have to look into paying for a static IP, which I really don't want to resort to.

johnpoz

The vpn is showing up because its pinging itself.. You can tell from his 0.311ms response time, that sure isn't the other end of the tunnel?

If you can not ping your gatetway - then your internet connection is down.. If pfsense has an IP - did you you try just setting it as always up or pick something else as the monitor.. But if pfsense thinks the gateway is down because dpinger gets no answer then yeah its not going to work.

phobes

@johnpoz Thanks for the reply! The gateway isn't reachable because the WAN randomly drops, and refuses to acquire a new lease. It's a problem that seems common for Verizon FiOS users specifically.

I've resolved this, albeit in a way I do not really like. I have pfSense running behind my Archer A7 successfully by disabling the DHCP server on the A7, changing it's IP to 10.0.0.1, changing pfSense's WAN to a static IP of 10.0.0.2, added 10.0.0.1 as the static gateway, and adding pfSense's IP to the router's DMZ.

All of my homelab services and website work now, but I'll have to get another device to utilize for WIFI as all Wi-Fi devices now bypass pfSense and subsequently my VPN.

I would still like to get to the bottom of the WAN issue, if anyone reading this has any insight!

johnpoz

@phobes checkout this old thread about dhcp issues with verizon fios

https://forum.netgate.com/post/903882

Sounds like some changes to the timings fixed his

He says he was using the freebsd defaults, which are different than the pfsense defaults

The top is freebsd preset and the bottom is the pfsense defaults

https://docs.netgate.com/pfsense/en/latest/interfaces/configure-ipv4.html#dhcp

phobes

@johnpoz I actually tinkered with this, it didn't seem to help unfortunately. I believe what I'm going to do is get a small managed switch and put it in front of the pfSense VM, I've seen a few people say that did the trick.

Thanks for the reply friend!