pfSense 2.7.2 loss of WAN connection shortly after IP assignment.

Aus_Karlos · Apr 10, 2024, 10:21 AM

Hey guys im new to use pfSense. I have a pfSense installed as a VM on Proxmox which is a SuperMicro server. The LAN interface is a virtual adapter and the WAN interface are from an Intel C621 chipset 1GB NIC card using PCI-E Passthrough.

Im currently on a Telstra (Australian ISP) Business plan with a StaticIP. Telstra however use a DHCP service to still give out StaticIPs. The issue im having is that when pfSense WAN is assigned the StaticIP the internet functions for about 5-10mins then stops responding. No packets are sent or received when using packet capture. If i refresh the WAN adapter I then get a random dynamic IP, after 5-10mins the connection will drop again and i'll have to rinse and repeat.

I have ruled out an ISP issue for now as using their supplied Telstra modem it will be assigned the IP and the internet functions fine. (its an ongoing investigation with my ISP as they said Business plans should not get a Dynamic IP)

At first I though it was a release/renew issue so what I did was get the StaticIP assigned to the Telstra Modem, cloned the MAC address so I don't have to release it. And configure the WAN port on pfsense to spoof the MAC (Thinking that a PCI-E passthrough I have full control over the device in the VM and dont need to change anything on the Host). However the same issue arises after about 5mins. I loose connection and get a dynamic IP (Even with the spoofed MAC in use). Plugging the Telstra modem back in I get a new dynamic IP indicating that the StaticIP is still leased. I have to wait 8hrs for the lease to expire on their end before I can try again if I can't release it manually (They don't have the tools to force an expiry).

As anyone experienced this? Im hoping its a configuration issue on my end.
Thanks guys.

stephenw10 · Apr 10, 2024, 1:13 PM

How long a lease is the ISP giving you? Check the dhcp logs.

The fact it pulls another lease and it's something random is odd. Is it possible something else is connected and trying to pull a lease too?

@Aus_Karlos said in pfSense 2.7.2 loss of WAN connection shortly after IP assignment.:

No packets are sent or received when using packet capture.

Nothing at all? Not ever the dhcp traffic?

Do the main pfSense system logs show anything at the 5min point when it stops?

Steve

Aus_Karlos · Apr 10, 2024, 1:13 PM

@stephenw10 Found the dhcp logs when the Static IP was assigned.

Apr 10 07:28:59 	dhclient 	17457 	PREINIT
Apr 10 07:28:51 	dhclient 	38420 	exiting.
Apr 10 07:28:51 	dhclient 	38420 	connection closed
Apr 10 07:18:54 	dhclient 	38420 	bound to 110.xxx.xx.xxx -- renewal in 1800 seconds.
Apr 10 07:18:54 	dhclient 	83756 	Creating resolv.conf
Apr 10 07:18:54 	dhclient 	83268 	Adding new routes to interface: ixl0
Apr 10 07:18:54 	dhclient 	82312 	New Routers (ixl0): 110.xxx.xx.x
Apr 10 07:18:54 	dhclient 	80991 	New Subnet Mask (ixl0): 255.255.255.0
Apr 10 07:18:54 	dhclient 	80482 	New IP Address (ixl0): 110.xxx.xx.xxx
Apr 10 07:18:54 	dhclient 	79908 	ifconfig ixl0 inet 110.xxx.xx.xxx netmask 255.255.255.0 broadcast 255.255.255.255
Apr 10 07:18:54 	dhclient 	79288 	Starting add_new_address()
Apr 10 07:18:54 	dhclient 	78504 	BOUND
Apr 10 07:18:54 	dhclient 	38420 	DHCPACK from 58.162.26.129
Apr 10 07:18:53 	dhclient 	38420 	DHCPREQUEST on ixl0 to 255.255.255.255 port 67
Apr 10 07:18:53 	dhclient 	76936 	ARPCHECK
Apr 10 07:18:51 	dhclient 	70966 	ARPSEND
Apr 10 07:18:51 	dhclient 	38420 	DHCPOFFER from 58.162.26.129
Apr 10 07:18:51 	dhclient 	38420 	DHCPDISCOVER on ixl0 to 255.255.255.255 port 67 interval 2
Apr 10 07:18:44 	dhclient 	38420 	DHCPREQUEST on ixl0 to 255.255.255.255 port 67
Apr 10 07:18:40 	dhclient 	38420 	DHCPREQUEST on ixl0 to 255.255.255.255 port 67
Apr 10 07:18:38 	dhclient 	38420 	DHCPREQUEST on ixl0 to 255.255.255.255 port 67
Apr 10 07:18:36 	dhclient 	38420 	DHCPREQUEST on ixl0 to 255.255.255.255 port 67
Apr 10 07:18:36 	dhclient 	46514 	ARPCHECK
Apr 10 07:18:34 	dhclient 	41034 	ARPSEND
Apr 10 07:18:33 	dhclient 	38420 	DHCPOFFER from 58.162.26.129
Apr 10 07:18:33 	dhclient 	38420 	DHCPDISCOVER on ixl0 to 255.255.255.255 port 67 interval 2
Apr 10 07:18:33 	dhclient 	38420 	DHCPNAK from 144.133.168.168
Apr 10 07:18:33 	dhclient 	38420 	DHCPREQUEST on ixl0 to 255.255.255.255 port 67
Apr 10 07:18:33 	dhclient 	39160 	PREINIT

It looks like it was only a few seconds before it closed the connection when I got my StaticIP.

But then after that with the dynamic IPs the I get the following in the dhcp logs.

bound to 60.225.56.64 -- renewal in 33993 seconds.

Which is about 9hrs. However I have timed it properly and my connection drops out at the 30min mark down to the second.

stephenw10 · Apr 10, 2024, 1:29 PM

@Aus_Karlos said in pfSense 2.7.2 loss of WAN connection shortly after IP assignment.:

Apr 10 07:18:54 dhclient 38420 bound to 110.xxx.xx.xxx -- renewal in 1800 seconds.

Hmm, only a 1h lease on the static IP. But the process is closed after 10mins. Did the link bounce?

Do both leases come from the same DHCP server?

Gertjan · Apr 10, 2024, 1:35 PM

@Aus_Karlos

What is your definition of a static IP ?

You've not selected a static IP setup on your WAN, but a DHCP (the D stands for Dynamic).

This :

is completely different.
So its the DHCP client that is used, "requesting" for upstream (ISP) if some server can give a DHCP lease.
144.133.168.168 said : not me ^^
But 58.162.26.129 gave a lease !
For some unknown reason, the dhcp client continued requesting ..... strange.
But our 58.162.26.129 insisted, and was finally accepted.

bound to 110.xxx.xx.xxx -- renewal in 1800 seconds.

You have a WAN IP, and it will get renewed in 30 minutes - the lease is valid for 1 hour, which is a more simple procedure.
Still a bit short for a IPv4 lease. But hey, why not, not all ISPs are created equal.
I'm curious to see what happened at this this "30 minutes" mark.

10 minutes in :

@Aus_Karlos said in pfSense 2.7.2 loss of WAN connection shortly after IP assignment.:

Apr 10 07:28:51 dhclient 38420 exiting.
Apr 10 07:28:51 dhclient 38420 connection closed

what happened ?
Check also the gateway logs (dpinger) -the connection became that bad that 'dpinger' pulled = reset the WAN - the plug ?
And if so, if the connetion is bad, DHCP restarts but can't 'pass' neither.

Aus_Karlos · Apr 10, 2024, 1:37 PM

@stephenw10 No, the StaticIPs come from a different gateway (144.something.something )than the Dynamic ones im getting (60.255.xxx.xx). The link doesn't bounce as there are no other WAN adapters currently setup. The system can see 2x 1GB adapters and a 10Gbe LAN adapter. The second 1gb adapter is not setup under interfaces and the 10Gbe is the LAN interface.

@Gertjan Telstra provide StaticIPs though DHCP for domestic home Business plans. They only provide true StaticIPs to commercial sites located in the CBD.

Aus_Karlos · Apr 10, 2024, 1:42 PM

@Gertjan

Apr 10 07:28:54 	dpinger 	52091 	exiting on signal 15
Apr 10 07:28:54 	dpinger 	52091 	WAN_DHCP 110.xxx.xx.x:: sendto error: 65
Apr 10 07:28:53 	dpinger 	52091 	WAN_DHCP 110.xxx.xx.x: sendto error: 65
Apr 10 07:27:57 	dpinger 	52091 	WAN_DHCP 110.xxx.xx.x: Alarm latency 0us stddev 0us loss 100%
Apr 10 07:27:55 	dpinger 	52091 	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 110.xxx.xx.x bind_addr 110.xxx.xx.xxx identifier "WAN_DHCP "
Apr 10 07:27:55 	dpinger 	26192 	exiting on signal 15
Apr 10 07:27:55 	dpinger 	95108 	exiting on signal 15
Apr 10 07:20:18 	dpinger 	95108 	WAN_DHCP 110.xxx.xx.x: Alarm latency 13842us stddev 2591us loss 22%
Apr 10 07:18:59 	dpinger 	95108 	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 110.xxx.xx.x bind_addr 110.xxx.xx.xxx identifier "WAN_DHCP "

So it looks like im getting packet loss shortly after connecting with the StaticIP.

stephenw10 · Apr 10, 2024, 2:17 PM

It could be objecting to the monitoring pings. Try disabling gateway monitoring in Sys > Routing > Gateways, edit the gateway.

If that works you can set the monitoring to some other IP.

Gertjan · Apr 10, 2024, 2:44 PM

@Aus_Karlos said in pfSense 2.7.2 loss of WAN connection shortly after IP assignment.:

Apr 10 07:18:59 dpinger 95108 send_interval 500ms .....

Compare with the further above :

Apr 10 07:18:54 dhclient 38420 bound to 110.xxx.xx.xxx -- renewal in 1800 seconds.

This is the very same moment that the dhcpclient has been its job : all is well, a WAN IP is alive.

Less then 2 minutes later : the first ICMPs get lost : the connection degrades fast :

Apr 10 07:20:18 dpinger 95108 WAN_DHCP 110.xxx.xx.x: Alarm latency 13842us stddev 2591us loss 22%

and less then 10 minutes later (note : dpinger stayed silent !) , dpinger exists :

Apr 10 07:27:55 dpinger 95108 exiting on signal 1

probably because the WAN got reset. Like some one ripped out the cable.
I even tend to think : if it is not dpinger who did this, it it the physical connection that goes bad ?
What is the upstream device ? a modem ? router ?

And who is this :

Apr 10 07:27:55 dpinger 26192 exiting on signal 15

26192 and 95108 are different dpinger processses.
Do you have more then one WAN ?

Anyway, another dpinger gets spawn : process 52091 :

Apr 10 07:27:55 dpinger 52091 send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 110.xxx.xx.x bind_addr 110.xxx.xx.xxx identifier "WAN_DHCP "

But it alarms-out right away with 100 % loss ...
And then the interface is pulled down, again, more like the link was pulled down like a cable disconnect :

Apr 10 07:28:53 dpinger 52091 WAN_DHCP 110.xxx.xx.x: sendto error: 65
Apr 10 07:28:54 dpinger 52091 WAN_DHCP 110.xxx.xx.x:: sendto error: 65

"sendto error" means : "the interface used has been removed from the system".

So, for dpinger, it's time to quiet as there is nothing more to do :

Apr 10 07:28:54 dpinger 52091 exiting on signal 15

Process 52091 stops ....

stephenw10 · Apr 10, 2024, 2:56 PM

That particular sendto error is 'no route': https://docs.netgate.com/pfsense/en/latest/troubleshooting/gateway-errors.html#sendto-error-65

So it lost the IP/gateway rather than the interface itself.

Aus_Karlos · Apr 11, 2024, 11:35 AM

Just waiting on my ISP report. The tech team was confused as even though the Business plan lists "StaticIP coming soon". The plan itself is only a StaticIP service and they have no clue as to why im getting Dynamic IPs. They have a theory that my service was somehow activated 2 weeks to early on their end and my account had yet to be configured. I got a message from the ISP on the 2nd of April saying the service is good to go. But it wasn't suppose to be active until the 12th tomorrow.

So hopefully my issue will be resolved soon.

Aus_Karlos · May 30, 2024, 1:37 AM

So its been a while but I got an update. It was Telstra. Which today is still unresolved. The lack of internal dashboard tools for their tech support makes it extremely difficult to identify any issues on their network something as simple as to view the current connected devices MAC address connected to the NTD they have to elevate it to NBNCo.

I switched over to Aussie Broadband and was setup in seconds with no issues. I even have framed Route setup and have a pool of static IP addresses I can use for online services. Not to mention the amount of tools at your disposal on the user dashboard portal.