Frequent disconnection when with WAN network when using static IP from ISP

[nischay]

Hi Everyone,

I am facing wierd issue. I have static public IPv4 address from my ISP. When I configure it on a NIC I see frequent disconnection. I see packet get dropped (ingress/egress) for 3 -5 minutes then again it start working. When I use same static IP on a Ubuntu laptop I am not seeing packer drop issue. I also observed no issue when I use PPPoE profile from ISP. I tried configuring static IP on different interfaces but observed same issue.

PfSense - 2.7.1-RELEASE

[stephenw10]

Do you still see the gateway in the ARP table when it fails?

Does is fail after the same time each time?

[nischay]

@stephenw10

Thanka for your reply. Disconnection is pretty frequent and consistent in timing. Happens every hour.

I configured same IP address on my OpenWRT router not seeing any issue.

[stephenw10]

Ok well check the ARP table when it fails.

What does the system log show?

[rajnetgen]

@stephenw10 Been facing same issue but for me the disconnection is random, 1-2 times per hour.
Checked ARP table and entry for gateway and IP was still there during disconnection.

System logs show packet loss for gateway IP(have tried adding google dns and cloudflare ip for uptime monitoring)

Jan 12 13:12:16 dpinger 92509 FTTHGW 122.12.173.1: Alarm latency 15381us stddev 1881us loss 22%
Jan 12 13:16:08 dpinger 92509 FTTHGW 122.12.173.1: Clear latency 15220us stddev 1567us loss 6%
Jan 12 14:39:53 dpinger 92509 FTTHGW 122.12.173.1: Alarm latency 15238us stddev 1317us loss 22%
Jan 12 14:41:06 dpinger 92509 FTTHGW 122.12.173.1: Alarm latency 1100683us stddev 911751us loss 94%
Jan 12 14:41:20 dpinger 92509 FTTHGW 122.12.173.1: Alarm latency 232348us stddev 595618us loss 70%
Jan 12 14:41:59 dpinger 92509 FTTHGW 122.12.173.1: Clear latency 82639us stddev 346324us loss 5%

[stephenw10]

It's recovering by itself?

If it's still monitoring the WAN gateway IP directly try setting an external monitoring IP like 8.8.8.8 or 1.1.1.1. That will give you better data and it may be simply that gateway stops responding to ping when it's loaded.
https://docs.netgate.com/pfsense/en/latest/routing/gateway-configure.html

[rajnetgen]

@stephenw10 have checked the ARP table for changes when there is gateway down issues.
I did saw that the when the gateway starts to show pack loss and goes down there is a time left under status for the gateway IP in ARP table.
FTTH 122.17.13.1 74:22:dd:01:51:4a broadband.com Expires in 43 seconds ethernet
FTTH 122.17.13.145 94:e1:bd:13:c6:20 broadband.com Permanent ethernet

If I delete the entry then the gateway starts to instantly recover and come back online.
Else I guess it will wait for the time to expire and then come back online.

But this does not happen everytime and this time rune for 1200 seconds and sometimes it goes to zero and then again start from 1200 seconds

[stephenw10]

You should still set the monitoring IP to something external. The ISPs gateway is under no obligation to respond to ping. It may just drop ping requests when it's loaded.

[rajnetgen]

@stephenw10
Yes I use 3rd party like google dns or cloudflare as default but had changed to gateway to test
The behaviour is same for both cases.
When the connection starts to get packet loss, if I go to ARP table and delete the gateway entry then it starts to come back online else it wait for timeout and then come back online automatically.

Can we reduce the ARP timeout?

[nischay]

@stephenw10

I tried both 1.1.1.1/8.8.8.8 even tried ip of my public VM running in digital ocean in Bengaluru, India as GW monitoring IP.

I configured same static IP in my OpenWRT box and strangley not seeing any issue runnning fine for more then 2 weeks without any downtime.

For now I have configured port forwarding to pfsense and added OpenWRT LAN ip as gateway in Pfsense.

[JonathanLee]

Have you checked your category 5, 6 or 7 the rj45 cable with a cable tester?

[nischay]

@JonathanLee

It is CAT6 5 Feet patch cable. Used same cable to connect OpenWRT to ISP ONT.

[JonathanLee]

@nischay they have speed ratings for each type and use. is it a shielded one? It could be getting interference from elevators, or florescent lights, many issues. Swap out that cable for an upgraded one

[stephenw10]

@rajnetgen said in Frequent disconnection when with WAN network when using static IP from ISP:

Can we reduce the ARP timeout?

Yes you can set it as a system tunable: net.link.ether.inet.max_age
pfSense/FreeBSD uses 1200s by default.

[nischay]

@stephenw10

Thanks for pointing it out. I have updated net.link.ether.inet.max_age to 60 seconds same as Linux default. I will update here if it help to resolve issue.

[root@pfsense-fw ~]# sysctl net.link.ether.inet.max_age
net.link.ether.inet.max_age: 60

[stephenw10]

I've once seen an ISP that would fail to arp for devices in that situation and so required a lower value there. So it's unlikely, but possible.

[nischay]

@stephenw10

Seems like changing ARP Cache TTL to 60 seconds fixed issue I was observing. Finally I have resolution to my issue such a relief.