WAN interface stops working every few days.

gawainxx

I just came to an anecdotal realization that this behavior may potentially occur within a couple of minutes after my PC having been powered on or waken from sleep (although I could be wrong), so I'm switching my PC from hardwired to WiFi thinking that the odd config may somehow be causing an issue? It goes PFSense > TP-Link 16 port POE switch > TP-Link AP > TP-Link switch (via opt1 on AP) > PC

stephenw10

Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.

You are running 2.4.5p1 right?

Steve

gawainxx

@stephenw10 said in WAN interface stops working every few days.:

Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.

You are running 2.4.5p1 right?

Steve
Yep, 2.4.5 -p1
I would be very surprised if something related to what I'm doing with the AP caused an issue with the WAN interface, It is however oddly coincidental that the issues seem to occur right around the times I'm using the system that's connected to the switch behind it. Could also be something else to do with the system. would like to rule the switch path being an issue out as it is an odd config...

No Mac Spoofing

System pfSense
Netgate Device ID: ff022c73b01fa88921e4
BIOS Vendor: Dell Inc.
Version: 2.10.0
Release Date: Thu May 24 2018
Version 2.4.5-RELEASE-p1 (amd64)
built on Tue Jun 02 17:51:17 EDT 2020
FreeBSD 11.3-STABLE

The system is on the latest version.
Version information updated at Mon Aug 31 15:14:55 PDT 2020
CPU Type Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
Current: 2300 MHz, Max: 2301 MHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM
Kernel PTI Enabled
MDS Mitigation Inactive

stephenw10

Hmm, there's just nothing that can introduce 2-3 seconds of latency in pfSense. Not without deliberately trying least. Limiters can do that.

2.4.5 had a bug in it that behaved similarly but that is fixed in 2.4.5p1.

Steve

akuma1x

@gawainxx said in WAN interface stops working every few days.:

Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?

If your network setup isn't too complicated, this is what I would have done by now.

If you choose this option, don't put ANYTHING into the default config. Just run it bare and see if it still fails. If it does, this is a good sign that something is wrong with your pfsense box itself.

Jeff

gawainxx

@akuma1x
What sort of hardware issues do you think could potentially cause this behavior?

I've ran a Memory and CPU torture test and no issues where, I've tried several different nics for the WAN. First one was onboard, second was a broadcom PCIE, current one is an Intel PCIE. I've however been using the onboard NIC for LAN VLAN's this entire time, could the broadcom onboard nic somehow be indirectly effecting WAN?

Restarting the pf sense router or the ONT will resolve the issue, I'm left scratching my head

.P.S. the server is on a Line-Interactive UPS.. (I did also test if the UPS was causing it)_

If the issue happens again with that AP and daisy chained switch disconnected, I'll grudgingly set the router back up from scratch with the exception of the firewall config (which I'll comb through by hand prior to importing)

gawainxx

@gawainxx

Could a NAT rule for a Nintendo switch cause any issues?

	<outbound>
		<mode>hybrid</mode>
		<rule>
			<source>
				<network>192.168.3.30/32</network>
			</source>
			<sourceport></sourceport>
			<descr><![CDATA[Nindento Switch|Static NAT]]></descr>
			<target></target>
			<targetip></targetip>
			<targetip_subnet></targetip_subnet>
			<interface>wan</interface>
			<poolopts></poolopts>
			<source_hash_key></source_hash_key>
			<staticnatport></staticnatport>
			<destination>
				<any></any>
			</destination>
			<updated>
				<time>1589685349</time>
				<username><![CDATA[admin@192.168.3.157 (Local Database)]]></username>
			</updated>
			<created>
				<time>1589685349</time>
				<username><![CDATA[admin@192.168.3.157 (Local Database)]]></username>
			</created>
		</rule>

I also notice there are some shaping rules burried in my config .xml which are not visible in the GUI.. Hmm

stephenw10

No, an outbound NAT rule will not be doing anything.

Traffic shaping is far more likely. Assuming it's anything config related.

Steve

gawainxx

Ok, I reloaded everything, with the exception that I imported the VPN config, certs and firewall rules because those would have been a royal PITA to rebuild.

Problem still persists.

There have been several times in the past few weeks where I suddenly got very high latency and packet loss but it resolved itself after a couple of minutes.

Somehow using my main workstation for the first time in a day seems like it could be attributing to the issue, it seems like the behavior occurs 5-10 minutes after I've powered that system on...? I can't think of why a single system could cause the WAN interface of pfsense to behave like this though?

I'm getting towards the end of my list of ideas and could desperately use some solutions.

I've just connected my centurylink C3000z in bridge mode and placed pfsense behind that, seeing if perhaps letting the centurylink "modem" handle the VLAN tagging makes some difference?

Here is a copy of my config, I have scrubbed anything cert or credential related from it.
1599534090821-config_scrubbed.xml

I'm getting down towards my last options which would be to purchase another desktop for the explicit purpose of temporarily running it as the pfsense sever to test if it's somehow a host issue or using my spare ASUS router (This would cause me a lot of headaches as I would have to reconfigure my entire home network, stripping out vlans and resubnetting all of my vms, devices.)

stephenw10

The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.

If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.

Steve

Cool_Corona

What version of pfsense??

stephenw10

It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5.

gawainxx

@stephenw10 said in WAN interface stops working every few days.:

The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.

If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.

Steve

I'm not using the ISP router for routing or dhcp atm, just handling the vlan tagged traffic to see if it has any influence...
I may have to suffer and try running a double NAT for a week or two though to see if the behaviour persists when ISP router handles traffic.

gawainxx

@stephenw10 said in WAN interface stops working every few days.:

It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5.

Interesting, I'll need to take a close look at that thread later. The webui does definately take several seconds to load when I initially try to access it while the gateway issues are occuring

stephenw10

If you are somehow hitting that still you would see high latency to the firewall itself from a LAN side client everytime you ran Status > Filter reload.

Steve

gawainxx

@stephenw10

hmm, manually doing a filter reload caused the ping to firewall to jump from 1 to 100ms for a single ping but nothing noteworthy aside from that.

I am however still using the ISP "Modem" (Truly a router) to handle the VLAN tagging, which i set up last night.

hugoeyng

@gawainxx I already faced this kind of trouble.
The latency starts growing until th Wan interface stops working.
A (not) solution was to turn monitoring IP to of.

Then I changed my ISP and both, the trouble and the ISP, desappeared.

gawainxx

@hugoeyng said in WAN interface stops working every few days.:

@gawainxx I already faced this kind of trouble.
The latency starts growing until th Wan interface stops working.
A (not) solution was to turn monitoring IP to of.

Then I changed my ISP and both, the trouble and the ISP, desappeared.

To verify, you changed the monitoring function/action off?
I'll add that to my to-do list.
Unfortunately the speed with my ISP is awesome and the pricing is reliable, unlike Comcast. I'd rather not go to Comcast if I can help it as it's tiresome to negotiate pricing once every year.

kuradeel

One more thing to check, is it possible that your machine is overheating? Specifically the network card? I've had a situation where the network card was malfunctioning and a reboot of the machine would not fix it. I physically had to shut the entire machine down (remove power) in order to fix the problem. I think it stems from the network card in particular overheating. If you have something blocking the airflow or the processors on the network card do not have heatsinks, then perhaps it's that. I had issues with a Broadcom card just generally so I switched to Intel and haven't looked back, except for when I have an overheating issue. Sounds like you have the issue at or around noon? Go on eBay and get something like the INTEL EXPI9404PTL PRO/1000 PT Quad Port PCIe card for like $30. Intel makes the best gigabit cards in my opinion.

gawainxx

@kuradeel
I doubt it, the r210 ii has excellent airflow and mobo, proc temps are all green.

This has happened with onboard as well as 3 different PCIe NICs 2x broadcom, 1x Intel all different chips