WAN interface stops working every few days.
-
Most of that taken when it was down?
Was something rebooted at some point in that log? When?
-
@stephenw10
The pings and tracerts where taken while the wan connection was acting up and I was unable to browse the web.About here begins where I manually disconnected and reconnected the PPOE interface from Status >Interface
Aug 31 11:18:08 ppp caught fatal signal TERM
I didn't reboot until ~11:28 or so.
This issue has been really aggravating as several times it's happened I've been in the middle of a work related meeting.. It's somewhat embarrassing to have to reconnect to a meeting regularly due to connection issues when you work in IT...
Sometimes meeting audio will continue but I won't see any video when the net goes out, will usually disconnect me entirely after a bit though.Thoughts?
Here are my nuclear options if I can't figure out anything else.
- Take one of my dell desktops and temporarily stand it up in place of the poweredge to see if it's some oddity with the poweredge (some weird PSU voltage spike maybe?)
- Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
- Seeing if it's possible to place the centurylink provided zyxel "Modem" in a bridge mode and let it handle the PPPoe
- Dropping my Spare Asus router in as the main nat provider (I really do NOT look forward to the prospect of changing the IP address configuration on all of my servers and switches when doing this).
-
I just came to an anecdotal realization that this behavior may potentially occur within a couple of minutes after my PC having been powered on or waken from sleep (although I could be wrong), so I'm switching my PC from hardwired to WiFi thinking that the odd config may somehow be causing an issue? It goes PFSense > TP-Link 16 port POE switch > TP-Link AP > TP-Link switch (via opt1 on AP) > PC
-
Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.
You are running 2.4.5p1 right?
Steve
-
@stephenw10 said in WAN interface stops working every few days.:
Hard to imagine that has anything to do with it. Unless you are spoofing a MAC address somewhere and have a conflict? It would be logged though.
You are running 2.4.5p1 right?
Steve
Yep, 2.4.5 -p1
I would be very surprised if something related to what I'm doing with the AP caused an issue with the WAN interface, It is however oddly coincidental that the issues seem to occur right around the times I'm using the system that's connected to the switch behind it. Could also be something else to do with the system. would like to rule the switch path being an issue out as it is an odd config...No Mac Spoofing
System pfSense
Netgate Device ID: ff022c73b01fa88921e4
BIOS Vendor: Dell Inc.
Version: 2.10.0
Release Date: Thu May 24 2018
Version 2.4.5-RELEASE-p1 (amd64)
built on Tue Jun 02 17:51:17 EDT 2020
FreeBSD 11.3-STABLEThe system is on the latest version.
Version information updated at Mon Aug 31 15:14:55 PDT 2020
CPU Type Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
Current: 2300 MHz, Max: 2301 MHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM
Kernel PTI Enabled
MDS Mitigation Inactive -
Hmm, there's just nothing that can introduce 2-3 seconds of latency in pfSense. Not without deliberately trying least. Limiters can do that.
2.4.5 had a bug in it that behaved similarly but that is fixed in 2.4.5p1.
Steve
-
@gawainxx said in WAN interface stops working every few days.:
- Reset to factory and rebuild the config from absolute scratch, by hand rather then importing it?
If your network setup isn't too complicated, this is what I would have done by now.
If you choose this option, don't put ANYTHING into the default config. Just run it bare and see if it still fails. If it does, this is a good sign that something is wrong with your pfsense box itself.
Jeff
-
@akuma1x
What sort of hardware issues do you think could potentially cause this behavior?I've ran a Memory and CPU torture test and no issues where, I've tried several different nics for the WAN. First one was onboard, second was a broadcom PCIE, current one is an Intel PCIE. I've however been using the onboard NIC for LAN VLAN's this entire time, could the broadcom onboard nic somehow be indirectly effecting WAN?
Restarting the pf sense router or the ONT will resolve the issue, I'm left scratching my head
.P.S. the server is on a Line-Interactive UPS.. (I did also test if the UPS was causing it)_
If the issue happens again with that AP and daisy chained switch disconnected, I'll grudgingly set the router back up from scratch with the exception of the firewall config (which I'll comb through by hand prior to importing)
-
Could a NAT rule for a Nintendo switch cause any issues?
<outbound> <mode>hybrid</mode> <rule> <source> <network>192.168.3.30/32</network> </source> <sourceport></sourceport> <descr><![CDATA[Nindento Switch|Static NAT]]></descr> <target></target> <targetip></targetip> <targetip_subnet></targetip_subnet> <interface>wan</interface> <poolopts></poolopts> <source_hash_key></source_hash_key> <staticnatport></staticnatport> <destination> <any></any> </destination> <updated> <time>1589685349</time> <username><![CDATA[admin@192.168.3.157 (Local Database)]]></username> </updated> <created> <time>1589685349</time> <username><![CDATA[admin@192.168.3.157 (Local Database)]]></username> </created> </rule>I also notice there are some shaping rules burried in my config .xml which are not visible in the GUI.. Hmm
-
No, an outbound NAT rule will not be doing anything.
Traffic shaping is far more likely. Assuming it's anything config related.
Steve
-
Ok, I reloaded everything, with the exception that I imported the VPN config, certs and firewall rules because those would have been a royal PITA to rebuild.
Problem still persists.
There have been several times in the past few weeks where I suddenly got very high latency and packet loss but it resolved itself after a couple of minutes.
Somehow using my main workstation for the first time in a day seems like it could be attributing to the issue, it seems like the behavior occurs 5-10 minutes after I've powered that system on...? I can't think of why a single system could cause the WAN interface of pfsense to behave like this though?
I'm getting towards the end of my list of ideas and could desperately use some solutions.
I've just connected my centurylink C3000z in bridge mode and placed pfsense behind that, seeing if perhaps letting the centurylink "modem" handle the VLAN tagging makes some difference?
Here is a copy of my config, I have scrubbed anything cert or credential related from it.
1599534090821-config_scrubbed.xmlI'm getting down towards my last options which would be to purchase another desktop for the explicit purpose of temporarily running it as the pfsense sever to test if it's somehow a host issue or using my spare ASUS router (This would cause me a lot of headaches as I would have to reconfigure my entire home network, stripping out vlans and resubnetting all of my vms, devices.)
-
The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.
If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.
Steve
-
What version of pfsense??
-
It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5.
-
@stephenw10 said in WAN interface stops working every few days.:
The TTL exceeded message you are seeing from upstream when it happens still makes it look like some upstream routing problem to me.
If you are able to use the ISP router in there as a test though that would rule out an obscure pfSense issue.
Steve
I'm not using the ISP router for routing or dhcp atm, just handling the vlan tagged traffic to see if it has any influence...
I may have to suffer and try running a double NAT for a week or two though to see if the behaviour persists when ISP router handles traffic. -
@stephenw10 said in WAN interface stops working every few days.:
It's 2.4.5p1. Because, yeah, this sure looks like #10414 in 2.4.5.
Interesting, I'll need to take a close look at that thread later. The webui does definately take several seconds to load when I initially try to access it while the gateway issues are occuring
-
If you are somehow hitting that still you would see high latency to the firewall itself from a LAN side client everytime you ran Status > Filter reload.
Steve
-
hmm, manually doing a filter reload caused the ping to firewall to jump from 1 to 100ms for a single ping but nothing noteworthy aside from that.
I am however still using the ISP "Modem" (Truly a router) to handle the VLAN tagging, which i set up last night.
-
@gawainxx I already faced this kind of trouble.
The latency starts growing until th Wan interface stops working.
A (not) solution was to turn monitoring IP to of.Then I changed my ISP and both, the trouble and the ISP, desappeared.
-
@hugoeyng said in WAN interface stops working every few days.:
@gawainxx I already faced this kind of trouble.
The latency starts growing until th Wan interface stops working.
A (not) solution was to turn monitoring IP to of.Then I changed my ISP and both, the trouble and the ISP, desappeared.
To verify, you changed the monitoring function/action off?
I'll add that to my to-do list.
Unfortunately the speed with my ISP is awesome and the pricing is reliable, unlike Comcast. I'd rather not go to Comcast if I can help it as it's tiresome to negotiate pricing once every year.
