Internet Connection Being Interrupted At Certain Intervals
-
Hi,
We are using pfSense CE version 2.8.1 and pfBlockerNG, and the latest patches have been applied. After applying these patches, we are experiencing interruptions on the pfSense connection. This problem occurred after the patches, and we have checked the following points, but the problem has not been completely resolved:
- DHCP Release Record Clearing
- pfBlockerNG Unbound Python mode
- pfBlockerNG was disabled and tested; the result was the same.
We kindly request your information and support regarding this matter.
-
pfSense isn't a switch, but a router, so every NIC has its own connection and isn't related to other NICs.
The connection goes away on your WAN interface ?
You can see that here : example :
If your WAN uplink stops working, then you'll notice that 'DNS' on your LAN also stops working, as the local pfSense DNS resolver needs the uplink to find IP addresses.
Any other uplink / downlink traffic will also be interrupted.pfBlockerng : can't be the issue because pfBlockerng disabled didn't change the behavior.
If the issue originates on the WAN interface, check if you can find lines like this :
where ix3 is your WAN interface.
If you find these lines, then 'someone' is pulling your WAN interface down.
It could be pfSense doing this, as it has decided to reset the interface because is was considered 'bad'.
See the first image : if there are to many ping requests who don't get an answer back, or very slowly, pfSense takes action :That where this comes into play :
System > Routing > Gateways > Edit (the WAN interface)
Normally, if the WAN goes bad, check your ISP 'router' or modem, status and don't stop there : follow the uplink up until the ISP equipment.
Or is your LAN going 'off line' (uplinks, downlinks), for example the link between your pfSense LAN plug and an intermediate device like a switch ? Or is your pfSense LAN cable directly connected to a device (less ideal) ?
Another important point is : your connection is purely IPv4 or IPv4 and IPv6 ?
-
Thank you so much for the detailed explanation and guidance!
Following your advice, I checked the System > General logs for any link state changed to DOWN messages on the WAN/LAN interfaces. I only saw those during manual system reboots, which thankfully rules out any physical cable, switch, or NIC hardware issues. I also checked the Monitoring graphs, and the system is virtually idling (average CPU utilization is under 1%), so the hardware is perfectly fine.
Digging deeper into the logs, I found the actual culprit:
DHCP Spam: I found a massive anomaly in the DHCP logs. A TP-Link Archer AX12 (used as a LAN AP) was stuck in a bugged loop, sending a DHCPREQUEST every 34 seconds! Also, my DHCP pool was at 87% utilization due to dynamic MAC addresses from some Hyper-V VMs.
The Chain Reaction: Because "DHCP Registration" was enabled in the DNS Resolver, this constant 34-second DHCP spam from the AP was triggering Unbound to restart continuously. This caused the DNS resolution to fail, making it feel like the internet was dropping.
I can completely disabled "DHCP Registration" in Unbound, assigned a static IP to the TP-Link AP outside the pfSense DHCP pool, and cleared all the DHCP leases. (Oh, and to answer your other question, my connection is purely IPv4).
The constant DHCP requests in the logs have finally stopped. I am going to test the network for the next 2-3 hours under normal load to be 100% sure, and I will report back here with the final result.
If you have any other suggestions, I would be happy to hear them.
-
The issue where every DHCP DNS registration was fully restarting Unbound was well known and documented in redmine #5413. I used to suffer greatly from that on a busy network!
That issue was marked "solved" a couple of years ago as of pfSense+ 24.11, but it isn't mentioned whether those changes were implemented for CE. I would have thought 2.8.1 would have contained that fix, but you might want to try 2.9.0 beta anyway as it's very close to a stable release.
Side note: there's also new bug at redmine #17003 that's being tracked, similar side effect.
-
You will need to switch the DHCP server from ISC to Kea in order to avoid the
unbounddaemon restarts associated with DHCP client lease renewals. Sounds like perhaps the OP still has ISC enabled as the DHCP server ifunboundis actually restarting with each lease renewal.Moving to Kea avoids the daemon restart required when registering DHCP client leases in the DNS Resolver. Kea has a process for directly injecting the updated client lease information into the DNS Resolver without needing to write to a file and then restart the DNS Resolver daemon process.
-
Hi @bmeeks
Thank you for your support.
I also tried the KEA DHCP settings you mentioned, but the problem persisted. As I mentioned before, it happened after the patch update, and I started checking the updates I made. I saw that one update was conditional and there was a note saying not to do it if another update had already been made, so I reverted that update.
I've also attached a screenshot. I will now test it this way. I will keep you informed of progress.
Thanks again to everyone for their support.
-
You can check if this applies to you :
@luckman212 said in Internet Connection Being Interrupted At Certain Intervals:
Side note: there's also new bug at redmine #17003 that's being tracked, similar side effect.
That 'bug' describes the source of the issue :
While the carrier is flapping, these events continue to accumulate and pfSense repeatedly enters the WAN reconfiguration path.
The carrier flapping : that's "an ISP", or more precise : an ISP connection (upstream ISP modem router or modem included) so pfSense sees a rapid stream of WAN 'down' and WAN 'up' events.
You can see these events coming, have a look, at the probably most important pfSense web GUI page (no joke) : Status > System Logs > System > General
@sacavusoglu : you saw these ? Have these ?
My example shows a lot of interface events, all in the same milli second range.
and the good news is : no need to understand what is happening here, but you, as an pfSense, you do not want to see this. This shouldn't happen, consider it "bad".
A ton of code gets executed when a link goes down, and even more if a link comes up. And be aware, this isn't a pfSense issue. get a tplink, dlink, or even Unifi or high end multi xxxk$ Cisco router, and you will have major issues. Cisco would probably just fire you to solve the issue.
So again : you don't want unstable connections. Not on your LANs, not on your WANs. Period.I get it, most of use WAN connection that cost less then 40 €/$ for a multiple "Gbit/sec" ISP connection, and it has no real guarantees. ISP contract mention it clearly : it's a 'best effort' connection. Great if it works, and if it doesn't, well, you weren’t paying for quality.
My solution, not 100 % perfect, but it will deal with it : put UPS powered switches on all sides of pfSense.
For myself : I keep my ISP router actif on pfSense, as it has to translate 'fiber' to 'RJ45 Ethernet' anyway, and this router is UPS powered. My pfSense WAN can't go down ** anymore, even if I rip the fiber out of the wall.Btw : yeah, I'm rambling, comes with the age I guess.
** nuance : the pfSense System > Routing > Gateways action on a gateway will flag the loss of a connection, and pfSense will try to re establish the connection by pulling it's WAN down and up thus resetting the interface so it can negotiate a new connection. This sequence will take multiple seconds, not sub ill seconds.
-
Hi @Gertjan
When I look at Status > System Logs > System > General, I almost never see this warning, but if you have time, I'm sending you the log link from Google Sheets for you to review.
As you said, network issues can create different situations; regardless of the company or operating system, a crash can occur with an update. Since this problem is within our own company, there's neither a complete internet outage nor a full connection issue. It happens occasionally, so honestly, even I sometimes encounter it while working :)
You never chatter. Experience is invaluable, so I'd always love to listen to your opinions for hours. Since I've worked more in software, hosting, and server backup systems, this area is relatively new to me.
I rolled back the patch as mentioned in my last message, but the result is the same.
Thank you again.
-
@sacavusoglu said in Internet Connection Being Interrupted At Certain Intervals:
I'm sending you the log link from Google Sheets for you to review.
I saw your log.
There are two pfSense boot sequences, each of them followed by two 'linkup' events, one for each interface, re0 (WAN) and re1 (LAN).
All these were normal, and aspected.There is a (2 actually, one for each interface) :
/rc.linkup: Ignoring link event during boot sequence.
During boot, interfaces are brought up (drivers get initialized, memory (for queues) gets allocated, interrupts are set up, etc etc) and during that moment the 'interface event' is discarded for the moment, as the system considers itself in an unknown, 'not ready' state. Like a person unlocking the door of the shop, you can't do that and handling the cash register at same moment.
Here :
Then admin drops by to say hello.
For 27 minutes : all is quiet, and the suddenly :
This one came out of the blue sky.
There were no WAN monitor (dpinger) messages, nothing, so change are more then great this WAN disconnect was initiated by the device to where pfSense is connected to.
Dono what that device is, but you should have a chat with it.
It shouldn't pull down the connection like that. It is of course OK that it does this, but some guy, namely you, will talk about it here on this forum, as you found out about the consquences : "Internet Connection Being Interrupted At Certain Intervals".
So I tend to say : whatever this device is or does, make it stop taking your (pfSense) WAN down down.
The reason could be anything, and depends on what type of connection you use.The story continues :
The WAN comes back a second later, your NIC drivers complais, the entire "link up" sequence barrels down, I see you use a VPN .. what ? Server ? Client ** ? and the very nasty
/rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1787174216] unbound[5891:0] error: bind: address already in use [1787174216] unbound[5891:0] fatal error: could not open ports'
shows up.
This error message means that unbound was about to start (pfSEnse restarted it because of a interface LINK event), but the ports it needs to 'bind' (listen) to (TCP UDP port 53 on all selected interface, typically 'all') couldn't succeed because 'some other' process was already holding the port '53'.
That would be the 'previous' unbound process instance that was told to stop, and wasn't finished shutting down yet ....
And yes, unbound needs some time to shut down, and start up, like hundreds of milli seconds. See the Status > System Logs > System > DNS Resolver for unbound related events and there time requirements.edit : extra info here : If you told unbound to 'forward' to an upstream resolver like 1.1.1.1 over TLS, and also SSL (TLS) aka 'openssl' comes into play. That one needs millions of software lines more == takes even more time.
( did I mention that logs, maybe cryptic, are always telling you what is going on, and when you know what they actually say, also tell you the why part... ? That's why the exist on every system on planet earth - even 'Windows' has a huge log system )
The unbound instance that was blocking the new unbound from starting will probably really stop moments later, and leave your pfSense without a running unbound process. this is not really a problem, but most (like 99,5 %) will see this as 'Internet Connection Being Interrupted' as host names can't be resolved any more. The issue is know as 'DNS is broken' (again) and you probably will see a :
The solution is simple : start it, you can do so in the GUI.
Your real mission (always the same) : get to the bottom of things, and stop making this situation from happing.
It's already know : I rinse and repeat : have a chat with the upstream device, the device WAN is hooked up to.Btw : whatever you do, wherever you find advise, do not - and I insist : do not use / install this one :
as that one is for software developers. The ones that can read scripts, settings, know how thinsg work because they know why it works as it does. These people never have Internet issues, and if they have, they did it on purpose for 'testing reasons'. For the common mortals like me and you : stay away from Service_Watchdog and just solve the issue.
If this means : change the way how you connects to the net, then so be it.
Start by swapping NICs, swap Ethernet cables, double check the power bricks, use an UPS.Btw ; personal advise : you use Realtek NICs.
Normally, you would reserve these NICs for your worst enemy, but don't be surprised you will get sued if you gave them these NICs, they won't appropriate it at all.
Do like everybody else : use Intel NICs.Footnote : realtek isn't bad per se. They did some great things for the 'PC clone community' back in the past (1997) like making a cheap audio chipset "AC 97" so we could ditch our "Sound Blaster sound card" as our PC's had from then on native sound capabilities on the motherboard. They also made NICs for PC clones, mostly build directly into the south bridge chip set (Windows support only) so a PC could have a cheap RJ45 NIC (3COM, Intel etc were way more expensive back then).
Realtek used the other Oses, like everything except Windows = .... a no go for me.
That said, true, things got better recently
@sacavusoglu said in Internet Connection Being Interrupted At Certain Intervals:
this area is relatively new to me
Area ? pfSense ? Believe me, or not, but pfSense just a router. What a router is, does, isn't really rocket sience. It's a rather recent invention and severally documented. Even better : Internet was invented so 'people' could talk about it, as everybody needed to be on the same page (= using the same hardware, protocols etc) to be part of it ... to talk about it.
pfSense exposes more things to the user who is called the admin. So, before, we had 4 things to break our Internet connection, with pfSense we have '97'.
Just apply the KIS rule, and don't touch what you don't know about, and by magic pfSense behaves like any other router out there : it works.@sacavusoglu said in Internet Connection Being Interrupted At Certain Intervals:
You never chatter.
I guess I just proved you wrong
-
Hi @Gertjan
Thank you again for your analysis and support. I'm replying a little late because I have new findings.
First of all, the new stable version of PFSense has been released and is installable. I haven't tested it yet, but I will test it as soon as possible. I will also test OPNSense, another distribution I will be testing.
I will also consider the Intel NIC issue. I plan to develop a suitable infrastructure for it in future upgrades.
Regarding the issues we discussed; as you mentioned, there were interruptions in DHCP, causing the connection to drop and reconnect for a few seconds. I also found the cause of this problem. Normally, all systems in our office operate behind an APC Backup 1200 kV power supply. However, when the air conditioner in the office, which doesn't have an inverter, enters active mode, there is a momentary power surge, affecting the modem. I noticed this interruption in the internet TV box provided by the same internet provider, and it was also happening on that device :)
I also created a service request with the internet provider regarding the DHCP disconnection, and they remotely connected to the modem and performed a firmware update. The firmware update didn't solve the problem, but we understand the issue isn't related to the service provider's fiber optic cable or modem.
As a side note, all the electrical wiring was completely renovated before we started working in this office, and our outlets have grounding. I might have a grounding check done, but I'll assess how well the grounding can tolerate electrical interference during sudden power surges.
That's all the information I have for now. Thank you again for your support.
Privacy Policy · Cookie Policy