Consistent Loss of Internet Connectivity With Wireless Clients

wmheath586

@nollipfsense I am ready to dump it because we really cannot keep having down time. I love the idea of pfSense and what I have see so far (compared to UI), but we cannot keep having issues like this. You are correct as far as DNS. DHCP and IP address reservation is handled by pfSense. The APs are just true APs.

JKnott

@wmheath586

If WiFi clients fail, but wired don't, the problem has nothing to do with pfsense. It's a WiFi issue. Is it only Apple clients? A little more info will help.

wmheath586

@jknott I strongly disagree. The wifi clients do drop, but wired clients are not issue free. The vast majority of wired clients are fine but some will not be able to access the web, while others can but cannot use local hostnames when the issue happens. Wireless Apple products are the only clients which will ask for the wifi password when the issue happens. Windows and Android clients just report the connection has no internet connectivity or sometimes get stuck in a loop of connecting and disconnecting from wifi. What is pointing me to pfSense is the fix. For a while the only fix was to force reload pfBlockerng. Recently, say the past 5 days or so, I have to force reload as well as use Unifi to restart the APs. For a bit more information on the setup:
APs addresses are static and in the 192.168.1.0/24 no VLAN

Clients are in 192.168.20.0/24 VLAN 20, 192.168.30.0/24 VLAN 30, and 192.168.88.0/24 VLAN 80.

pfSense is using Lagg0 as a trunk to carry the VLANs to the switches. Switches are also in the 192.168.1.0/24.

DNS and DHCP servers/settings match and are correct for their respective networks. All clients display/report the correct DNS, gateway, and IP addresses when working.

Gertjan

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

They will attempt to connect and receive the "No Internet Connection" error, or oddly enough, be asked to enter the wifi password (seems to only be an Apple product error.) There seems to be no pattern to the issues. Sometimes it is mobile clients, sometimes it is all wireless clients, sometimes it is only Apple, sometimes only Windows, and some times it is only Android.

Question : and all wired connections are fine, right ?
Take note : the wifi AP is for pfSense just another wired device. What the device is doing with the traffic, is up to the AP. pfSense (the Ethernet protocol for that matter) doesn't know anything about radio devices.

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

Force reload of pfBlockerng

loop:
Related, not related, if any issue exists, go 'vanilla' right away.
This includes DNS settings.
pfSense itself is fine.
Proof of concept : re install, change nothing and see for yourself : it works.
It doesn't ? => Goto loop:

edit : I've some serious bad-ass low bud access points here, like de Linksys "Sisco" E1200 with DD-WRT firmware. They support the ancient "b", "g" and something they call "n".
My iPad and iPhone never had issues with them.
And if they did, I would change the settings right away and/or go back to previous settings.

Not a joke : I've seen a situation where Wifi stopped working several times a day. Nothing was possible any more. Weeks later we found out that a neighbour was using a micro wave without the front door - the glass plate in it was missing.
Again : no joke - these people actually exist.

Gertjan

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

but wired clients are not issue free. The vast majority of wired clients are fine but some will not be able to access the web, while others can but cannot use local hostnames when the issue happens. .......

Yeah, we know.
People keep on using Realtek for example.
Or partially ripped out network cables.
Or try to pump gigabits over what actually is a polished phone cable "bought on Aliexpress".
Etc.
It's true :

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

Wireless Apple products are the only clients which will ask for the wifi password when the issue happens.

That's interesting info.
I invite you to visit the apple.com manual page that explains when an iDevice asks for a wifi password.
You know the answer : when it doesn't recognize / it is a new network.
Ones entered, it never asks that password againfor that Wifi network.
Look up for yourself why it start to ask the the password again.
Hint : Wifi SSID changed - Wifi MAC changed - Wifi encryption changed - Wifi ..... changed
Maybe it also asks again if the gateway (or DNS) -> these are pfSense settings - changed - dit you change these ?
Btw : I didn't test all these cases as I don't need to.

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

VLAN

Oh.
That changes a lot.
There are VLAN's.
I pass.

wmheath586

@gertjan The microwave neighbor is terrifying. If you get a chance check out the "wireless power" guy that cut a hole in his and added a horn in his kitchen.

I understand what you and others are saying about APs. These are Unifi AC Pros, UAP AC Lites, and UAP AR LR units. They have been rock solid. I am not seeing any loss of connectivity from the AP controller to the APs or any errors or alerts on their side. I also do not get any warnings about wireless congestion or radar detection.

As far as going vanilla, I have attempted that. pfSense with no packages, same issue. With pfBlockerng same issue just more frequent. Roll back to a previous version, same issue.

Gertjan

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

These are Unifi AC Pros, UAP AC Lites, and UAP AR LR units. They have been rock solid.

I hope so, I'm planning on buying a couple of them, the Pros or the Lites, very soon myself, as the optic fibre is in front of the door.

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

I am not seeing any loss of connectivity from the AP controller to the APs or any errors or alerts on their side.

Necaue you're using that 'controller' : when an iPhone has to 'authenticity' again to an already known network, is this not logged on the "Unifi side" ?
Something must have changed so that these devices start seeing a previous known network as unknown ....

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

pfSense with no packages, same issue

pfSense vanilla is actually just 'as good' as any other ISP type off the shelves ISP router.
Actually better as pfSense is free, and these Netgear/Linksys/TPLink/etc stuff.
Better because it's using worlds first DHCP server - one of the the best IP routing stack kernels : FreeBSD.

Add a web server in front to make it look like "less difficult", centralize all the settings in one config file for easy re setup and replication, add 3 tons of bells and whistles.

My clients - I work for a hotel - use pfSense without knowing it. If the Wifi was breaking every time would have know it by now. I'm using pfSense for more then a decade.
And yes, I 'test' my captive portal Wifi nearly every day.
Note : because I have a captive portal, I do not use SSID encryption. The network is "open". Not really a big deal, as every connection is TLS these days. Nobody retrieves mail any more over port 110, send mail over port 25, or looks at sites using port 80.
Captive portal authentication is done over https (TLS) of course.

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

As far as going vanilla, I have attempted that. pfSense with no packages, same issue. With pfBlockerng same issue just more frequent. Roll back to a previous version, same issue.

So, pfSense using 'any possible setup' seems no to work well.
That's close to saying : it's not pfSense, your issue.

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

With pfBlockerng same issue just more frequent

If pfBlockerng blocks sites like "http://captive.apple.com/hotspot-detect.html" then you actually might trigger a big issue.

I advise you to start using packages when everything else is fine.

NollipfSense

What seems obvious in my first reading of OP post is something going with his WIFI AP ... appears conflicting somehow. If I were he, I would let the WIFI AP do DHCP to see whether that resolves the issue.

I am a Mac person and have several Apple devices on my network and never experienced any issue. In my case though, I have a Mikrotik that does DHCP while pfSense do the DNS. My WIFI APs are Apple Extreme AC as well as an Apple Extreme N for guest.

JKnott

@wmheath586

I have a separate Unifi AP and Cisco switch connected to pfsense. I have never seen the issues you're talking about. As far as WiFi clients go, there is no way pfsense can know they're connected via WiFi, All it sees are Ethernet frames carrying IP packets. Is there anything else on your network that might affect this, such as an authentication server?

JKnott

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

Wireless Apple products are the only clients which will ask for the wifi password

This brings up a point that someone else mentioned. Apple and Android now have "privacy" MAC addresses that change. This can mess up WiFi connections. In Android, it's simple enough to disable that, but I don't know about Apple devices.

wmheath586

@jknott I understand that. Again, it is not just wireless clients. Wireless clients are the only ones which report back no internet connection.

wmheath586

@gertjan said in Consistent Loss of Internet Connectivity With Wireless Clients:

I hope so, I'm planning on buying a couple of them, the Pros or the Lites, very soon myself, as the optic fibre is in front of the door.

They are pretty good. You will need the Unifi Controller to set them up and do the initial provisioning. I would stick with a locally hosted controller if you can. You do not need much in terms of horsepower to run it.

Necaue you're using that 'controller' : when an iPhone has to 'authenticity' again to an already known network, is this not logged on the "Unifi side" ?

Something must have changed so that these devices start seeing a previous known network as unknown ....

The logs on the Unifi Controller show me that the client has disconnected. No authentication errors/timeouts, no DHCP issues, nothing like that. Just that the client has disconnected. I agree something has had to have changed, it just finding a starting point to track that down.

So, pfSense using 'any possible setup' seems no to work well.

That's close to saying : it's not pfSense, your issue.

Close, but not the same. The original network setup was based on Ubiquiti's EdgeMax line of gear. Unfortunately, due to a hardware failure (and a global pandemic which devastated the industry I work in) we moved to pfSense. This was because at the time we simply could not get our hands on the hardware to replace the EdgeRouter. We needed a solution and pfSense fit the bill. Again, I really do like pfSense, and if offers a good bit more than EdgeMax (in most ways), I just cannot get it to be stable. It is fine to say "pfSense is not your issue", but you have to be willing to provide a starting point for resolution in some fashion. From my point of view, the lowest common denominator with my setup is pfSense. I have followed the configuration guides and setup guides and now, this is where I sit. I have no indication of anything else being the issue, and to me the fix directly involves pfSense. So when people keep saying its not pfSense, it is like having a car not working and the mechanic saying the engine is fine, its just the components within the engine that don't work. That is a valid explanation, it just does not give me much of a path to move to start to solve the problem.

@NollipfSense

What seems obvious in my first reading of OP post is something going with his WIFI AP ... appears conflicting somehow. If I were he, I would let the WIFI AP do DHCP to see whether that resolves the issue.

I was thinking that same thing. Maybe some how things are not releasing or renewing correctly, or there is a stuck setting somewhere. I have tried setting static IP for for the APs (this was the original configuration of the network), I have set some APs to DHCP, and some to DHCP with MAC reservation. All APs and connected device request and receive the correct settings, but the issue still happens. Las Saturday I completely shutdown the network and all devices and rebooted. That fixed things from Saturday to about 12pm Monday afternoon. That was the longest things have been stable.

At this point, to me, there seems to be some conflict between Unifi APs, and pfSense. Where that conflict lies is not known to me. My bets would be on something to do with DNS resolution. I have been looking around on line and in the community for a while now and I do not seem to be the only one with issues between UAPs and pfSense. Maybe there is something funky with the way the APs handle traffic, maybe there is something funky with the way pfSense handles DNS. Maybe it is a combination, or something entirely unrelated. From my seat, if I slap the EdgeMax back in and remove pfSense, everything works without issue. Therefore, to me, the issue lies within pfSense. I understand network infrastructure and how networks work and why people are biting at the wifi issue. Yes it is a problem, but it is a symptom of a larger issue I cannot track down. I truly do not mean to come across as rude or ungrateful for the suggestions, I am just at a point of frustration which there seems to be no solution. If posting my config is helpful, I am more than happy to do so.

JKnott

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

At this point, to me, there seems to be some conflict between Unifi APs, and pfSense. Where that conflict lies is not known to me.

There is none. On the pfsense side of the AP, there is no difference whatsoever between a WiFi client and a wired client. Any WiFi issues are strictly between the client and AP, unless you also have an authentication server that you haven't mentioned.

bingo600

@wmheath586

I have 4 sites w. AP AC Pro's on pfSense 2.4.5-p1 (still)
And have no issues at all ,but mostly Windows Clients (Win10)

I had issues (short stalls) - When registering DHCP in DNS, and have as many others had to disable that feature in unbound.

I'm currently having 2 SSID's (Vlans) active on 3 sites , and 4 SSID's on one.
No issues at all.

Else my AP's are Rock-Solid. w pfSense.
I'm using a Debian10 VM , for my unifi controller.

I'm not using pfBlocker though.

Edit:
There was somthing w. a DHCP snooping feature on the AP settings , that could cause an issue. I can't remember what , and don't have access to them right now. But that would just affect wireless.

/Bingo

johnpoz

@wmheath586 said in Consistent Loss of Internet Connectivity With Wireless Clients:

These are Unifi AC Pros, UAP AC Lites, and UAP AR LR units

I have 1 each of these on my home network - and have had zero issues with them.. I am running current beta firmware 5.60.3

Have all kinds of different devices. iphones, alexa, roku, harmony hub, smart bulbs and switches, etc..

And have not seen any sort of issues at all.. Can tell you for damn sure it has zero to do with pfsense if your not also seeing issues with wired devices.

wmheath586

@bingo600 I have disabled these settings and there is no change. What AP firmware and controller version are you using? It is not just wifi clients, they just seem to have issues more frequently. At least, wifi clients show the user there is an issue more readily than wired.

@jknott There is no authentication server. It is not only wifi clients which have the issue.

bingo600

@wmheath586

No access to the system right now , but the latest debian package from the unifi repos. And a fw from about a month ago - Not allowing AP FW upfrade automatically.

ThatGuy

Hey wmheah586, really sorry to hear about your struggles with pfSense and your APs. I’m gonna start with the not-so-good news first and then move on to the good news.

Not-So-Good News

Dude, you’re all over the place with problems.

Wired clients can’t access local DNS and sometimes the Internet. Ah yeah, that’s a big problem. You have a DNS issue which has nothing to do with Wireless Clients disconnecting like those Apple devices.
Wireless Clients can’t connect or have to re-authenticate to the APs or if they are connected, no Internet. Are they getting IP addresses when they can't get to the Internet? Can they ping IP addresses like pfSense or Google (8.8.8.8)?

I honestly don’t think these two problems are related. It sounds like you have two completely different problems. As lots of others have stated, pfSense and UniFi are rock solid. I’ve lost count on how many of those installs I’ve done ranging from 5 Wireless Devices connected to 500. If everything is configured correctly, it works and is rock solid.

So I’m inclined to think you have configured something incorrectly in pfSense (or UniFi). You may also have hardware out there still holding onto something from the EdgeRouter.

What managed switches are you using?
Are you using Managed switches? They could have something to do with your issue.
Do you have a rouge DHCP Server out there?
How is your WAN configured with your ISPs gateway (modem). Is it bridged or set to pass-thru?
Do you have "fast roaming" turned on in the UniFi Controller?
Is this a Windows AD environment?

Good News

This can be fixed. The hardest part is isolating where the problems really are. How do you do that? Use the KISS method (Keep it Simple Stupid). Here is what I would start with:

Get rid of pfBlockerNG. Heck, I’d even do a fresh install of pfSense and configure everything from scratch. Or, edit a backup XML config file and get rid of anything pertaining to the pfBlockerNG package. However, since you’re new to pfSense I’d start from a fresh install and do everything from scratch. pfBlockerNG when uninstalled from pfSense can still leave things behind that you can’t see in the GUI. A lot of pfSense packages do this, not just pfBlockerNG. Start FRESH! pfBlockerNG is heavily integrated to DNS and I sense this could be your DNS issue.
Get DNS working FIRST! I know the wireless issue is pressing but if you’ve got DNS problems things are only going to get worse from there.
Set up another UniFi controller from Scratch, hard reset one or a few of the APs and adopt them to that controller. You can have two controllers running in the same environment. Resetting APs and starting from scratch would be one way to isolate things.
If there is any way you can put in a small unmanaged switch from pfSense’s LAN port before going into any other switch that would be great. You could then hook up devices to that unmanaged switch like a couple wired computers and those APs you reset and see if DNS is flowing correctly between those devices. (Obviously you'll need to power those WAPs with a POE injector.) Some may say the unmanaged switch won’t pass VLANs. Some unmanaged switches like TP-Link unmanaged switches WILL pass the VLANs. Others won’t. I typically stick with TP-Link switches because I can use VLANs especially with UniFi APs. Devices on the VLANs will be able to communicate with devices on the LAN but you can traffic shape if needed.

Hang in there wmheah586. Yes, this job is hard. But it can be lots of fun too….especially when you fix a problem like you’re having.

JKnott

@thatguy said in Consistent Loss of Internet Connectivity With Wireless Clients:

Some may say the unmanaged switch won’t pass VLANs. Some unmanaged switches like TP-Link unmanaged switches WILL pass the VLANs. Others won’t. I typically stick with TP-Link switches because I can use VLANs especially with UniFi APs.

There is absolutely no reason why an unmanaged switch won't pass VLAN frames. The only significant difference between a VLAN frame and any other is the contents of the Ethertype/Length field. There are many types of Ethernet frames and any switch that can't pass all of them is defective.

Also, some TP-Link managed switches don't handle VLANs properly.

wmheath586

@thatguy

Thanks for reaching out. Here are some answers. Hopefully they shine a bit more light on the issue.

Here we go:

Wired clients can’t access local DNS and sometimes the Internet. Ah yeah, that’s a big problem. You have a DNS issue which has nothing to do with Wireless Clients disconnecting like those Apple devices.

The DNS issue with wired clients and the wireless client disconnections happen at the same time. It is not a one or the other situation. However sometimes it is a specific group of machines. By that I mean all apple, or all windows, or apple and windows but not android. It is a mixed bag, and sometimes it is just everything in general. When wireless clients are dropped, wired clients (some) loose DNS functionality for local hostname resolution and some websites. I really do understand how APs, routers, switches and networking works, I promise. I am simply stating (or trying to) that when X fails in pfSense, then X fails with APs. What that X is, is what I am trying to sort out. I never only lose wireless clients or only lose wired clients.

Wireless Clients can’t connect or have to re-authenticate to the APs or if they are connected, no Internet. Are they getting IP addresses when they can't get to the Internet? Can they ping IP addresses like pfSense or Google (8.8.8.8)?

Wireless clients do not connect to wifi at all when this happens. Apple devices (other than iPhones) will just disconnect and not re-attempt to connect unless manually told to do so. Windows clients will continue to re-attempt to connect until it displays "Cannot Connect to This Network." Android clients get stuck in a loop of "Connecting....", "Wifi Networks are Available", "Connecting..." and repeat. iPhones (newer models) will show the network as available, but will prompt the user to enter the wifi password, older models will just disconnect and not reconnect. They do not receive IP addresses. When finally connected they receive the correct IP, DNS (192.168.x.1, pfSense), and gateway. At the same time, wired clients will do one of two things.

Continue to work normally as if nothing is wrong, including local DNS resolution.
Present users with "No Internet Connection" warnings, local DNS resolution fails, cannot ping external IP addressed, but can ping internal.

In the second above case, local DNS resolution fails and only some websites function. An example would be attempting to reach pfsense.atlas-nc.lan fails, but 192.168.1.1 works. From inside pfSense name resolution works using the DNS Lookup tool, and the pfSense machine can ping external WAN addresses. Unifi, UNMS (UISP now I think), Unifi NVR, 3CX, and VPN connections (two of which reside in separate VLANs, one of which is a 1:1 NAT situation) connections are all functional when this happens, and do not drop internet connectivity. VoIP phones continue to function as normal and do not drop calls. VPN connections can continue to use local hostnames when this is all happening. I have attempted to capture traffic from wireless clients during this reconnection phase, but nothing seems to be transmitted. I may have a setting in wireshark wrong, honestly. I do see traffic when things come back up.

What managed switches are you using?

The managed switches are UI EdgeMax 24 Port non-Poe. They are three total. One Lagg group feed SW1, then SW1 & SW2 are trunked together via another lagg interface. Switch 3 is fed from S2 with an untagged trunk line. They are EdgeMax and not Unifi, so they do not interact with the Unifi Controller at all. They are connected to UNMS for monitoring. I dont know if it is important but I do NOT get disconnection notices or warnings from UNMS or any of the connected gear (including remote devices and off-site locations), it keeps trucking.

Are you using Managed switches? They could have something to do with your issue.

Yes. There does not seem to be any issues as far as I can see with the switches, VLANs, and lagg groups. They are all functioning as expected.

Do you have a rouge DHCP Server out there?

I have looked for this, and if I do, I cannot find it. I do not see any indication of another DHCP server on the network. All DHCP provided IP addresses are correct. That includes all DHCP pools and MAC reservation pools.

How is your WAN configured with your ISPs gateway (modem). Is it bridged or set to pass-thru?

The ISP modem is in bridge mode I believe. The first hop from the modem is the pfSense box. pfSense handles the static WAN gateway and static/alias WAN IP addresses.

Do you have "fast roaming" turned on in the UniFi Controller?

Fast romaming is disabled. So is the "High Performance Devices" option, PMF, and Wifi AI.

Is this a Windows AD environment?

It is not.

Get rid of pfBlockerNG. Heck, I’d even do a fresh install of pfSense and configure everything from scratch. Or, edit a backup XML config file and get rid of anything pertaining to the pfBlockerNG package. However, since you’re new to pfSense I’d start from a fresh install and do everything from scratch. pfBlockerNG when uninstalled from pfSense can still leave things behind that you can’t see in the GUI. A lot of pfSense packages do this, not just pfBlockerNG. Start FRESH! pfBlockerNG is heavily integrated to DNS and I sense this could be your DNS issue.

I have tried this. So initially everything is working fine without any plugins (packages). However, after a few days this issue will reappear. Even without any other packages installed. With packages installed the issue does not seem to be any more or less frequent. I was running to some issues with DNSBL and unbound, but after disabling DNSBL, that went away. No the original issue, but the unbound crashing issue.

Set up another UniFi controller from Scratch, hard reset one or a few of the APs and adopt them to that controller. You can have two controllers running in the same environment. Resetting APs and starting from scratch would be one way to isolate things.

The unifi controller was moved to a dedicated machine when pfSense went in. So that configuration is new with this system. I cannot physically access some of the APs without a lift, and most definitely not during the business week. If it comes to it I may be able find an open weekend and give it a shot.

If there is any way you can put in a small unmanaged switch from pfSense’s LAN port before going into any other switch that would be great. You could then hook up devices to that unmanaged switch like a couple wired computers and those APs you reset and see if DNS is flowing correctly between those devices. (Obviously you'll need to power those WAPs with a POE injector.) Some may say the unmanaged switch won’t pass VLANs. Some unmanaged switches like TP-Link unmanaged switches WILL pass the VLANs. Others won’t. I typically stick with TP-Link switches because I can use VLANs especially with UniFi APs. Devices on the VLANs will be able to communicate with devices on the LAN but you can traffic shape if needed.

I do not have another switch to test this with, but I will see what I can find. I do know I have an untagged trunk line running to a 3rd EdgeMax switch in the back office. Clients connected directly to that switch do not have issues when the rest of the network does. There is an AP passing VLAN assigned networks connected directly to that switch, and those wireless clients DO suffer the same issues as the rest of the network.