Consistent Loss of Internet Connectivity With Wireless Clients

bingo600 · Apr 27, 2021, 4:54 PM

No access to the system right now , but the latest debian package from the unifi repos. And a fw from about a month ago - Not allowing AP FW upfrade automatically.

ThatGuy · Apr 27, 2021, 6:19 PM

Hey wmheah586, really sorry to hear about your struggles with pfSense and your APs. I’m gonna start with the not-so-good news first and then move on to the good news.

Not-So-Good News

Dude, you’re all over the place with problems.

Wired clients can’t access local DNS and sometimes the Internet. Ah yeah, that’s a big problem. You have a DNS issue which has nothing to do with Wireless Clients disconnecting like those Apple devices.
Wireless Clients can’t connect or have to re-authenticate to the APs or if they are connected, no Internet. Are they getting IP addresses when they can't get to the Internet? Can they ping IP addresses like pfSense or Google (8.8.8.8)?

I honestly don’t think these two problems are related. It sounds like you have two completely different problems. As lots of others have stated, pfSense and UniFi are rock solid. I’ve lost count on how many of those installs I’ve done ranging from 5 Wireless Devices connected to 500. If everything is configured correctly, it works and is rock solid.

So I’m inclined to think you have configured something incorrectly in pfSense (or UniFi). You may also have hardware out there still holding onto something from the EdgeRouter.

What managed switches are you using?
Are you using Managed switches? They could have something to do with your issue.
Do you have a rouge DHCP Server out there?
How is your WAN configured with your ISPs gateway (modem). Is it bridged or set to pass-thru?
Do you have "fast roaming" turned on in the UniFi Controller?
Is this a Windows AD environment?

Good News

This can be fixed. The hardest part is isolating where the problems really are. How do you do that? Use the KISS method (Keep it Simple Stupid). Here is what I would start with:

Get rid of pfBlockerNG. Heck, I’d even do a fresh install of pfSense and configure everything from scratch. Or, edit a backup XML config file and get rid of anything pertaining to the pfBlockerNG package. However, since you’re new to pfSense I’d start from a fresh install and do everything from scratch. pfBlockerNG when uninstalled from pfSense can still leave things behind that you can’t see in the GUI. A lot of pfSense packages do this, not just pfBlockerNG. Start FRESH! pfBlockerNG is heavily integrated to DNS and I sense this could be your DNS issue.
Get DNS working FIRST! I know the wireless issue is pressing but if you’ve got DNS problems things are only going to get worse from there.
Set up another UniFi controller from Scratch, hard reset one or a few of the APs and adopt them to that controller. You can have two controllers running in the same environment. Resetting APs and starting from scratch would be one way to isolate things.
If there is any way you can put in a small unmanaged switch from pfSense’s LAN port before going into any other switch that would be great. You could then hook up devices to that unmanaged switch like a couple wired computers and those APs you reset and see if DNS is flowing correctly between those devices. (Obviously you'll need to power those WAPs with a POE injector.) Some may say the unmanaged switch won’t pass VLANs. Some unmanaged switches like TP-Link unmanaged switches WILL pass the VLANs. Others won’t. I typically stick with TP-Link switches because I can use VLANs especially with UniFi APs. Devices on the VLANs will be able to communicate with devices on the LAN but you can traffic shape if needed.

Hang in there wmheah586. Yes, this job is hard. But it can be lots of fun too….especially when you fix a problem like you’re having.

JKnott · Apr 27, 2021, 7:01 PM

@thatguy said in Consistent Loss of Internet Connectivity With Wireless Clients:

Some may say the unmanaged switch won’t pass VLANs. Some unmanaged switches like TP-Link unmanaged switches WILL pass the VLANs. Others won’t. I typically stick with TP-Link switches because I can use VLANs especially with UniFi APs.

There is absolutely no reason why an unmanaged switch won't pass VLAN frames. The only significant difference between a VLAN frame and any other is the contents of the Ethertype/Length field. There are many types of Ethernet frames and any switch that can't pass all of them is defective.

Also, some TP-Link managed switches don't handle VLANs properly.

wmheath586 · Apr 27, 2021, 7:19 PM

@thatguy

Thanks for reaching out. Here are some answers. Hopefully they shine a bit more light on the issue.

Here we go:

Wired clients can’t access local DNS and sometimes the Internet. Ah yeah, that’s a big problem. You have a DNS issue which has nothing to do with Wireless Clients disconnecting like those Apple devices.

The DNS issue with wired clients and the wireless client disconnections happen at the same time. It is not a one or the other situation. However sometimes it is a specific group of machines. By that I mean all apple, or all windows, or apple and windows but not android. It is a mixed bag, and sometimes it is just everything in general. When wireless clients are dropped, wired clients (some) loose DNS functionality for local hostname resolution and some websites. I really do understand how APs, routers, switches and networking works, I promise. I am simply stating (or trying to) that when X fails in pfSense, then X fails with APs. What that X is, is what I am trying to sort out. I never only lose wireless clients or only lose wired clients.

Wireless Clients can’t connect or have to re-authenticate to the APs or if they are connected, no Internet. Are they getting IP addresses when they can't get to the Internet? Can they ping IP addresses like pfSense or Google (8.8.8.8)?

Wireless clients do not connect to wifi at all when this happens. Apple devices (other than iPhones) will just disconnect and not re-attempt to connect unless manually told to do so. Windows clients will continue to re-attempt to connect until it displays "Cannot Connect to This Network." Android clients get stuck in a loop of "Connecting....", "Wifi Networks are Available", "Connecting..." and repeat. iPhones (newer models) will show the network as available, but will prompt the user to enter the wifi password, older models will just disconnect and not reconnect. They do not receive IP addresses. When finally connected they receive the correct IP, DNS (192.168.x.1, pfSense), and gateway. At the same time, wired clients will do one of two things.

Continue to work normally as if nothing is wrong, including local DNS resolution.
Present users with "No Internet Connection" warnings, local DNS resolution fails, cannot ping external IP addressed, but can ping internal.

In the second above case, local DNS resolution fails and only some websites function. An example would be attempting to reach pfsense.atlas-nc.lan fails, but 192.168.1.1 works. From inside pfSense name resolution works using the DNS Lookup tool, and the pfSense machine can ping external WAN addresses. Unifi, UNMS (UISP now I think), Unifi NVR, 3CX, and VPN connections (two of which reside in separate VLANs, one of which is a 1:1 NAT situation) connections are all functional when this happens, and do not drop internet connectivity. VoIP phones continue to function as normal and do not drop calls. VPN connections can continue to use local hostnames when this is all happening. I have attempted to capture traffic from wireless clients during this reconnection phase, but nothing seems to be transmitted. I may have a setting in wireshark wrong, honestly. I do see traffic when things come back up.

What managed switches are you using?

The managed switches are UI EdgeMax 24 Port non-Poe. They are three total. One Lagg group feed SW1, then SW1 & SW2 are trunked together via another lagg interface. Switch 3 is fed from S2 with an untagged trunk line. They are EdgeMax and not Unifi, so they do not interact with the Unifi Controller at all. They are connected to UNMS for monitoring. I dont know if it is important but I do NOT get disconnection notices or warnings from UNMS or any of the connected gear (including remote devices and off-site locations), it keeps trucking.

Are you using Managed switches? They could have something to do with your issue.

Yes. There does not seem to be any issues as far as I can see with the switches, VLANs, and lagg groups. They are all functioning as expected.

Do you have a rouge DHCP Server out there?

I have looked for this, and if I do, I cannot find it. I do not see any indication of another DHCP server on the network. All DHCP provided IP addresses are correct. That includes all DHCP pools and MAC reservation pools.

How is your WAN configured with your ISPs gateway (modem). Is it bridged or set to pass-thru?

The ISP modem is in bridge mode I believe. The first hop from the modem is the pfSense box. pfSense handles the static WAN gateway and static/alias WAN IP addresses.

Do you have "fast roaming" turned on in the UniFi Controller?

Fast romaming is disabled. So is the "High Performance Devices" option, PMF, and Wifi AI.

Is this a Windows AD environment?

It is not.

Get rid of pfBlockerNG. Heck, I’d even do a fresh install of pfSense and configure everything from scratch. Or, edit a backup XML config file and get rid of anything pertaining to the pfBlockerNG package. However, since you’re new to pfSense I’d start from a fresh install and do everything from scratch. pfBlockerNG when uninstalled from pfSense can still leave things behind that you can’t see in the GUI. A lot of pfSense packages do this, not just pfBlockerNG. Start FRESH! pfBlockerNG is heavily integrated to DNS and I sense this could be your DNS issue.

I have tried this. So initially everything is working fine without any plugins (packages). However, after a few days this issue will reappear. Even without any other packages installed. With packages installed the issue does not seem to be any more or less frequent. I was running to some issues with DNSBL and unbound, but after disabling DNSBL, that went away. No the original issue, but the unbound crashing issue.

Set up another UniFi controller from Scratch, hard reset one or a few of the APs and adopt them to that controller. You can have two controllers running in the same environment. Resetting APs and starting from scratch would be one way to isolate things.

The unifi controller was moved to a dedicated machine when pfSense went in. So that configuration is new with this system. I cannot physically access some of the APs without a lift, and most definitely not during the business week. If it comes to it I may be able find an open weekend and give it a shot.

If there is any way you can put in a small unmanaged switch from pfSense’s LAN port before going into any other switch that would be great. You could then hook up devices to that unmanaged switch like a couple wired computers and those APs you reset and see if DNS is flowing correctly between those devices. (Obviously you'll need to power those WAPs with a POE injector.) Some may say the unmanaged switch won’t pass VLANs. Some unmanaged switches like TP-Link unmanaged switches WILL pass the VLANs. Others won’t. I typically stick with TP-Link switches because I can use VLANs especially with UniFi APs. Devices on the VLANs will be able to communicate with devices on the LAN but you can traffic shape if needed.

I do not have another switch to test this with, but I will see what I can find. I do know I have an untagged trunk line running to a 3rd EdgeMax switch in the back office. Clients connected directly to that switch do not have issues when the rest of the network does. There is an AP passing VLAN assigned networks connected directly to that switch, and those wireless clients DO suffer the same issues as the rest of the network.

ThatGuy · Apr 27, 2021, 8:23 PM

@wmheath586,

I'd say it's probably one or more of your managed switches. I'd back up the configs, reset them, and then set them back up from scratch sans config. You could also have issues with your DNS Resolver in pfSense. I'd like to see a screenshot of your DNS Resolver settings. If you had to restore the config, you could but of course be stuck with this same issue. For some crazy reason have you checked to make sure in your UniFi controller you don't have DHCP enabled. I've never tried to turn it on because I don't use the USGs.

JKnott, I can't remember what make it is but there is an unmanaged switch that doesn't do VLANs. I think it's either Netgear or D-Link. Haven't used them in years but I remember one of those not working with VLANs. I just stuck with the TP-Link ones. Yeah, their managed switches aren't too hot. Not enough horsepower. If I have to use a managed or POE switch, spend the dough and go with Ubiquiti.

JKnott · Apr 27, 2021, 8:26 PM

@thatguy

All an unmanaged switch can do with VLAN frames is pass them. It takes a managed switch to "do" VLANs, that is assign ports to VLANs etc.. The TP-Link problem, which also affects some APs is multicasts can leak from the main LAN to a VLAN.

Gertjan · Apr 29, 2021, 6:38 AM

@jknott said in Consistent Loss of Internet Connectivity With Wireless Clients:

In Android, it's simple enough to disable that, but I don't know about Apple devices.

Consider it also simple enough for Apple devices :

login-to-view

But ... switching this option on, on a new new, will create a new profile ones, the password has to be entered, and then that 'random' MAC will get used for that SSID.
Until you 'forget' the network, of get back to 'no private random MAC' profile.
So, both options work.

@jknott said in Consistent Loss of Internet Connectivity With Wireless Clients:

Apple and Android now have "privacy" MAC addresses that change

Because this works well for one, the other just 'copied' the functionality. That's how things are done these days ^^

wmheath586 · May 7, 2021, 6:13 PM

@thatguy Sorry for taking so long to reply. My job requires me to be away from the office for a while at times, so I am just now able to get you the config settings. They are posted below.

login-to-view login-to-view login-to-view

ThatGuy · May 7, 2021, 6:44 PM

@wmheath586 ,

Make a backup config before you make these changes just in case.

In your DNS Resolver settings try making these changes.

Network Interfaces: ALL
Outgoing Network Interfaces: ALL
DNS Query Forwarding-->Enable Forwarding Mode: CHECKED
DHCP Registration-->Register DHCP leases in the DNS Resolver: CHECKED
Static DHCP-->Register DHCP static mappings in the DNS Resolver: CHECKED (I don't know if you have any Static Mappings to Clients but it shouldn't hurt to check it even if you don't have static mappings to printers, servers, etc).

Keep in mind we're only making DNS changes and that wouldn't explain much to me why some of your client devices aren't pulling IP addresses. More likely that would be a DHCP/routing issue. But give these DNS Resolver changes a try and see what happens.

wmheath586 · May 7, 2021, 7:20 PM

@thatguy The settings suggested have been tried. They were modified about a month ago. The "DNS Query Forwarding", "DHCP Registration", and "Static DHCP" options were unchecked due to an issue with Unbound failing and having to be restarted.

Update: The original issue has become less frequent. Now it only seems to happen between the 16:00 and 18:00 hour block, and after about 15 minuets the clients are able to reconnect on their own. Oddly enough I do not see anything going on with pfSense or Unifi at that time.

On the other hand, we are moving back to the EdgeRouter. Management and the person that signs my pay check want the original equipment and stability we had before moving to pfSense. The only reason we were trying to stick it out was for the VPN server included, but we found a better option with the VPN appliance from OpenVPN. I appreciate the help and suggestions everyone has given, and maybe we will give it another shot in the future. Until then, thanks again for the help!

SteveITS · May 19, 2021, 2:28 PM

@jknott said in Consistent Loss of Internet Connectivity With Wireless Clients:

"privacy" MAC addresses that change. This can mess up WiFi connections. In Android, it's simple enough to disable that, but I don't know about Apple devices.

For reference, the Private Address setting is via: Settings, Wi-Fi, tap the active Wi-Fi connection, tap the (i) info icon, look for Private Address partway down that page and turn it on or off.

Also FYI, I've had issues with my iPhone disconnecting at home since installing 14.5. Suspect it's related to the eero and roaming. Others have it also, per Reddit. Had no issue with any prior iOS or other devices already on 14.5. Workaround is to toggle the Private Address setting (on or off) which I believe just changes away from the apparently blocked MAC.

EDIT: turning off WPA3 on the eero resolved it, so despite WPA3 working for six months apparently it has a problem with iOS 14.5.

AKEGEC · May 7, 2021, 11:19 PM

@wmheath586 I know that you are very frustration about this matter, but please do know that pfsense is base on freeBSD. This means there are limitation, e.g. some type of WiFi card/device doesn't work properly. That being said I am sure in the near future devs will fix this issue...but then again it's freeBSD after all. Anyway I can't wait for pfsense version 2.6.

papdee · May 12, 2021, 8:33 AM

@wmheath586 assuming everything else configured correctly I would point to an IP address conflict or corrupted ARP table, possibly one or more of your devices is obtaining an IP address that is the same as your pfSense box or the ARP table being corrupted by one of your devices.

you can go to shell and run arp -a to inspect your ARP tables.

For DHCP leases go into the GUI under the Status menu.

papdee · May 14, 2021, 12:38 PM

@wmheath586 you might also want to drill down further to the MAC address tables in your router. If you are using a managed switch you should be able to telnet into your router and inspect the MAC address table. This would be relevant if you are running multiple VMs and have left the MAC addresses at their defaults.