PFSense hangs when certain device connects to network

Sikh

So I'm having an interesting problem that started about 3 weeks ago. Whenever my doorbell decides to reconnect to my network, pfsense will stop responding. All traffic on the LAN just stops responding and I can't load the web UI at all.

I can access pfsense locally (monitor and mkb) but remotely (over LAN) I cant. I tried to do a packet capture but nothing was coming through. Once I block the doorbell on my wireless network (ubiquiti AP's and local controller), then my network goes back to normal in about 20 seconds. I am able to load pfsense remotely again and everything on the network starts working.

The only thing thats changed is I upgraded my pfsense version from 2.5.0 to 2.5.2 and I also upgraded my Wifi AP's firmware. I've since then rolled back the AP's Firmware to the "known good" firmware and the issue keeps happening.

I have even gone as far as getting a completely new doorbell(same model) hoping that the old one was going bad and that didnt fix the issue either.

I have not rolled back pfsense yet but im thinking there's something to this issue that im not seeing and hoping you folks can help me out.

JKnott

@sikh said in PFSense hangs when certain device connects to network:

I tried to do a packet capture but nothing was coming through.

Do you have a managed switch so that you can port mirror and run Wireshark?

Sikh

@jknott unfortunately I do not.

This is definitely one thing I thought about but I just have a dumb 24 port switch in my rack. My next upgrade is a managed switch but that’s for next year :(

JKnott

@sikh

You could buy a cheap managed switch and create a data tap.

Sikh

@jknott woah this is a great idea. Do you have a cheap managed switch you would recommend? Would this work?

stephenw10

@sikh said in PFSense hangs when certain device connects to network:

TL-SG105E

The TP-Link site shows it can do port mirroring. I've never used one myself though.

When this happens can you still connect out from pfSense at the console? The WAN still works?

Steve

JKnott

@sikh

I have one of those TP-Link switches. It works fine for this application, but doesn't handle tagged VLANs properly. So, if you're only going to use it for a data tap, go ahead. But if you're going to use it as a switch with tagged VLANs, go for another make. There are a few others in that price range and, as far as I know, only TP-Link has the VLAN issue.

JKnott

@stephenw10

The data tap is tranparent. You just insert it between the devices and connect a computer running Wireshark. I describe all this in that article.

Sikh

@stephenw10

That’s one thing I forgot to mention. When this happens, it’s only the LAN that’s choking. I can sit at the shell and do a speedtest and I can go back to the main menu and do a ping out to anything and it works fine. I can also run a trade route and no issues at all.

So during this WAN works fine but LAN is completely choking. I checked top and idle was 97% like it always is. Memory was also mostly free like it always is.

I have a feeling this either has something to do with DHCP leases and when the doorbell goes to renew it OR something with this update and some piece of traffic the doorbell is sending is choking the NIC. At first I thought it was the doorbell but after getting a brand new one and having the issue, I’m thinking it might be something in 2.5.2 but I also know rolling back means reinstalling the entire OS and there’s no other way to downgrade.

I also thought of something else I can do. I’m going to start a packet capture from the shell and then have my doorbell connect and see if I’m able to capture the traffic it sends until my LAN is choked. Hopefully I can see something before my LAN gets choked and I have to block it on my unifi controller.

mr.rosh

@sikh Apply a static ip address, via DHCP to the door bell. and increase the dhcp issue time. monitor the outcome please.

stephenw10

Hmm, this seems a lot more like something on the LAN side, guessing the doorbell, causing the APs or a switch to shut off the ports. Like a flood detection / prevention for example.

You see nothing at all coming into the pfSense LAN in a packet capture. Yet it is able to communicate over the WAN no problem at all. If it was something in pfSense it would have to be at the NIC hardware level to prevent a pcap seeing anything. And if that was the case you would probably need to at least reboot pfSense to clear it, probably actually power cycle it. And you do not.

Steve

johnpoz

And how do you have this all wired up?

Pfsense lan into this dumb 24 port switch, then your AP connected to the switch. Do you have other wired devices, have to assume so or why would you have a 24 port switch? Can they talk to each other?

example

What specifically can not talk? Can computer A talk to B, can C talk to A.. Can devices like laptops or phones talk to anything on the wire, or other wireless devices?

If it was pfsense issue - then A could talk to B for example. But A or B or C couldn't talk to pfsense but could talk to each other.

Pfsense IP is what exactly? 192.168.1.1? What is the IP of this doorbell when it comes online? You sure its just not the doorbell with same IP as pfsense lan?

Does say computer A, show the mac address of pfsense or computer B, C ?

It just really seems unlikely that 1 device could take out a whole network.. Even if was spewing nothing but garbage.. And loads of it.. It shouldn't be able to stop A from talking to B..

bmeeks

@johnpoz said in PFSense hangs when certain device connects to network:

Pfsense IP is what exactly? 192.168.1.1? What is the IP of this doorbell when it comes online? You sure its just not the doorbell with same IP as pfsense lan?

I'm with @johnpoz here. My first guess is the Ring doorbell has the same IP address as the pfSense firewall's LAN interface. So when the Ring comes online, it will "hide" the default gateway for all of your other LAN hosts and they will appear not to be working if you are using them to reach something on the Internet. That would not prevent them from talking to each other, though.

Further proof here is that you seem to be able to login to the Unifi Controller to "ban" the Ring device and disconnect it from the network. When you do that, you say things come back to normal. To me that hints strongly that the Ring is usurping someone elses IP address, and it sounds like it's the firewall's LAN IP that is getting usurped.

Are you 100% positive the Ring is set for DHCP?

johnpoz

@bmeeks said in PFSense hangs when certain device connects to network:

seem to be able to login to the Unifi Controller

But is he doing that remote from another device on the network - or is he local on the device running the controller?

Can not really tell from the info given so far.

bmeeks

Yeah, not enough information to make a more definitive hypothesis. But the only two things I can imagine that would account for the symptoms he posted are a network loop or an IP address conflict. The fact he has tried two different Ring devices with the same result would tend to rule out a defective Ring device.

johnpoz

@bmeeks Kind of also rules out a static IP set wrong? But not exactly sure how ring sets IP - might be like unifi when you set a device to a static IP, it learns that from the controller?

Would be moronic for such a device to default to .1 or .254 that would most likely conflict with router on network.

bmeeks

@johnpoz said in PFSense hangs when certain device connects to network:

@bmeeks Kind of also rules out a static IP set wrong? But not exactly sure how ring sets IP - might be like unifi when you set a device to a static IP, it learns that from the controller?

I've never configured one of those Ring devices, so not sure how it defaults out of the box. I would assume some kind of wireless SSID of its own that you connect to with your phone to initially configure it by giving it the credentials for your own wireless network.

I set up a Wyze camera a while back for someone and that's how it worked. Out of the box it had its own little wireless that you connected to (or it might have been Bluetooth), and then you configured it for your network from your phone.

coolspot

@Sikh said in PFSense hangs when certain device connects to network:

@stephenw10

That’s one thing I forgot to mention. When this happens, it’s only the LAN that’s choking. I can sit at the shell and do a speedtest and I can go back to the main menu and do a ping out to anything and it works fine. I can also run a trade route and no issues at all.

So during this WAN works fine but LAN is completely choking. I checked top and idle was 97% like it always is. Memory was also mostly free like it always is.

I have a feeling this either has something to do with DHCP leases and when the doorbell goes to renew it OR something with this update and some piece of traffic the doorbell is sending is choking the NIC. At first I thought it was the doorbell but after getting a brand new one and having the issue, I’m thinking it might be something in 2.5.2 but I also know rolling back means reinstalling the entire OS and there’s no other way to downgrade.

I also thought of something else I can do. I’m going to start a packet capture from the shell and then have my doorbell connect and see if I’m able to capture the traffic it sends until my LAN is choked. Hopefully I can see something before my LAN gets choked and I have to block it on my unifi controller.

Did you ever solve this? My LAN is also choking randomly every few hours - pfSense LAN interface becomes unresponsive. WAN works fine, no errors in my logs either.

stephenw10

@coolspot said in PFSense hangs when certain device connects to network:

pfSense LAN interface becomes unresponsive.

In what way? How are you testing it?

Are other LAN side devices able to connect to each other?

coolspot

@stephenw10 said in PFSense hangs when certain device connects to network:

@coolspot said in PFSense hangs when certain device connects to network:

pfSense LAN interface becomes unresponsive.

In what way? How are you testing it?

Are other LAN side devices able to connect to each other?

It is very odd - I've be struggling with this for the past few weeks:

LAN interface becomes unresponsive and drops packets
Console remains functional and I can ping to WAN (next time I'm going to run to the console to ping LAN as well)
I can ping other LAN devices from my PC - no dropped packets, so I have isolated the issue to pfSense
Smart Netgear Switch reports no errors
No obvious errors in logs or dmesg, closest perhaps is unbound has a HUP exit and restart
System recovers after 1-2 minutes but that is enough to interrupt streams
Ran memtest86 no issues
Swapped three different NICs and ports
Swapped CAT6 cables
Swapped Network ports and Switch
Did a reinstall of CE and still see the issue albeit much less frequently
System has Intel I219 and I225-LM NICs - it used to be virtualized and ran solid for years. My issues seem to have cropped up when I reverted to baremetal

Nothing is logged in pfSense which I find very odd.

As a side note, is there a way to get timestamps from dmesg?