Network becomes totally unusable when my parents connect
-
My parents are visiting an the whole time they have been here connected to my network it's been unusable (websites don't load). My dad says that anytime he loads any page it fails, then the second time it works. They each have a google pixel (9 pro an 8), 1 Lenovo Yoga tablet and 1 Apple iPad. When they are connected there seems to be what my layman understanding is a NAT or DHCP issue? many of my smart devices are bounced off and on the network and web pages just don't load. I turned off "randomize" mac address on the tablet and maybe that helped? But not all the way. For now they are off, but I really need to figure this out.
Here's a log: https://pastebin.com/u4EBbehe
If anyone has any idea what's going on that would be very very helpful...
-
Hmm, there are a few things there.
Do you know what these are:
arp: 10.10.10.6 moved from 00:25:90:ed:a6:f4 to f4:52:14:80:ce:30 on bce1You have a lot of gateway alarm events due to latency. If you only have one WAN I would disable the monitoring action (not monitoring all together) for the gateway to prevent unecessary reloads. I would also set some external monitoring IP since the gateway itself there seems very variable.
There are a number of events there where one NIC actually lost link like:
Feb 28 16:16:43 kernel bce1: link state changed to DOWNI assume that is the LAN side NIC. Did something downstream get rebooted there?
-
@stephenw10 thank you @stephenw10!
I'm not 100% sure but 10.10.10.6 is my unraid box.
@stephenw10 said in Network becomes totally unusable when my parents connect:
If you only have one WAN I would disable the monitoring action (not monitoring all together) for the gateway to prevent unecessary reloads. I would also set some external monitoring IP since the gateway itself there seems very variable.
I only have one WAN, but have 2 vlans (IoT and Cams) as well as OpenVPN:
How would I disable the monitoring action? How would I set external monitoring IP?@stephenw10 said in Network becomes totally unusable when my parents connect:
I assume that is the LAN side NIC. Did something downstream get rebooted there?
bce1, as you can see above is the LAN. I think lots of things were being rebooted. I rebooted the unraid server. The Google Home Max kept losing connecting an reconnected and probably a lot of our clients as well.
I told my parents to turn off WiFi on all their 4 devices. Instantly the internet went back to being usable. We've slowly been adding them back in and it seems teh Yoga tablet is the MAIN offender. They've had wifi turned on both thier phones for a few hours now and no real issues...Though my Tidbyt (wifi display) in the kitchen is just not reconnected. but i can search webpages and they come up et al.
-
@RickyBaker said in Network becomes totally unusable when my parents connect:
How would I disable the monitoring action? How would I set external monitoring IP?
Edit the WAN gateway in System > Routing > Gateways. Both settings are there.
Also make sure the default gateway isnt set to auto there because it may try to use the ovpn gateway as default.
What is bce1 actually connected to? Only rebooting that should cause it to lose link.
-
@stephenw10 said in Network becomes totally unusable when my parents connect:
Edit the WAN gateway in System > Routing > Gateways. Both settings are there.
I found and ticked this box to disable the monitoring of the gateway (should i disable the action box as well?)
Didn't see the external monitoring option but starting googling it and wasn't really sure what external IP I could use to monitor the gateway. I'm far from a power user and don't have access to a personal external server
@stephenw10 said in Network becomes totally unusable when my parents connect:
Also make sure the default gateway isnt set to auto there because it may try to use the ovpn gateway as default.
Its set to WANGW but the Default gateway IPv6 was set to "Auto" so I changed to "none". I was previously having NAT/DNS issues that persisted until i disabled ipv6You can follow the whole saga here if you're a masochist but the last page has my eventual "solution" which was adding no-ip6 to the advancedsettings of the DNSResolver.
@stephenw10 said in Network becomes totally unusable when my parents connect:
What is bce1 actually connected to? Only rebooting that should cause it to lose link.
bce1 is a LAN port connected to the first port of a Unifi 48 port switch (a Unifi 48-500w poe switch is connected to that via SFP+. I'm sorry but I don't understand what you mean by rebooting should cause it to lose link...I DID reset the switches twice to try to solve the issue, but once was while the pfsense was halted.
-
The only reason bce1 should lose link is if the switch port goes down. So if the switch is rebooted. Or, potentially, if it drops the port to prevent a loop or packet storm etc. The switch logs may show something.
You should disable only the Gatway Monitoring Action nd leave monitoring enabled on the WAN gateway. That way you still get WAN quality data but if it triggers an alarm it doesn't restart services and VPNs etc.
-
Is one of your parents devices configured for a static IP, which potentially conflicts with one of your existing network devices and/or inside your DHCP pool?
Might want to consider putting them on an isolated guest network in case there is some malware on one of those devices. Sorry mom and dad, but you have been quarantined.
-
@stephenw10 said in Network becomes totally unusable when my parents connect:
The only reason bce1 should lose link is if the switch port goes down. So if the switch is rebooted. Or, potentially, if it drops the port to prevent a loop or packet storm etc. The switch logs may show something.
You should disable only the Gatway Monitoring Action nd leave monitoring enabled on the WAN gateway. That way you still get WAN quality data but if it triggers an alarm it doesn't restart services and VPNs etc.
got it, i dunno what a loop or packet storm is (though i can guess). I'll look into logs on unifi, thanks for the suggestion. How many drops did you see in my logs? You're pretty confident this is an issue that needs to be chased down it sounds...
Now that I enabled Gateway Monitoring and disabled Gateway Action I see the input field for Monitor IP. I assume this is what you were referring to earlier. Any suggestions on something to put in here?
@elvisimprsntr said in Network becomes totally unusable when my parents connect:
Is one of your parents devices configured for a static IP, which potentially conflicts with one of your existing network devices and/or inside your DHCP pool?
Might want to consider putting them on an isolated guest network in case there is some malware on one of those devices. Sorry mom and dad, but you have been quarantined.
Reply
great question and something I immediately checked. My dad is kind of bad about security so I'm concerned about malware
-
There were quite a few link state changes in the logs and I wouldn't expect to see any in normal running between pfSense and a switch.
Yes most managed switches have some ability to detect bad traffic (a loop, packet storm/flood) and disconnect a port to prevent it. So potentially it might be doing that but it should log it if so.
The the Monitor IP is where I would add an external IP to give better, more accurate data. Many people use 8.8.8.8 or 1.1.1.1 there since they automatically route to something local (anycast).
-
@stephenw10 said in Network becomes totally unusable when my parents connect:
Yes most managed switches have some ability to detect bad traffic (a loop, packet storm/flood) and disconnect a port to prevent it. So potentially it might be doing that but it should log it if so.
ok i'll look into pulling a log from Unifi, thanks
@stephenw10 said in Network becomes totally unusable when my parents connect:
The the Monitor IP is where I would add an external IP to give better, more accurate data. Many people use 8.8.8.8 or 1.1.1.1 there since they automatically route to something local (anycast).
do you recommend that I do that? set it to 8.8.8.8 (that's google's dns server right?)
-
@RickyBaker I'm not sure if this helps troubleshooting, but my parents report that, though the internet is "usuable" they have to load every webpage twice. Once it stalls out and says address can't be reached or what not and then on reload it loads. The offending tablet has been offline since yesterday. (and as a reminder, the minute he turned off the tablet, i was able to access webpages et al)
-
Well this jumped out at me in the critical section. There are a number of dockers running, could that be it?
edit: Looks like it's my unraid server is double connected, once through ethernet and once with a SFP cable. I'd like the speed of the sfp, sort of assumed it would be able to deal with the redundancy itself but i guess not?
-
Yup using 8.8.8.8 is fine. At least for a test. Technically Google is under no obligation to reply pings there but it has done for years.
Connecting unraid twice might create a loop there. I'm sure it could be connected using a lagg but it would require some config on the sever.
-
@stephenw10 There's def a whole Networking submenu of Unraid that seems to imply i can set it up for redundancy but I would be a liar if I implied I had any confidence I set it up right or that I'm even accurate in what I'm saying. I'll pull one of the connection in the interim
-
@RickyBaker said in Network becomes totally unusable when my parents connect:
@stephenw10 There's def a whole Networking submenu of Unraid that seems to imply i can set it up for redundancy but I would be a liar if I implied I had any confidence I set it up right or that I'm even accurate in what I'm saying. I'll pull one of the connection in the interim
Did that help? FWIW, what you're talking about there is called link aggregation and it typically does need some special configuration both on the host system (your unraid in this case) and on the switch your host system is connected to. I find that it's usually an aggravation to deal with and I don't bother unless 1gbps internal bandwidth somehow isn't enough (so in other words...never :D)
I did notice you mentioned Ubiquiti hardware and it jogged my memory on something. We use pfSense / Ubiquiti at my company and I administer all of it across multiple sites. We ended up having a problem with somewhat similar symptoms you describe when we attached some equipment to our network that uses Dante protocol for communication (if you're curious, Dante is an Audio over IP protocol; we are using it with conference room speakers, microphones, and an audio mixer). Do your parents have anything maybe at their house that could possibly use an audio or video protocol like Dante, or something similar?
We would get some odd network behavior when we connected our Dante enabled audio mixer to our network, as I said, similar to what you are describing when you parents attach to your network. We solved it by turning on IGMP Snooping in the Unifi controller / cloud key's global network settings. Fixed our issue straight away, so it could possibly be some kind of multicast traffic your parents PCs are broadcasting.
-
@bp81 said in Network becomes totally unusable when my parents connect:
Did that help?
It doesn't appear to have helped. Making my Dad turn off that offending laptop (and them eventually going back to Florida) made the internet usable but it's still pretty slow to load any websites with no issues in Wifi AP signal strength. My wife has informed me that she also witnessed it before my parents arrival so it could have been a problem that has existed for some time, that was somehow exacerbated by the arrival of my parents devices.
@bp81 said in Network becomes totally unusable when my parents connect:
Do your parents have anything maybe at their house that could possibly use an audio or video protocol like Dante, or something similar?
I really don't think so but oddly enough i'm very familiar with Audinate/Dante. I don't have any either though. I have always suspected the VSSL A.6 zoned amplifiers and their 14 static IP's are the source of many an invisible problem...
@bp81 said in Network becomes totally unusable when my parents connect:
We solved it by turning on IGMP Snooping in the Unifi controller / cloud key's global network settings. Fixed our issue straight away, so it could possibly be some kind of multicast traffic your parents PCs are broadcasting.
I've done a fair amount of Multicast tinkering and UDP Broadcast Relay to get certain services to communicate across the VLAN's, including the VSSL service. Could some of this be causing the issues? Anything I should disable or check? Any specific log I should post?
-
@RickyBaker said in Network becomes totally unusable when my parents connect:
I've done a fair amount of Multicast tinkering and UDP Broadcast Relay to get certain services to communicate across the VLAN's, including the VSSL service. Could some of this be causing the issues? Anything I should disable or check? Any specific log I should post?
Straight up, I am very new to UDP broadcast type stuff because until recently I didn't have to contend with much of it. It might be worth just turning off your UDP broadcast relay just to see if it helps, just on the idea that if you have a complex network and you're having issues, turning off features one at a time until performance improves might at least point you to the culprit.
The symptoms you describe do kind of sound like a possible network loop as well, though most switches these days run spanning tree protocol and protect themselves by disabling ports to break the loop. Might be worth checking to see if any of your network switches have spanning tree turned off and is allowing a loop.
Sorry I couldn't be more help.
-
@bp81 said in Network becomes totally unusable when my parents connect:
The symptoms you describe do kind of sound like a possible network loop as well, though most switches these days run spanning tree protocol and protect themselves by disabling ports to break the loop. Might be worth checking to see if any of your network switches have spanning tree turned off and is allowing a loop.
very helpful suggestion thanks, i'll google but any chance you know offhand where to find in ubiquiti?
-
@RickyBaker said in Network becomes totally unusable when my parents connect:
@bp81 said in Network becomes totally unusable when my parents connect:
The symptoms you describe do kind of sound like a possible network loop as well, though most switches these days run spanning tree protocol and protect themselves by disabling ports to break the loop. Might be worth checking to see if any of your network switches have spanning tree turned off and is allowing a loop.
very helpful suggestion thanks, i'll google but any chance you know offhand where to find in ubiquiti?
I’m not sitting in front of my controller right now; I know it’s in Settings screen for a particular site. I’ll see if I can look that up this weekend.
-
@RickyBaker It's under Settings - Networks.
Simplest way to find it, at least in the new UI, is to click Settings and type in the search field at the top...
