Complete loss of network; where to find info on what happened?
-
Evening-
I have 3 netgate hardware appliances at work and run the software at home. This evening the entire network went belly up, complete with all ports of the switch spamming and no connectivity working. (edit: The software pfsense went down; not all the hardware)
Power cycled the pfsense-running computer didn't help. I had to dismantle the dumb switch and only then did the spamming stop. There are 4 APs (2x winstar and 2x linksys). The PFsense box handles everything.
Searching lastlog showed nothing, and nothing is present except the reboot. Any ideas as to what I could do to try and reconstruct what caused this blowup? This is very unlike any other previous run of pfsense.
Thanks much,
-
If your switch was spamming and it didn't stop until you removed your unmanaged switch, it sounds like there was a loop in your network somewhere.
-
@purduephotog said in Complete loss of network; where to find info on what happened?:
complete with all ports of the switch spamming
You mean they were all just blinking like crazy? Sure sounds like a loop.. Another advantage of smart switch that can run stp ;)
-
@johnpoz said in Complete loss of network; where to find info on what happened?:
@purduephotog said in Complete loss of network; where to find info on what happened?:
complete with all ports of the switch spamming
You mean they were all just blinking like crazy? Sure sounds like a loop.. Another advantage of smart switch that can run stp ;)
Yeah. I've got one slated to go in and plugged in next to it awaiting cutover.
Honestly tho... What could dynamically trigger a loop? Or do you think it just happened as opposed to a new piece of hardware
-
Is this a work network?? Have had users plug say an IP phone both plugs into the same switch..
Users are stupid ;) That is how loops happen on a work network... On a home network did someone plug the same switch into itself?
-
@purduephotog said in Complete loss of network; where to find info on what happened?:
Searching lastlog showed nothing, and nothing is present except the reboot. Any ideas as to what I could do to try and reconstruct what caused this blowup? This is very unlike any other previous run of pfsense.
Thanks much,
Yeah man , I agree with you. Even if I don't understand very clear you mean.
Anyway I know only one thing to do when strangers blown happen like this.
So I mean, for "logging network behaviors" you need to fire up , first of all, a network monitoring tool like Wireshark
And you need at least one managed switch with port mirroring feature, (or older HUB, do the same but I wouldn't write about difference between switch and hub, if you are old aged enough, you well know this)
Port mirroring allow you to sniffing traffic between one of your upstream or downstream ethernet trunk.
Basically if you set (indeed on a dumb managed switch you hold) mirroring at LAN port number of your pfsense box LAN iterface, you can see very clear informations on simple gui (Wireshark run as windows or linux host, connected to mirroring port reserved for monitoring purpose) Everything is printed at your screen in real time, trough LAN net flows while Wireshark app can log it and store on file for "future" reference.So hurry, go to learn wireshark's basics and make your way every it is!
Just mind to "start" sniffing task before of crash :D lool
-
@johnpoz no it's a home network. No one in the family can even walk to the switch. But now I'm getting an idea how something might have happened to the mesh aps when or if the network upstream crashed. I wonder if the other apps tried to take over.
-
@purduephotog said in Complete loss of network; where to find info on what happened?:
I'm getting an idea how something might have happened to the mesh aps when or if the network upstream crashed. I wonder if the other apps tried to take over.
Also side note for Wireshark can do sniffing with wireless lan, but you need to set it (your wlan net) open, otherwise you grab only crypted traffic) (Easy way).
Bye.
P.S.
Oh, for all pfSense history logs, more advanced users can take useful infos by reading directly some logfiles into /var/log subfolder.
Here is the pfSense place where is storing past log hystory. Sometimes will be more quick to avoid messing with regular log file viewer
So you can recall from pfSense web ui logfile you want look . at any time by browsing /var/log folder with Diagnostic / Edit File tool like this example below:
Or by CLI with Less command, if you like Ssh console managing:
[2.4.4-RELEASE][root@pfSense.babiz]/var/log: less routing.log
In conclusion, additional useful feature is made by ** Packet Capture" ** built into pfSense, allowing you to record a kind of *.cap file, get sniffing from pfSense interface you wand and read it later with wireshark "open .cap files" option for data analytics and filtering jobs, is more convenient and not require any mirroring stuff.
Just configure it as your need and start capture will write this special file
/root/packetcapture.capIt's readable with any .cap file reader capable , Wireshark or just pfSense built-in reader.
-
@babiz Thank you kindly. The /var/log was empty except for the reboot; there were no past logs. That bugged me and I wondered if they were somewhere else I hadn't thought to look for.
I have wireshark, but honestly hadn't thought to try it while everything was flipping out.As far as I can tell it looks as if the WAPs attempted to take over when something happened to PFSense. There was a cable modem reboot in there, too, which once triggered some strange IP issues. But without the log (and I was desperately trying to get everything back up) I've got nothing to go back and look at.
Thank you for taking the time to read and respond to my post, and point out ways I can in the future better analyze issues. That's a very welcoming approach you have there, and I'm quite thankful for it.
~J
