Logging Internet Dropouts
-
Hey Guys,
Backstory: I have a quick question for you. I woke up at 6:45am this morning to no internet. I have Verizon FiOS with their router and ONT in my basement. Of course the first thing I tried was unplugging it and plugging it back in. That didn't work so I called. They ran me though about 45 mins of troubleshooting and still couldn't get it to work. So they said they would send out a tech between 1-3…Around noon the internet randomly started working on and off. 2pm comes and I get a text that they fixed the outage and were taking the liberty of canceling my appointment without asking me if I'd still like a tech and to pretty much kick rocks....Internet was still dropping around 3:30-4...They said their stuff is showing it's good when they look at it but they are unable to ping my Verizon router intermittently...
TLDR: I want to see how I can bring up a log of my pfsense firewall that shows the drops in connection so I can show the tech when he gets here tomorrow. I work in telecom and know the whole "It was working when I got there, I'm outta here" BS. I keep getting notifications when I login to pfsense that say:
"There were error(s) loading the rules: /tmp/rules.debug:136: syntax error - The line in question reads [136]: pass out route-to ( re3 192.168.1.1 ) from 192.168.1.254 to !/ tracker 1000002761 keep state allow-opts label "let out anything from firewall host itself""
Is there anyway to filter these dropoffs out in the logs so that I can show them?Thanks, ToXZiN 1
-
One thing I've done is write a shell script that pinged my ISPs gateway at regular intervals and logged the failures.
-
I log this sort of things a few ways. There are plenty of outside services that will do it for FREE.. Status Cake for example does a nice job.. And you get emails when stuff is down, etc. I have it monitoring my VPSes around the globe as well, etc.
But for example you want to know exactly when and how long your internet was down.. Look at the pretty breakdown you can get.. Plus I get a daily update email from them every morning all my stuff have it monitoring.. All free ;)
I also run domotz on my network (not free) as a VM that I get push notifications to my phone when it can not phone home (internet down) or any of my other systems that I have flagged as important go offline.. Or for example when either of my grown sons stop over to the house and their phones connect to the network.. This lets me know when my one son is there in the afternoon to pick up my grandson, etc.
While pfsense can do a shitton of stuff - it is not the get all end all one stop box for everything.. While if outage was recent you could look at your quality monitor, do you send the logs to syslog where you could parse it and even run a report on specific log items, etc. When it comes to actually "monitoring" the network there are other tools.
-
Awesome. Thanks for the replies guys!
-
The pfSense quality graph is also quite valuable for detection in the outbound direction.
If you have gateway monitoring on WAN (the default setting), the system is automatically keeping track of two pings per second in Status > Monitoring.
From there select settings, change the left axis to Quality / WANGW (or the local equivalent).
A good place to start with Options: 8 hours, Resolution: 1 minute.
Another place to check is in Status > System Logs, Gateways. Any events there with "Alarm" in them are times when the ping monitor had excessive loss or latency.
A failure will look something like this: Jan 7 15:05:31 dpinger WANGW 8.8.8.8: Alarm latency 0us stddev 0us loss 100%
Lines like this are just the dpinger process starting or reloading and are normal:
dpinger send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.4.4 bind_addr 198.51.0.16 identifier "DSLGW "
Sometimes it is beneficial to change your monitoring address to something further out. In that example you can see that I am monitoring a google DNS server there. In general, monitoring the ISP gateway is fine if it reliably responds to pings. Changes to the monitor IP address can be made in System > Routing and editing the appropriate gateway.
-
Are you running suricata?
-
I don't believe I am. I originally built the pfsense rig for load balancing. Since then gigabit fiber came thru and I haven't needed to load balance anything. The tech came out and changed the ONT and since then I haven't had a drop off.
