Pfsense internet goes down all the time

unf0rg0tt3n

Hi guys,

First of all, excuse me; no idea where to post this.
I am running pfsense inside a VM for a while now and it was quite stable. But since a few months my internet goes down all the time.

All of the sudden, the ISP's modem goes in some sort of soft reboot with all the leds flickering, ofcourse they say, not our problem because they already replaced the modem, cables and even the cable to the junction box down the street.

I am out of ideas, I have tried hooking up directly to the modem, that works because the soft reboot won't last that long... but the VM takes a while to recover. Which logs do I need to check in order to find out?

Which more information do you guys need?

Thanks!

stephenw10

The system log. The gateway log.

If the modem is actually rebooting there's nothing pfSense can do to cause that. Nothing we can do to fix it. Does the modem uptime confirm it rebooted?

Steve

Gertjan

@unf0rg0tt3n said in Pfsense internet goes down all the time:

I have tried hooking up directly to the modem, that works because the soft reboot won't last that long...

So, the modem reboots, no matter how and with what, you use it.
From what you're saying, not a "VM" or pfSense issue.

unf0rg0tt3n

@stephenw10 Thanks Steve and @Gertjan . I actually can't login to the modem anymore since it's in bridge mode.

The problem might be with the cat 6e (which is over 20meter) no idea if that is the problem; but the log: https://pastebin.com/GmH0TCSP
Do you see strange stuff? I only see that it goes down.

edit:
Gateway log: https://pastebin.com/zcpKa6be

Gertjan

@unf0rg0tt3n said in Pfsense internet goes down all the time:

https://pastebin.com/zcpKa6be

By any chance, the WAN wasn't 'saturated' at that moment ? 'ping' is a low priority protocol.
if not, your uplink is just 'very bad'.

unf0rg0tt3n

@gertjan So it could be a nicked or damaged cat 6a cable? My pfsense box is in the attic and the modem is downstairs.

Gertjan

@unf0rg0tt3n
Time to go to the attic and take pfSense next to the modem.
Using another cable of course.

unf0rg0tt3n

@gertjan Hm, I have a spare pc laying around. I need to duplicatie my config to that pc. I hope it isn't the cable, so much work to replace that one.

Gertjan

@unf0rg0tt3n
I hope it's the cable ;)
The modem, or any cables beyond that, is not up to you, so chances are you have to 'live with it'.

Cool_Corona

Is the ISP modem in bridgemode??

If not, start there.

unf0rg0tt3n

@gertjan That would be the easiest solution. Because "ziggo" dug open 40 meters of street and replaced the whole cable already; that was damaged though.
Thanks! I'll post my findings

@Cool_Corona it is

JKnott

@unf0rg0tt3n said in Pfsense internet goes down all the time:

So it could be a nicked or damaged cat 6a cable?

Trying a new cable is the easiest thing to do. However, if the cable is bad, you will have no connection, connection at 100 Mb or lots of errors & poor throughput. You can get cheap testers to verify the cable is wired properly.

unf0rg0tt3n

@jknott Thanks for your reply!
The cable is good (according to the tester I used), about the errors:

enp7s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:25:90:dc:0f:3a  txqueuelen 1000  (Ethernet)
        RX packets 621161334  bytes 713693829981 (664.6 GiB)
        RX errors 1  dropped 6  overruns 0  frame 1
        TX packets 411474807  bytes 208509699544 (194.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xdfd00000-dfd20000

There arent many, 1RX error with 6 dropped.
This is on physical hypervisor interface.
Pfsense report 0 errors

stephenw10

Yeah, that seems fine. You could also check the Status > Interfaces page in pfSense in case it's some intermittent fault that test didn't show.
Perhaps the cable is wrapped around your AC compressor and it's suppression has failed for example.

The gateway logs like this are bad:

Jun 21 07:13:45	dpinger	84199	WAN_DHCP 1.1.1.1: Alarm latency 14389us stddev 2120us loss 22%
Jun 21 07:21:25	dpinger	84199	WAN_DHCP 1.1.1.1: Clear latency 14821us stddev 2999us loss 5%
Jun 21 07:26:40	dpinger	84199	WAN_DHCP 1.1.1.1: Alarm latency 15642us stddev 6878us loss 21%
Jun 21 07:34:18	dpinger	84199	WAN_DHCP 1.1.1.1: Clear latency 14966us stddev 3562us loss 5%
Jun 21 07:37:33	dpinger	84199	WAN_DHCP 1.1.1.1: Alarm latency 13933us stddev 1854us loss 21%
Jun 21 07:45:12	dpinger	84199	WAN_DHCP 1.1.1.1: Clear latency 15617us stddev 4664us loss 6%

But note that is packet loss only. Typically you would see a large increase in latency too if it was a saturation issue.

Your modem may still have an IP you can connect to in bridge mode, just not in the WAN subnet. You might need to add a VIP on WAN to connect to it.

You are monitoring 1.1.1.1 which is generally a good idea but it might be interesting to switch that back to the gateway IP to see if that also fails.
If you have only one WAN you should disable the gateway monitoring action on that gateway ( not the monitoring itself) there is no point restarting everything when there is no secondary WAN to failover to.

Steve

stephenw10

Mmm, I also note those packet loss events are almost exactly the same length, ~7m40s.

That seems too consistent to be something like a bad cable.

Steve

unf0rg0tt3n

@stephenw10 said in Pfsense internet goes down all the time:

Mmm, I also note those packet loss events are almost exactly the same length, ~7m40s.

That seems too consistent to be something like a bad cable.

Steve

Yeah, I guess soft reboot times.

@stephenw10 said in Pfsense internet goes down all the time:

Yeah, that seems fine. You could also check the Status > Interfaces page in pfSense in case it's some intermittent fault that test didn't show.
Perhaps the cable is wrapped around your AC compressor and it's suppression has failed for example.

The gateway logs like this are bad:
Jun 21 07:13:45	dpinger	84199	WAN_DHCP 1.1.1.1: Alarm latency 14389us stddev 2120us loss 22%
Jun 21 07:21:25	dpinger	84199	WAN_DHCP 1.1.1.1: Clear latency 14821us stddev 2999us loss 5%
Jun 21 07:26:40	dpinger	84199	WAN_DHCP 1.1.1.1: Alarm latency 15642us stddev 6878us loss 21%
Jun 21 07:34:18	dpinger	84199	WAN_DHCP 1.1.1.1: Clear latency 14966us stddev 3562us loss 5%
Jun 21 07:37:33	dpinger	84199	WAN_DHCP 1.1.1.1: Alarm latency 13933us stddev 1854us loss 21%
Jun 21 07:45:12	dpinger	84199	WAN_DHCP 1.1.1.1: Clear latency 15617us stddev 4664us loss 6%
But note that is packet loss only. Typically you would see a large increase in latency too if it was a saturation issue.

Your modem may still have an IP you can connect to in bridge mode, just not in the WAN subnet. You might need to add a VIP on WAN to connect to it.

You are monitoring 1.1.1.1 which is generally a good idea but it might be interesting to switch that back to the gateway IP to see if that also fails.
If you have only one WAN you should disable the gateway monitoring action on that gateway ( not the monitoring itself) there is no point restarting everything when there is no secondary WAN to failover to.

Steve

AC compressor? We don't have AC haha. But the cable is bundled with 2 other ones linked to AP's.

stephenw10

I have no AC either but you get the idea.
Whatever is causing a problem may not be present all the time. The test you ran looks good but the pfSense interface stats are monitoring it all the time so if it's something intermittent it should show there.

@unf0rg0tt3n said in Pfsense internet goes down all the time:

Yeah, I guess soft reboot times.

That seems like a very long time for anything to reboot. The modem?
I would want to be sure the modem is or isn't rebooting.

Steve

unf0rg0tt3n

@stephenw10 I don't know if that might take that long. It feels forever https://imgur.com/a/jMjwwez

stephenw10

It may be it really takes that long if it loses upstream sync. It's a cable modem?

It could be it's not rebooting at all and only loses upstream sync. pfSense it connected to it directly, not via a switch?
The logs you posted do not show it losing link on the WAN just the gateway going down. It it really rebooted I would expect to see the link go down.

Have you checked to see if the modem has an admin interface IP you might be able to access?

Steve

unf0rg0tt3n

@stephenw10 There isn't an interface IP. it's a Arris TG2492LG (cable modem)
The link might not go down because it's a VM? In that case there is always link?
I might be wrong.

According to the webpage it takes 5 minutes to fully reboot the modem. PFsense might take another 2, to see that it's actually up.

Also, no switch; directly attached to the modem via UTP