pfsense cant be accessed, internet gone, must reboot
-
Hello,
I have a protectli vault fw4c. It has worked very well for some time. This weekend it started giving the appearance of freezing up. I have to pull power to get it to restart and it works fine for a time. I am not finding errors in logs.
Please help to troubleshoot and get this resolved. As stated, I am not recognizing errors in the logs, so not sure what to share. -
This has all the hallmarks of failing hardware. I would run a memory module test as a first step. But your issue could be any of the hardware in the box. If you have made no change and this just started happening out of the blue, then hardware is about the only thing it could reasonably be.
-
@gems said in pfsense cant be accessed, internet gone, must reboot:
it started giving the appearance of freezing up
Just how frozen is it? No output on either console?
-
@stephenw10 It does not respond to ping nor ssh. When I connect a screen and keyboard/mouse, I do not see anything. It also does not respond to the web gui attempting to connect from my laptop, but I suspect that is more due to the lost dns services when it hangs. I have not thought of another way to attempt to connect or validate.
-
@bmeeks Thanks, I suspected this, but I have not found any of the commands within the freebsd to show any errors, nor do I see any errors in syslog after the reboot.
-
@gems I caught some logs between dropping internet and losing web gui access.
Here are some of the initial lines when the internet dropped....
Apr 15 22:34:31 check_reload_status 441 Linkup starting igc0
Apr 15 22:34:31 kernel igc0: link state changed to DOWN
Apr 15 22:34:32 php-fpm 97244 /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp, 6: dhcp6)
Apr 15 22:34:32 php-fpm 97244 /rc.linkup: DEVD Ethernet detached event for wan
Apr 15 22:34:35 check_reload_status 441 Reloading filter
Apr 15 22:34:35 rc.gateway_alarm 9149 >>> Gateway alarm: WAN_DHCP (Addr:23.252.245.1 Alarm:down RTT:0ms RTTsd:0ms Loss:100%)
Apr 15 22:34:35 check_reload_status 441 updating dyndns WAN_DHCP
Apr 15 22:34:35 check_reload_status 441 Restarting IPsec tunnels
Apr 15 22:34:35 check_reload_status 441 Restarting OpenVPN tunnels/interfaces
Apr 15 22:34:35 check_reload_status 441 Reloading filter
Apr 15 22:34:37 check_reload_status 441 Linkup starting igc0
Apr 15 22:34:37 kernel igc0: link state changed to UP
Apr 15 22:34:39 php-fpm 24 /rc.linkup: Hotplug event detected for WAN(wan) dynamic IP address (4: dhcp, 6: dhcp6)
Apr 15 22:34:39 php-fpm 24 /rc.linkup: DEVD Ethernet attached event for wan
Apr 15 22:34:39 php-fpm 24 /rc.linkup: HOTPLUG: Configuring interface wan
Apr 15 22:34:39 check_reload_status 441 rc.newwanip starting igc0
Apr 15 22:34:39 php-fpm 24 /rc.linkup: calling interface_dhcpv6_configure.
Apr 15 22:34:39 php-fpm 24 /rc.linkup: Accept router advertisements on interface igc0
Apr 15 22:34:39 php-fpm 24 /rc.linkup: Starting DHCP6 client for interfaces igc0 in DHCP6 without RA mode
Apr 15 22:34:39 php-fpm 24 /rc.linkup: Starting rtsold process on wan(igc0) -
@gems
Is igc0 your WAN ?
If so, it start by going up at 15 22:34:31 - and go down again moments (ms) later.Have a "talk" with the device that's connected to your pfSese, and figure out why it flaps the WAN line like that : why it pulls down its LAN interface that is connected to the pfSense WAN interface ?
That said, it could also be pfSense that 'restes' the line.
This happens when the WAN quality monitoring detectes that the WAN line is bad.
This is done by a process called "dpinger" that pings the/a WAN upstream gateway, and mesres the delay.
You can see that delay here :where it shows my IPv4 and IPv6 connectivity "quality".
If the ping requests don't come back anymore == bad connection ? then dpinger can 'reset' (that's called the "action") the WAN interface. This provokes a WAN down event followed by a WAN UP event, and this will recreate (rebuild) the WAN connection.
As per your instructions : System >Routing > Gateways > Edit :If dpinger starts to detect that ping requests stop to come back, then this even is also logged :
( this shows the start of a dpinger process for the the IPv4 WAN part )
-
@gems said in pfsense cant be accessed, internet gone, must reboot:
When I connect a screen and keyboard/mouse, I do not see anything.
Do you see something when it's running normally? Or when you reboot? You might be using the wrong console if it's serial console only.
Seeing no output at the console (assuming it works normally) and no crash report after rebooting starts to look like a hardware issue.
If the console is working normally you might catch something on it just before it hangs. If it's a failing disk for example it can show there and be unable to log anything.
-
@stephenw10 Mine has 2 hdmi ports, I have tried both.
I have also contacted the internet provider and they believe that there may be an issue with the card that converts the fiber to ethernet and are scheduled to replace later today.. -
protectli vault fw4c ... also has a serial port, which could also be used.
Btw : afaik, its the serial prt, or the hdmi port, not both of them.But it does not have any 'fiber' plug as far as I can see ... so how do you hook up this box ?
I get it, with an RJ45 going to the ISP equipment. So : why wait, that wire : instead of hooking up the Protocli as the WAN line, hook up your PC or whatever has a RJ45, and test the connection. You'll know right away if its a ISP issue, or something else.Although I never saw a Protocli in my live, something tells me that thousands are using that device with pfSense.
You mission is : make the console work - serial = this port:
🔒 Log in to viewor, more obvious, any of these two :
( and then you need an usb keyboard )
Both type of ports, the serial, or the HDMI, if configured correctly = check this with the protocli doc, will show text as soon as the BIOS boots.
No need to install Linux, Windows, or even pfSense on it.You've said that it worked for some time : so you know how to access the console, as in the beginning, when there was no OS on the internal drive, none of the 4 the Ethernetport could work. You need the console access to do some initial setup to make pfSense - or any other OS - work.
If that console doesn't work work anymore, then the issue isn't the installed OS, as that one will start up later on, when the BIOS loaded and starts the OS (pfSense in this case - but it could be any access). -
my bad, I was not clear.
My internet provider runs fiber to the outside of the house. The provider also own the card that converts the fiber to ethernet. The cable from that card plugs into an ethernet port labelled WAN on the vault. So it is possible that this is a hardwatre issue, but with the equipment from the internet provider and not the vault. -
You should see something on at least one of the HDMI ports even if it's just the POST output at boot.
Most pfSense installs boot dual console but only one can be primary.
But, yes, check the serial console. It's generally more useful than video anyway because you can log the output or connect to it via some other local host. So worth while getting that setup even if it not necessary to solve this.
-
@stephenw10 Quick update, my isp provider came out yesterday about lunch time an replaced the card that converts the fiber to ethernet and replaced the rj45 socket. We tested with no errors and i have not seen a hang on the vault yet and it is now almost 30 hours running.