Preventing connectivity issues after loss of power?



  • Recently I had a short power outage(5-10seconds) and when power came back on, so did my ESXi host and pfSense VM.  I lost network and internet connectivity on both of my home networks and it took me several hours of swapping cables around, setting statics on stuff, etc in order to bounce back and forth between vSphere and pfSense CLI/GUI access before finally getting everything back online.  Both of the NICs in my main rig were getting 196.254.x.x addresses since pfSense(and by extension my DHCP server) was broken, so this meant no accessing anything on 192.168.1x or 10.0.0.x which are my home networks.  Apparently I just had to delete and redo my interfaces and then reboot pfSense…I think.  At this point I'm still not sure what broke or what exactly fixed it.

    Once I get new batteries for my APC smartUPS-1000 it will help in situations where I only lose power for a few seconds or a few minutes, but if the power outage outlasts the UPS batteries then I'll be thrown in to the same situation as before.  Even if I set up ESXi to do a graceful shutdown it still won't prevent the pfSense VM from experiencing a hard shutdown.  This is just a bandaid for the problem anyway.

    What I want to know is what exactly happened, why did it happen, and how can it be properly resolved?  I now have a pfSense config backup and a pfSense VM snapshot, but that doesn't address the actual problem here.  I'm also not even sure how easily I can restore those backups because without network connectivity I can't access the webGUI or run vSphere without moving cables around and changing static IPs until I find a combination that gives me some sort of connectivity.



  • @opjohnny:

    Even if I set up ESXi to do a graceful shutdown it still won't prevent the pfSense VM from experiencing a hard shutdown.  This is just a bandaid for the problem anyway.

    ...

    Hard shutdown doesn't have to be the case.  If you set up auto startup/shutdown of VMs on your ESXi host and have VM tools installed on your VMs, those VMs can be shutdown gracefully by ESXi before it shuts itself down.



  • Unless ESXi knows how to send a shutdown command to pfSense that pfSense sees and understands as a normal shutdown command, it will still suffer a hard shutdown.  Basically what ESXi is doing is hitting the power button to the VM to turn it on and off.  That's still a hard shut down.

    I want to know what specifically breaks in pfSense on a hard shutdown and if anyone has fixed it within pfSense.  Once I can figure this out then I'll be more than happy to set up my UPS and configure the graceful shutdown of everything.



  • @opjohnny:

    I want to know what specifically breaks in pfSense on a hard shutdown and if anyone has fixed it within pfSense.  Once I can figure this out then I'll be more than happy to set up my UPS and configure the graceful shutdown of everything.

    … seems odd your pfSense instance is breaking due to a hard shutdown. I don't think this is supposed to happen.

    When you rebooted your ESXi, were the network interfaces all functioning correctly?



  • The thing that was most odd was when I was able to access the webGUI everything looked fine except for the fact it wasn't handing out DHCP leases to the NICs on my main rig.  This is why I had to set static IPs and move patch cables around so I could move between vSphere access and webGUI access trying to fix the problem.  I wiped and redid my three interfaces(WAN, LAN1, LAN2) and most of the settings returned on their own(saved in the background maybe?) and then after rebooting it ran fine and still is running fine.

    I rebooted my main rig, the pfSense VM, and my ESXi host before doing anything else so it wasn't just a reboot that fixed it.  I'm guessing that reconfiguring the interfaces + a pfsense reboot was the solution.  It's really bothering me that I don't know what actually happened and how I can permanently fix it because I won't always be there when the power goes out and other people use these networks.



  • Honestly, I don't remember if the network config in ESXi was fine or not after that outage.  I think I did have to wipe/reinstall those configs too.  I wonder if those are what broke which then broke the interfaces on pfSense?  ???



  • @opjohnny:

    Unless ESXi knows how to send a shutdown command to pfSense …

    It does.  You just need to select the right shutdown mode:

    ![2013-12-31 09-13-55.png](/public/imported_attachments/1/2013-12-31 09-13-55.png)
    ![2013-12-31 09-13-55.png_thumb](/public/imported_attachments/1/2013-12-31 09-13-55.png_thumb)



  • @biggsy:

    @opjohnny:

    Unless ESXi knows how to send a shutdown command to pfSense …

    It does.  You just need to select the right shutdown mode:

    That simply just "cuts power" to the VM before ESXi continues its shutdown process.  It's like holding the power button on a PC or laptop until it turns off.

    As far as I know the only way to get pfSense to shutdown is from CLI(option 5?) or webGUI(diagnostics : halt system), or possibly via the command shell using a shutdown command.  ESXi can't do any of these tasks.  All it knows is Power On and Power Off for the virtual machines.

    ETA:  And those settings you screencapped are already set up on mine.  That's why pfSense came back on with ESXi after the power outage.



  • @opjohnny:

    @biggsy:

    @opjohnny:

    Unless ESXi knows how to send a shutdown command to pfSense …

    It does.  You just need to select the right shutdown mode:

    That simply just "cuts power" to the VM before ESXi continues its shutdown process.  It's like holding the power button on a PC or laptop until it turns off.

    No, it does not just cut the power to the VM.  For me, this is the key benefit of VMware tools.  I used it, quite deliberately, yesterday to shut down a whole host without having to shut down each of the individual VMs beforehand.

    Of course, if ESXi isn't told to shut down gracefully, as happens in a power-fail with no UPS …


Log in to reply