2.0 to 2.0.1 update failed - complete meltdown



  • I'm mainly posting this at the request of Chris Buechler. (Who has now stated that I'm a Jackass and has told me to "f off")
    As he's sure that I'm screwing this up, and no way the update could have a problem….

    I have 9 pfSense firewalls in use. Most of them are running 2.0.1, a few are 2.1.  Two were still on 2.0.
    They have been running perfectly for over a year, so I've never bothered updating them.
    This morning I got a call that my IPSec VPN's were down, so I tried to restart them, when they wouldn't restart, I rebooted both units.
    When the VPN's didn't come back, I figured I just go ahead and update the units.

    3 minutes later I got this on the console

    Broadcast Message from root@pfsense.xxx.local                         
            (no tty) at 11:43 PST...                                               
    
    Firmware upgrade in progress...                                                
    
    ELF: not found
    kill: @@HHHHPHP?p??P?P???@?@P?td@@@/libexec/ld-elf.so.FreeBSD?8
    
                                                                   : not found
    kill: 3: Syntax error: "(" unexpected
    Connection to 97.65.xxx.xxx closed.
    

    and this from the GUI:

    
    An HTTP_REFERER was detected other than what is defined in System -> Advanced (https://10.103.71.1/system_firmware.php). You can disable this check if needed in System -> Advanced -> Admin.
    

    So, yeah, I think the update failed, BIG TIME.  Both units have the exact same errors. They both had 99% the same configuration, only difference being the subnets.


  • Netgate Administrator

    @Transeau:

    I'm mainly posting this at the request of Chris Buechler. (Who has now stated that I'm a Jackass and has told me to "f off")

    :D I shall wait to pass judgement!

    I assume you already updated your other units with no problems?
    Are these two any different from the others?
    Did you have any packages installed? This would be my first suspect.
    My second suspect would be that the units had a problem already, since you couldn't bring the VPNs back up, and that updating them has just shown this up.

    Steve


  • Rebel Alliance Developer Netgate

    Most likely problems:

    • Storage failure, perhaps all 9 units had SSDs or HDDs from the same malfunctioning batch and they're all melting down (Not unheard of, just happened to a customer in the last couple weeks)
    • Accidentally mixing 32-bit and 64-bit binaries/files in some weird way

    Without more info it's hard to speculate any more beyond general hardware issues.

    Things that would help to know:
    1. What exact hardware is in use
    2. What exact image was used to install them, and how, including the kernel you're running
    3. What exact file was used to upgrade the systems



  • I asked you to do this so we could help, and you refused and just wanted to send me nasty emails, after several of which you got a nasty response. Shocking, right? ::) Thank you for posting, you could have just done this from the get go.

    In addition to what Jim mentioned, you're sure you were pointing to the default update URL? Assuming you were using auto-update. Checking a backup of your config would show. If you follow some instructions out there on various sites other than ours you can end up getting pointed to some non-official source.



  • Your box was broken, you upgraded, and it broke more.  That doesn't sound like the underlying issue was the upgrade.

    Anyway, are these the same boxes you were complaining about last fall where you were getting disk errors and the system was crashing?



  • Boxes wen't broken.

    One of the two boxes came back to life with a simple power cycle. Updated and all.
    The other required a fresh install. Haven't had another issue with either box.
    I ran the manufacture's diags on both drives, both checked out 100%.
    and no, these were not the box that I was "complaining" about. That issue went away after moving the box to 2.0.

    I doubt it was a hardware issue, and I've found other people (4) that have reported the same 2.0 to 2.0.1 issue.  Maybe it has to do with the update being done over an OpenVPN link? Yes, I know I'm reaching on that, I'm just curious as to what could have caused it.



  • OpenVPN UDP stream? No sense check on file integrity on the upload system?

    Speculation, here to learn



  • Great, i did also get this magnificent hdd failure club..
    That hdd were working perfectly before upgrade attempt.

    –> to the computer hardware shop to get one hdd


Locked