System botched after upgrade attempt to 2.3.3



  • This just happened on my APU1, trying to upgrade from 2.3.2_1.
    The first part of the upgrade seemed to go well (install new kernel), but after the reboot the unit is dead right after entering multiuser mode.
    The console log shows:

    
    ***** FILE SYSTEM MARKED CLEAN *****                                            
    Filesystems are clean, continuing...                                            
    Mounting filesystems...                                                         
    Setting up memory disks...mdmfs: newfs exited with error code 255               
    mdmfs: newfs exited with error code 255                                         
     done.                                                                          
    
         ___                                                                        
     ___/ f \                                                                       
    / p \___/ Sense                                                                 
    \___/   \                                                                       
        \___/                                                                       
    
    Welcome to pfSense 2.3.2-RELEASE (Patch 1) on the 'pfSense' platform...         
    
    Creating symlinks......done.                                                    
    Failed to write core file for process pkg (error 14)                            
    pid 134 (pkg), uid 0: exited on signal 11                                       
    Segmentation fault                                                              
    Failed to write core file for process pkg (error 14)                            
    pid 136 (pkg), uid 0: exited on signal 11                                       
    Segmentation fault                                                              
    Failed to write core file for process pkg (error 14)                            
    pid 137 (pkg), uid 0: exited on signal 11                                       
    Segmentation fault                                                              
    ERROR: It was not possible to identify which pfSense kernel is installed        
    Failed to write core file for process php-cgi (error 14)                        
    pid 159 (php-cgi), uid 0: exited on signal 11                                   
    Failed to write core file for process php-cgi (error 14)                        
    pid 277 (php-cgi), uid 0: exited on signal 11                                   
    Failed to write core file for process php-fpm (error 14)                        
    pid 396 (php-fpm), uid 0: exited on signal 11                                   
    Segmentation fault                                                              
    fcgicli: Could not connect to server(/var/run/php-fpm.socket).                  
    Launching the init system...Failed to write core file for process php-cgi (erro)
    pid 401 (php-cgi), uid 0: exited on signal 11                                   
    Segmentation fault                                                              
    Starting CRON... Failed to write core file for process cron (error 14)          
    pid 403 (cron), uid 0: exited on signal 11                                      
    Segmentation fault                                                              
    done.                                                                           
    fcgicli: Could not connect to server(/var/run/php-fpm.socket).                  
    Failed to write core file for process php (error 14)                            
    pid 440 (php), uid 0: exited on signal 11                                       
    Segmentation fault                                                              
    Failed to write core file for process pkg (error 14)                            
    pid 445 (pkg), uid 0: exited on signal 11                                       
    Segmentation fault                                                              
    Failed to write core file for process pkg (error 14)                            
    pid 447 (pkg), uid 0: exited on signal 11                                       
    Segmentation fault                                                              
    Failed to write core file for process pkg (error 14)                            
    pid 448 (pkg), uid 0: exited on signal 11                                       
    Segmentation fault                                                              
    ERROR: It was not possible to identify which pfSense kernel is installed        
    Failed to write core file for process php (error 14)                            
    pid 453 (php), uid 0: exited on signal 11                                       
    Segmentation fault                                                              
    pfSense (pfSense) 2.3.2-RELEASE (Patch 1) amd64 Tue Sep 27 12:13:07 CDT 2016    
    Bootup complete                                                                 
    
    

    Just about any process seems to segfault. The exact same thing happens when I boot with the old kernel.
    I can get a prompt in single-user mode, but more segfaults:

    
    # vi upgrade_log.txt                                                            
    pid 25 (vi), uid 0: exited on signal 11                                         
    Segmentation fault                                                              
    
    

    The upgrad log itself shows that it apparently started with the package and base system upgrades:

    
    [84/88] Extracting pfSense-default-config-2.3.3: .... done                      
    [85/88] Upgrading pfSense-base from 2.3.2_1 to 2.3.3...                         
    [85/88] Extracting pfSense-base-2.3.3: ..... done                               
    ===> Keeping a copy of current version mtree                                    
    ===> Removing schg flag from base files                                         
    ===> Extracting new base tarball                                                
    ERROR: It was not possible to identify which pfSense kernel is installed        
    ...
    
    

    Short of a clean re-install, is there anything I could try from the console, any way to roll back the botched update?



  • I just upgraded an APU with 4G nanoBSD on an SD card in it and that worked fine.
    I guess APU1 is different to "vanilla APU"?



  • @phil.davis:

    I just upgraded an APU with 4G nanoBSD on an SD card in it and that worked fine.
    I guess APU1 is different to "vanilla APU"?

    I mean the APU (PC Engines AMD G-T40E); I wrote "APU1" because there is now a newer version, the "APU2".


  • Rebel Alliance Developer Netgate

    Personally, I'd start with a fresh full install image. Even if you were using NanoBSD, switch to a full install on that hardware.



  • @jimp:

    Personally, I'd start with a fresh full install image. Even if you were using NanoBSD, switch to a full install on that hardware.

    Yes, salvaging this hosed system is not worth it.
    When the problems happened, I took another retired APU that still had 2.3.2 on it. The configuration was differernt, so I had to change the LAN IP address via a serial console and then restored the latest backup.
    So, I was back in business soon enough, and the broken unit will now become the backup.
    The troublesome update was already using a full install, I neglected to mention this.

    I'll try an update again with the replacement unit, but I'll use the console, so I should be better able to see what goes wrong, if it happens again.



  • After a clean install of 2.3.3 on my backup APU I made another attempt to upgrade the currently active unit from 2.3.2_1 to 2.3.3; this time I did it via the serial console, command "pfSense-upgrade".
    Not surprisingly, it worked perfectly this time  ;)


Log in to reply