[SOLVED - manual upgrd] Upgraded to 2.1.4 and now my WG Firebox X700 won't boot!



  • Just upgraded to newest 2.1.4 and it gives me error gallore..!

    
    ad0: 1919MB <smi 20110407="" model="">at ata0-master UDMA33
    ad0: FAILURE - READ_DMA timed out LBA=3931197
    ad0: FAILURE - READ_DMA timed out LBA=3931183
    ad0: FAILURE - READ_DMA timed out LBA=3931196
    ad0: FAILURE - READ_DMA timed out LBA=3931199
    ad0: FAILURE - READ_DMA timed out LBA=3931199
    ad0: FAILURE - READ_DMA timed out LBA=3931137
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=128
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=128
    ad0: FAILURE - READ_DMA timed out LBA=128
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=16
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=16
    ad0: FAILURE - READ_DMA timed out LBA=16
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=0
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=0
    ad0: FAILURE - READ_DMA timed out LBA=0
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=512
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=512
    ad0: FAILURE - READ_DMA timed out LBA=512
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=64
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=64
    ad0: FAILURE - READ_DMA timed out LBA=64
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=0
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=0
    ad0: FAILURE - READ_DMA timed out LBA=0
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=2
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=2
    ad0: FAILURE - READ_DMA timed out LBA=2
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=16
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=16
    ad0: FAILURE - READ_DMA timed out LBA=16
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=128
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=128
    ad0: FAILURE - READ_DMA timed out LBA=128
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=0
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=0
    ad0: FAILURE - READ_DMA timed out LBA=0
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=0
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=0
    ad0: FAILURE - READ_DMA timed out LBA=0
    ad0: TIMEOUT - READ_DMA retrying (1 retry left) LBA=1
    ad0: TIMEOUT - READ_DMA retrying (0 retries left) LBA=1
    ad0: FAILURE - READ_DMA timed out LBA=1</smi> 
    


  • I managed to boot to 2.1.3 by selecting 2 on serial boot…  Any ideas???



  • I just attempted the update and after 20 minutes it is still spinning its' wheels on "an upgrade is currently in progress"….  not a good sign.

    ok... it finally finished.. wooh, had me worried.



  • Are you running nano on cf-card?

    EDIT:  Tried a 2nd time, same result.  Not sure if it's my CF-card, but the CF-card was brand new when I first installed pfSense on my WG Firebox X700.  Any other WG Firebox users experiencing this?



  • no, i'm running it on a HP G7 N54L..

    took close to 25 minutes to update, but it finished.. few packages are working, but i read the notes so… i guess i got some re-installs to do.



  • Yes, also have issues. Did also push upgrade from (nanobsd) 2.1.3 -> 2.1.4 on my spare WG x1250e, and it is now stuck in a boot loop. I don't even get to a bsd boot option (slice selection) :(
    Guess I'll need to yank out the CF and see what has happened…  ::)



  • did this update break the console, 2 things seem broken, first when u boot the alix the message that says u got 5 seconds to hit any key for shell and second when it says bootup complete, the rest of the part that shows console commands and interface ips never show



  • to make my previous post complete:
    system was a nanobsd 2G i386 on a kingston CF, minimal packages installed (OpenVPN Client export only), running on a WG x1250e with 1G ram.
    First impression I have is that the filesystem has become corrupted or so?
    Exact actions take: initiate upgrade from the gui, waited like 40minutes or so (then pulled the plug because it didn't come up).
    Connected serially, and saw it was booting in loops… So or the upgrade screwed something, or I did not wait long enough. More analyses later today when I have more time, all suggestions welcome in meantime.



  • I get the same boot loops (sounds like a cereal?  LOL) but mine says it can't find the boot device that it will restart in 15 seconds and then it gives me the menu to select 1 or 2 (1 being the 2.1.4 and 2 being the 2.1.3 backup) so only way I got it back running was selecting 2 and running off of that backup now)  Won't try again until it's solved I guess.


  • Netgate Administrator

    Ha! I'm ready for all new Frosted Boot Loops!
    That DMA error looks like what you see if you boot the Nano_VGA image. The difference is that DMA is not disabled by default in the vga flavour. Could it be that the new 2.1.4 Nano images don't have DMA disabled? That would make it fail on the Alix box which several developers use so that seems like a case that should have been tested.

    Reading this post:
    https://forum.pfsense.org/index.php?topic=78603.0
    It looks like maybe the upgrade image is defaulting to vga and hence DMA still enabled.

    Steve



  • My X750E unit suffered from this same bug…. Luckily I have 2 units in high availability mode so I didn't have to come running into the office last night when the unit didn't come back to life on its own.

    So I was able to get it to boot off the other 2.1.3 image and then cloned my 2.1.3 image to the other boot partition until I see confirmation that the update works properly. I was running the 4g version FYI.



  • I'm with you on the wait….  If anybody is nice enough to let us know when this issue is fixed we will be even more grateful than giving us a super awesome firewall solution!  Thanks pfSense!


  • Netgate Administrator

    I very nearly vpn'd into my home box to upgrade it but I think I'll hold for a bit. The box is in London but I'm currently in Texas and I don't fancy trying to talk anyone through a remote slice switch.
    Keep an eye on the mailing lists. Nothing there yet and if this really is going to nano_vga I would expect it to bite plenty of users.

    Steve



  • So I'm screwed. Exit the FW from the serverroom, it's on the desk now  >:(
    Checked fast the CF, I still see the 2 partitions. (so my 2 slices are still there)
    Booting the box, I only see output in 115200 baud. Nothing shows when I set it to 9600.
    But if it upgraded to the vga image, it kind of explains why I get no serial output I suppose?

    @phatty, how did you get it to boot the alternate slice without serial output??

    As it is already on the desk, the easy route would of course be just flash the latest image… but there's little fun in that, so I'll continue the search for now ;D

    @steve; it seems that alix is also affected (if I understood it correctly), see post from "xbipin" earlier in this thread.



  • yes alix affected only regarding the console, the rest still updates and reboots fine also the rest works also



  • Okay, tnx for the feedback, and good that it comes up.
    That might explain why others don't report issues (I like to assume I'm like most, and I don't look at the console unless there's issues ;))
    Making no progress here with that darn CF. Any hints on how to safely mount that BSD partition in windows?
    (sorry, bsd noob here :-[  Still learning…)



  • @bennyc:

    So I'm screwed. Exit the FW from the serverroom, it's on the desk now  >:(
    Checked fast the CF, I still see the 2 partitions. (so my 2 slices are still there)
    Booting the box, I only see output in 115200 baud. Nothing shows when I set it to 9600.
    But if it upgraded to the vga image, it kind of explains why I get no serial output I suppose?

    @phatty, how did you get it to boot the alternate slice without serial output??

    As it is already on the desk, the easy route would of course be just flash the latest image… but there's little fun in that, so I'll continue the search for now ;D

    @steve; it seems that alix is also affected (if I understood it correctly), see post from "xbipin" earlier in this thread.

    Benny, the bios output, and the pfsense output for my setup is 115200, but at 9600, for a very short second, I was able to wait and eventually see the boot options for which slice to boot off of.. It took a couple tries for me to time my button pushing just right to get it to boot off the older image.


  • Banned

    This is exactly the reason why I run it in VM's…



  • This is exactly the reason why I run it in VM's…

    +1.

    I can't be bothered playing around with finicky hardware, CF cards and all that nonsense.


  • Banned

    No if uptime is important, then I see no alternative….and its a lot more flexible as well.



  • It's clever bu how do you do when you must routing more than 7/8 networks ?? You need necessarily a major hardware layer… (machine with 7/8 network devices...)


  • Banned

    No….I use VLAN's.... and only have 3 physical interfaces...



  • With the watchguard sitting on my desk I attempted to go ahead and update again today. This time I manually retrieved pfSense-2.1.4-RELEASE-4g-i386-nanobsd-upgrade.img.gz from the Bluegrass Mirror. This time update applied and system rebooted properly.  Not sure if it was a problem with the auto updater selecting the wrong update, or if they fixed something with the files. I noticed the time stamp on the Bluegrass Mirror where I downloaded the update from was newer than when I had attempted the upgrade last.


  • Netgate Administrator

    @bennyc:

    Making no progress here with that darn CF. Any hints on how to safely mount that BSD partition in windows?

    The easiest way to do this, by far, is to do it from a BSD install. If you don't have a BSD machine then run a FreeBSD live CD like GhostBSD.

    Steve



  • @stephenw10:

    @bennyc:

    Making no progress here with that darn CF. Any hints on how to safely mount that BSD partition in windows?

    The easiest way to do this, by far, is to do it from a BSD install. If you don't have a BSD machine then run a FreeBSD live CD like GhostBSD.

    Steve

    Virtual machine with FreeBSD installed and connect the disk to it with an USB to SATA adapter, just about any USB enclosure will do.



  • Not that I'm surprised, but I thought it was worth reporting I'm now 2 for 2 on upgrading my Watchguard X750's using the manual update method.



  • @phatty:

    Not that I'm surprised, but I thought it was worth reporting I'm now 2 for 2 on upgrading my Watchguard X750's using the manual update method.

    Thank you for your almost realtime feedback on your update attempts. I would like to clarify one point before I am going to give my Alix backup machine a try: What do you mean by "manual update method"?
    1.) From the webGUI: "System -> Firmware -> Manual Update" or 2.) From the console menu: "13) Upgrade from console"?

    I have never used method 1.) before. Instead I used method 2.) or "Auto Update" from the webGUI.

    I strongly suppose that update method does not matter as long as all selectable mirrors carry the same update files. Can you confirm the BluegrassNet mirror having had a more recent time stamp compared to the auto-selected mirror?

    Regards,
    Peter



  • @phatty:

    for a very short second, I was able to wait and eventually see the boot options for which slice to boot off of…

    Thank you for that. While downloading the GhostBSD iso (also tnx at Steve for the hint), took a shot at this and after a couple of tries I got to boot slice 2. Now I have again console access, much better ;D

    I will repeat upgrade attempt tomorrow evening. Are there logs kept on the system about past upgrade attempt? I'm curious to know where it went wrong, or what went wrong… (there's always something to learn)



  • It is repeatable. I tried twice upgrading again, with exact same outcome as described here below.

    Booted in 2.1.3, nano 2g image. Went to 'auto-update'

    A new version is now available
    
    Current version: 2.1.3-RELEASE
      NanoBSD Size : 2g
           Built On: Thu May 01 15:52:17 EDT 2014
        New version: 2.1.4-RELEASE
    
      Update source: https://updates.pfsense.org/_updaters 
    

    -> invoke auto upgrade

      Auto Update Download Status
    ----------------------------------------------------
      Current Version : 2.1.3-RELEASE
      Latest Version  : 2.1.4-RELEASE
      File size       : 76070249
      Downloaded      : 24739580
      Percent         : 33%
    ----------------------------------------------------
    

    File size reflects here : https://updates.pfsense.org/_updaters/

    latest-nanobsd-2g.img.gz                           25-Jun-2014 21:26            76070249
    

    Seems to get the correct image in my case…

    I tried following in console: this is partial output:

    
    Broadcast Message from root@pfsense.localdomain
            (no tty) at 11:06 CEST...
    
    Installing /root/latest.tgz.
    
    Broadcast Message from root@pfsense.localdomain
            (no tty) at 11:13 CEST...
    
    NanoBSD Firmware upgrade is complete.  Rebooting in 10 seconds.
    
    *** FINAL System shutdown message from root@pfsense.localdomain ***
    
    System going down IMMEDIATELY
    
    pfSense is now shutting down ...
    
    Jun 28 11:13:40 ovpns3: link state changed to DOWN
    lighttpd[53693]:ovpns1: link state changed to DOWN
     (server.c.1558) server stopped pflog0: promiscuous mode enabled
    Bump sched buckets to 256 (was 0)
    Bump sched buckets to 256 (was 0)
    Bump sched buckets to 256 (was 0)
    Bump sched buckets to 256 (was 0)
    Bump sched buckets to 256 (was 0)
    Bump sched buckets to 256 (was 0)
    Bump sched buckets to 256 (was 0)
    Bump sched buckets to 256 (was 0)
    pflog0: promiscuous mode disabled
    Waiting (max 60 seconds) for system process `vnlru' to stop...done
    Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
    Waiting (max 60 seconds) for system process `syncer' to stop...
    Syncing disks, vnodes remaining...0 0 done
    All buffers synced.
    Uptime: 2h5m18s
    usbus0: Controller shutdown
    uhub0: at usbus0, port 1, addr 1 (disconnected)
    usbus0: Controller shutdown complete
    usbus1: Controller shutdown
    uhub1: at usbus1, port 1, addr 1 (disconnected)
    usbus1: Controller shutdown complete
    usbus2: Controller shutdown
    uhub2: at usbus2, port 1, addr 1 (disconnected)
    usbus2: Controller shutdown complete
    usbus3: Controller shutdown
    uhub3: at usbus3, port 1, addr 1 (disconnected)
    usbus3: Controller shutdown complete
    usbus4: Controller shutdown
    uhub4: at usbus4, port 1, addr 1 (disconnected)
    usbus4: Controller shutdown complete
    Rebooting...
    þþþþÿ
    1  pfSense
    2  pfSense
    
    F6 PXE
    Boot:  1
    

    After that, we were back at a dead box.
    Interestingly, I never got the (very brief) serial console again until I cycled the box.
    Could it be there is something wrong with this 2g image? Or am I missing something here?

    For a small version upgrade, this is requiring way more attention then anticipated. ::)


  • Netgate Administrator

    Look at the update files here:
    http://files.bgn.pfsense.org/mirror/updates/

    Now look here:
    https://updates.pfsense.org/_updaters/

    Notice that the standard Nano upodate images are a different size to the VGA updates but at the _upaters URL they are the same (except the 512MB image). It seems likely they are all the VGA images.
    Edit: Finally mangaed to open the files on Android.  ::) Both have the same MD5 and both are decsribed as the VGA update in the MD5 file.

    Go the manual update method by downloadibng the update file manually from the mirrors and uploading that to your box.

    Steve



  • @stephenw10:

    Edit: Finally mangaed to open the files on Android.  ::) Both have the same MD5 and both are decsribed as the VGA update in the MD5 file.

    Go the manual update method by downloadibng the update file manually from the mirrors and uploading that to your box.

    That explains a lot. I've seen your new thread, so I'll hold off a bit with an upgrade, I'm not in a rush and who knows I can help others by testing once the situation is a bit more clear.



  • I'm surprised they haven't fixed this issue yet.  Usually they are very good at fixing upgrade issues.  There is no way of telling they have fixed it or not unless someone keeps trying and be a guinea pig… Who will make the stand?  :D

    EDIT:  Better yet, what is the link to the nano 2gb so I can manually upgrade it?



  • @m4f1050:

    I'm surprised they haven't fixed this issue yet.  Usually they are very good at fixing upgrade issues.  There is no way of telling they have fixed it or not unless someone keeps trying and be a guinea pig… Who will make the stand?  :D

    EDIT:  Better yet, what is the link to the nano 2gb so I can manually upgrade it?

    You can find all neccessary information about the manual update method here: https://forum.pfsense.org/index.php?topic=78713.msg429345#msg429345
    I have meanwhile updated my Alix and my Soekris with this method.

    Regards,
    Peter



  • @pvoigt:

    @m4f1050:

    I'm surprised they haven't fixed this issue yet.  Usually they are very good at fixing upgrade issues.  There is no way of telling they have fixed it or not unless someone keeps trying and be a guinea pig… Who will make the stand?  :D

    EDIT:  Better yet, what is the link to the nano 2gb so I can manually upgrade it?

    You can find all neccessary information about the manual update method here: https://forum.pfsense.org/index.php?topic=78713.msg429345#msg429345
    I have meanwhile updated my Alix and my Soekris with this method.

    Regards,
    Peter

    Actually, just rebooted my firebox and booted it from 2nd mount point (2.1.3), it's the actual image that's bad.

    Where is the NYI mirror link?

    EDIT:  NVM, found it..  https://www.pfsense.org/download/mirror.php?section=updates



  • i upgraded again the alix using console and upgrade from url method and pointed it to a upgrade file on the mirror and all went perfect and everything works fine



  • Success, finally!  Thanks pvoigt for pointing me to the right direction.

    
    Broadcast Message from root@xxx.xxx.net
            (no tty) at 8:50 EDT...
    
    NanoBSD Firmware upgrade in progress...
    
    Broadcast Message from root@xxx.xxx.net
            (no tty) at 8:50 EDT...
    
    Installing /root/firmware.tgz.
    
    Broadcast Message from root@xxx.xxx.net
            (no tty) at 8:55 EDT...
    
    NanoBSD Firmware upgrade is complete.  Rebooting in 10 seconds.
    
    *** FINAL System shutdown message from root@xxx.xxx.net ***
    
    System going down IMMEDIATELY
    


  • @m4f1050:

    I'm surprised they haven't fixed this issue yet.  Usually they are very good at fixing upgrade issues.  There is no way of telling they have fixed it or not unless someone keeps trying and be a guinea pig… Who will make the stand?  :D

    I too find it very strange, not 1 reaction from ESF. One would think that when releasing a new version, there would be some follow-up? Steve even made a separate thread for it….
    No problem testing it again on my test-firebox once there is a new image available....



  • As stated here: https://forum.pfsense.org/index.php?topic=78713.0
    Issue is fixed. I just upgraded my Firebox from 2.1.3 -> 2.1.4, using auto-upgrade (gui-easy-way), and all is fine!

    Thanks for the fix guys! :)