Howto reinstall pfSense setup on new HDD smoothly ?

Reiner030

Hi,

pitily there seems no documentation yet for this case.

We use SuperMicro servers with SATA DOM for about a year (and some other since half a year) and every time I make "bigger" writings (updates) on these they crashes…/ lost their SATA DOMs until powered off/on again.

So we try to switch to normal HDDs but I just tried first one and I'm not very glad with moving process.

1) dd memstick image to USB stick
2) setup new HDD into server
3) stick in USB Stick
4) reboot

5) switch to "3 run from US"
6) select "I"nstall
7) setup HDD with new partitions
8) wait for reboot/setup with "empty" image
overgone vlan config, setup "dummy" interfaces for WAN/LAN so that I can access console

9) mount /dev/ad8s1a (old SATA DOM /) to /mnt
10) cp -r /mnt/cf/conf/* /cf/conf/
to get old configs on new HDD
11) remove /root/first_run (or how this file is named^^)
12) reboot

so now my problems begin:

a) ... now it come up and "reinstall" all packages... let me count it... 1 ... :( (OpenVPN TAP fix).
All other packages weren't installed, even I try to reload config over webinterface
(and I dislike to setup several firewalls with a dozen packages manually ^^)

b) ssh key has changed - seems not to saved into config file ?

Is there an overview what files should be copied/removed /touched to get an equal running copy ?

Bests

Reiner

wallabybob

@Reiner030:

pitily there seems no documentation yet for this case.

Did you see http://doc.pfsense.org/index.php/Automatically_Restore_During_Install Is there some reason why you can't apply this document to your circumstances?

Is the size of your new hard drive equal to or greater than the size of the hard drive? If so, you could image clone your old drive onto the new drive and edit /etc/fstab to mount / from the correct drive but if the drives are very much bigger than the size of the files you want copied over (perhaps you have a squid cache you don't care to preserve) this could take a long time.

@Reiner030:

We use SuperMicro servers with SATA DOM for about a year (and some other since half a year) and every time I make "bigger" writings (updates) on these they crashes…/ lost their SATA DOMs until powered off/on again.

Please describe more precisely what you mean by the drives "crash". What updates do you mean - firmware only? What brand and model drives are they?

In your step 10 it should have been sufficient to copy /cf/conf/config.xml to the new drive.

Why did you perform step 11?

stephenw10

An important question here is if you are changing from DOMs to HDs are you also switching install types from 'Nano' to 'Full'?

Steve

Reiner030

@wallabybob:

@Reiner030:

pitily there seems no documentation yet for this case.

Did you see http://doc.pfsense.org/index.php/Automatically_Restore_During_Install Is there some reason why you can't apply this document to your circumstances?

I remember that there was such auto-restore but it was'nt working.

Could be the problem that I forgot the point
On a DOS/FAT formatted USB drive, make a directory called "conf"
since I was assuming that he find his config on SATA dom and reuse it when booting from HDD.

@wallabybob:

Is the size of your new hard drive equal to or greater than the size of the hard drive? If so, you could image clone your old drive onto the new drive and edit /etc/fstab to mount / from the correct drive but if the drives are very much bigger than the size of the files you want copied over (perhaps you have a squid cache you don't care to preserve) this could take a long time.

Yes (320 GB >> 4 GB) but since the system on SATA DOM has failures I don't wont clone it ;)

@wallabybob:

@Reiner030:

We use SuperMicro servers with SATA DOM for about a year (and some other since half a year) and every time I make "bigger" writings (updates) on these they crashes…/ lost their SATA DOMs until powered off/on again.

Please describe more precisely what you mean by the drives "crash". What updates do you mean - firmware only? What brand and model drives are they?

You can see the crash picture attached which I still found - I've always reported the crash reports to pfsense dev per gui.
We use
2x SuperMicro X8SCM-F (without KVM, onboard controller used)
6x SuperMicro X9SCM-F (with KVM, onboard controller used)

There are often timeouts… which sounds like this problem
http://freebsd.1045724.n5.nabble.com/ahcich-Timeouts-SATA-SSD-td5752011.html

(and we have several times full SATA DOMs disconnects - often BIOS /boot helped but sometimes I must power off/on the server to find it again)

@wallabybob:

In your step 10 it should have been sufficient to copy /cf/conf/config.xml to the new drive.

Why did you perform step 11?

mmh, that was not working as written,

But I would try it then with config on USB stick fopr next "updates" (from last of the 2.1-BETA1 relased to the RC0 relase).

Step 11 - removing file first_run was because I hoped it did not try the automatic setup/question me what to do.

@stephenw10:

An important question here is if you are changing from DOMs to HDs are you also switching install types from 'Nano' to 'Full'?

Steve

SATA DOM is equal to HDDs so we used memstick installation before.

Bests

Reiner

gw1-jws1-crash-after-bgp-problem-sata-timeout.jpg_thumb

crash-2013-05-08-gw1-zws8.jpg_thumb

crash_fw1-zws8.jpg_thumb

stephenw10

Running a full install on a DOM, which is flash media, may have damaged it if you're running packages etc. That could be what is causing your crashes.
That doesn't help you here though.

Steve

Reiner030

@stephenw10:

Running a full install on a DOM, which is flash media, may have damaged it if you're running packages etc. That could be what is causing your crashes.
That doesn't help you here though.

Steve

mmh, I don't think so (other admin is initally setup/administrating it… I mostly do setup for BGP/OSPF and like to play with 2.1 BETAs for IPv6 support;))

SMART status says on all devices: OK (even with intensive tests).
such dongle works on a DELL server with VMware ESX setup very nicely.
Fresh setups of Firewalls/Gateways with "very new hardware" had this problem after a few updates of pre-Beta versions without logging much onto SATA DOMs.
This was the reason I have to discuss over half a year that the SATA DOM must be damanged / not functional with FreeBSD/Supermicro board combination.

So... now my actual upgrade Problem to HDDs...
Now I've tried/done it as written in
http://doc.pfsense.org/index.php/Automatically_Restore_During_Install

but same problem as my manual installation under 2.0.3 ...
a) It "reinstalls" all packages by deleting them only :(
b) it changes ssh keys of firewall when setting up new system.
but rest of system came up nicely quick configured.

NEW (now tested first):
c) With "pfSense-memstick-2.1-RC0-amd64-20130614-0450.img.gz" I came only to the installer menue to setup video display.
After selecting "Accept these settings" I got blue screen (with <f10 menue="">bar at top) and nothing happens anymore
(even plug-in/out messages from system itself). :( If someone is interested / if it's helpful I've made a video snapshot of boot
sequence (15 MB AVI file).
d) interesting "featured" problem: I can do what I want selecting boot order / disabling SATA DOM in boot order/HDD recognition
(without removing SATA DOM physically)... the newer SuperMicro board with 2.1-RC0 always start from my SATA DOM instead from HDD ???

so... time for bed ;)

Bests

Reiner</f10>

wallabybob

@Reiner030:

but same problem as my manual installation under 2.0.3 …
a) It "reinstalls" all packages by deleting them only :(
b) it changes ssh keys of firewall when setting up new system.
but rest of system came up nicely quick configured.

a: a fresh install starts with no packages, but if it truly inherits the previous configuration file from a correctly formatted USB stick I would have thought it would force the necessary installs BUT perhaps that failed (due to inability to access the internet?).
b: The config.xml file is supposed to hold all the system configuration information. I would expect expect that to include keys for ssh server but not necessarily any client keys. A quick scan of a 2.0.1 config.xml file didn't show up anything I recognised as ssh server keys. What did you see that lead you to conclude ssh keys were changed?

@Reiner030:

If someone is interested / if it's helpful I've made a video snapshot of boot sequence (15 MB AVI file).

How about putting it in a public place and posting a link to it?

@Reiner030:

d) interesting "featured" problem: I can do what I want selecting boot order / disabling SATA DOM in boot order/HDD recognition
(without removing SATA DOM physically)… the newer SuperMicro board with 2.1-RC0 always start from my SATA DOM instead from HDD ???

Seems like a BIOS problem UNLESS the hard drive is not bootable (doesn't have a slice marked bootable) and so its going to the DOM which is bootable.

Reiner030

@wallabybob:

@Reiner030:

but same problem as my manual installation under 2.0.3 …
a) It "reinstalls" all packages by deleting them only :(
b) it changes ssh keys of firewall when setting up new system.
but rest of system came up nicely quick configured.

a: a fresh install starts with no packages, but if it truly inherits the previous configuration file from a correctly formatted USB stick I would have thought it would force the necessary installs BUT perhaps that failed (due to inability to access the internet?).
b: The config.xml file is supposed to hold all the system configuration information. I would expect expect that to include keys for ssh server but not necessarily any client keys. A quick scan of a 2.0.1 config.xml file didn't show up anything I recognised as ssh server keys. What did you see that lead you to conclude ssh keys were changed?

Thanks for the ideas… but I knew what I do (normally ;))

a) The gateway has clearly access to internet - before update and just after update.
It just not reinstalls the package from list (even not manually as written in first message of this thread)...
Manually it works fine.

b) Missing SSH Host Keys is more a bug/feature request to include it in config.xml, too...
How I found the change... good question ;)

reiner.keller@rkeller-thinkstation:~$ ssh 192.168.45.2
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
36:08:9b:b4:34:a1:56:28:61:87:18:a7:bc:68:77:5f.
Please contact your system administrator.
Add correct host key in /home/reiner.keller/.ssh/known_hosts to get rid of this message.
Offending key in /home/reiner.keller/.ssh/known_hosts:144
RSA host key for 192.168.45.2 has changed and you have requested strict checking.
Host key verification failed.

@wallabybob:

@Reiner030:

If someone is interested / if it's helpful I've made a video snapshot of boot sequence (15 MB AVI file).

How about putting it in a public place and posting a link to it?

ok… let's search ... a I have a dropbox account for testing only ^^...
https://www.dropbox.com/s/u0at0ccn2na4ng1/boot.avi

@wallabybob:

@Reiner030:

d) interesting "featured" problem: I can do what I want selecting boot order / disabling SATA DOM in boot order/HDD recognition
(without removing SATA DOM physically)… the newer SuperMicro board with 2.1-RC0 always start from my SATA DOM instead from HDD ???

Seems like a BIOS problem UNLESS the hard drive is not bootable (doesn't have a slice marked bootable) and so its going to the DOM which is bootable.

mmh, should be bootable (I gone through hole installation procedure when setup HDD).
I would try it next days again. Problem is only that these task can be done only at "night" .

Bests

Reiner

Reiner030

So… today it crashed again - with still running SATA DOM .
I have the problem on this server that pfSense only boot to SATA DOM even when I select F5 for "Drive 1" and then (let) F1 for "pfSense"...

Attached the crash "dump" to screen as image.
And here the output after reboot (and upgrade to RC0 - before there was Image from April 22th running):

[2.1-RC0][root@gw1.jws1.local]/root(1): atacontrol list
ATA channel 2:
Master: ad4 <atp 20111212="" ig="" sata="" dom="">SATA revision 2.x
Slave: no device present
ATA channel 3:
Master: ad6 <wdc wd3200bekt-08pvmt1="" 02.01a02="">SATA revision 1.x
Slave: no device present

To the remark of
@stephenw10:

Running a full install on a DOM, which is flash media, may have damaged it if you're running packages etc. That could be what is causing your crashes.
That doesn't help you here though.

Steve

dan be disagreed by also following answer from these days:

@wallabybob:

DOMs usually have flash with many more write cycles than consumer grade flash intended for cameras, phones etc. Therefore you can run the full version of pfSense on them. I have run my home pfSense from the same 1GB DOM for over 4 years.

If you want to run NanoBSD why pay extra for a DOM when a cheap CF card would do?

But if you really want to run the nanoBSD version off a DOM then the flash procedure would be essentially the same as for a CF except you would need to connect the DOM to the motherboard on the appropriate socket (SATA or IDE) for the DOM and take care that you specify the destination device correctly. The exact details will depend a bit on what equipment you have available for flashing the DOM.

Bests

crash-gw1-jws1-05-06-13.jpg_thumb </wdc></atp>

kejianshi

Bad news and good news.

First the bad news. I have had same experience as you most of the time. I back up my config, wipe a drive, install pfsense from live disk, restore my config and several of my packages are missing after the reboot.

Now the good news. The settings are still there. Just install the missing packages from package manager and they will most like be configured exactly as they were before you changed drives. This has been the case with me 100% of the time.

wallabybob

The report before the Fatal Trap 12 suggests your drive ad4 went on an extended vacation (well more than 15 seconds) in which it refused to respond to the host controller. I remember Anandtech reporting some early SSDs (Solid State Disks) did this sort of thing and it was suspected they were wear leveling. That the crash happened after a write timeout failure suggests you might be seeing something similar. If that is the case it should be possible to clone the drive under another OS: while it is not being written to it shouldn't be attempting wear leveling. Thus (assuming the DOM isn't bigger than the hard drive) you should be able to boot a Linux Live USB stick (say Ubuntu) and dd the DOM to the hard drive, switch the DOM and hard drive connectors (so the hard drive becomes ad4) and boot into pfSense.

Reiner030

1) missing reinstall of packages:
@kejianshi:

Now the good news. The settings are still there. Just install the missing packages from package manager and they will most like be configured exactly as they were before you changed drives. This has been the case with me 100% of the time.

yes, clear, but its not practical to install several packages by hand on 2 firewalls.
And the pfsense is not functional but take over master when it got up… that's not nice.

Yesterday night I setup next 2 firewalls (still one gateway is missing) ...
I saw again in console that the installer try to reinstall all packages.
Because I take a more intensive look onto it I think I found an Interesting behavior.

It seems that the "reinstall package" part is called 3-5 times.
Weeks before I thought the lines are for each package install (when not looking intensive on it because I sit in other room then firewalls)

=> Could it be that the installer tries to remove the packages... fails, because they are not installed ...
And therefore didn't try to install them "again" ?

Pitily there is no /conf/upgrade_log.txt (or /cf/conf/upgrade_log.txt) written so that I can support here with more informations.

Device Mountpoint FStype Options Dump Pass#

/dev/ad6s1a / ufs rw 1 1
/dev/ad6s1b none swap sw 0 0</wdc></atp>

Reiner030

@Reiner030:

It seems that the "reinstall package" part is called 3-5 times.
Weeks before I thought the lines are for each package install (when not looking intensive on it because I sit in other room then firewalls)

=> Could it be that the installer tries to remove the packages… fails, because they are not installed ...
And therefore didn't try to install them "again" ?

Pitily there is no /conf/upgrade_log.txt (or /cf/conf/upgrade_log.txt) written so that I can support here with more informations.

mmh, I try to cut/clean the video I made of it with our last firewall switch…

I found the log in System Logs => System for package updating process in the other firewalls
(the FTP crash bug needs update to latest firmware) but it's still not very helpful when setting up new install with HDD:

Jul 9 22:05:18 sshlockout[77839]: sshlockout/webConfigurator v3.0 starting up
Jul 9 22:05:18 login: login on ttyv0 as root
Jul 9 22:05:14 php: rc.start_packages: The System Patches package is missing its configuration file and must be reinstalled.
Jul 9 20:05:12 check_reload_status: Syncing firewall
Jul 9 22:05:12 php: rc.start_packages: The bacula-client package is missing its configuration file and must be reinstalled.
Jul 9 22:05:10 php: rc.start_packages: The Cron package is missing its configuration file and must be reinstalled.
Jul 9 22:05:08 php: rc.start_packages: The pfBlocker package is missing its configuration file and must be reinstalled.
Jul 9 22:05:06 kernel: <<118>
Jul 9 22:05:06 php: rc.start_packages: The OpenBGPD package is missing its configuration file and must be reinstalled.
Jul 9 22:05:04 php: rc.start_packages: The mtr-nox11 package is missing its configuration file and must be reinstalled.
Jul 9 20:05:02 check_reload_status: Syncing firewall
Jul 9 22:05:02 php: rc.start_packages: The iperf package is missing its configuration file and must be reinstalled.
Jul 9 22:04:59 php: rc.start_packages: The nut package is missing its configuration file and must be reinstalled.
Jul 9 22:04:59 php: rc.start_packages: Restarting/Starting all packages.
Jul 9 22:04:59 syslogd: kernel boot file is /boot/kernel/kernel

@Reiner030:

3) boot to SATA DOM instead of HDD
=>
[2.1-RC0][root@gw1.jws1.local]/root(4): mount /dev/ad6s1a /mnt
[2.1-RC0][root@gw1.jws1.local]/root(5): vi /mnt/etc/fstab
[2.1-RC0][root@gw1.jws1.local]/root(6): cp -p /conf/config.xml /mnt/conf/config.xml

(change there the 4 to 6 )… => fixed ;) ... I would try it then this night.

[2.1-RC0][root@gw1.jws1.local]/conf(7): cat /mnt/etc/fstab

Device Mountpoint FStype Options Dump Pass#

/dev/ad6s1a / ufs rw 1 1
/dev/ad6s1b none swap sw 0 0

was not very helpful. It really boots from SATA DOM even I select HDD onto loader…

=> additional fix: modified /etc/fstab on SATA DOM to same mount points ;D

wallabybob

@Reiner030:

was not very helpful. It really boots from SATA DOM even I select HDD onto loader…

The BIOS boots from ad4? The BIOS boots from ad6 to FreeBSD initial bootloader which then loads kernel from ad4?

You should be able to remove ad4 (SATA DOM) so only ad6 is available.
It is possible that (due to some limitation in the early boot code or BIOS) you can boot from ad4 but not from ad6. In that case you should be able to move the hard drive to the ad4 connector (so that FreeBSD names it ad4), edit /etc/fstab to change ad6 to ad4 and have everything work.

Reiner030

@wallabybob:

@Reiner030:

was not very helpful. It really boots from SATA DOM even I select HDD onto loader…

The BIOS boots from ad4? The BIOS boots from ad6 to FreeBSD initial bootloader which then loads kernel from ad4?

You should be able to remove ad4 (SATA DOM) so only ad6 is available.
It is possible that (due to some limitation in the early boot code or BIOS) you can boot from ad4 but not from ad6. In that case you should be able to move the hard drive to the ad4 connector (so that FreeBSD names it ad4), edit /etc/fstab to change ad6 to ad4 and have everything work.

no thats not a problem… in BIOS I selected the HDD as boot device but it make no difference if I boot them from "Drive 0" or "Drive 1" ... on both settings it boots from SATA DOM... why ever.

Removing SATA DOM is not so easy... a full rack, and a small server room... I dislike to try it "alone in the dark" ;)
But this way it's working till we have in the next month time for removing SATA DOMs ^^

Reiner030

@Reiner030:

mmh, I try to cut/clean the video I made of it with our last firewall switch…

Gateway just crashed because of TRAP 12 bug…. The good thing: I found in crashdump the console output while re-installing package ;)

<118>
<118>
<118>
<118>
<118>
<118>
<118>done.
<118> Starting package iperf...
<118>
<118>
<118>
<118>
<118>
<118>
<118>done.
<118> Starting package OpenBGPD...
<118>
<118>
<118>
<118>
<118>
<118>
<118>done.
<118> Starting package pfBlocker...
<118>
<118>
<118>
<118>
<118>
<118>
<118>done.
<118> Starting package Cron...
<118>
<118>
<118>
<118>
<118>
<118>
<118>done.
<118> Starting package bacula-client...
<118>
<118>
<118>
<118>
<118>
<118>
<118>done.
<118> Starting package System Patches...
<118>
<118>
<118>
<118>
<118><118>
<118>
<118>done.
<118>Bootup complete