Transfer image from one CF to another

mastermindpro · Aug 4, 2006, 6:19 PM

I found the Wiki page for resizing images. I have successfully resized the RC2 image to 128MB using the automatic script (Thanks to the author!).

I've finalized my config on that image, and now I want to extract an image from it so I can write it to ~9 other devices. I tried using RawDisk to read the finalized image back to my hard drive. I then wrote that image back to another CF card using physdiskwrite. There were no errors reported along the way.

I took the CF copy and put it in a WRAP board. It booted and appeared to work just fine until about half-way through the boot process. It got to the point where it was trying to mount /dev/ufs/pfSenseCfg, and then it claimed that didn't exist. The rest of the boot was a series of "thrown rods". ;D

Has anyone else used RawDisk successfully? What's the secret?

sullrich · Aug 4, 2006, 6:59 PM

Take a look at this page for installation help:

http://doc.pfsense.org/index.php/Installing_pfSense

mastermindpro · Aug 4, 2006, 7:57 PM

I wasn't exactly asking about installation. I can do an install just fine, it's just that I haven't been able to extract an image from a functioning install to use to clone a device. Nothing on the install page discusses this.

sullrich · Aug 4, 2006, 8:18 PM

You don't need to extract a functioning image. You can simply start with the images that we posted.

Or maybe I don't really understand. Either way you do know that this is not supported.

mastermindpro · Aug 4, 2006, 8:38 PM

I could start with the provided images, but I'd have to:

Install an expanded image on a CF
Boot it and set the interfaces so I could get to the GUI
Restore settings from a backup of the master system
Change interface parameters after restore
Copy additional files (captive portal) from master system CF to clone CF

If I could read a fully-configured image off of a CF card as the Wiki indicates I can, then I could reduce the above process to this:

Write custom expanded image to CF
Boot it and re-assign interfaces in clone device

While the first method doesn't take too long, I have to do this to 15 devices. That coupled with the fact that my 15 device network is purely a small testbed for a 50-200 device network. As you can see, that'd add up pretty fast.

Besides, why would copying a fully functional image from one CF card to another not be supported? Seems like a no-brainer to me…it's like hard drive cloning. Given the Wiki topic on this is rather small (as it should be), I wouldn't think there's any voodoo going on.

sullrich · Aug 4, 2006, 8:44 PM

Its not supported meaning the developers are not going to go out of our way to make sure that what you are doing works.

We have limited resources and our hands are full already supported the official releases. We just don't have the manpower to support custom environments at the present time.

mastermindpro · Aug 4, 2006, 9:31 PM

I realize all of that, but what makes this a custom environment?

hoba · Aug 4, 2006, 10:22 PM

You somehow want to roll out your own image and that is not supported by the devs. However, maybe somone else can answer this as we don't have time to test this out.

mastermindpro · Aug 4, 2006, 10:30 PM

Then I suggest someone remove the topic "HowTo: Read Flash Image" from the Wiki, since it's clear the devs don't want anyone to do this. :-[

hoba · Aug 4, 2006, 10:39 PM

It's not supported, but anybody can do anything with it. If you want to swim in the deep water you have to know how to swim ;D

mastermindpro · Aug 4, 2006, 11:12 PM

Please mail me a gun so I can shoot myself. Also, please promote use of the gun in your documentation but don't explain it's proper use. Also, when someone asks about proper use of the gun, please inform them they are on their own.

See how frickin' stupid that sounds? Notice any parallels to your statement(s) above? I'm not trying to be mean, I'm trying to point out some easily-resolvable issues with mentioning RawDisk in the Wiki. It only has 3 or 4 command-line options, so it can't be too easy to screw up. :o Read, write, disk enumeration… What's not to understand about that?

If there's something that may (even theoretically) need to be done following such a cloning process, such as change a disk label or something, then clue me in. I'm assuming this CAN work, since it's on the Wiki. I'm looking for those who can help explain what needs to happen to make it work.

So, please don't reply to me if you:

Think I'm stupid
Don't have a solution or any thoughts that might begin to point me in the direction of a solution
Want to respond saying how busy you are
Can't definitively say "this will never work"

Cause guess what? None of those help anyone. If you don't have a solution, don't waste your time replying to me. Please ignore me. I don't want to spoil any more dev time than this thread has already done.

lsf · Aug 5, 2006, 2:11 AM

Free software comes with no implied warranty what so ever. The wiki has tons of experimental stuff, so has HEAD code, if you like to use it, by all means do so. But do not complain about stuff not working, and do not expect others already sharing their work for free, to actually comply to the demands of users !!

A lot of devs feels like, hey I give something away, so that others may benefit from my research/work, and what I get in return is more "work".

Why should we even support pfSense at all ? We do aprechiate bug reports, but lately everything seems to change into demands !

If you really need help then post a bounty and put your money where your mouth is.
I'm sure that your need to put the same setup on 9 boxes means this is somehow connected to a comercial project of some sort, so take the money you saved by getting a free firewall and put some of them up as a bounty and see if you can get help.

mastermindpro · Aug 5, 2006, 2:45 AM

You guys just don't get what I'm trying to say, do you? I ask a few questions, and it devolves into "everyone's picking on me" whining.

I'm not demanding anything, and I never have. The devs are wasting everyone's time by replying "That's not supported" or similar. Whether a dev will support it or not is actually COMPLETELY IRRELEVANT to the issue. I don't expect support from the devs. I post here because this is typically referred to as a "user forum". I'm posting to see if another "user" has done something similar to what I'm trying to do. If none has, then so be it…maybe I'll be the first to post the method here.

Seriously, you devs need to take a chill pill and quit trying to be Superman. You're burning yourselves out. I may post a bounty just so some of you can raise some beer money. ;)

My advice to the devs, if any of them gave a flyin' rip, would be to NOT try to be the pfSense tier 1 support department. Spend your precious time on true problems or issues with the software, not on average joe's like me just trying to get things to work. In short, if it ain't broke, shut up and walk away. Believe me, I mean this with all possible respect.

jeroen234 · Aug 5, 2006, 8:09 AM

on windows xp open the pfsense image with winrar
make youre chanes now close winrar
and burn youre chaned image to cd or cf as normal

rsw686 · Aug 20, 2006, 2:11 AM

Why don't you just use dd to read the image and write it back. Theres a version for windows xp if you need that. I've used it before to backup an image and write it back when I was testing out some things.

Just did a quick google search and this looks like the page I used.

http://www.chrysocome.net/dd

pcatiprodotnet · Oct 13, 2006, 4:47 AM

I use "dd" for Windows to dup[licate] my pfSense CF cards (of same brand/model/size); works great.
(although, it's just as easy to do a pfSense backup, edit the .xml with new IPs, restore into newly flashed CF).

dd –list

My flash card on this PC is always "6", and "6" is the same disk number I use in "physdiskwrite.exe".

Copy from CF to hd...

dd bs=16k if=\?\Device\Harddisk6\Partition0 of=foo.img.gz

Copy from hd to CF...

dd bs=16k if=foo.img.gz of=\?\Device\Harddisk6\Partition0

rsw686 · Oct 5, 2006, 11:08 PM

I have created some scripts that I use to create a custom image to flash to my compact flash card. It starts with the base RC3 release, adds the RC3a, RC3b, RC3c, RC3d patches, replaces the stock config with mine, and adds in miniupnpd.

You have to use it on a freebsd system and I personally use freesbie in vmware. I run the one file, it downloads the files from my webserver and goes through all the steps to create the image. When its done I upload the file and flash it on my windows box with physdiskwrite.

Here are the files. Look in each one for a description of what it does. sh-custom-image.sh calls them all and does the dirty work. If you use miniupnpd you will need to add that cfg-add-miniupnpd to your config file <installedpackages>section to make it work.

Slightly modification of the below files should get you a working way to setup your images.

http://wgnrs.dynalias.com:81/pfsense/sh-custom-image.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-patch.sh
http://wgnrs.dynalias.com:81/pfsense/sh-replace-config.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-miniupnpd.sh

http://wgnrs.dynalias.com:81/pfsense/miniupnpd.tar.gz
http://wgnrs.dynalias.com:81/pfsense/cfg-add-miniupnpd.xml</installedpackages>

sullrich · Oct 5, 2006, 11:10 PM

@rsw686:

I have created some scripts that I use to create a custom image to flash to my compact flash card. It starts with the base RC3 release, adds the RC3a, RC3b, RC3c, RC3d patches, replaces the stock config with mine, and adds in miniupnpd.

You have to use it on a freebsd system and I personally use freesbie in vmware. I run the one file, it downloads the files from my webserver and goes through all the steps to create the image. When its done I upload the file and flash it on my windows box with physdiskwrite.

Here are the files. Look in each one for a description of what it does. sh-custom-image.sh calls them all and does the dirty work. If you use miniupnpd you will need to add that cfg-add-miniupnpd to your config file <installedpackages>section to make it work.

Slightly modification of the below files should get you a working way to setup your images.

http://wgnrs.dynalias.com:81/pfsense/sh-custom-image.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-patch.sh
http://wgnrs.dynalias.com:81/pfsense/sh-replace-config.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-miniupnpd.sh

http://wgnrs.dynalias.com:81/pfsense/miniupnpd.tar.gz
http://wgnrs.dynalias.com:81/pfsense/cfg-add-miniupnpd.xml</installedpackages>

Neat-o ;D

rsw686 · Oct 6, 2006, 12:46 AM

@rsw686:

I have created some scripts that I use to create a custom image to flash to my compact flash card. It starts with the base RC3 release, adds the RC3a, RC3b, RC3c, RC3d patches, replaces the stock config with mine, and adds in miniupnpd.

You have to use it on a freebsd system and I personally use freesbie in vmware. I run the one file, it downloads the files from my webserver and goes through all the steps to create the image. When its done I upload the file and flash it on my windows box with physdiskwrite.

Here are the files. Look in each one for a description of what it does. sh-custom-image.sh calls them all and does the dirty work. If you use miniupnpd you will need to add that cfg-add-miniupnpd to your config file <installedpackages>section to make it work.

Slightly modification of the below files should get you a working way to setup your images.

http://wgnrs.dynalias.com:81/pfsense/sh-custom-image.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-patch.sh
http://wgnrs.dynalias.com:81/pfsense/sh-replace-config.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-miniupnpd.sh

http://wgnrs.dynalias.com:81/pfsense/miniupnpd.tar.gz
http://wgnrs.dynalias.com:81/pfsense/cfg-add-miniupnpd.xml</installedpackages>

Forgot to mention…

The miniupnpd.tar.gz files are the latest files with the 20060924 binary. I tried out the 20060930 binary and it seems to have regressed with errors filling the system log. You can read about here

http://forum.pfsense.org/index.php/topic,551.msg13483.html#msg13483

Miniupnpd uses the miniupnp.sh script to start the service. This file contains your ip address and nic interface hardcoded. Not a big deal as the sync_package_miniupnpd() recreates it. However it doesn't seem to run this when syncing packages at startup. I looked in the code and it looks like the miniupnpd.xml file should have a custom_php_resync_config_command section which calles sync_package_miniupnpd(). I think the thought was that once its configured theres really no need to keep recreating the same startup file.

All I did was go to the miniupnpd settings page and click change (this calles sync_package_miniupnpd) and the service restarted and it worked from then on out. The ideal way, which I do now is to modify the miniupnpd.sh file which is in miniupnpd.tar.gz link above. Find the below line. Change fxp2 to your wan interface name and the 10.10.1.1 to your lan ip address.

killall miniupnpd; /usr/local/sbin/miniupnpd -p 2869 -i fxp2 -a 10.10.1.1

Now when you build and flash your image you can stick it in and your good to go.

rsw686 · Oct 7, 2006, 9:40 PM

@rsw686:

Forgot to mention…

The miniupnpd.tar.gz files are the latest files with the 20060924 binary. I tried out the 20060930 binary and it seems to have regressed with errors filling the system log. You can read about here

http://forum.pfsense.org/index.php/topic,551.msg13483.html#msg13483

Miniupnpd uses the miniupnp.sh script to start the service. This file contains your ip address and nic interface hardcoded. Not a big deal as the sync_package_miniupnpd() recreates it. However it doesn't seem to run this when syncing packages at startup. I looked in the code and it looks like the miniupnpd.xml file should have a custom_php_resync_config_command section which calles sync_package_miniupnpd(). I think the thought was that once its configured theres really no need to keep recreating the same startup file.

All I did was go to the miniupnpd settings page and click change (this calles sync_package_miniupnpd) and the service restarted and it worked from then on out. The ideal way, which I do now is to modify the miniupnpd.sh file which is in miniupnpd.tar.gz link above. Find the below line. Change fxp2 to your wan interface name and the 10.10.1.1 to your lan ip address.

killall miniupnpd; /usr/local/sbin/miniupnpd -p 2869 -i fxp2 -a 10.10.1.1

Now when you build and flash your image you can stick it in and your good to go.

With the recent comitted changes it is no longer necessary to configure the miniupnpd.sh file. It will be automatically created upon first boot. It's as simple as using my scripts with the included files and your good to go.