Transfer image from one CF to another

mastermindpro

You guys just don't get what I'm trying to say, do you? I ask a few questions, and it devolves into "everyone's picking on me" whining.

I'm not demanding anything, and I never have. The devs are wasting everyone's time by replying "That's not supported" or similar. Whether a dev will support it or not is actually COMPLETELY IRRELEVANT to the issue. I don't expect support from the devs. I post here because this is typically referred to as a "user forum". I'm posting to see if another "user" has done something similar to what I'm trying to do. If none has, then so be it…maybe I'll be the first to post the method here.

Seriously, you devs need to take a chill pill and quit trying to be Superman. You're burning yourselves out. I may post a bounty just so some of you can raise some beer money. ;)

My advice to the devs, if any of them gave a flyin' rip, would be to NOT try to be the pfSense tier 1 support department. Spend your precious time on true problems or issues with the software, not on average joe's like me just trying to get things to work. In short, if it ain't broke, shut up and walk away. Believe me, I mean this with all possible respect.

jeroen234

on windows xp open the pfsense image with winrar
make youre chanes now close winrar
and burn youre chaned image to cd or cf as normal

rsw686

Why don't you just use dd to read the image and write it back. Theres a version for windows xp if you need that. I've used it before to backup an image and write it back when I was testing out some things.

Just did a quick google search and this looks like the page I used.

http://www.chrysocome.net/dd

pcatiprodotnet

I use "dd" for Windows to dup[licate] my pfSense CF cards (of same brand/model/size); works great.
(although, it's just as easy to do a pfSense backup, edit the .xml with new IPs, restore into newly flashed CF).

dd –list

My flash card on this PC is always "6", and "6" is the same disk number I use in "physdiskwrite.exe".

Copy from CF to hd...

dd bs=16k if=\?\Device\Harddisk6\Partition0 of=foo.img.gz

Copy from hd to CF...

dd bs=16k if=foo.img.gz of=\?\Device\Harddisk6\Partition0

rsw686

I have created some scripts that I use to create a custom image to flash to my compact flash card. It starts with the base RC3 release, adds the RC3a, RC3b, RC3c, RC3d patches, replaces the stock config with mine, and adds in miniupnpd.

You have to use it on a freebsd system and I personally use freesbie in vmware. I run the one file, it downloads the files from my webserver and goes through all the steps to create the image. When its done I upload the file and flash it on my windows box with physdiskwrite.

Here are the files. Look in each one for a description of what it does. sh-custom-image.sh calls them all and does the dirty work. If you use miniupnpd you will need to add that cfg-add-miniupnpd to your config file <installedpackages>section to make it work.

Slightly modification of the below files should get you a working way to setup your images.

http://wgnrs.dynalias.com:81/pfsense/sh-custom-image.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-patch.sh
http://wgnrs.dynalias.com:81/pfsense/sh-replace-config.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-miniupnpd.sh

http://wgnrs.dynalias.com:81/pfsense/miniupnpd.tar.gz
http://wgnrs.dynalias.com:81/pfsense/cfg-add-miniupnpd.xml</installedpackages>

sullrich

@rsw686:

I have created some scripts that I use to create a custom image to flash to my compact flash card. It starts with the base RC3 release, adds the RC3a, RC3b, RC3c, RC3d patches, replaces the stock config with mine, and adds in miniupnpd.

You have to use it on a freebsd system and I personally use freesbie in vmware. I run the one file, it downloads the files from my webserver and goes through all the steps to create the image. When its done I upload the file and flash it on my windows box with physdiskwrite.

Here are the files. Look in each one for a description of what it does. sh-custom-image.sh calls them all and does the dirty work. If you use miniupnpd you will need to add that cfg-add-miniupnpd to your config file <installedpackages>section to make it work.

Slightly modification of the below files should get you a working way to setup your images.

http://wgnrs.dynalias.com:81/pfsense/sh-custom-image.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-patch.sh
http://wgnrs.dynalias.com:81/pfsense/sh-replace-config.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-miniupnpd.sh

http://wgnrs.dynalias.com:81/pfsense/miniupnpd.tar.gz
http://wgnrs.dynalias.com:81/pfsense/cfg-add-miniupnpd.xml</installedpackages>

Neat-o ;D

rsw686

@rsw686:

I have created some scripts that I use to create a custom image to flash to my compact flash card. It starts with the base RC3 release, adds the RC3a, RC3b, RC3c, RC3d patches, replaces the stock config with mine, and adds in miniupnpd.

You have to use it on a freebsd system and I personally use freesbie in vmware. I run the one file, it downloads the files from my webserver and goes through all the steps to create the image. When its done I upload the file and flash it on my windows box with physdiskwrite.

Here are the files. Look in each one for a description of what it does. sh-custom-image.sh calls them all and does the dirty work. If you use miniupnpd you will need to add that cfg-add-miniupnpd to your config file <installedpackages>section to make it work.

Slightly modification of the below files should get you a working way to setup your images.

http://wgnrs.dynalias.com:81/pfsense/sh-custom-image.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-patch.sh
http://wgnrs.dynalias.com:81/pfsense/sh-replace-config.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-miniupnpd.sh

http://wgnrs.dynalias.com:81/pfsense/miniupnpd.tar.gz
http://wgnrs.dynalias.com:81/pfsense/cfg-add-miniupnpd.xml</installedpackages>

Forgot to mention…

The miniupnpd.tar.gz files are the latest files with the 20060924 binary. I tried out the 20060930 binary and it seems to have regressed with errors filling the system log. You can read about here

http://forum.pfsense.org/index.php/topic,551.msg13483.html#msg13483

Miniupnpd uses the miniupnp.sh script to start the service. This file contains your ip address and nic interface hardcoded. Not a big deal as the sync_package_miniupnpd() recreates it. However it doesn't seem to run this when syncing packages at startup. I looked in the code and it looks like the miniupnpd.xml file should have a custom_php_resync_config_command section which calles sync_package_miniupnpd(). I think the thought was that once its configured theres really no need to keep recreating the same startup file.

All I did was go to the miniupnpd settings page and click change (this calles sync_package_miniupnpd) and the service restarted and it worked from then on out. The ideal way, which I do now is to modify the miniupnpd.sh file which is in miniupnpd.tar.gz link above. Find the below line. Change fxp2 to your wan interface name and the 10.10.1.1 to your lan ip address.

killall miniupnpd; /usr/local/sbin/miniupnpd -p 2869 -i fxp2 -a 10.10.1.1

Now when you build and flash your image you can stick it in and your good to go.

rsw686

@rsw686:

Forgot to mention…

The miniupnpd.tar.gz files are the latest files with the 20060924 binary. I tried out the 20060930 binary and it seems to have regressed with errors filling the system log. You can read about here

http://forum.pfsense.org/index.php/topic,551.msg13483.html#msg13483

Miniupnpd uses the miniupnp.sh script to start the service. This file contains your ip address and nic interface hardcoded. Not a big deal as the sync_package_miniupnpd() recreates it. However it doesn't seem to run this when syncing packages at startup. I looked in the code and it looks like the miniupnpd.xml file should have a custom_php_resync_config_command section which calles sync_package_miniupnpd(). I think the thought was that once its configured theres really no need to keep recreating the same startup file.

All I did was go to the miniupnpd settings page and click change (this calles sync_package_miniupnpd) and the service restarted and it worked from then on out. The ideal way, which I do now is to modify the miniupnpd.sh file which is in miniupnpd.tar.gz link above. Find the below line. Change fxp2 to your wan interface name and the 10.10.1.1 to your lan ip address.

killall miniupnpd; /usr/local/sbin/miniupnpd -p 2869 -i fxp2 -a 10.10.1.1

Now when you build and flash your image you can stick it in and your good to go.

With the recent comitted changes it is no longer necessary to configure the miniupnpd.sh file. It will be automatically created upon first boot. It's as simple as using my scripts with the included files and your good to go.

ZPrime

Your method didn't work for me, I tried to embed my config but it didn't seem to take. I edited in the appropriate XML stuff first and no worky…

Maybe it's because it couldn't find an RC3f patch to apply? All that is on the mirrors is RC3e, at least as of yesterday night.

jeroen234

here is the link to rc3f
http://www.pfsense.com/~sullrich/pfSense-1.0-RC3f-Full-Embedded.tgz

rsw686

@bradenmcg:

Your method didn't work for me, I tried to embed my config but it didn't seem to take. I edited in the appropriate XML stuff first and no worky…

Maybe it's because it couldn't find an RC3f patch to apply? All that is on the mirrors is RC3e, at least as of yesterday night.

RC3f was in the sullrich folder or on my site with the scripts. Although now it doesn't matter as 1.0 is out. Check out the following files they should do what you want.

http://wgnrs.dynalias.com:81/pfsense/sh-replace-config.sh
http://wgnrs.dynalias.com:81/pfsense/sh-add-miniupnpd.sh
http://wgnrs.dynalias.com:81/pfsense/miniupnpd.tar.gz
http://wgnrs.dynalias.com:81/pfsense/cfg-add-miniupnpd.xml

OR

http://wgnrs.dynalias.com:81/pfsense/sh-custom-image.sh

As I rewrote the the above script. It is updated for 1.0-RELEASE, includes error checking, and the ability to specify a config file at the command line. If no config file is specified it will use the default config file with the miniupnpd package section added.

Download file to freebsd machine
chmod +x sh-custom-image.sh
./sh-custom-image.sh
or
./sh-custom-image.sh yourconfig.xml

Try the above and let me know if it doesn't work.

jeroen234

and more for the colection: sh-add-files.sh
this file will read the file filelist and add the files listed there to the image
start the file with sh-add-files.sh pfSense.img
use the complete filepath to the file in file filelist
like this: /bin/routed

sh-add-files.sh.txt
filelist.txt

jeroen234

added some text to sh-add-miniupnpd.sh
it will now check of the config for miniupnp is in the config.xml
if its not it will download the file cfg-add-miniupnpd.xml
and add it to the correct place in config.xml


#!/bin/sh
# This script adds the miniupnpd files to image specified.
# Make sure the miniupnpd.tar.gz file is in the script directory.
#
#
# The miniupnpd.tar.gz file was made by creating a directory named
# miniupnpd. In there should be the following files:
# miniupnpd, miniupnpd.sh, miniupnpd.inc, miniupnpd.xml, status_upnp.php
# Then you execute the command "tar cfz miniupnpd.tar.gz miniupnpd/"
#
# Usage {miniupnpd} {imagefile}
# Example ./sh-add-miniupnpd.sh miniupnpd.tar.gz pfSense.img

TARFILE="$1"
IMGFILE="$2"
WORKDIR=`pwd`
UPNPDIR="$WORKDIR/miniupnpd"
MD=`mdconfig -a -t vnode -f $WORKDIR/$IMGFILE`
mkdir $WORKDIR/a
mount /dev/${MD}a $WORKDIR/a

tar zxf $TARFILE

cp $UPNPDIR/miniupnpd.inc $WORKDIR/a/usr/local/pkg/miniupnpd.inc
chmod 755 $WORKDIR/a/usr/local/pkg/miniupnpd.inc

cp $UPNPDIR/miniupnpd.xml $WORKDIR/a/usr/local/pkg/miniupnpd.xml
chmod 644 $WORKDIR/a/usr/local/pkg/miniupnpd.xml

cp $UPNPDIR/miniupnpd $WORKDIR/a/usr/local/sbin/miniupnpd
chmod 755 $WORKDIR/a/usr/local/sbin/miniupnpd

cp $UPNPDIR/status_upnp.php $WORKDIR/a/usr/local/www/status_upnp.php
chmod 755 $WORKDIR/a/usr/local/www/status_upnp.php
cd $UPNPDIR
echo -n "looking for miniupnp config in config.xml.."
if grep -q miniupnp $WORKDIR/a/conf/config.xml
then echo "..miniupnp config found."
else
echo "..miniupnp config not found."
cp $WORKDIR/a/conf/config.xml $UPNPDIR/test.xml
echo "downloading miniupnp xml config"
fetch -q -o cfg_add_miniupnp.xml  http://wgnrs.dynalias.com:81/pfsense/cfg-add-m
iniupnpd.xml

# the downloaded file has <installedpackages>and</installedpackages> #the test.xml has them to so we remove them from the downloaded file
echo "removing not needed text"
 sed -e '/<installedpackages>/ d' -e '/<\/installedpackages>/ d' cfg_add_miniupn
p.xml >upnp.xml

# now we add upnp.xml to test.xml afther <installedpackages>echo  "adding miniupnp config."

sed '/<installedpackages>/ r upnp.xml' test.xml > output.xml
mv output.xml $WORKDIR/a/conf/config.xml

fi
echo “all done”

rm -rf miniupnpd

umount $WORKDIR/a
rm -rf $WORKDIR/a
mdconfig -d -u ${MD}</installedpackages></installedpackages></installedpackages>

rsw686

@jeroen234:

added some text to sh-add-miniupnpd.sh
it will now check of the config for miniupnp is in the config.xml
if its not it will download the file cfg-add-miniupnpd.xml
and add it to the correct place in config.xml


#!/bin/sh
# This script adds the miniupnpd files to image specified.
# Make sure the miniupnpd.tar.gz file is in the script directory.
#
#
# The miniupnpd.tar.gz file was made by creating a directory named
# miniupnpd. In there should be the following files:
# miniupnpd, miniupnpd.sh, miniupnpd.inc, miniupnpd.xml, status_upnp.php
# Then you execute the command "tar cfz miniupnpd.tar.gz miniupnpd/"
#
# Usage {miniupnpd} {imagefile}
# Example ./sh-add-miniupnpd.sh miniupnpd.tar.gz pfSense.img

TARFILE="$1"
IMGFILE="$2"
WORKDIR=`pwd`
UPNPDIR="$WORKDIR/miniupnpd"
MD=`mdconfig -a -t vnode -f $WORKDIR/$IMGFILE`
mkdir $WORKDIR/a
mount /dev/${MD}a $WORKDIR/a

tar zxf $TARFILE

cp $UPNPDIR/miniupnpd.inc $WORKDIR/a/usr/local/pkg/miniupnpd.inc
chmod 755 $WORKDIR/a/usr/local/pkg/miniupnpd.inc

cp $UPNPDIR/miniupnpd.xml $WORKDIR/a/usr/local/pkg/miniupnpd.xml
chmod 644 $WORKDIR/a/usr/local/pkg/miniupnpd.xml

cp $UPNPDIR/miniupnpd $WORKDIR/a/usr/local/sbin/miniupnpd
chmod 755 $WORKDIR/a/usr/local/sbin/miniupnpd

cp $UPNPDIR/status_upnp.php $WORKDIR/a/usr/local/www/status_upnp.php
chmod 755 $WORKDIR/a/usr/local/www/status_upnp.php
cd $UPNPDIR
echo -n "looking for miniupnp config in config.xml.."
if grep -q miniupnp $WORKDIR/a/conf/config.xml
then echo "..miniupnp config found."
else
echo "..miniupnp config not found."
cp $WORKDIR/a/conf/config.xml $UPNPDIR/test.xml
echo "downloading miniupnp xml config"
fetch -q -o cfg_add_miniupnp.xml  http://wgnrs.dynalias.com:81/pfsense/cfg-add-m
iniupnpd.xml

# the downloaded file has <installedpackages>and</installedpackages> #the test.xml has them to so we remove them from the downloaded file
echo "removing not needed text"
 sed -e '/<installedpackages>/ d' -e '/<\/installedpackages>/ d' cfg_add_miniupn
p.xml >upnp.xml

# now we add upnp.xml to test.xml afther <installedpackages>echo  "adding miniupnp config."

sed '/<installedpackages>/ r upnp.xml' test.xml > output.xml
mv output.xml $WORKDIR/a/conf/config.xml

fi
echo “all done”

rm -rf miniupnpd

umount $WORKDIR/a
rm -rf $WORKDIR/a
mdconfig -d -u ${MD}</installedpackages></installedpackages></installedpackages>

Have you tested this script out?? As you need to mount the other partition in the image as thats where the config is stored. Look at my sh-mount-image.sh or sh-replace-config.sh

http://wgnrs.dynalias.com:81/pfsense/sh-mount-image.sh
http://wgnrs.dynalias.com:81/pfsense/sh-replace-config.sh

Regardless I like the script. Billm had suggested that the script automatically add the required section last night and you beat me to writing it :). I will update mine with the changes. Thanks!

jeroen234

hmm i have only tested the part that add's the miniupnp code to config.xml on my soekris
will look at the replace config file
[edit]
oke the config is on party D not on A

took from the replace config file what was needed


#!/bin/sh
# This script adds the miniupnpd files to image specified.
# Make sure the miniupnpd.tar.gz file is in the script directory.
#
#
# The miniupnpd.tar.gz file was made by creating a directory named
# miniupnpd. In there should be the following files:
# miniupnpd, miniupnpd.sh, miniupnpd.inc, miniupnpd.xml, status_upnp.php
# Then you execute the command "tar cfz miniupnpd.tar.gz miniupnpd/"
#
# Usage {miniupnpd} {imagefile}
# Example ./sh-add-miniupnpd.sh miniupnpd.tar.gz pfSense.img

TARFILE="$1"
IMGFILE="$2"
WORKDIR=`pwd`
UPNPDIR="$WORKDIR/miniupnpd"
MD=`mdconfig -a -t vnode -f $WORKDIR/$IMGFILE`
mkdir $WORKDIR/a
mount /dev/${MD}a $WORKDIR/a

tar zxf $TARFILE

cp $UPNPDIR/miniupnpd.inc $WORKDIR/a/usr/local/pkg/miniupnpd.inc
chmod 755 $WORKDIR/a/usr/local/pkg/miniupnpd.inc

cp $UPNPDIR/miniupnpd.xml $WORKDIR/a/usr/local/pkg/miniupnpd.xml
chmod 644 $WORKDIR/a/usr/local/pkg/miniupnpd.xml

cp $UPNPDIR/miniupnpd $WORKDIR/a/usr/local/sbin/miniupnpd
chmod 755 $WORKDIR/a/usr/local/sbin/miniupnpd

cp $UPNPDIR/status_upnp.php $WORKDIR/a/usr/local/www/status_upnp.php
chmod 755 $WORKDIR/a/usr/local/www/status_upnp.php

cd $UPNPDIR
MD1=`mdconfig -a -t vnode -f $WORKDIR/$IMGFILE`
mkdir $WORKDIR/d
mount /dev/${MD1}d $WORKDIR/d

echo -n "looking for miniupnp config in config.xml.."
if grep -q miniupnp $WORKDIR/d/conf/config.xml
then echo "..miniupnp config found."
else
echo "..miniupnp config not found."
cp $WORKDIR/d/conf/config.xml $UPNPDIR/test.xml
echo "downloading miniupnp xml config"
fetch -q -o cfg_add_miniupnp.xml  http://wgnrs.dynalias.com:81/pfsense/cfg-add-m
iniupnpd.xml

# the downloaded file has <installedpackages>and</installedpackages> #the test.xml has them to so we remove them from the downloaded file
echo "removing not needed text"
 sed -e '/<installedpackages>/ d' -e '/<\/installedpackages>/ d' cfg_add_miniupn
p.xml >upnp.xml

# now we add upnp.xml to test.xml afther <installedpackages>echo  "adding miniupnp config."

sed '/<installedpackages>/ r upnp.xml' test.xml > output.xml
mv output.xml $WORKDIR/d/conf/config.xml
chmod 640 $WORKDIR/d/conf/config.xml
fi
echo “all done”

rm -rf miniupnpd

umount $WORKDIR/a
rm -rf $WORKDIR/a
mdconfig -d -u ${MD}
umount $WORKDIR/d
rm -rf $WORKDIR/d
mdconfig -d -u ${MD1}</installedpackages></installedpackages></installedpackages>

rsw686

Since we are downloading the current config section might as well get the current miniupnpd files as well.


#!/bin/sh
# This script adds the miniupnpd files to image specified.
#
# Usage {imagefile}
# Example ./sh-add-miniupnpd.sh pfSense.img

IMGFILE="$1"
WORKDIR=`pwd`

MD=`mdconfig -a -t vnode -f $WORKDIR/$IMGFILE`
mkdir $WORKDIR/d
mount /dev/${MD}d $WORKDIR/d
mkdir $WORKDIR/a
mount /dev/${MD}a $WORKDIR/a

echo "downloading miniupnp files..."

fetch -q -o $WORKDIR/a/usr/local/sbin/miniupnpd "http://pfsense.com/packages/config/packages/miniupnpd/sbin/miniupnpd"
fetch -q -o $WORKDIR/a/usr/local/pkg/miniupnpd.inc "http://pfsense.com/packages/config/packages/miniupnpd/miniupnpd.inc"
fetch -q -o $WORKDIR/a/usr/local/pkg/miniupnpd.xml "http://pfsense.com/packages/config/packages/miniupnpd/miniupnpd.xml"
fetch -q -o $WORKDIR/a/usr/local/www/status_upnp.php "http://pfsense.com/packages/config/packages/miniupnpd/status_upnp.php"

chmod 755 $WORKDIR/a/usr/local/pkg/miniupnpd.inc
chmod 644 $WORKDIR/a/usr/local/pkg/miniupnpd.xml
chmod 755 $WORKDIR/a/usr/local/sbin/miniupnpd
chmod 755 $WORKDIR/a/usr/local/www/status_upnp.php

echo "looking for miniupnp config in config.xml..."

if grep -q miniupnp $WORKDIR/d/conf/config.xml
	then echo "  miniupnpd config already exists!"
else
	cp $WORKDIR/d/conf/config.xml $WORKDIR/config.xml

	echo "  downloading miniupnp config..."
	fetch -q "http://wgnrs.dynalias.com:81/pfsense/cfg-add-miniupnpd.xml"

	# lets remove <installedpackages>and</installedpackages> 
	sed -e '/<installedpackages>/ d' -e '/<\/installedpackages>/ d' cfg-add-miniupnpd.xml > cfg-miniupnpd.xml

	echo  "  adding miniupnp config..."
	# add miniupnpd section after <installedpackages>sed '/<installedpackages>/ r cfg-miniupnpd.xml' config.xml > $WORKDIR/d/conf/config.xml
	chmod 640 $WORKDIR/d/conf/config.xml

	rm -f cfg-add-miniupnpd.xml cfg-miniupnpd.xml config.xml
fi

echo "miniupnpd successfully added!"

umount $WORKDIR/d
rm -rf $WORKDIR/d
umount $WORKDIR/a
rm -rf $WORKDIR/a
mdconfig -d -u ${MD}</installedpackages></installedpackages></installedpackages>

jeroen234

looking nice
is it me or is the file now less lines ?

rsw686

@jeroen234:

looking nice
is it me or is the file now less lines ?

Yes, there is less lines now. Downloading the miniupnpd files saved space. I consolidated some of the file copying back and forth. Also removed the second mdconfig as it was unneeded. I verified the script works and will be using it to build my images once I modify my other script. Thanks.

shreckbull

maybe, with a "ghost like program" and with IDE to CF adapter you can create a image of the old and you can reinject it on new CF …

OR, with a soft as "winimage" you can create and image from CF in your usb card reader ...
no ?

PS : i use a 1GB CF in a IDE to CF adapter. So, i don't use embedded, but full install from CD Live.

falcon

Heres a tip

what ever you do DONT use ghost!

I just tried making a ghost image, ghost made a 1 gig image from a 1gig cf card, (install wasnt 1gig) i then rebooted and it no longer boots, it just beeps at me at Default: F1

I then tried to ghost on the image i had made and it wont it says theres not enough space

So yeah pretty much a disaster.