Booting pfSense 2.1 cdrom ISO platform with VMware Tools PBI functionality

[gi-minni]

To enhance security in my ESXi infrastructure I love to boot from a read-only environment reducing the persistence layer to a small image having only a few files to manage(1x scriptfile, 1x OVA, 1x config.xml and 1 pbi file). I am just using pfSense since a week so I am a novice regarding pfSense.  pfSense is a great tool I want to support and I love it so much to share some insights with you. This can be only a recipe how to do complex stuff. This implementation need in-deep knowledge how to build a development infrastructure, build a custom layer and add additional stuff not found in vanilla pfSense distribution.

After reading, testing,, probing around a little bit. I got a working and stable environment.  Please find a rough explanation how  I have accomplished this for the 2.1 release. I do not know if this is working in other, I guess (Push Button Installation) PBI would inhibit this.

Here some insights/issues I had to solve myself:

1. ISO has no persistence layer

• pfSense has the capability to move the custom.xml file to a separate partition with (option 98), there create a separate partition for custom.xml.
  I build a 100MB FAT32 partition, rebooted the ISO and executed Option 98. Now every time the system starts the PFI (Pre-Flight Installer) is looking for the additional partition (/dev/da0s1) and mount it under /tmp/mnt/cf. Two other mount points exists /tmp/mnt/cf -> /cf and /tmp/mnt/cf/conf -> /conf.

2. Add custom layer for missing files
   This is a painful way, because you must have some knowledge how to get things running. There is a good explanation in the Dev wiki how to build
   the pfSense layer. What I have missed is the clear explanation how a custom layer works. There are some old examples around (pfDNS, pfFTP, …). You need to adapt it to 2.1. There is currently also a file in the GIT repo that overwrites things. Get prepared to setup your own shell script for building stuff, circumventing errors, etc.

3. ISO has no Open VM Tools installed
   With 2.1 and PBI (Push Button Installation) there is a fantastic and easy way to install software packages without compilation. The problem was the small memory footprint. The ISO works as intended and for testing it and install it on an underlying persistence platform it's ok. In my case I had to do some modifications.

• create e separate /etc/rc.conf.local and change some variables, like tmpsize="500m" and varsize="100m". Tmpsize is important while it will be the base for all the other mount points. Varsize is only important if you need to hold some logs to look for details.
• create near the end of the rc file a user exit like rc.custom_boot_late and execute a script in /conf/scripts/ to load open-vm-tools-nox11-425873_3-amd64-pbi from /conf/repos/. In my case I have a 64bit environment otherwise use i386 instead. Get the file from pfSense files repository. The PBI reside of course in the persistence layer created before. Keep in mind this is ONLY possible AFTER partition detection from PFI!
• If you like add also the Open VM Tools drivers in boot stage to load the VMXNET driver allowing to use VMXNET2 (Enhanced). Look in this forum around to find how to do this.
• If you like add also in the rc.conf.local your favorite keymap "keymap="xxx.iso.kbd"" and call at the end of rc "/etc/rc.d/syscons start" to load your keymap in the late boot process. I love to work from the ssh command line without having the keymap translation in my head :-)

**I hope I get dev people (Chris and others) convinced to expand the ISO standard capability to …:

1. ... change easily tmpsize, varsize, etcsize (there is a GUI way for this but is only defined for embedded platform)
2. ... have a hook to put some PBI images to start in rc.custom_boot_early and late process. I think if a hook like /conf/scripts/rc.custom_boot_late exists it helps to start some extra stuff if needed.**
   Screenshots (Login to view):
3. VMware console view after final boot showing Open VM Tools PBI late install process
4. VMware console view after final boot showing disk layout with mounted /dev/da0s1 persistence
   layer
5. VMware console view after final boot showing running vmware-guest daemon & loaded kernel modules
   
   
   
   
   
   

[doktornotor]

@gi-minni:

1. … change easily tmpsize, varsize, etcsize (there is a GUI way for this but is only defined for embedded platform)

Actually not true with 2.1