PfSense in a VMware cloud environment - The power of vmtoolsd & xmllint

  • I want to share my ideas how I intend to use pfSense as a flexible dynamic router in a VMware
    cloud environment. Dynamic injection and provisioning of pfSense is required.
    Here my goals:

    1) Filter outbound connectivity & fence internal zone from other virtual environments
    Use two private VLAN's. The promiscous primary PVLAN-ID for WAN, the community
    secondary PVLAND-ID for LAN. PfSense is managing both side using bi-NAT (1:1) and
    virtual IP configuration. This is another story not explained here, while off-topic.

    2) Rock solid & secure environment
    This is accomplished by booting the Live-CD and a small persistent partition holding config.xml
    and open VM tools.  This sandbox environment is secured by checksums and less vulnerable
    being a read-only environment in volatile memory.

    3) Automated Deployment
    Packed as a OVA file it can be deployed as a single file from a local source or even HTTP, FTP
    repositories into an ESXi, vSphere Center, Workstation environment.
    It has only few files to handle:
    OVA file is a tarred file containing the OVF shipment scheme below:

    • pfSense.ovf - The description file holding configuration information
    • pfSense.iso - The pfSense Live Cd
    • pfSense.vmdk - The persistent disk holding config.xml & open VM tools
    • - The manifest files for checksum & integrity validation

    4) It must configure itself without human intervention
    Here comes the power of vmtoolsd & xmllint.
    vmtoolsd is the vmware guest daemon. it has two fantastic way how to read things.

    a) - vmtoolsd –cmd
    I defined inside a pfSense.vmx file (the VMware configuration file) a key named = "<pfsense>…</pfsense>"

    Few adjustment are needed: remove newlines and create a single config.xml line, change the
    xml-tag from double to single quote (See above) and don't include pictures. I guess the key is a string type with the limit of 65535 bytes. Include only the necessary stuff
    stay below 64 KByte and everything goes smooths.

    Consider also the fact to use the compressing feature of xmllint. This reduce size with the impact
    to have an unreadable transportation form. Create  a compressed config.xml:
    xmllint –compress /conf/config.xml --output /conf/config.gz.xml  or use openssl as an
    encryption platform if you need to obfuscate the pfSense configuration.

    This can be used on single ESXi server non-cloud environment, shipping only the pfSense.vmx,
    pfSense.vmdk and pfSense.iso files and adding the pfSense.vmx file to inventory. This shipment
    scheme is different from the OVA/OVF scheme above.

    b) - vmtoolsd –cmd "info-get guestinfo.ovfenv"
    This is the powerful way to embed the config.xml file inside an OVF descriptor and reading it
    at boot time. Use xmllint to extract key=value pairs (see xmllint explanation below)

    There are two OVF transport mechanism:

    1. Read OVF descriptor with open VM tools (Option 4b)
    2. Include an ISO file with ovf-env.xml file (the OVF descriptor) and rui.crt file (certificate file of ESXi
      server or vSphere Center Server). I prefer Option 4b

    5) Automatic variable extraction & validation

    • xmllint is shipped with pfsense and checks not only the config.xml integrity
      xmllint –valid /conf/config.xml - Test the config.xml

    Here comes the power of xmllint
    xmllint –xpath "//interfaces//lan//ipaddr//text()" /conf/config.xml - Using the
    XPATH functionality you can get everything out of the config.xml file. The above
    statement read the key internal ip address (ipaddr) of the router and return the value
    stored with the key e.g. ""

    Of course some of you will ask me is this enough to run inside a cloud environment? I say no,
    this is not the whole story, whats about the glue code to handle dynamic configuration and
    dynamic injection of config.xml and how can this be implemented?

    I am on the road to fit every thing together one appliance & other tools are needed to build a
    working environment. Stay tuned more to come …

  • Interesting idea gi-minni

    Have you also looked at VMware NSX?

    what do you think your solution would bring to the table over integrating the router/firewall directly into the Hypervisor level?

    maybe you could also look at using pfSense + the vmware API for layer 3+ features from pfSense and turn pfSense into a awesome competitor for the juniper and arista networks gear.

    most of the NSX is based on Nicira's Open vSwitch so here's hoping for compatibility :)

Log in to reply