Re-implementing the entire backend/frontend of pfSense

MrMoo · Jun 10, 2006, 3:24 AM

I need a PDC/LDAP/TFTPD/PXEboot server for a LTSP project and after not having much luck with flashboot for OpenBSD so I am using pfSense as a starting point. Having following the m0n0wall and pfSense progress I am aware of the need for better separation of the frontend & backend components and am willing to try out some possibilities. As I see it the easiest and free option is to use XML-RPC between a frontend PHP/Lighty webserver and a PHP standalone configuration/automation process.

Is there a reason for sticking to PHP4? With PHP5 I can use SimpleXML for configuration, and the Zend framework to use their XML-RPC and other utilities.

From analysing the rc/rc.boot process it looks like it would be more productive to introduce a private config.xml that handles internal configuration of pfSense minus the high end functionality specified by the current config.xml. For instance this would include the entire init list in rc.bootup, i.e. which processes are started, parameters, etc.

Another possibly controversial area would be splitting the update image into two, one for the core BSD & automation process, and another for the webapp frontend. This could allow different translations on smaller storage units, makes it easier to update the webapp, and re-affirms the separation between frontend & backend.

An XML-RPC process has the benefit of allowing remote management by a central console, but if the overhead is too much I will investigate other options such as JSON, and XDR compact messages. A big question mark remains over delivering of latent messages from callbacks to a busy process.

I see the wiki is rather absent in core details, although Scott Ullrich is fiendishly updating almost everyday the code in CVS. I've started to note on paper instead, I have a blog/wiki too but waiting on some filing paperwork for the domain ::)

sullrich · Jun 10, 2006, 4:18 AM

@MrMoo:

I need a PDC/LDAP/TFTPD/PXEboot server for a LTSP project and after not having much luck with flashboot for OpenBSD so I am using pfSense as a starting point. Having following the m0n0wall and pfSense progress I am aware of the need for better separation of the frontend & backend components and am willing to try out some possibilities. As I see it the easiest and free option is to use XML-RPC between a frontend PHP/Lighty webserver and a PHP standalone configuration/automation process.

I am mighty glad you showed up. We have been discussing this for some time. My wiki page on wiki.pfsense.com has some of my goals for this year and this is part of it.

@MrMoo:

Is there a reason for sticking to PHP4? With PHP5 I can use SimpleXML for configuration, and the Zend framework to use their XML-RPC and other utilities.

We discussed this as well. We want the next version to use php 5.

@MrMoo:

From analysing the rc/rc.boot process it looks like it would be more productive to introduce a private config.xml that handles internal configuration of pfSense minus the high end functionality specified by the current config.xml. For instance this would include the entire init list in rc.bootup, i.e. which processes are started, parameters, etc.

Very good idea. So you could basically decouple the process. This file would be generated after changes? If so, it would be really nice for the system to be able to basically be able to change only the needed bits. For example if the traffic shaper changes, only the necessary rules would change in the file instead of regenerating the entire configuration each time for ALTQ. This would also apply to standard pf rules as well.

@MrMoo:

Another possibly controversial area would be splitting the update image into two, one for the core BSD & automation process, and another for the webapp frontend. This could allow different translations on smaller storage units, makes it easier to update the webapp, and re-affirms the separation between frontend & backend.

Yes, we actually have code in place that seperates the two. We also have code in place for binary diff updates but this code wasn't tested well enough to be included in 1.0. Also, we have been controversial from the start so why stop now :)

@MrMoo:

An XML-RPC process has the benefit of allowing remote management by a central console, but if the overhead is too much I will investigate other options such as JSON, and XDR compact messages. A big question mark remains over delivering of latent messages from callbacks to a busy process.

Yep. Bill had already planned on yanking out the webConfigurator except for XMLRpc and driving the backend from a custom frontend. He's spoken about in in various places but I cannot recall exactly where now. I'm sure he'll be along for the fun soon on this thread.

@MrMoo:

I see the wiki is rather absent in core details, although Scott Ullrich is fiendishly updating almost everyday the code in CVS. I've started to note on paper instead, I have a blog/wiki too but waiting on some filing paperwork for the domain ::)

Heh, why thank you. I need to start working with Chris (cmb) more because I hate docs. He's the doc person :)

BTW: you seem pretty interested and we all seem to be on the same page for the most part. What's it going to take to sucker you into helping us code? :P

PS: I extended the columns on the forum so I could reply to long posts better. Hope nobody dislikes.

billm · Jun 10, 2006, 4:23 AM

@MrMoo:

I need a PDC/LDAP/TFTPD/PXEboot server for a LTSP project and after not having much luck with flashboot for OpenBSD so I am using pfSense as a starting point. Having following the m0n0wall and pfSense progress I am aware of the need for better separation of the frontend & backend components and am willing to try out some possibilities. As I see it the easiest and free option is to use XML-RPC between a frontend PHP/Lighty webserver and a PHP standalone configuration/automation process.

Is there a reason for sticking to PHP4? With PHP5 I can use SimpleXML for configuration, and the Zend framework to use their XML-RPC and other utilities.

From analysing the rc/rc.boot process it looks like it would be more productive to introduce a private config.xml that handles internal configuration of pfSense minus the high end functionality specified by the current config.xml. For instance this would include the entire init list in rc.bootup, i.e. which processes are started, parameters, etc.

Another possibly controversial area would be splitting the update image into two, one for the core BSD & automation process, and another for the webapp frontend. This could allow different translations on smaller storage units, makes it easier to update the webapp, and re-affirms the separation between frontend & backend.

An XML-RPC process has the benefit of allowing remote management by a central console, but if the overhead is too much I will investigate other options such as JSON, and XDR compact messages. A big question mark remains over delivering of latent messages from callbacks to a busy process.

As a really really quick and dirty hack (and a good way to start seperating the frontend from the backend) you can put the webgui on another box and convert write_config() to make an xmlrpc call to the appropriate machine and call the function that saves the configs (don't recall what it is offhand, carpsync does this). Cleaning up the webui to allow for this split would be the direction I'd take. XMLRPC is certainly the right way to approach this IMO.

Also, in HEAD we're using sessions now and we can make use of shared memory in php (I was planning on taking advantage of some of this to remove some of the temp files we create to communicate between php and C).

–Bill

MrMoo · Jun 10, 2006, 4:54 AM

@sullrich:

BTW: you seem pretty interested and we all seem to be on the same page for the most part. What's it going to take to sucker you into helping us code? :P

My company (which is just me :D ) is dedicated to building two PDC and a printer/scanner/fax units before I can work on my real set of products (messaging stuff). I have a stack of various hardware and VMWare Server working busy already. My first step is trying to get a simple image with only networking and SSHD working.

If I really need to I will write a custom messaging core with glib and shared memory/sockets but i'm hoping to delay that and port it in after I have the two platforms in a working state.

The list of features I have planned is grandiose, but I have to keep things in check and actually get something working quickly. For added entertainment I need to get a English/Simplified Chinese/Traditional Chinese GUI working on it too :'(

sullrich · Jun 10, 2006, 5:00 AM

@MrMoo:

My company (which is just me :D ) is dedicated to building two PDC and a printer/scanner/fax units before I can work on my real set of products (messaging stuff). I have a stack of various hardware and VMWare Server working busy already. My first step is trying to get a simple image with only networking and SSHD working.

Interesting. I was just considering wrapping everything with SSH.

@MrMoo:

If I really need to I will write a custom messaging core with glib and shared memory/sockets but i'm hoping to delay that and port it in after I have the two platforms in a working state.

What would this be used for?

@MrMoo:

The list of features I have planned is grandiose, but I have to keep things in check and actually get something working quickly. For added entertainment I need to get a English/Simplified Chinese/Traditional Chinese GUI working on it too :'(

-CURRENT already has been gettext()'d. You're well on your way.

MrMoo · Jun 10, 2006, 5:30 AM

@sullrich:

@MrMoo:

If I really need to I will write a custom messaging core with glib and shared memory/sockets but i'm hoping to delay that and port it in after I have the two platforms in a working state.

What would this be used for?

Replacing "check_reload_status" with a more generic system so that scripts like /etc/rc.linkup end up as simple message sends and then the actual processing of the events is centralised and not subject to parallel running issues. Its more of an enterprise messaging bus way of thinking, so here's a very basic diagram with two system publishers: "rc.linkup" for an interface coming up, and "ps auxw" for a periodic poll providing the current process list.

![](http://fnjordy.umcus.org/tmp/messaging core.png)

sullrich · Jun 10, 2006, 5:31 AM

@MrMoo:

Replacing "check_reload_status" with a more generic system so that scripts like /etc/rc.linkup end up as simple message sends and then the actual processing of the events is centralised and not subject to parallel running issues. Its more of an enterprise messaging bus way of thinking, so here's a very basic diagram with two system publishers: "rc.linkup" for an interface coming up, and "ps auxw" for a periodic poll providing the current process list.

![](http://fnjordy.umcus.org/tmp/messaging core.png)

Beauty!!

MrMoo · Jun 12, 2006, 7:41 AM

Ok, things start to get messy. Here's a simple view of the most basic core:

The FreeBSD "rc" script starts up the "automator" which reads the public & private XML configuration. The private configuration specifies a "sshd" module and loads it in. The module creates a ModuleManager to manage basic state information, and a ProcessManager to look after the actual "sshd" main process.

Taking benefit of PHP5 the configuration is all SimpleXML, this allows trivial XML to Array conversion and very simple xpath load & save. It gets complicated with the concept of multiple configuraiton stores, here listed as runtime, public, and private. The Config class would first inspect runtime, then public, and finally private for any configuration data required. The question here is over the required functionality of a device such as pfSense. Currently with m0n0wall and pfSense there is no such concept of a runtime configuration, i.e. a transient configuration state that is lost when you reboot. At the development level there raises a question of the means of querying for data from the combined repository. While I would like to simply overide [] and -> on a base config object PHP does not allow this in a sufficiently powerful form as required. This means a namespacing solution is required of the form:
[code]
$config['modules.sshd.command'];

Or if a specific tree version is required:


$config->runtime['modules.sshd.command'] = '/sbin/sshd';

I have defined the ModuleManager as a fixed state engine, ideally it should be a base class for a module but I await demands of further integration. The modules themselves are required to implement a defined interface from the "Automator." The ProcessManager uses the popen suite of calls to manage all child processes similar to init on Linux. The goal is to remove the need for periodic 'ps' polling by maintaining the parent child link of each server. In this state stdout & stderr can be recorded for each process in a delayed pipe, and the actual process state can be defined as either not running, running, or trying to terminate. The additional benefit here is the ability to dissassociate the startup and kill penalty of some servers, especiallly those that have turned rogue and require a SIGKILL. Recording a state of "terminate" for a process allows us to check later for whether it has actually died and send further signals as necessary.

There's a question here that does this make syslog redundant, or should the process output be ignored and only syslog / application log file be used.

The next steps on my todo list are:

XML-RPC server in order to initiate:
Configuration or module recycling
Network configuration in order to test a non-developer image

Anything missing here?

MrMoo · Jun 12, 2006, 10:44 AM

Resolved the namespace issue, however a logical error with runtime/public, it needs to be an exclusive or operation in order for items removed from the public configuration to work.

Implementing load with xpath is non-trivial with SimpleXML API I need to investigate the DOM instead. Adding a SimpleXMLElement as a child throws an error when using a reference to keep the head of the XML document :-\

Oh well, you can enjoy the mistakes I made here:

http://fnjordy.umcus.org/tmp/miru-config.tar

MrMoo · Jun 13, 2006, 3:06 AM

In order to make xpathing in and out of the DOM easier I had to add the php5-dom module, but appears to work ok. I haven't implemented creating the parent nodes if they are missing from the destination xpath but not needed yet.

http://fnjordy.umcus.org/tmp/miru-config2.tar

The simpe test for xpathing is:

require_once 'Config.php';

$config = new Config();
$config->load_public('/conf/config.xml');
$config->load_private('/conf/private.xml');

$config->load_public('/conf/sshd.xml', '/miru/modules/sshd');

So this allows you to save and load specific nodes of the configuration, e.g. IPSEC, OpenVPN, traffic shaper and import them to a different configuration.

sullrich · Jun 13, 2006, 3:18 AM

I understand the topic "Re-implementing the entire backend/frontend of pfSense" but you are talking about rewriting large parts of the existing code that can claim a subnode pretty easily as an array, etc?

Just want to try and feel the entire scope of this project. Our hackathon is coming up soon and I need to finalize my list.

Example of getting a xml subnode how it is now:

require("guiconfig.inc");
$a_filter = &$config['filter']['rule'];

After looking at your code, how can it be retrofitted to work with the above example easily?

Or in your example $config is simply $config(an array)? Are there any downsides to this? Speed?

MrMoo · Jun 13, 2006, 4:57 AM

I'm working on process/module control and working out the requirements, firstly I think the big one is dependencies similar to Gentoo's runscript, here from svscan (supervise):


depend() {
  need net
  after net
  before ntpd ntp-client
  before spamd
  before apache apache2
}

The important area of functionality is what is required to reconfigure a process, usually one that cannot be simply SIGHUPed. The process needs to be stopped, the configuration file re-written, and the process started. If this process and dependencies that need this process they need to be stopped & started. And an awkward issue of if it takes a minute to restart an IPSEC connection should you allow other things to occur at the same time.

Here is the basic process state machine:

![](http://fnjordy.umcus.org/tmp/process manager.png)

So to reconfigure something like sshd you have to be aware of the current process state, i.e.


switch (sshd->state) {
case NONE:
case FAILED_TO_START:
case KILLED:
case DIED:
case EXITED:
case STOPPED:
  start sshd;
  break;
case STARTING:
case RUNNING:
  stop sshd;
  on( stop, { start sshd } );
  break;
case STOPPING:
  on( stop, { start sshd } );
  break;
case ZOMBIE:
  exit( "need to reboot to continue." );
}

MrMoo · Jun 13, 2006, 6:35 AM

@sullrich:

Example of getting a xml subnode how it is now:

require("guiconfig.inc");
$a_filter = &$config['filter']['rule'];

After looking at your code, how can it be retrofitted to work with the above example easily?

Or in your example $config is simply $config(an array)? Are there any downsides to this? Speed?

Its a non-trivial re-architecture in order to split functionality at the cost of speed. On the backend the above code would transform into this:


global $config;
$a_filter = $config->filter->rule;

I have absolutely no idea how it would appear on the frontend as a request to the configuration process would be needed to retrieve the data. The benefit is the data is live in the configuration process as opposed to re-read from file every time.

This is basically a big investigation into whether the benefits gained are sufficient to down play the side effects. When looking at a more generic appliance framework it would be nice to have the extra functionality. :)

(edit) The underlying adjustment is making the rather synchronous functionality of the current setup to be asynchronous. Is it possible to make pfSense/m0n0wall more responsive to events that occur when they occur.

MrMoo · Jun 13, 2006, 8:52 AM

@sullrich:

Just want to try and feel the entire scope of this project. Our hackathon is coming up soon and I need to finalize my list.

Sure, considering the duplication of work going on with CVS-head I will start working from the other end and fitting samba, ldap, cups, hylafax, and sane into the current image. By then hopefully you will have a release of your XML-RPC based code and I can help out with that instead.

Project 1: Domain controller for LTSP

Add OpenLDAP server + SASL
Add SAMBA PDC

Project 2: Printer, Scanner, Fax server

Add CUPS, hpijs, hplip, gimp-print, foomatic, foomatic-db, ghostscript
Add Hylafax
Add Sane-backends & Sane-frontends

By basing of pfSense I could remove the routing/firewall specific components and GUI and replace with parts specific to the components for each project. Extra time allows me to update the pfSense framework, and if time becomes short I will simply have to treat each project as a series of regular pfSense packages. 8)

MrMoo · Jun 13, 2006, 10:33 AM

Ok, bad day, neither openldap-sasl-client, openldap-sasl-server, or samba install through freesbie, however cyrus-sasl made it. Hopefully better luck tomorrow. ???

sullrich · Jun 13, 2006, 9:52 PM

@MrMoo:

@sullrich:

Just want to try and feel the entire scope of this project. Our hackathon is coming up soon and I need to finalize my list.

Sure, considering the duplication of work going on with CVS-head I will start working from the other end and fitting samba, ldap, cups, hylafax, and sane into the current image. By then hopefully you will have a release of your XML-RPC based code and I can help out with that instead.

Project 1: Domain controller for LTSP

Add OpenLDAP server + SASL

Add SAMBA PDC

Project 2: Printer, Scanner, Fax server

Add CUPS, hpijs, hplip, gimp-print, foomatic, foomatic-db, ghostscript

Add Hylafax

Add Sane-backends & Sane-frontends

By basing of pfSense I could remove the routing/firewall specific components and GUI and replace with parts specific to the components for each project. Extra time allows me to update the pfSense framework, and if time becomes short I will simply have to treat each project as a series of regular pfSense packages. 8)

This very much falls into my long term goals. Everything you've said so far makes a lot of sense (no pun intended, hah!)

Although all of this is great on paper, it's going to be one hell-of-a-hike to get all of this done. We need a plan of attack so that I can started retrofitting head in this direction.

Any suggestions of how I can help get this started from my end?

edit: minor spelling fixes.

MrMoo · Jun 24, 2006, 3:03 PM

Update: Slight change of track. In order to get something working I tried running with Voyage Linux (Debian/WRAP) and got everything up working in the morning through their Knoppix install; a copy of FreeNAS because i'm waiting back from a Taiwan developer on some NFS issues with a NAS appliance, some slight modifications in order to allow root file permissions; Ubuntu desktop install with LVM and then copy everything over to NAS for a diskless server following on with the thin client setup of LTSP MueKow implementation. Hopefully tomorrow I can test that, see that Chinese input works correctly, and think about moving from VMWare Server to the real hardware boxes, then planning moving back to BSD for the directory server.

I"m off to Korea this week, then onto Macau, so no updates for a couple of weeks :-[

To the left of the layer 4 switch is this stack of hardware, hopefully be useful:

[img]http://fnjordy.umcus.org/tmp/stack-of-boxes.jpg

I guess I should be asking you which area of pfSense would you like to see improved the most?

MrMoo · Jun 24, 2006, 3:13 PM

Oh no, Korea lost to Switzerland. Oh well :'(

I had a quick look at MultiSeat LTSP and with Linux VServer it looks overly complicated. I'm either going to look at web infrastructure for an embedded PDC or try moving from Voyage Linux to the FreeNAS configuration. ???

I like the admin page for WordPress however it doesn't appear easily transferable to the amount of information required with a router or PDC package. Has the core pfSense been updated completely to the new XML GUI-Builder, or is it only packages?

sullrich · Jun 28, 2006, 5:20 PM

Only packages for the most part.

MrMoo · Jun 29, 2006, 1:52 PM

Well I moved over to FreeNAS core and things are working with a web interface now. This gets me a little closer to actually helping out. I have an image with Samba 4, TFTPD, DHCPD booting LTSP server and clients. Here's the TODO list:

:D

Kernel

Remove options / modules not used: need to track down better details of all options.
Add embedded specific modules, performance options, etc.
Create an embedded (no-HDD) kernel, and a generic (IDE & SCSI HDD) kernel.

O/S

Keep tftpboot files over an upgrade, requires different method to gzip|dd for imaging.
Replace OpenSSH with DropBear to save some space (worth the effort?).
Remove PAM (possible?).

Configuration

Private / Public / Runtime configuration split to allow testing new and different options.
Management daemon to co-ordinate system tasks.

Application

Support VLAN interface configuration.
VLAN DHCPD support.
Smaller Samba 4 binaries by removing uneeded modules, there are a lot statically compiled.
Various Windows stuff as I have no experience using AD.

Frontend

i18n/l10n support like pfSenses gettext().
AJAX edits and status feedback like pfSense.
Larger, clearer interface like WordPress, mainly CSS work I would believe.
Administration interface users like pfsense (is there a use with a AD server?).

I'm using Samba 4 to see how stable and big it is, its one process compared to using both OpenLDAP & Samba 3.