Documenting Package System

[raj2569]

Hi,

I am trying to write a package for clamav and havp. I have setup a test repository locally and trying to understand the manifest and package xml formats. Rather than just ask the questions, I will document what I understand and ask what I do not know, In the process making a crude documentation of package system. I will wikify it  once I get enough information.  Pl correct any errors

pfSense Packaging System

The basic idea behind pfSense packaging system is to install the normal FreeBSD packages, but keep the configuration inside the pfSense xml. This is achieved by creating the config file of application from scratch from xml duing installation and when it is changed in web interface. pfSense provides infrastructure to create the web interface and to store it in it's xml config. The package writer is expected to convert the data from xml to the native format of the application.

Creating a Local Repository.

To test local packages you need to create a local repository of packages. The local repository can be any web server accessible from wan interface with 2 main subdirectories. They are packages and pfSense. packages contain the main package manifest file pkg_config.xml. This also has config subdirectory for package specefic configuration files. The pfSense directory has xmlrpc code. Get the xmlrpc code from http://www.pfsense.com/xmlrpc.tgz and untar it in the directory. To change the default repository directory from pfsense.com to your webserver, edit the /etc/inc/globals.inc and change xmlrpcbaseurl to your repository. Now click the System -> Packages from the pfsense web interface and you should see packages from your local repository. You are set to go now.

Package System

pfSense package are composed of

• Manifest File

• Package Configuration files

• Actual Binaries from FreeBSD

Manifest File

Manifest file is the packages/pkg_config.xml It contains list of packages available for installation and some basic information about packages. The format of manifest xml is as follows:

 <pfsensepkgs><packages><package><name></name>

  <category></category>

  <depends_on_package_base_url></depends_on_package_base_url>
  <depends_on_package></depends_on_package>

  <status></status>
  <maintainer></maintainer>
  <configurationfile></configurationfile>
  <logging><facilityname></facilityname>
   <logfilename></logfilename></logging></package> 
 <packages></packages></packages></pfsensepkgs> 

Package configuration files
Package manifest file for each package specifies a configuration file for each package via config_file tag. The convention is  to keep this file inside the packages/config/ directory. The format is:

 <packagegui><name></name>

	<backup_file></backup_file>
	<aftersaveredirect></aftersaveredirect>

	<configpath></configpath>

<menu>
		<name></name>

		<url>/pkg_edit.php?xml=package.xml&act=edit&id=0</url>
	</menu>

	 <service><name></name>

		<executable></executable></service> 

	 <tabs><tab><active></active></tab></tabs> 

	 <additional_files_needed></additional_files_needed> 

	 <adddeleteeditpagefields><columnitem><fieldname></fieldname></columnitem></adddeleteeditpagefields> 

	 <fields><field><fieldname></fieldname></field></fields> 

	 <custom_php_global_functions></custom_php_global_functions> 

	<custom_php_install_command></custom_php_install_command> 
	 <custom_php_deinstall_command></custom_php_deinstall_command> 

	 <custom_add_php_command></custom_add_php_command> 
	 <custom_add_php_command_late></custom_add_php_command_late> 
	 <custom_delete_php_command></custom_delete_php_command> 
	 <custom_php_resync_config_command></custom_php_resync_config_command> 

	 <start_command></start_command> 

	 <process_kill_command></process_kill_command></packagegui> 

Binaries from FreeBSD
The actual bineries are the normal FreeBSD packages for that particular program. That can be downloaded directly from FreeBSD FTP Server.

Misc

I will expand this as I proceed with my package and understand more about package system, In the mean time pl feel free to add more info as comments.

With Regards,

raj

[raj2569]

Adding my questions here, if some one know the answers to them pl reply.

• what is  the meaning of logging along with facilityname and logfilename in manifest xml

• What is the difference between config_file and configurationfile in manifest xml

• http://pfsense.com/packages/pkg_config.xml has package tag inside packages tab and ouside it. for example spamd comes inside packages tag,
  while pure-ftpd comes directly under pfsensepkgs. What's the difference?

• Which of the tags are mandatory for manifest xml

• Is the version tag in package xml the same as the one in manifest xml?

• What types of file can be included via include_file ? (Only php or another xml file also?)

• How is include_file different from additional_files_needed

• When tabs are used what should be the format of externel xml files that are included?

• How can I display the output of commands executed during installation to the output window?

• How can I make one pfSense package dependent on another pfSense package? For example havp package depends on clamav package, and that package needs to be installed and configured

Apparently this is not possible

[hoba]

Feel free to start a new section "Package Developement" at http://doc.pfsense.com  ;)

[raj2569]

@hoba:

Feel free to start a new section "Package Developement" at http://doc.pfsense.com  ;)

This will go there, if you notice I am updating the first post with new material as soon as I get them. But these needs to be peer reviewd to be posted as "Documentation" :) So post the answers ;)

raj

[lsf]

You might find this interesting if you didn't already see it.

http://wiki.pfsense.com/wikka.php?wakka=CoreGUI

[billm]

@raj2569:

Adding my questions here, if some one know the answers to them pl reply.

• http://pfsense.com/packages/pkg_config.xml has package tag inside packages tab and ouside it. for example spamd comes inside packages tag,
  while pure-ftpd comes directly under pfsensepkgs. What's the difference?

Disabled (non-working generally) packages are placed outside the packages tag.  Only question I know the answer to I'm afraid.  Wish I could get this running on localhost :-/  My box is being pissy with me ATM.

–Bill

[billm]

One note…if using a developers pfSense install to host the package files and the xmlrpc server, you will need to comment out line 394 in xmlrpc_server.inc.

It's the line that reads:
header($this->server_headers);

For some reason it seems to clash with lighttpd and I don't feel like debugging it.  Works fine with Apache apparently (and works fine w/out that header being sent).  With lighty, multiple headers appear to be sent.

--Bill

[raj2569]

@lsf:

You might find this interesting if you didn't already see it.

http://wiki.pfsense.com/wikka.php?wakka=CoreGUI

Thanks for the link.

raj

[raj2569]

if you mess up thepkg_config.xml you will get a totally misleading error when you take System -> Packages. which says to check your network connection.

Unable to communicate to my_server.com. Please check DNS, default gateway, etc.

But this error can also come if your pkg_config.xml  is bad. I have spend the whole day today to track this :(

raj

[raj2569]

http://your_server.com/pfSense/pkg_tester.php is your friend!

raj

[sullrich]

@raj2569:

http://your_server.com/pfSense/pkg_tester.php is your friend!

raj

Say what!?

[raj2569]

@sullrich:

@raj2569:

http://your_server.com/pfSense/pkg_tester.php is your friend!

raj

Say what!?

I was using pkg_tester.php while I was debugging the the problem caused by incorrect pkg_config.xml. The problem exact problem was that I put two log statements in logging section. Like this:

          <logging><facilityname>HAVP Access Log</facilityname> 
                <logfilename>access.log</logfilename>
                <facilityname>HAVP Error Log</facilityname> 
                <logfilename>havp.log</logfilename></logging> 

I was just trying out things by trial and error, and if I put two <facilityname>and <logfilename>inside <logging>the xml_parse($xml_parser, $data, feof($fp))) inside parse_xml_config_raw fails to return. To track this problem I used pkg_tester.php. that's why I said pkg_tester.php is your friend. If some one else is struggling with similar problems again this just gives them one more idea to work with.

raj </logging></logfilename></facilityname>

[sullrich]

For my records in the future:

Warning:  Header may not contain more than a single header, new line detected. in /usr/local/www/pfsense/Etomite0.6/pfSense/xmlrpc_server.inc on line 394

This fixes this issue.  Just ran into this on the pfSense.com website serving packages.

Woops.

[Thowie]

Hello,
I made a local "Package server" on my Network for Develop on packages. I do all this what you write in your config, and it works. But if in the pkg_config.xml a package has more than one <depends_on>lines, like squid or clamav, i became an errormessage on my pfsensebox. He says: "cant connect to xxx.xxx.xxx.xxx. Cached data yoused." If i only use packages that had only one <depends_on>line, all works fine. Why is that so???
The Packageserver is a Apache 2 Server with the latest xmlrpc-code.
Do i somthing wrong???
So please help me, because my dsl-connection somtimes break down for hours. Not good :(
So thanks and a nice day for you…
Bye
Thomas

p.s. sorry, for my worst english ;)</depends_on></depends_on>

[captaintrip]

hi thowie,

you have to checkout the xmlrpc stuff from the cvs (www folder) - the xmlrpc.tgz in the first post is an older version.
then it works with more than one depency per package.

greets
captaintrip

[eri--]

@raj2569:

Adding my questions here, if some one know the answers to them pl reply.

• what is  the meaning of logging along with facilityname and logfilename in manifest xml

• What is the difference between config_file and configurationfile in manifest xml

I cannot tell you a clear answer to this i have to look at the code.

• http://pfsense.com/packages/pkg_config.xml has package tag inside packages tab and ouside it. for example spamd comes inside packages tag,
  while pure-ftpd comes directly under pfsensepkgs. What's the difference?

Some packages are removed from that list to not be allowed to install iirc.

• Which of the tags are mandatory for manifest xml

Only logging/descr is optional, while missing the others might break things.

• Is the version tag in package xml the same as the one in manifest xml?

Yes.

• What types of file can be included via include_file ? (Only php or another xml file also?)

include_file is just for including a file that is needed and it need to be php files.

• How is include_file different from additional_files_needed

additional_files is used during installation/syncing of package while include is for needed dependencies of php code.

• When tabs are used what should be the format of externel xml files that are included?

XML the same as the xml file you use for one package.

• How can I display the output of commands executed during installation to the output window?

There is a global variable $static_output to which you can add.

• How can I make one pfSense package dependent on another pfSense package? For example havp package depends on clamav package, and that package needs to be installed and configured

Apparently this is not possible

Its through additional_files_needed tag but i think it is not really finished or has some culprits to use.
Basically you specify the xml files in that tag that are needed.