new package / best practices

mmetc

Hello!

We at CrowdSec are preparing a first release of a pfSense package.

The basic functionality is pretty much ok, but we have a few doubts regarding integration and packaging. We already did an opnsense version but some concepts don't translate directly, of course.

The "pfSense-pkg-crowdsec" package requires the preexisting freebsd packages "crowdsec" (threat detection and sharing with community) and "crowdsec-firewall-bouncer" (remediation). The latter needs two pf tables and related rules for ipv4/ipv6 addresses. So far so good, upon install we register them to config.xml and all the parts are working.

The package also has a settings page, which is used to (re)configure the other two crowdsec* packages and possibly start or stop them.

If this is accepted as upstream, the packages crowdsec and crowdsec-firewall-bouncer will need to be built in the pfSense repository but it's not an issue at the moment.

Now the questions

1 - we have YAML configuration files pretty much for everything. To edit them via php, we found no better way than to embed symfony/yaml (vendored with composer). This can create some incompatibility with future pfSense releases, but is there a better way?

2 - service management: the pfSense-pkg-crowdsec package should start/stop the others according to their "enabled" values on the settings page. But as far as I see, we are discouraged from using "service crowdsec enable" to make it stick, which would make us resort to "onestart/onestop" but my understanding may be way off here.

3 - we keep persistent data in /var/db/crowdsec, which does not work well when /var is on a RAM disk. Should we advise users to disable the RAM disk (that's what we do now) or would it make sense to move the persistent data somewhere else (/root for example, or shiver /conf). If the ram disk was for /var/log only it would not be an issue, but we really need to keep a small database somewhere.

For reference, the (still pre-beta) package is at https://github.com/crowdsecurity/pfSense-pkg-crowdsec. We hope to have a testable version in a few days and fixing the above issues would certainly help. Other minor issues can wait.

Thanks!

bmeeks

I'm not an employee of Netgate, but I do maintain the Snort and Suricata packages for pfSense. I created the Suricata package from scratch and picked up maintenance of the existing Snort package. I will try to answer some of your questions.

First of all, in case you have not already, you should review the offical Netgate docs here: https://docs.netgate.com/pfsense/en/latest/development/develop-packages.html. There are two main types of GUI packages on pfSense. One makes use of the XML Template format described in the link docs. The other is more free-form PHP. There is not a lot of great documentation for the free-form variety. Learning by looking at others' package code is perhaps the best teacher. Links to some of that are included farther down below.

There are some "hooks" that the package installation process in pfSense will honor so that you can perform certain actions within your package code during post-install, uninstall, and package sync. These hooks are called, when present in your package, by pfSense during certain system operations.

Here are some more specific answers to your questions.

On pfSense, all configuration information is expected to reside in the config.xml file for the firewall. That file resides in /conf/. There is a section within the file for installed packages, and each installed package has its own subsection within the larger packages section. All the user configuration values for your program should reside in the config.xml file. Storing things on the filesystem in custom files is not good because those will NOT be backed up as part of a routine configuration backup by pfSense.
For configuration files specific to your application (for example, the YAML file), most of us package maintainers read data from the appropriate section of config.xml and then format it and write it to the appropriate application conf file on the filesystem. In the pfSense world, pretty much everything lives in config.xml, and local conf files for applications and services are recreated on-the-fly when those applications and services are started. Any edits done directly on the filesystem by a user will be overwritten the next time the package is stopped and restarted. The Suricata package needs a suricata.yaml conf file for each configured interface. That file is created as part of starting the package by reading the parameters from config.xml and then formatting them into the required YAML conf file for the Suricata binary to ingest. If you install the Suricata package, you can examine how this is done. The main code is in file suricata_generate_yaml.php. That code makes use of a matching template file named suricata_yaml_template.inc. These files can be found in /usr/local/pkg/suricata/ once the Suricata package is installed on the firewall. You can also inspect them here on GitHub: https://github.com/pfsense/FreeBSD-ports/tree/devel/security/pfSense-pkg-suricata/files/usr/local/pkg/suricata. This will give you an idea of how things are done.
For packages that have underlying binary services that need to be stopped/started by pfSense as part of its shutdown or boot up, you create a shell script in /usr/local/etc/rc.d/ with a name that matches your package application. That shell script should include code for handling command-line stop, start, and restart commands. pfSense will then call that shell script at the appropriate times and with the correct argument during system shutdown, start-up, or reboot. The shell script, if used, is specified in the package manifest file that must be created as part of the overall package.
Your package should be built by Netgate on their systems and stored in their signed repository. That will make things vastly easier, secure, and reliable for your package users. To get your package in the Netgate/pfSense repo, submit a Pull Request here: https://github.com/pfsense/FreeBSD-ports. Place your package in the appropriate sub-directory (likely "security" for CrowdSec).
RAM Disks are a real bear. Not only do users tend to make them small, but any data stored there is wiped out upon a system restart. But if you store data in config.xml (as Base64 encoded strings for anything that might be "binary" in nature), it will be persisted across reboots. You would simply read the data from config.xml at startup and write it back to system files if you must. But it's better to get accustomed to keeping stuff in config.xml.

mmetc

@bmeeks thanks a lot for the reply!

1/5) I am familiar with the configuration workflow. I've been maintaining a similar plugin, although not for pfsense, for some time and the platform has the same config.xml logic as pfSense. Our users need to be able to extend the configuration and apply the concepts they are familiar with from the Linux version. A comprehensive plugin like you describe is just not possible with the current resources and pace of development, and likely need non-trivial changes to the main application too.

The persistent data is required but does not have to be maintained for the long term. Even if the database does not survive a major upgrade and needs to be recreated, it's still preferable than adding complexity or locking down the configuration options. A the moment I'm telling users that they can disable the RAM disk, or use a second machine for the persistent data. I will explicitly warn that a reconfiguration is required in case of backup/restore.

I understand that most packages don't need to edit yaml because they write it from scratch. But if we want to, we have no option to include a yq parser and there is no php package manager in pfsense. Maybe a dependency on go-yq that we can call from php would be acceptable? That has been a life saver in shell scripts and cron jobs too.
I'm not sure if I need /usr/local/etc/rc.d/pfsense-crowdsec.sh when I already have rc.d/crowdsec and rc.d/crowdsec_firewall, that come from the respective packages. Thanks, I'll test and try to keep it as simple as possible.
yes, we already have a repository at https://github.com/crowdsecurity/pfSense-pkg-crowdsec and are polishing before having a few weeks of beta

Thanks again

bmeeks

@mmetc said in new package / best practices:

Maybe a dependency on go-yq that we can call from php would be acceptable? That has been a life saver in shell scripts and cron jobs too.

You can put runtime library dependencies in the Makefile for your package and the pkg utility used to install packages will resolve them when installing your package. Any dependent library packages must be in the same pfSense repo. But pretty much anything currently in the FreeBSD Ports tree can be "activated" on pfSense. The only caveat would be some package that is known to be incompatible with an existing library, but that should be very rare.

@mmetc said in new package / best practices:

I'm not sure if I need /usr/local/etc/rc.d/pfsense-crowdsec.sh when I already have rc.d/crowdsec and rc.d/crowdsec_firewall, that come from the respective packages. Thanks, I'll test and try to keep it as simple as possible.

I think the /usr/local/etc/rc.d/ path is currently hard-coded into the parts of pfSense that start packages at boot or when executing a "restart all packages" command. I do not believe it will find a script in /etc/rc.d/. At least not when doing things from the GUI. Of course you could just use a simple one-liner shell script in /usr/local/etc/rc.d/ that simply calls the other script in /etc/rc.d/.

I'm tagging @jimp in this thread. He is a Netgate developer and may have some helpful input.

mmetc

@bmeeks said in new package / best practices:

I think the /usr/local/etc/rc.d/ path is currently hard-coded into the parts of pfSense that start packages at boot or when executing a "restart all packages" command. I do not believe it will find a script in /etc/rc.d/.

Of course, sorry, I meant /usr/local/etc/rc.d/crowdsec and crowdsec_firewall. Meaning I don't need to wrap them from another script. They are supposed to work in vanilla freebsd after all. I can just enable them in /usr/local/etc/rc.conf.d/*

I'm tagging @jimp in this thread. He is a Netgate developer and may have some helpful input.

Great, thanks