Feedback on challenges with the pfSense package development process

[andrew_cb]

I'm posting here, but if pfSense Packages is a better place, feel free to move.

There seem to be challenges with the timeliness and update of existing packages and the adoption of new packages for pfSense. Notably, Zabbix 7.0 was released over five months ago, and the target pfSense release has slipped several times and is now at 25.01. It will likely be several more months before it becomes available.

Third-party packages are a major selling point of pfSense, and the homepage of netgate.com boasts:
"Highly extensible with 3rd party packages to support block lists, content filtering, intrusion prevention, policy-based routing and more".

The significant delays in releasing updated packages are at odds with this.
This isn't an issue of Plus vs. CE - all these delays are all with pfSense Plus, so free vs. paid has no direct influence.

Users compiling and updating packages on their own seems to be a Herculean effort, as demonstrated in this post for compiling miniupnpd:

https://forum.netgate.com/topic/169749/pfsense-compile-requirements-for-3rd-party-software/101

Among the issues:

• complexities with Poudriere jails
• dependencies on private Gitlab files and libraries
• numerous "mystery" values that need to be changed

It boils down to Are packages a core feature of pfSense, and Netgate is committed to keeping them up-to-date, or are packages a burden that gets updated when there's time and users shouldn't depend on them?

Wrapping up, I have the following suggestions for Netgate:

• Commit to release schedules for package updates, e.g. 3 months after a new package version is released upstream.
• Categorize packages into "primary" and "secondary" priority with corresponding schedules so that expectations can be correctly set and development efforts can be prioritized accordingly.
• Introduce voting on packages to help Netgate prioritize package updates.
• Simplify the package development process. The ideal solution would be for Netgate to release package-building environments as VM images.
• Put together a complete guide to the package build process so users can develop packages more easily.
• Address the "mystery" values that need to be changed by documenting the values that need to be changed or by releasing a "clean" version of the code that compiles as CE.

[marcosm]

Thanks for the constructive feedback.

Regarding the comments:

The significant delays in releasing updated packages are at odds with this.

[...] is committed to keeping them up-to-date [...]

It should be mentioned that the Zabbix 7.0 package was added upstream on June 2024 (after the current stable release of pfSense Plus 24.03). Notably, it's a new package and not simply an update to an existing package. Regarding miniupnpd, that's a system package (similar to unbound) which is handled differently from packages listed in the package manager.

[michmoor]

@marcosm Marcos, are packages dependent on point releases of pfsense plus?
What constitutes a new package?
Whats a realistic timeframe new packages like Zabbix can be added to the pfsense repo?

[andrew_cb]

@marcosm
The questions asked @michmoor are a good summary of the basic information needed for pfSense packages. I'm not sure of the best way to track that - a single table in the pfSense documentation would be the most convenient for reference. Another option would be to put the current package version and next scheduled update on the package's documentation page, but not all packages have pages. Redmine is another option, but it's less convenient to navigate. Maybe something in the packages list in Pfsense itself?

If the default policy is that packages are normally only updated with each new pfSense release, then I think that needs to be more clearly stated.

Going back to the Zabbix example, when there was an update to 6.4.10, it was only made available in 24.03 and not in 23.09 or 23.05.
There have been no further mainstream updates to the Zabbix packages since April 2024. The issue is still open and "confirmed".
https://redmine.pfsense.org/issues/15313

24.11 RC contains Zabbix 6.4.19 from Sept 2024, so it is relatively up to date.

This is becoming urgent because Zabbix's End of Full Support for 6.4 was 30 June 2024, and End of Limited Support is 31 December, 2024.
https://www.zabbix.com/life_cycle_and_release_policy

It's frustrating that multiple comments and posts, both on the forums and on Redmine, have asked about updates to Zabbix 6.4 and a package for 7.0, and Netgate has not responded other than changing the target version on the 7.0 Redmine.

I have no idea how easy or difficult it is to release a package. Can you comment on roughly how much time/effort is required for a knowledgeable person (i.e Netgate staff) to release a Zabbix package for:

• a) an existing version (i.e. 6.0, 6.4), and
• b) a new version (i.e. 7.0)

p.s. I appreciate that you are responding to the discussion and I do not mean for this post to come across as hostile.

[bmeeks]

As a volunteer package developer/maintainer, let me chime in here with a point.

One thing I believe many users do not grasp is that most packages used with pfSense consist of two very distinct parts. There is a binary component that you see posted in FreeBSD-ports and this piece is installed by generic FreeBSD users. For these packages, users do their own configuration and set up via the command line interface (meaning no GUI).

But on pfSense, most packages also have a customized GUI component written in PHP. That GUI piece is created and then maintained by a developer familiar with the GUI interface of pfSense. The user then "interacts with" and "configures" the said package using the customized GUI on pfSense. The configuration parameters of the package are stored in the firewall's config.xml file and are only written to any file system based configuration file used by the package's binary piece when the user "saves" the setting in pfSense. This "save" operation writes a new conf file for the package (and also generally will overwrite anything the user attempted to do via the CLI outside of the pfSense GUI). This type of operation also means that the PHP GUI code must be intimately familiar with all of the exact parameters and their associated values required by the package binary portion. So, if a new version of the binary package requires new or updated conf parameters in order to start, the PHP GUI code that creates the conf file may need modification.

This can many times mean that when a new version of the package binary comes out (let's use Zabbix 7 as an example), updating to the new version means potentially rewriting or at least modifying the customized PHP code that constitutes the GUI interface. And therein lies the potential issue. The PHP code used to make the package integrate into pfSense is not something the upstream maintainers of the binary (Zabbix in this example) have knowledge of nor do they care to learn. Thus someone on the pfSense side of the equation must edit the PHP GUI code to make sure everything still works with the updated binary piece from upstream.

The packages subsystem in pfSense began as volunteer contributions. Most packages are still supported by non-paid volunteers not associated with Netgate in any manner. Over the years, as these volunteer maintainers have come and gone, either their package dies or in a few very rare cases Netgate takes over its maintenance. The disconnect I see is that many users seem to think Netgate develops and maintains all the third-party packages available in the pfSense Package Manager menu. That is not the case at all.

[michmoor]

@bmeeks
To be fair the disconnect is introduced by the advertisement on the product website

https://www.netgate.com/supported-pfsense-plus-packages

Support is not clearly defined. It’s understandable to me why most see a package in a pfsense repo is supported by pfsense dev team.
The reality is that only pf in pfsense is supported. Nothing else.

[bmeeks]

@michmoor said in Feedback on challenges with the pfSense package development process:

@bmeeks
To be fair the disconnect is introduced by the advertisement on the product website

https://www.netgate.com/supported-pfsense-plus-packages

Support is not clearly defined. It’s understandable to me why most see a package in a pfsense repo is supported by pfsense dev team.
The reality is that only pf in pfsense is supported. Nothing else.

Yeah, I did not get into "why" many users think Netgate developers support all the packages. Just stating that the perception is not always reality .

To be fair to Netgate, it is clearly spelled out on that page which packages they directly support. That would be all those marked "Maintained by Netgate".

Since Zabbix is mentioned in this thread, note that it is not listed at all on the linked page.

[michmoor]

@bmeeks I would just say that the package support list aids in the confusion. It should only list the ones maintained by Netgate. Including others is , in my opinion, bad marketing as they are not supported. I’m not directing this to you Bill, just “talking out loud”. That page needs clarification/ cleanup more so because it’s mentioned byTAC within Reddit forums when they communicate supported packages which just isn’t true for most of them.

[andrew_cb]

I don't recall seeing https://www.netgate.com/supported-pfsense-plus-packages. Is this the only place with the package support status? I don't see it linked anywhere on the website. Regardless, there are numerous references to non-Netgate maintained packages throughout the website, which give the appearance of at least being endorsed by Netgate, and no mention of their "risky", unofficial status.

Here's a few just from browsing around:

https://docs.netgate.com/pfsense/en/latest/packages/list.html < The first two paragraphs at the top of the page are sort of a disclaimer, but still doesn't make it clear about the support status of packages. There's no link to the official page.
"Zabbix-proxy
Zabbix Agent proxy. Collects performance and availability data on behalf of the Zabbix server, lowering the burden on the server. Available in multiple versions."

https://www.netgate.com/pfsense-plus-applications/firewall
"Observe key operating metrics like network utilization, CPU load and disk space usage with built-in Zabbix monitoring"

https://www.netgate.com/blog/alternatives-to-ntopng
"pfBlockerNG: DNS-based Blocking and Filtering
pfBlockerNG is a pfSense Plus software package that allows you to block access to malicious domains, ads, and other unwanted content at the DNS level. It uses various lists and feeds to stay up-to-date with the latest threats and can be customized to block specific categories or domains based on your organization's policies. pfBlockerNG integrates with pfSense Plus software's firewall and routing capabilities, providing an additional layer of security."

AND

"Network Monitoring Tools
When it comes to network monitoring, open-source tools like Nagios, Zabbix, and Prometheus are widely used in various organizations.
...
There is a pfSense Plus package called NRPE which allows a Nagios server to monitor pfSense Plus software, and
The node_exporter package in pfSense will export the Prometheus metrics to that external server."

https://www.netgate.com/pfsense-plus-technology
"Telegraf
An agent written in Go for collecting, processing, aggregating, and writing metrics."

"Zabbix-agent
Zabbix Monitoring agent. The agent gathers operational information locally and reports data to Zabbix server for further processing. The agent can also generate alerts in case of failures. Available in multiple versions."

The Packages menu in pfSense similarly doesn't include any disclaimers or link to the official package status page.

According to the package lists, NO monitoring packages are supported (Prometheus, Nagios, Zabbix, Telegraf). Is SNMP really the only official option?

You can see how this creates a very confusing experience for an end-user when trying to use packages.

[andrew_cb]

@bmeeks Thanks for your insight. The complexities with creating/maintaining a GUI for packages makes total sense.

Again with Zabbix, based on my use of it with other systems, no changes to the GUI configuration parameters are necessary as the basic parameters are the same. Currently, pfSense has Zabbix 5.0, 6.0, and 6.4 packages and associated configuration values/pages. If it can accommodate 3 versions, adding a 4th that requires no/minimal modifications should be relatively simple.

If there is a complex issue that is delaying the update or release of a package, then at least some communication about that would go a long way.

If Netgate is not the maintainer of a package, and building packages for pfSense requires a very particular FreeBSD build environment, then are those essentially orphaned from a pfSense perspective? What is the trigger and motivator for them to be updated?

Netgate staff have do not respond to these posts saying "Package xyz is not maintained by pfSense, please ask the maintainer to submit an updated version to be reviewed for inclusion in the package repository", rather they respond to some of the posts and set a target version in redmine. If Netgate is truly not maintaining the FreeBSD/pfSense port builds of these packages, then this behavior further adds to the confusion.

Another point I want to mention - Netgate staff caution against installing packages from the FreeBSD repository as this can break things, and instead recommend to only install packages from the pfSense repository. Again, this further reinforces the impression that Netgate is maintaining and curating the packages available in Packages.

[michmoor]

@andrew_cb
Just want to reiterate what was stated in these forums awhile ago.
Do not install any package you cannot directly support yourself. This is for Suricata/Snort,pfblocker,zabbix.
Once you make these tools part of your production system, you run the risk that at any point in time these packages stop receiving support. Then what?
As I stated above, the ONLY thing that Netgate supports is 'pf'. Nothing more nothing less.

[bmeeks]

@andrew_cb said in Feedback on challenges with the pfSense package development process:

@bmeeks Thanks for your insight. The complexities with creating/maintaining a GUI for packages makes total sense.

Again with Zabbix, based on my use of it with other systems, no changes to the GUI configuration parameters are necessary as the basic parameters are the same. Currently, pfSense has Zabbix 5.0, 6.0, and 6.4 packages and associated configuration values/pages. If it can accommodate 3 versions, adding a 4th that requires no/minimal modifications should be relatively simple.

If there is a complex issue that is delaying the update or release of a package, then at least some communication about that would go a long way.

If Netgate is not the maintainer of a package, and building packages for pfSense requires a very particular FreeBSD build environment, then are those essentially orphaned from a pfSense perspective? What is the trigger and motivator for them to be updated?

Netgate staff have do not respond to these posts saying "Package xyz is not maintained by pfSense, please ask the maintainer to submit an updated version to be reviewed for inclusion in the package repository", rather they respond to some of the posts and set a target version in redmine. If Netgate is truly not maintaining the FreeBSD/pfSense port builds of these packages, then this behavior further adds to the confusion.

Another point I want to mention - Netgate staff caution against installing packages from the FreeBSD repository as this can break things, and instead recommend to only install packages from the pfSense repository. Again, this further reinforces the impression that Netgate is maintaining and curating the packages available in Packages.

There are complexities invovled in building packages for FreeBSD (and by extension, pfSense) that are not always appreciated by those that do not have extensive experience in the area. Let me try to provide a high level overview.

The binary portions of packages must be built using the exact same FreeBSD kernel that the package will ultimately be installed on. That means, since pfSense uses a customized/modified FreeBSD kernel, any packages installed on pfSense need to be built in a builder that uses the same customized FreeBSD kernel and any shared library versions. That's where the warning about installling packages from the standard FreeBSD repo comes from. Those packages will be built for stock FreeBSD kernels and are also likely configured for and expecting to find particular versions of shared libraries. Those exact conditions may not exist on a given pfSense version. The kernel code is different from stock FreeBSD and the versions of shared libaries can well be different. When packages are installed, they automatically pull in the dependent library versions they were built with. That may overwrite something that pfSense needs and thus can break pfSense itself.

Netgate has their own set of package builder servers. Those servers build (meaning compile) the binary and PHP portions of packages using an enviroment created from a particular pfSense version (CE 2.7.2, or CE 2.8.0 DEVEL, or 24.03 Plus). That means the packages built by pfSense servers and served up in the pfSense GUI will have shared dependencies that match what is already installed by pfSense. Thus no conflict and no possibility of breaking pfSense.

When pfSense itself is updated, the packages are all rebuilt using the new kernel and shared library tree.

The binary portions of a package are tied to the underlying kernel version and thus the pfSense version. The PHP portion of a package is usually NOT tied to the kernel version althought it might have some dependency on the exact version of PHP used in the pfSense GUI.

The pfSense package repo source code is hosted on GitHub here: https://github.com/pfsense/FreeBSD-ports. Anyone is free to submit Pull Requests that contain updates for existing packages or that add entirely new packages to the pfSense list. That has happened many times over the years. Over the course of history, someone has had need of some particular function or package and created it themselves. They then may decide to "share it" with the larger community by creating a Pull Request and posting their package code to the GitHub link I provided. If the Netgate developers think the Pull Request has merit, they merge it into the tree. That's how most of the packages found their way into pfSense in the first place. Only a very few were created by Netgate themselves.

If you look at some of the recently closed Pull Requests on GitHub you will see updates I've provided for Suricata and updates that the pfBlockerNG developer has provided for that package: https://github.com/pfsense/FreeBSD-ports/pulls?q=is%3Apr+is%3Aclosed. Note that neither he nor I are Netgate employees. We are just pfSense users like you that decided to produce a package (or update an existing one) and shared our work with the larger community. For some of the really popular packages, the Netgate team may step up and fix small bugs to help out the larger community and to aid the volunteer package maintainer. That has happened occasionally for the Snort and Suricata packages I maintain. It has also happened from time to time with the pfBlockerNG package. But generally speaking Netgate will not take it upon themselves to update the binary portion of a package that they do not directly maintain. They wait on the volunteer maintainer who has more detailed knowledge of the inner workings of a package to submit any update via a Pull Request at the GitHub link posted above.

[michmoor]

@bmeeks
Curious, do you think the current system of 3rd party packages and support of a volunteer community is scalable?
If you look at other vendor firewalls, the entirety of the system is under control of the company. Unifi may run Suricata in the background but if something breaks on their system they take ownership of it and not wait for a fix to hopefully come upstream.

[SteveITS]

@michmoor On one hand, arguably no, but OTOH there’s no free version of UniFi AFAIK.

Open source is a weird middle ground of lots of people wanting updates, but few have the skill and time and will to do the work for free.

[bmeeks]

@SteveITS said in Feedback on challenges with the pfSense package development process:

Open source is a weird middle ground of lots of people wanting updates, but few have the skill and time and will to do the work for free.

This is most certainly true and very well describes the overall situation with regards to packages on pfSense and any other similar open source platform.

[bmeeks]

@michmoor said in Feedback on challenges with the pfSense package development process:

@bmeeks
Curious, do you think the current system of 3rd party packages and support of a volunteer community is scalable?
If you look at other vendor firewalls, the entirety of the system is under control of the company. Unifi may run Suricata in the background but if something breaks on their system they take ownership of it and not wait for a fix to hopefully come upstream.

@michmoor, not trying to pick on you with this reply, but your questions and remarks make it appear you seem to expect the same level of support from pfSense and Netgate as you get from the other firewall vendors I see in your signature (Palo Alto and Juniper).

At the price you are paying for pfSense versus what I suspect you are paying for Palo Alto and Juniper, that is just never going to happen . Especially so for the free open source edition of pfSense (and remember the packages tree is essentially the same for both the free and paid editions of pfSense, with the one exception I recall being an AWS package for Plus that is not available in CE).

Netgate could certainly follow the lead of a Palo Alto or Juniper or Cisco and bring all the packages in house and maintain them. But the price of Plus would have to increase substantially and I would not expect to see that support spread to the open source side for free.

[marcosm]

@michmoor said in Feedback on challenges with the pfSense package development process:

@marcosm Marcos, are packages dependent on point releases of pfsense plus?
What constitutes a new package?
Whats a realistic timeframe new packages like Zabbix can be added to the pfsense repo?

I doubt this answer is satisfactory but it's what I've got...

Sometimes. Generally anything with its own page at https://www.freshports.org/. Depends on the situation.

Off-topic: I personally hesitate to spend much effort browsing/responding in social platforms because it often tends to result in additional stress that I prefer to avoid in my life. That's become more and more true as the years pass. Anyway, I say this to offer some insight into why I may not respond to things. I imagine I'm not alone in that.

[michmoor]

@marcosm No worries Marcos. I appreciate you even responding.
Your answer is satisfactory. I will leave it at that :)

[michmoor]

@bmeeks Always picking on me..why is that ( kidding).

Im not comparing any level of support between vendors in my profile for example and Netgate. There is a gulf between them but that's not the issue (or my issue) in particular.
For some background, I have been using PfSense for about five years now. When I started using the platform, I assumed that everything running on it or installed on it was supported. Much like the OP here, I figured it was the total package, and if you needed white glove support, you could get a TAC support contract.

I later found out that there are nuances in what is supported and by whom. The website's marketing and documentation do lead anyone to believe that there is support for third-party packages, but I found out years ago that is not true. Hence why i would advocate for cleaning up the package support list page and only list the packages that are supported by Netgate. Documentation can state otherwise.

[tinfoilmatt]

@michmoor said in Feedback on challenges with the pfSense package development process:

There is a gulf between them but that's not the issue (or my issue) in particular.

Speaking for myself and rather confidently on behalf of others, many of us feel like that "gulf" exists in the opposite direction from what you suggest and are nothing but grateful that Netgate continues to maintain a CE and a free Plus which can utilize third-party packages at all.

@michmoor said in Feedback on challenges with the pfSense package development process:

The website's marketing and documentation do lead anyone to believe that there is support for third-party packages

That's just not true unless you're claiming to speak on behalf of literally everyone.