Working with Packages on 2.3 and Converting to Bootstrap


  • Rebel Alliance Developer Netgate

    I have collected and combined a lot of information and added some more to come up with what I hope is a good set of info to help others make the jump into working with packages on 2.3 and converting things to bootstrap as needed.

    Basic info for those who have not yet done any development work:
    https://doc.pfsense.org/index.php/Getting_Started_with_pfSense_Development
    https://doc.pfsense.org/index.php/Contributor_License_Agreement_for_Developers

    Overview of the new pfSense Package port directory structure:
    https://doc.pfsense.org/index.php/Package_Port_Directory_Structure

    A list of current packages on 2.3 and where they are located inside the ports repository:
    https://doc.pfsense.org/index.php/Package_Port_List

    Updated doc about Developing Packages with 2.3-specific info:
    https://doc.pfsense.org/index.php/Developing_Packages

    Notes about converting packages to bootstrap:
    https://doc.pfsense.org/index.php/Converting_Packages_to_Bootstrap

    2.3 Package Conversion and Operation Status
    Here is a Google Docs sheet with the current list of 2.3 packages and their status. Many use XML forms and do not need any help (which is good!) but still need testing to confirm they function as expected on 2.3.

    The master ticket in Redmine for the conversion of packages is here: https://redmine.pfsense.org/issues/5568

    Let me know if anything seems to be missing or if there are any specific topics missed that need covered.

    Thanks to all for their effort!


  • Administrator

    bmeeks, I was thinking of taking a shot at converting Suricata so I had somewhere to dive in if you need a hand.


  • Rebel Alliance Developer Netgate

    I added a link above to a Google docs sheet with the current status of the packages we have moved to 2.3 already. Many do not need conversion but still could use testing.

    Feel free to report success or failure here in this thread but if you do find a problem please start a new thread to investigate the problem to avoid crosstalk this thread.



  • @jdillard:

    bmeeks, I was thinking of taking a shot at converting Suricata so I had somewhere to dive in if you need a hand.

    Thank you!  I will be glad for the help and to help you.  Suricata has not been started. I had been working on Snort until I took a break to work on the netmap stuff.  Send me a PM and we can swap e-mail addresses (if you don't have mine already).  Will be easier probably to correspond that way or by phone even.

    Bill


  • Administrator

    @bmeeks:

    @jdillard:

    bmeeks, I was thinking of taking a shot at converting Suricata so I had somewhere to dive in if you need a hand.

    Thank you!  I will be glad to help.  Suricata has not been started. I had been working on Snort until I took a break to work on the netmap stuff.  Send me a PM and we can swap e-mail addresses (if you don't have mine already).  Will be easier probably to correspond that way or by phone even.

    Bill

    Will do. I figured it would be best to attempt a smaller package, like cron, so I can get the process nailed down before moving onto something larger like Suricata. I'll be in touch once I get that done and we can come up with a plan of attack.


  • Banned

    Guys… those packages that are adding their own user/group accounts - is that now safe to remove that code?


  • Rebel Alliance Developer Netgate

    Such as?

    If it's for a binary package, at least in theory, the pkg code should be handling that properly. I know PBIs had some issues there.  When in doubt, pkg add just the underlying program (not the pfSense pkg) and check for the user after.


  • Banned

    @jimp:

    Such as?

    If it's for a binary package, at least in theory, the pkg code should be handling that properly. I know PBIs had some issues there.  When in doubt, pkg add just the underlying program (not the pfSense pkg) and check for the user after.

    Well, the first one I hit was Avahi. Then, there's the infamous Squid thing with the ClamAV stuff where PBI was adding something with wrong UID/GID and the proxy user/group confusion/mess. Also, I recall there vaguely there was some issue related to the UID/GIDs handling with certain range being reserved for pfSense user manager and the accounts removed on reboot???


  • Banned

    Another note - I don't think this package makes any sense? Never had any GUI.

    https://github.com/pfsense/FreeBSD-ports/tree/devel/net-mgmt/pfSense-pkg-iftop


  • Rebel Alliance Developer Netgate

    @doktornotor:

    Another note - I don't think this package makes any sense? Never had any GUI.

    https://github.com/pfsense/FreeBSD-ports/tree/devel/net-mgmt/pfSense-pkg-iftop

    It's just there so it gets built and available in the repo. It's console only and very handy to have available.


  • Rebel Alliance Developer Netgate

    @doktornotor:

    @jimp:

    Such as?

    If it's for a binary package, at least in theory, the pkg code should be handling that properly. I know PBIs had some issues there.  When in doubt, pkg add just the underlying program (not the pfSense pkg) and check for the user after.

    Well, the first one I hit was Avahi. Then, there's the infamous Squid thing with the ClamAV stuff where PBI was adding something with wrong UID/GID and the proxy user/group confusion/mess. Also, I recall there vaguely there was some issue related to the UID/GIDs handling with certain range being reserved for pfSense user manager and the accounts removed on reboot???

    None of the "official" users would collide with pfSense:

    https://github.com/pfsense/FreeBSD-ports/blob/devel/UIDs
    https://github.com/pfsense/FreeBSD-ports/blob/devel/GIDs

    Packages in FreeBSD are supposed to specify their users in the Makefile, and the specifics of UID/GID/etc are pulled from the above files. For example:
    https://github.com/pfsense/FreeBSD-ports/blob/devel/www/squid/Makefile#L36

    I don't think we should have any code in the .inc files or elsewhere to manually make users any longer, if one isn't in the above list we should add it in and also tweak the pfSense package Makefile appropriately.


  • Banned

    @jimp:

    I don't think we should have any code in the .inc files or elsewhere to manually make users any longer, if one isn't in the above list we should add it in and also tweak the pfSense package Makefile appropriately.

    OK, sounds good. Junk nuked. :)



  • I had a bit of a look at this task and didn't notice any mention of anything relating to REST..

    I noticed the pfSense blog post about converting to REST in 3.0 (https://blog.pfsense.org/?p=1588) - its quite old though.. Is this still the plan?

    While in there changing the PHP pages it could be good if people were taking this into account as much as possible..

    Something like:

    • Split functional code out into a [file]_rest.php that returns JSON

    • Change HTML page to make REST HTTP requests to execute their functions

    Doing this now could get the process of converting to REST early (unless 3 is the next release directly after 2.3?).

    I was taking a look at the sipproxd and it seems relatively small and thinking - depending on the time frame that packages need to be converted in - I might be able to take a look at converting that one in this manner? Could give a basic template to start moving to the REST structure..

    I'm primary a web developer and work a lot with REST APIs (built in Django) and web front ends (the Ember JavaScript framework)


  • Rebel Alliance Developer Netgate

    REST will be for 3.0 only and some time off (we haven't even began drafting the API) – Seems like overkill to start all that now when we haven't decided on what we'll be using for any of that.

    We'll be on 2.3 for a while, most likely there will even be a 2.4, that blog post is long-term goals.


  • Administrator

    @jmurphyau:

    I had a bit of a look at this task and didn't notice any mention of anything relating to REST..

    I noticed the pfSense blog post about converting to REST in 3.0 (https://blog.pfsense.org/?p=1588) - its quite old though.. Is this still the plan?

    While in there changing the PHP pages it could be good if people were taking this into account as much as possible..

    Something like:

    • Split functional code out into a [file]_rest.php that returns JSON

    • Change HTML page to make REST HTTP requests to execute their functions

    Doing this now could get the process of converting to REST early (unless 3 is the next release directly after 2.3?).

    I was taking a look at the sipproxd and it seems relatively small and thinking - depending on the time frame that packages need to be converted in - I might be able to take a look at converting that one in this manner? Could give a basic template to start moving to the REST structure..

    I'm primary a web developer and work a lot with REST APIs (built in Django) and web front ends (the Ember JavaScript framework)

    I agree with Jimp, we should focus on getting 2.3 out to prevent what could be massive scope creep. Not saying it isn't important though. I was actually looking at make graph.php utilize an API over the weekend so that I could convert the traffic graphs to d3.j3 using JSON for the datasource. If you want to take a stab at converting various small data points to JSON APIs and need someone to bounce ideas off of let me know. I could probably squeeze a little bit of time in the week to hack on stuff like that.



  • NUT?

    @jimp:

    Let me know if anything seems to be missing or if there are any specific topics missed that need covered.


  • Rebel Alliance Developer Netgate

    Not addressing missing packages yet. The ones we have listed must all be complete first before we can consider expanding that list.



  • Okay, thanks Jim.

    @jimp:

    Not addressing missing packages yet. The ones we have listed must all be complete first before we can consider expanding that list.



  • @jimp:

    Overview of the new pfSense Package port directory structure:
    https://doc.pfsense.org/index.php/Package_Port_Directory_Structure

    Looks great!!!

    I've send the first pull request to postfix package. Structure looks extremely easier to port packages to pfsense. Excelent work!!!!!!



  • @marcelloc:

    I've send the first pull request to postfix package.

    Still not showing up under Available Packages



  • @biggsy:

    @marcelloc:

    I've send the first pull request to postfix package.

    Still not showing up under Available Packages

    it hasn't been merged yet


  • Netgate

    @jimp:

    REST will be for 3.0 only and some time off (we haven't even began drafting the API) – Seems like overkill to start all that now when we haven't decided on what we'll be using for any of that.

    We'll be on 2.3 for a while, most likely there will even be a 2.4, that blog post is long-term goals.

    Maybe a year to an initial release, but yeah.



  • Bind package is needed too..



  • Dear marcelloc and doktornotor, please consider making a independent ClamAV package with default clamav permissions. Its a hell of a ride to make SquidClamAV and Mailscanner run side by side and after each pfSense update all modifications are gone and needs fixing again.

    At the moment, I've a working 2.2.6 system with:

    Postfix + latest Mailscanner (SA with all modules, DCC pyzor etc.) + SPF marking + ClamD 0.99 (all via pkg install)
    Squid + SquidGuard + Squidclamav + c-icap + ClamD 0.99

    ClamD is running fine, both Squidclamav and Mailscanner use it at the same time without conflicts.


  • Rebel Alliance Developer Netgate

    I updated another batch of packages, testing is appreciated:

    • Backup
    • siproxd (status page only)
    • FreeRADIUS2 (just needed the config viewer)
    • LADVD (status page only)
    • OpenBGPD (raw config and status pages)

    Of those the most significant one that needs testing is OpenBGPD since I don't have an active BGP connection to test against. Some of the status fields are empty but I'm not certain if it's my lack of working setup or a problem on the page. Feedback is definitely appreciated for that one, and all the rest as well.


  • Rebel Alliance Developer Netgate

    I pushed updates for Squid, squidGuard, Lightsquid, and some associated fixes in the base system that should at least make things functional for most people. I'll start a fresh thread for those, though.



  • I noticed that ntopng is in the Google Docs list of packages, but when I check on my 2.3 box, it doesn't appear in the list. Just wondering if there's a reason why this is, or if it's just an oversight somewhere.

    Thanks! :)


  • Rebel Alliance Developer Netgate

    @virgiliomi:

    I noticed that ntopng is in the Google Docs list of packages, but when I check on my 2.3 box, it doesn't appear in the list. Just wondering if there's a reason why this is, or if it's just an oversight somewhere.

    It's a work in progress. In order to accommodate ntopng we have to upgrade to rrdtool 1.4, but to get to that point in the base system we need to revamp our RRD code to work without the graph option so that we can avoid adding a couple hundred MB in dependencies. Once the base system RRD code is reworked then we can add ntopng to the packages.



  • I know that the more popular packages deserve priority but is there actually a defined process for reviewing and making packages available for install through the GUI?  If that process exists, is it something that can be published?

    My reason for asking is that I'm very much a fan of the postfix package and eager to test it out on 2.3.  It would be nice to know its current progress through the process.

    A package readiness spreadsheet, like the one that was available for 2.2 (I think), would be really useful.  Maybe even shared with and editable by package developers/maintainers, so they could say where they're up to.



  • Look at the ToDo issue in GitHub https://redmine.pfsense.org/issues/5568
    Then you can find the issue for any package that is in the process of conversion (or done).
    As people progress on each conversion those issues should be kept up-to-date.


  • Rebel Alliance Developer Netgate

    @biggsy:

    I know that the more popular packages deserve priority but is there actually a defined process for reviewing and making packages available for install through the GUI?  If that process exists, is it something that can be published?

    We haven't defined that process yet.

    @biggsy:

    A package readiness spreadsheet, like the one that was available for 2.2 (I think), would be really useful.  Maybe even shared with and editable by package developers/maintainers, so they could say where they're up to.

    See the first post in this thread, it already has a link to the sheet for 2.3.



  • If anybody could please pick up sarg, that would be great.



  • @ulcha:

    Bind package is needed too..

    I'm happy to help test bind, if required.
    At the moment, I'm holding off upgrading to 2.3 as I use bind for DNS.



  • can we get LCDproc-dev added to the list of packages that need conversion?

    I've spent a few hours looking at all this info, and looking at the code of 2.2.x packages and 2.3 packages and i'm totally lost.

    This guy posted instructions on how to manually get it working on 2.3, but we really need a package.



  • please pfsense is 2.3.2 not package  postfix & mailscanner. ????????????????????



  • I noticed pfsense has the freebsd ports tree in the git repo, so all these requests for binaries could simply be honoured by allowing people to compile ports on a pfsense box?


  • Rebel Alliance Developer Netgate

    @chrcoluk:

    I noticed pfsense has the freebsd ports tree in the git repo, so all these requests for binaries could simply be honoured by allowing people to compile ports on a pfsense box?

    Negative. Putting a compiler environment on a firewall is extremely dangerous.



  • well obviously there is a security risk if its implemented in a bad way, I see worse things on pfsense tho then a compiler.  I was just thinking out loud on how to make this a whole lot more flexible in packages.

    I been reading the documentation linked to from the first post and am left confused in regards to specific steps, e.g. if i am making a pkg to use on pfsense, is it ok to build on a FreeBSD box? or is it preferred to use the pre built FreeBSD binary? or is it required to use a special build environment?

    Also is there a requirement for a pkg to have GUI integration?


  • Rebel Alliance Developer Netgate

    Those are topics for a different thread.



  • so this topic is nothing to do with making packages for 2.3 as the topic title says?


Locked