How to convert a functional bug into a technical pfSense/FreeBSD bug report

[louis2]

Hello,

I am trying to get PIMD running on pfSense (2.5 dev), however there are bugs which prevent it from working (at least on my system / my configuration).

• Something goes wrong during boot and I exactly know where it goes wrong during the boot sequence
• and PIMD is running (a bit) I exactly know how to force a crash (and I have crash dumps, many)

However that functional level is not exact/good enough for NetGate to accept it as a bug / and to start a repair action.

That is the reason for this post!

I somehow need to close the gap between the functional problems and the exact technical problem cause NetGate would like to see

Note that I do have technical knowledge and skills, however I am not at all a FreeBSD- or a pfSense developer.

Never the less I looked on the internet and found remarks like:

• pkg install pfSense-kernel-debug.
  So I tried:
  Updating pfSense-core repository catalogue...
  pfSense-core repository is up to date.
  Updating pfSense repository catalogue...
  pfSense repository is up to date.
  All repositories are up to date.
  pkg: No packages available to install matching 'pfSense-kernel-debug' have been found in the repositories
• Debugging a Kernel Crash Dump with kgdb. So I tried:
  kgdb -n last (kgdb not found)
• Debugging a crash dump with DDD. So I tried:
  ddd -k /var/crash/kernel.0 /var/crash/vmcore.0
  sh: ddd: not found
• I could go to github downloading the PIMD code trying to understand under which circumstances it does generate the strange bootlog message
• inserting extra debuglines in the code
• it might be that there is a deveolper option during install
• etc

Whatever …. It is all …. Not the way to go I think !!

Does anyone knows how to close that gap between “functional issue” and “technical issue” can be bridged!!??
(apart from the question who should close that gap!).

Louis
PS for the issue itself see my posts on the 2.5 dev forum, but I do regard this a generic issue

[DaddyGo]

@louis2

Hi,

I recommend it to you, before you start anything:
https://forum.netgate.com/topic/149909/new-package-pimd

++++edit:
this too:
https://redmine.pfsense.org/projects/pfsense/roadmap

[louis2]

Thanks,

I did the first topic again, tja what to say ongoing discussion, not related to bugs

However under the second link, I found a bug which is surely relevant!

• Bug #9796: kernel panic after removing interfaces!! That seems related !!!
  I was glad to see that Luckman212 did put a remark there just 14 hours ago, however not so glad seeing how long it is already in the list !
  Not sure as well to which extend disable and remove is triggering the same kernel thing, but good change it does. However the example crash reports are not identical)

I really would not be surprised if that is the second bug I noticed (and which make may system crash an crash again). I really hope that this issue will be fixed on short term!

Never the less, my problem is and stays that I can clearly see that something is not working because of a), b) and c) and that NetGate does not accept that I describe the problem that way, and that I am not capable to generate three bugs describing what is the exact technical issue causing it.

For example when v2.5 started there where at least 4 issues affecting PIMD. One has been solved by the PIMD-maintainer, one is solved by Vicktor (PIMD package 0.03 not yet installable), one it perhaps bug #9796 and I am almost sure there is another one related to interface handling by PIMD (or probably the layer below PIMD). However …. There must be a ticket related to that otherwise it will never be fixed (or it will be fixed because an updated FeeBSD package fix it automatically).

I also have a feeling that that bug is not inside PIMD and not affecting PIMD only. We will see, some day .....

Louis
PS other subject! I discovered that CPU microcode is loaded, See that topic

[DaddyGo]

@louis2

Hi,

Pls note that, 2.5 is a developer version, not a production environment version.
Yes, its development will last as long as necessary.

@louis2 "PS other subject! I discovered that CPU microcode is loaded, See that topic"

Yes I saw.
The fact is that CPU microcodes are handled first by CPU vendors, then by BIOS / MOBO manufacturers and then by OS developers.

The order is this, what I wrote anno is that it’s up to the developers task (by kernel or etc...), it wouldn't be too expedient for everyone to modify CPU microcodes (this is already a higher level).

[louis2]

Yep,

I know, I did install the developer version because e.g. PIMD did not work in the regular version as wel Hoping ... it would work in that version.

Still hoping all bugs I noticed will be fixed. ..... just hope and pray ......

Louis

[bmeeks]

@louis2:
The pfSense bug reporting Redmine site is here: https://redmine.pfsense.org/. You can sign up and then create bug reports for the developer team.

Realize that their first priority is the basic pfSense firewall engine and associated GUI. Add-on things such as packages (and pimd is a package) are lower priority. But that does not mean it won't get looked at.

You are also certainly free to dig into the issue on your own and contribute bug fixes in the form of Github source code Pull Requests against the pfSense-2.5 DEVEL branch. The Github repository for the GUI is here: https://github.com/pfsense/pfsense. The Github repository for packages, which is clone of FreeBSD ports, is here: https://github.com/pfsense/FreeBSD-ports. You will need to have working knowledge of Git as well as programming savvy in PHP for the GUI code and in PHP and C for the packages code.

It is highly likely the problems you are experiencing with pimd are actually in FreeBSD-12.1 instead of being just a pfSense issue. That actually makes debugging easier as then you can simply create a FreeBSD-12.1 machine (a virtual machine is ideal) and install pimd and all of the FreeBSD development and debugging tools on it and get to work. Creating a pfSense development and debugging environment is much more difficult because so many of the required libraries and tools are obviously not a part of the regular firewall installation.

[louis2]

Thanks @bmeeks ,

I completely agree with your sentence "It is highly likely the problems you are experiencing with pimd are actually in FreeBSD-12.1 instead of being just a pfSense issue." Moreover I would not be surprised at all, if more issues are related to the same bug.

However I do not so much agree with "that actually makes debugging easier as then you can simply create a FreeBSD-12.1 machine (a virtual machine is ideal) and install pimd and all of the FreeBSD development and debugging tools on it and get to work."

That, because it extremely helps if you can reproduce the bug on the "development machine". And here on my router I can (easily) reproduce the problem, and I would "~ never" be able to emulate my network situation on a virtual machine.

I do have the luxury that the firewall is my private home firewall. That allows me to do testing never possible on a production environment.

So the maximum I am willing to is to add extra debugging aid to my router to identify the issue, not to solve the issue.

Louis

[DaddyGo]

@louis2 said in How to convert a functional bug into a technical pfSense/FreeBSD bug report:

I do have the luxury that the firewall is my private home firewall. That allows me to do testing never possible on a production environment.
So the maximum I am willing to is to add extra debugging aid to my router to identify the issue, not to solve the issue.

well, I don't agree with these lines of text above ...

whatever firewall we are talking about, production or home SOHO!

the firewall is for network protection, not to experiment
you can develop and experiment in a VM environment, because you do not risk your privacy or your company's data

if you find a serious error / issue with pimd, it will not be returned only by your network topology

otherwise, in this case, it can only be considered a unique special defect that does not affect others

ERGO: it is not a system-wide issue, so it is expected to be resolved (fixed) later

[louis2]

@DaddyGo

No I do not agree, of course I will explain why

• the firewall is for protection, and that should always be there we agree, however the network configuration, firewall and firewall settings I have here is by far better that in 99% of SOHO
• However since it is only for SOHO use, it is not a big issue if it is going down for a short period or even if it crashes (as long as I can recover it fast, what is possible due to prepared USB-sticks)
• and the alternative setting up a whole test lab, is a huge undertaking. Not realistic at all.
• and the suggestion that it is not a system-wide issue cannot be concluded as well. I am not saying or proved that you cannot reproduce it in a test lab, I just intended to say that that action is far more complex!!

Hope you understand

Louis

[bmeeks]

@louis2 said in How to convert a functional bug into a technical pfSense/FreeBSD bug report:

However I do not so much agree with "that actually makes debugging easier as then you can simply create a FreeBSD-12.1 machine (a virtual machine is ideal) and install pimd and all of the FreeBSD development and debugging tools on it and get to work."

That, because it extremely helps if you can reproduce the bug on the "development machine". And here on my router I can (easily) reproduce the problem, and I would "~ never" be able to emulate my network situation on a virtual machine.

You will have two issues with trying to debug on an actual pfSense firewall. The first one is that the necessary software tools are not there. And depending on the particular tool, installing it can break the firewall due to library versioning conflicts. The second issue is you will not have access to a debug-enabled kernel build. That's going to make tracing code execution through pimd and the kernel very difficult. The environment for "building" a pfSense kernel is not easy to set up.

By using the stock FreeBSD source code, you can build a debug-enabled system that has all the other debugging software tools on it and has a build environment where you can compile a debug-enabled version of the pimd package. Compiling and using the stock FreeBSD setup for debugging is well documented in various tutorials on Google.

So the maximum I am willing to is to add extra debugging aid to my router to identify the issue, not to solve the issue.

Louis

There is essentially no debugging you can add to your pfSense router that will be helpful. You could install the gdb package, but without all the source code for FreeBSD and the pfSense modifications also installed, and a debug version of pimd compiled and installed, there is nothing much to be gained because all you would be able to do is step through machine code. But without the C source modules and a debug-enabled build of pimd, there will be no "context" around those lines of machine code to help figure out where the issues are.

You originally asked how to convert a functional bug into a technical pfSense/FreeBSD bug report. I gave you the steps most folks would follow. You then responded by listing some things you would not be willing to do. Unfortunately, your list of "nots" is going to likely prevent you from achieving your stated goal of creating a technical bug report. That is if you meant by "technical" a detailed report listing which function or lines of code are the source of the problem.

[louis2]

@bmeeks

You are probably right. I did not see any realistic options as well. That is the reason, I did start this topic, just hoping anyone had a solution.

However, I did write a lot of software over time, and I do have a lot of technical knowledge, but all not related to vm, github, C, FreeBSD etc. And even if I did have that, the effort of creating:

• vm for pfsense
• vm's for network, servers and pc 's as source and destination
• configuration
• testcases
• etc

Just not realistic

The only thing perhaps possible for the boot issue, is downloading the pimd source to have a look under which conditions it does generate the messages I see in the boot log.

I assume that just before it generates those messages, it does perform a function call towards an OS-layer just below PIMD. That would than my No-1 verdict

Louis