25.03-BETA: pftop core dump

patient0

While trying to see the connections when testing PREF64 pftop did crash quite often. But not always. And if it did crash it was when I set the filter to icmp or started it with that filter pftop -f icmp.

pfSense+ 25.03-BETA is running as a VM on Proxmox, interfaces are vtnetX.

Attached two core dumps:

pftop.core_20250216-1010CET.tgz
pftop.core_20250216-0935CET.tgz

Additional infos: it failes with the message "Bus error (cored dumped)". And I run pftop from the shell.

stephenw10

So always when running manually at the CLI? Not in the GUI? or from the console menu?

patient0

@stephenw10 after @fireodo comments on the "25.03-BETA NAT64 / PREF64 Success and feedback" thread I tested it on 2.7.2 CE too. And it does fail there too while having a ping 1.1.1.1 running on a client.

Same behavior in 25.03-BETA and 2.7.2 CE:
No crash from the GUI. But pretty reliable crashing when calling pftop -f icmp from the shell (right now everytime). Calling only pftop and then setting the filter to icmp crashes it sometimes (right now all the time) but at times not very often (earlier this day).

Testing the console menu the behavior is the same as calling it from the shell, it crashes quite often when setting the filter to icmp while running a ping on a client. The ssh is then not usable anymore for me.

Below the screenshot of the moment when the pftop started from the console crashes (two screenshots merged from the VNC of the VM, not everything fitted on the screen).

Screenshot 2025-02-17 at 19.23.14.jpeg

patient0

Addition: just tested on my 24.11 "production hardware" (OPNsense DEC740), same thing happens.

pftop -f icmp core dumps right away and running pftop with filter set to icmp crashes it eventually. The WAN dpinger are enough, don't have to run a ping myself manually.

stephenw10

Mmm, OK that's pretty easy to replicate! And, yeah, seems to affect a number of versions.

stephenw10

Looks like it's covered by this: https://redmine.pfsense.org/issues/15595

patient0

@stephenw10 does indeed sound like it. The linked FreeBSD bug is interesting, don't think pftop go patched with what Kristof suggested.

Addition: I was able to reproduce it on a 15-CURRENT from 20250213 with a minimal pf.conf. It does take longer when started without arguments and setting the filter to icmp but it crashes after several minutes while running two ping-s. Calling pftop -f icmp crashes right away.

For reference the lldb backtrace

pfTop: Up State no entries (1), View: default, Order: none, Cache: 1000010:34:44
Process 5397 stopped
* thread #1, name = 'pftop', stop reason = signal SIGBUS: object-specific hardware error
    frame #0: 0x00002ef884a7dfae pftop`print_state(s=0x00003e14818312b0, ent=0xa5a5a5a5a5a5a5a5) at pftop.c:990:31
   987          }
   988          print_fld_uint(FLD_RULE, s->rule);
   989          if (cachestates && ent != NULL) {
-> 990                  print_fld_rate(FLD_SI, ent->rate);
   991                  print_fld_rate(FLD_SP, ent->peak);
   992          }
   993
(lldb) bt
* thread #1, name = 'pftop', stop reason = signal SIGBUS: object-specific hardware error
  * frame #0: 0x00002ef884a7dfae pftop`print_state(s=0x00003e14818312b0, ent=0xa5a5a5a5a5a5a5a5) at pftop.c:990:31
    frame #1: 0x00002ef884a7cbc6 pftop`print_states at pftop.c:1004:12
    frame #2: 0x00002ef884a84050 pftop`disp_update at engine.c:883:4
    frame #3: 0x00002ef884a84a68 pftop`engine_loop(countmax=0) at engine.c:1213:4
    frame #4: 0x00002ef884a81139 pftop`main(argc=0, argv=0x00002f00a510afc0) at pftop.c:2322:2
    frame #5: 0x00002f00a8a5a404 libc.so.7`__libc_start1(argc=1, argv=0x00002f00a510afb8, env=0x00002f00a510afc8, cleanup=<unavailable>, mainX=(pftop`main at pftop.c:2178)) at libc_start1.c:172:7
    frame #6: 0x00002ef884a7be2d pftop`_start at crt1_s.S:83

stephenw10

Fix is incoming.

pst

@stephenw10 I'm starting to see frequent core dumps in the May1 beta. The reason is different though, here's a snippet from running /status.php:

Firewall-pftop Rules

<jemalloc>: jemalloc_extent.c:1195: Failed assertion: "p[i] == 0"

stephenw10

Any more to go on? Like it does that every time you try to generate the status_output?

What about if you check the Rules view in Diag > pfTop?

Or directly run the command?: /usr/local/sbin/pftop -w 150 -a -b -v rules

pst

@stephenw10 said in 25.03-BETA: pftop core dump:

/usr/local/sbin/pftop -w 150 -a -b -v rules

I have used /status.php a lot recently so the entries in the system log most likely relates to those runs.

When I run the command directly I get

 605  90       Any     Q !igb0  sprite-rp          0        0        *       pass inet6 from <pfB_CF_IP_Whitelist_v6> to any port = http user group  f
Bus error (core dumped)

but I'm not sure if its related to a specific rule, if I pipe the command through tail I get a different result:

595  12       Any       igb0   tcp                0        0        0      Bus error (core dumped)
[25.03-BETA][root@pfsense.local.lan]/root:

stephenw10

Mmm, OK I replicated that. I don't think it's rule specific. Seems like a bug in pftop when called with those options.

stephenw10

Hmm, nope more specific than that. I can only replicate it on two test boxes. Most 25.03 instances don't hit this.

pst

@stephenw10 pftop seems a bit flakey. I actually managed to view the rules a couple of times today by just starting pftop and then navigate to rules. But now that doesn't work any more. Running the full command gives randomly either bus error, segmentation fault or the assert