NDP Table Data Missing (GUI)

RobbieTT · Sep 6, 2023, 4:13 PM

23.09.a.20230905.0551

No NDP data:

Sep 6 17:07:38	kernel		pid 79239 (ndp), jid 0, uid 0: exited on signal 11 (core dumped)

The error persists through reboots.

CLI works as normal with ndp -a so the issue is confined to the GUI alone.

️

jimp · Sep 6, 2023, 5:00 PM

Curious I was able to make this happen just once on one system here but then I refreshed and it worked, and now I can't make it happen again.

The GUI page just runs NDP same as you would from the CLI, but it uses ndp -na

jimp · Sep 6, 2023, 5:25 PM

I opened https://redmine.pfsense.org/issues/14755 to track this, though I tried on quite a few different systems and couldn't make it happen again.

RobbieTT · Sep 6, 2023, 5:49 PM

@jimp
Just checked and both ndp -a and ndp -na work just fine via the CLI.

I also noticed this at the end of the CLI table this time around:

Segmentation fault (core dumped)

I don't recall seeing that fault message before.

<118>Bootup complete
<6>pid 54956 (ndp), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 98799 (ndp), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 36036 (ndp), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 52178 (ndp), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 79239 (ndp), jid 0, uid 0: exited on signal 11 (core dumped)
<6>pid 95436 (ndp), jid 0, uid 0: exited on signal 11 (core dumped)
<6>ng0: changing name to 'pppoe0'
pf_test6: kif == NULL, if_xname pppoe0
<6>ng0: changing name to 'pppoe0'

️

RobbieTT · Sep 7, 2023, 6:54 PM

@jimp

For reasons unexplained the NDP table became populated again. Nothing has changed since yesterday or this morning but the data is back:

Weird but yes, it is intermittent, if you give it long enough.

[edit: Post reboot the NDP Table is empty again.]

️

RobbieTT · Sep 13, 2023, 6:31 PM

@jimp

Ok, after a few days of testing it seems the NDP Table only becomes populated after around 24 hours post-boot. This may be linked to previous observations that the idle/offline flags on the Status/DHCPv6 Leases page are invariably incorrect and that MAC and Hostnames are typically missing.

It is like the whole IPV6 stack has a gap between its normal functioning and the GUI monitoring and display pages, as the raw data via the CLI is correct.

[Issue remains present with v23.09.a.20230907.0600 too.]

️

cmcdonald · Sep 13, 2023, 6:31 PM

@RobbieTT

I have built ndp with debug symbols for testing. It would be great to get a core dump from this version

https://redmine.pfsense.org/issues/14755#note-3

RobbieTT · Sep 13, 2023, 8:15 PM

@cmcdonald

I guess that is on me but I've no real idea how to install this or recover back to normal again.

I've downloaded the file I can see that a couple of move files / mv command's are needed but don't presume I know what I am doing - both pfSense and BSD are new to me. Oh and I only have my in-production Netgate devise to test with (gulp!).

That said, my NDP table has vanished again this evening...

...so perhaps that is a sign I should crack-on with this.

️

cmcdonald · Sep 13, 2023, 8:44 PM

@RobbieTT

Download the ndp.full file from Redmine (this is the ndp binary built without compiler optimizations and with debugging symbols).
Navigate to Diagnostics > Command Prompt and upload ndp.full to pfSense, it should report Uploaded file to /tmp/ndp.full.
Still on Diagnostics > Command Prompt, enter mv /usr/sbin/ndp /usr/sbin/ndp.orig , This just renames your existing ndp binary so that the next step doesn't clobber it.
Now run mv /tmp/ndp.full /usr/sbin/ndp and then chmod +x /usr/sbin/ndp. The first command moves the new binary into the well-known location and the second command ticks the execute bit so that it can be, well, executed.

I would go ahead too and purge any existing core files with rm -f /root/*.core. Once you get an empty NDP Table again, there is a great chance that you've hit the issue and a new ndp.core file will be at /root/ndp.core, download this file and share it with me cmcdonald<at>netgate<dot>com

Thanks

RobbieTT · Sep 13, 2023, 8:44 PM

@cmcdonald
Perfect.

️

cmcdonald · Sep 15, 2023, 1:23 PM

@RobbieTT Any updates? We are eager to obtain a core dump from anyone who can replicate using the debugging build of ndp binary

RobbieTT · Sep 15, 2023, 1:31 PM

@cmcdonald

Not yet as this is my only pfSense device so I have to wait until the dark hours of the weekend to attempt it (remote working and all that).

I've kinda got used to having 2 routers when I was running EdgeRouters; it's more difficult to test things with only 1. I'm sure I will get another device at some point.

️

RobbieTT · Sep 16, 2023, 2:19 PM

@cmcdonald said in NDP Table Data Missing (GUI):

@RobbieTT Any updates? We are eager to obtain a core dump from anyone who can replicate using the debugging build of ndp binary

You should now have 2 emails - first with the core dump as per instructions and the second taken after a system reboot.

Good luck.

️

cmcdonald · Sep 18, 2023, 6:24 PM

@RobbieTT Thanks for the core. We were able to track this one down!

RobbieTT · Oct 17, 2023, 2:41 AM

@cmcdonald said in NDP Table Data Missing (GUI):

@RobbieTT Thanks for the core. We were able to track this one down!

Glad to help you and the team.

️

marcosm · Oct 17, 2023, 2:38 AM

@RobbieTT Has this happened since the fix?

RobbieTT · Oct 17, 2023, 2:38 AM

@marcosm
No, the NDP table has been fully populated in the GUI since the fix.

️