pfSense has detected a crash report or programming bug

rufus33

Hi i need help cause i really dunno whats causing this

Gertjan

You have a HA set up ?

...
pfsync_state_export() at pfsync_state_export+0x26/frame 0xfffffe00005bd950
pfsync_sendout() at pfsync_sendout+0x280/frame 0xfffffe00005bda00
pfsyncintr() at pfsyncintr+0xd1/frame 0xfffffe00005bda50
...

stephenw10

Mmm, that backtrace is not very revealing:

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100112 td 0xfffff80006179740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00005bd5f0
vpanic() at vpanic+0x197/frame 0xfffffe00005bd640
panic() at panic+0x43/frame 0xfffffe00005bd6a0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00005bd700
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00005bd750
trap() at trap+0x286/frame 0xfffffe00005bd860
calltrap() at calltrap+0x8/frame 0xfffffe00005bd860
--- trap 0xc, rip = 0xffffffff8109a0c6, rsp = 0xfffffe00005bd930, rbp = 0xfffffe00005bd950 ---
pfsync_state_export() at pfsync_state_export+0x26/frame 0xfffffe00005bd950
pfsync_sendout() at pfsync_sendout+0x280/frame 0xfffffe00005bda00
pfsyncintr() at pfsyncintr+0xd1/frame 0xfffffe00005bda50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00005bdab0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00005bdaf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005bdaf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Nothing much in the message buffer either.

More info needed. Did this just start happening? Did you make config changes? Update pfSense?

Steve

rufus33

@gertjan No Sir, Its just a simple setup which dials into a PPOE connection to the provider.

@stephenw10 this problem happens only when i restart PFSense. only way i can get it booting to normal condition is to turn off the Provider's ONU unit, wait for PFSense to boot successfully, and then turn ON the ONU back to dial PPOE. So its like, if the ONU Modem is on, PFsense goes into an infinite boot loop, ONU Off, everything works normally. Pfsense PC dials via Ethernet connection to the ONU Unit

stephenw10

And it has always done that? This didn't start happening when you upgraded pfSense for example?

rufus33

@stephenw10 Yes, this problem started on the recent update from pfsense, i even did a clean install and configuration of pfsense but the same problem remains.

Hardware wise, its running a Intel(R) Xeon(R) CPU E3-1225 v5 @ 3.30GHz with 16GB of ECC RAM. I have also cleaned the computer inside out including all the gold pins on various connections internally

stephenw10

Ok. What version did you upgrade from where it was working as expected?

Which NIC are you using for the PPPoE link?
Does it also work if you just unplug that link and then connect it after boot?

What NICs are in the bridge?

It looks like you have 4 ix NICs that are not being configured correctly:

ix0: <Intel(R) X540-AT2> mem 0x1fffc00000-0x1fffdfffff,0x1fffe04000-0x1fffe07fff at device 0.0 on pci3
ix0: The EEPROM Checksum Is Not Valid
ix0: IFDI_ATTACH_PRE failed 5
device_attach: ix0 attach returned 5
ix0: <Intel(R) X540-AT2> mem 0x1fffa00000-0x1fffbfffff,0x1fffe00000-0x1fffe03fff at device 0.1 on pci3
ix0: The EEPROM Checksum Is Not Valid
ix0: IFDI_ATTACH_PRE failed 5
device_attach: ix0 attach returned 5
ix0: <Intel(R) X540-AT2> mem 0x1fffc00000-0x1fffdfffff,0x1fffe04000-0x1fffe07fff at device 0.0 on pci3
ix0: The EEPROM Checksum Is Not Valid
ix0: IFDI_ATTACH_PRE failed 5
device_attach: ix0 attach returned 5
ix0: <Intel(R) X540-AT2> mem 0x1fffa00000-0x1fffbfffff,0x1fffe00000-0x1fffe03fff at device 0.1 on pci3
ix0: The EEPROM Checksum Is Not Valid
ix0: IFDI_ATTACH_PRE failed 5
device_attach: ix0 attach returned 5

Have those ever worked?

Steve

rufus33

@stephenw10 I cant really remember the version which worked but, it was the one before the following one:
2.6.0-RELEASE (amd64) | built on Mon Jan 31 19:57:53 UTC 2022 | FreeBSD 12.3-STABLE
The system is on the latest version. Version information updated at Tue Sep 13 1:20:03 +08 2022

PPOE im using the one built in to the motherboard (em0 with vlan 621). Unplugging this cable does work, meaning it can boot without infinite bootlooping. I have also tried with the other gigabit cards but the problem still occurs.

NICs bridged are the 2 IGB Intel Gigabit Cards, each card has 2 Gigabit ports used as local lan with dhcp.

the last card is a 10Gigabit card from Intel (X540-AT2) 2 ports, which is another issue where I'm trying to get this card working in PFsense. I'm not able to compile the latest driver from Intel within pfsense cause the tools does not exist, still figuring out on how to get this working, it works like a charm under a windows 10 install.

stephenw10

Hmm, OK.

Are you using VLAN 621 because the ISP requires it? Is em0 connected directly to the ONT?

The ix NIC is not connecting because the eeprom checksum is bad as shown in the error. A newer driver won't help there. The Windows driver either ignores it or just doesn't check. But you can correct that using Intel's NIC utility, probably in Windows but I've seen it done from FreeDOS etc.

Steve

rufus33

@stephenw10 Yes youre right, my provider requires me to place this vlan 621 for internet, and vlan822 for voip. Without this i wont be able to access the internet.

Yes, port em0 is directly connected to the ONU device via a RJ45 cat6 cable to ONU's RJ45 port 2,

AHh thank you very much for that tip on the NIC, i will try that and see what i get.

stephenw10

Hmm, are you able to try putting a switch between the em NIC and the ONU?

Or try using a different NIC as WAN?

rufus33

@stephenw10 I did use the other 3 more gigabit ports but the same problem exist.

I will try putting a switch in between today and let you know the result.

rufus33

@stephenw10 ok, placing a 5port Gigabit switch in between pfsense pc and the ONU device seems to solve the issue. No more boot looping. Now the weird thing is, i remove the switch and tried again with direct cable from pfsense pc to the onu device, now that also works without a problem. So confusing on why. So i tried restarting pfsense for almost 8 times now, all the runs boots up perfectly.

After the above test, regarding the Intel NIC issue, i removed the nic and used a windows pc to troubleshoot and managed to fix the EEPROM problem using intel's tools. Will try that today back in pfsense and let you know the result.

stephenw10

Hmm, well I can't explain the ONU boot issue but if the NIC utility found and fixed the checksum error I expect that to be recognised correctly in pfSense now.

Steve

rufus33

@stephenw10 Hi, everything works perfect now. No more boot loops and even the intels cards are now working in pfsense.. Thank you so much for your help. Really appreciate it.

stephenw10

Nice result!