CARP/Load Balance on secondary firewall error
-
Hello all~
Running pfSense 2.1.5
We’re getting the following error on our secondary firewall. I think this is involving CARP and load balancing, but haven’t narrowed down exactly where the error is coming from. We are load balancing web servers using the pfSense built-in load balancing system. Each server uses a redundant CARP vlan. I performed a packet capture and found our load balanced servers were using the secondary firewall IP sometimes (not the CARP IP). I suspect this might be the cause of the issue. I tried putting the load balanced IP on a different VLAN / network per suggestions in a different forum, but the problem is still not resolved.
Also, We ran memtest86+ overnight on the secondary firewall to confirm that there were not memory issues. This is the second time our secondary firewall has crashed with the same error.
I have screen shots of the fatal trap 12 / back trace sections below, happy to email screenshots to anyone who needs.
opt19_vip61 2 link states coalesced
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address = 0x308
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff8077ad3a
stack pointer = 0x28:0xffffff8091c40770
frame pointer = 0x28:0xffffff8091c407a0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 23396 (php)
[ thread pid 23396 tid 100922 ]
Stopped at _rw_wlock_hard+0x6a: movl 0x308(%rcx),%edx
db>Also did back trace for more information:
db>bt
Tracing pid 23396 tid 100922 td 0xffffff008fa17000
_rw_wlock_hard() at _rq_wlock_hard+0x6a
in6_setscope() at in6_setscope+0x23a
in6_selectsrc() at in6_selectsrc+0x2cf
in6_pcbladdr() at in6_pcbladdr+0x149
in6_pcbconnect() at in6_pcbconnect+0x3f
udp6_connect() at udp6_connect+0x1f3
kern_connect() at kern_connect+0xae
connect() at connect+0x41
amd64_syscall() at amd64_syscall+0x1f4
Xfast_syscall() at Xfast_syscall+0xfc
–- syscall (98, FreeBSD ELF64, connect) rip = 0x8013da11c, rsp 8, rbp = 0x10 --
db>I can email the dmesg output to anyone who wants it, I cant seem to attach to this thread. Would guess its a bad char in the in txt file.
Any suggestions would be appreciated!