Page fault while in kernel mode
I have two HP Proliant 360 G7 machines runnig 2.0.1 amd64 release.
They are configured to use WAN, SYNC and LAN interface with 30 VLAN interfaces configured on it.
The CARP is configured on WAN, LAN and all VLAN interfaces.
These machines are supposed to provide high-availability service under heavy load so I am doing failover and load tests to ensure that they are able to cope with it.
Unfortunately I was experiencing kernel panics from very beginning of my testing.
The machines panicked under various conditions:
- master server interfaces were disconnected from network to simulate network connectivity issue and first master panicked and slave server followed shortly and panicked as well
- master server was restarted and slave panicked
- carp was disabled on master and master panicked shortly followed by slave
- jmeter was setup to simulate heavy network traffic and after tens of minutes both machines panicked at some point
As the issue seemd to be in interrupt handling of bce driver, I decided to try it with different network cards so I used quad ports Intel PRO/1000 Network card and disabled onboard Broadcom cards. After this change the servers are more stable but I am still able to trigger kernel panic of master server by disabling carp on it if there is high network interface utilisation.
Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x290 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff806fb793 stack pointer = 0x28:0xffffff80000faa90 frame pointer = 0x28:0xffffff80000faab0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 <5>vip64: link state changed to DOWN processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (em0 taskq) db:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xffffff807ef0f300 curthread = 0xffffff00025e2000: pid 0 "em0 taskq" curpcb = 0xffffff80000fad40 fpcurthread = none idlethread = 0xffffff00023c6000: pid 11 "idle: cpu1" curpmap = 0 tssp = 0xffffffff811dab68 commontssp = 0xffffffff811dab68 rsp0 = 0xffffff80000fad40 gs32p = 0xffffffff811d99a0 ldt = 0xffffffff811d99e0 tss = 0xffffffff811d99d0 db:0:kdb.enter.default> bt Tracing pid 0 tid 64037 td 0xffffff00025e2000 _rw_rlock() at _rw_rlock+0x83 carp_forus() at carp_forus+0x58 ether_input() at ether_input+0x144 em_rxeof() at em_rxeof+0x1d0 em_handle_que() at em_handle_que+0x4d taskqueue_run() at taskqueue_run+0x93 taskqueue_thread_loop() at taskqueue_thread_loop+0x46 fork_exit() at fork_exit+0x118 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000fad30, rbp = 0 ---
I also tried to change mbuf configuration according to Tuning_and_Troubleshooting_Network_Cards guide but it didn't change a thing.
Definitely something specific to that combination of hardware regardless of NIC. Couple potential things I'd suggest trying one or both of.
- If you don't have > 4 GB RAM, try i386. We've seen a small number of scenarios where driver problems only exist in amd64.
- try 2.1 since it has a newer FreeBSD base that at times works much better with new servers.
I'd go with #2 first personally, from my experience with some new servers.
I should have mentioned that I tried 2.0.1 i386 as well with the same results.
So I went for the second option and upgraded both firewalls to 2.1-BETA0-amd64-20121106-0059.
This time when I disabled the CARP the master FW panicked and backup one hanged (even console wasn't working any more so I had to force reboot it) so I lost network completely. The kernel panic on master was little bit different but that might be due to changes in freebsd:
Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x308 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80768bd2 stack pointer = 0x28:0xffffff80000fba00 <5>opt164_vip64: link state changed to DOWN frame pointer = 0x28:0xffffff80000fba40 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, <5>opt165_vip65: link state changed to DOWN IOPL = 0 current process = 0 (em0 que) 0xffffff00048b0588: tag ufs, type VDIR usecount 1, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xffffff00048070d8 ref 0 pages 1 lock type ufs: EXCL by thread 0xffffff00047c0460 (pid 61142) ino 10386432, on dev da0s1a 0xffffff001bc85000: tag ufs, type VREG usecount 2, writecount 1, refcount 2 mountedhere 0 flags () v_object 0xffffff0012f65870 ref 0 pages 0 lock type ufs: EXCL by thread 0xffffff00047c0460 (pid 61142) ino 10386521, on dev da0s1a db:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xffffff807ece5180 curthread = 0xffffff00025798c0: pid 0 "em0 que" curpcb = 0xffffff80000fbd10 fpcurthread = none idlethread = 0xffffff00023d68c0: tid 64006 "idle: cpu1" curpmap = 0xffffffff8137c6d0 tssp = 0xffffffff814011e8 commontssp = 0xffffffff814011e8 rsp0 = 0xffffff80000fbd10 gs32p = 0xffffffff81400020 ldt = 0xffffffff81400060 tss = 0xffffffff81400050 db:0:kdb.enter.default> bt Tracing pid 0 tid 64038 td 0xffffff00025798c0 _rw_rlock() at _rw_rlock+0x92 carp_forus() at carp_forus+0x5c ether_input() at ether_input+0x15f em_rxeof() at em_rxeof+0x1c2 em_handle_que() at em_handle_que+0x5b taskqueue_run_locked() at taskqueue_run_locked+0x85 taskqueue_thread_loop() at taskqueue_thread_loop+0x4e fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe
As it is all somewhere in network card/driver or interface utilisation, I used one of the Broadcom interfaces for pfSync and left just WAN and LAN to use intel card. Just to spread the load a little. Now the machines run for around 12 hours and I'm generating network utilisation around 900Mbit/s through them.
During that time I was able to disable/enable CARP 10 times but I wasn't able to reproduce the issue.
Do you think that it is safe to use 2.1 BETA or better to use 2.0.1?
Do you think that it is safe to use 2.1 BETA or better to use 2.0.1?
Every production system we run internally (3 colo datacenters, office, all of our boxes at home) are on 2.1. The biggest risk is that which is inherent in any nightly snapshot builds of anything, upgrading. If everything works on the particular snapshot you're on, it's not going to break. Unless you follow development very closely, there's always risk in upgrading to snapshot builds. Though when you're running a pair you can mitigate that, upgrade the secondary, disable CARP on the primary, after verifying the secondary is good, upgrade the primary. Or if possible, just don't upgrade at all until a RC or release comes out, since those are QAed and automatic snapshot builds aren't.