Crash Report - Fatal trap 12: page fault while in kernel mode (lsof)
-
@stephenw10
I am not running it manually, so honestly I am not really sure what is using it. I do have a couple manual scripts running to populate some Grafana dashboards that collect a fair amount of stats, but I do not remember installing lsof for it's use. Reviewing the bash scripts I do not see it listed.Running ps -ax does not show a lsof as of now
Packages:
Acme
apcupsd
iperf
nmap
ntopng
pfblockerng
service_watchdog
suricata
telegraf
wireguard -
Hmm, it appears lsof is a dependency of Telegraf: https://github.com/pfsense/FreeBSD-ports/blob/devel/net-mgmt/pfSense-pkg-Telegraf/Makefile#L17
How do you have it configured?
-
@stephenw10
Sure here is what I have. I will note that at the bottom there is a listed Github project, and files that should have been called; however, I did not replace them when I migrated to the Topton mini-pc from my old 1U atom that I had been using for the last +8 years.But from what you have shown it should be localized to the Telegraf package, so that really helps.
[[inputs.net]] interfaces = ["igc0", "igc1", "igc2", "igc3","tun_wg0", "tun_wg1", "tun_wg2", "tun_wg3"] [[inputs.conntrack]] [[inputs.filestat]] [[inputs.internal]] [[inputs.interrupts]] [[inputs.linux_sysctl_fs]] [[inputs.net]] [[inputs.net_response]] protocol = "tcp" address = "localhost:443" [[inputs.netstat]] [[inputs.nstat]] [[inputs.procstat]] pattern = "." prefix = "pgrep_serviceprocess" [[inputs.dns_query]] # ## servers to query # servers = ["8.8.8.8"] servers = ["208.67.222.222"] [[inputs.netstat]] # # no configuration # Read metrics about swap memory usage [[inputs.swap]] # no configuration [[inputs.ping]] # ## Hosts to send ping packets to. urls = ["208.67.222.222"] # # ## Method used for sending pings, can be either "exec" or "native". When set # ## to "exec" the systems ping command will be executed. When set to "native" # ## the plugin will send pings directly. # ## # ## While the default is "exec" for backwards compatibility, new deployments # ## are encouraged to use the "native" method for improved compatibility and # ## performance. # # method = "exec" # # ## Number of ping packets to send per interval. Corresponds to the "-c" # ## option of the ping command. # # count = 1 # # ## Time to wait between sending ping packets in seconds. Operates like the # ## "-i" option of the ping command. # # ping_interval = 1.0 # # ## If set, the time to wait for a ping response in seconds. Operates like # ## the "-W" option of the ping command. # # timeout = 1.0 # # ## If set, the total ping deadline, in seconds. Operates like the -w option # ## of the ping command. # # deadline = 10 # # ## Interface or source address to send ping from. Operates like the -I or -S # ## option of the ping command. # # interface = "" # # ## Specify the ping executable binary. # # binary = "ping" # # ## Arguments for ping command. When arguments is not empty, the command from # ## the binary option will be used and other options (ping_interval, timeout, # ## etc) will be ignored. # # arguments = ["-c", "3"] # # ## Use only IPv6 addresses when resolving a hostname. # # ipv6 = false #################### ## GIT: https://github.com/VictorRobellini/pfSense-Dashboard [[inputs.exec]] commands = [ "/usr/local/bin/telegraf_pfinterface.php", "/usr/local/bin/telegraf_gateways.py", "/usr/local/bin/telegraf_pfifgw.php", "sh /usr/local/bin/telegraf_temperature.sh", "sh /usr/local/bin/telegraf_pinger_loss.sh" ] data_format = "influx" [[inputs.logparser]] files = ["/var/log/pfblockerng/dnsbl.log"] from_beginning=true [inputs.logparser.grok] measurement = "dnsbl_log" patterns = ["^%{WORD:BlockType}-%{WORD:BlockSubType},%{SYSLOGTIMESTAMP:timestamp:ts-syslog},%{IPORHOST:destination:tag},%{IPORHOST:source:tag},%{GREEDYDATA:call},%{WORD:BlockMethod},%{WORD:BlockList},%{IPORHOST:tld:tag},%{WORD:DefinedList:tag},%{GREEDYDATA:hitormiss}"] timezone = "Local" [inputs.logparser.tags] value = "1" [[inputs.logparser]] files = ["/var/log/pfblockerng/ip_block.log"] from_beginning=true [inputs.logparser.grok] measurement = "ip_block_log" patterns = ["^%{SYSLOGTIMESTAMP:timestamp:ts-syslog},%{NUMBER:TrackerID},%{GREEDYDATA:Interface},%{WORD:InterfaceName},%{WORD:action},%{NUMBER:IPVersion},%{NUMBER:ProtocolID},%{GREEDYDATA:Protocol},%{IPORHOST:SrcIP:tag},%{IPORHOST:DstIP:tag},%{NUMBER:SrcPort},%{NUMBER:DstPort},%{WORD:Dir},%{WORD:GeoIP:tag},%{GREEDYDATA:AliasName},%{GREEDYDATA:IPEvaluated},%{GREEDYDATA:FeedName:tag},%{HOSTNAME:ResolvedHostname},%{HOSTNAME:ClientHostname},%{GREEDYDATA:ASN},%{GREEDYDATA:DuplicateEventStatus}"] timezone = "Local" [[inputs.unbound]] server = "127.0.0.1:953" binary = "/usr/local/bin/telegraf_unbound.sh" -
Which of those are custom scripts you've imported?
Can you see how lsof is being called? Or disable that as a test?
-
@stephenw10
This is the part of the config that called the custom scripts. But as they are not currently present I have removed this part of the config. Looking at the scrips I do not see any lsof reference.The second device had that same configs, and after I removed it the box rebooted for some reason. Now I cannot get to it, because it sometimes does not want to bring up the WAN correctly. Once online, I will have to see it if there was a crash report or not.
#################### ## GIT: https://github.com/VictorRobellini/pfSense-Dashboard [[inputs.exec]] commands = [ "/usr/local/bin/telegraf_pfinterface.php", "/usr/local/bin/telegraf_gateways.py", "/usr/local/bin/telegraf_pfifgw.php", "sh /usr/local/bin/telegraf_temperature.sh", "sh /usr/local/bin/telegraf_pinger_loss.sh" -
Mmm, it seem like it must be the
input.filestatcall. What does that actually report? Can you comment it out to test? -
@stephenw10 - Roger that, I really do appreciate the help. I see no reason to have that in the config as I am not using it. It is not commented out. I will have to better look at the others to confirm I am using.
One more question, if I could: After these crashes I usually see push notifications of the reboot and Pushover web API notifications. So it has internet access for a while, then the device goes unreachable with this type of error.
arpresolve: can't allocate llinfo for x.x.x.x (WAN IP GW) on igc0I have seen other posts, but I did not think I found a good resolution for the issue. I suppose have them stop crashing, but... Yeah, just thought I would ask.
-
@stephenw10 OK I kept looking at these as I did have another crash but this time with clock. Looking down the my list I am seeing another using lsof:
[[inputs.netstat]]- https://github.com/influxdata/telegraf/tree/master/plugins/inputs/netstat
Will keep looking and see if I use these specific network collection. Network is my specific use-case, so will just have to try.
-
Those arpresolve errors are usually nothing to worry about. It's trying to create an arp entry for the gateway but no longer has an interface in that subnet because it lost the WAN. As soon as the WAN comes back up it clears. You should only ever see it temporarily when that happens.
-
@stephenw10
I had another instance of a crash and reboot. It always seems to happen when my modem reboots, or maybe just when changes in state/connectivity of the WAN interface? Should I ask out on Telegraf's forum?I would post more, but I am getting flagged as spam?
-
Fatal trap 9: general protection fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0xffffffff80d4caa4 stack pointer = 0x28:0xfffffe0084131c00 frame pointer = 0x28:0xfffffe0084131c40 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 2 (clock (0))db:0:kdb.enter.default> show pcpu cpuid = 0 dynamic pcpu = 0x111bf80 curthread = 0xfffffe0011faa560: pid 2 tid 100041 critnest 1 "clock (0)" curpcb = 0xfffffe0011faaa80 fpcurthread = none idlethread = 0xfffffe0011ee63a0: tid 100003 "idle: cpu0" self = 0xffffffff84010000 curpmap = 0xffffffff83020ab0 tssp = 0xffffffff84010384 rsp0 = 0xfffffe0084132000 kcr3 = 0xffffffffffffffff ucr3 = 0xffffffffffffffff scr3 = 0x0 gs32p = 0xffffffff84010404 ldt = 0xffffffff84010444 tss = 0xffffffff84010434 curvnet = 0xfffff800012004c0 db:0:kdb.enter.default> bt Tracing pid 2 tid 100041 td 0xfffffe0011faa560 kdb_enter() at kdb_enter+0x32/frame 0xfffffe0084131940 vpanic() at vpanic+0x163/frame 0xfffffe0084131a70 panic() at panic+0x43/frame 0xfffffe0084131ad0 trap_fatal() at trap_fatal+0x40c/frame 0xfffffe0084131b30 calltrap() at calltrap+0x8/frame 0xfffffe0084131b30 --- trap 0x9, rip = 0xffffffff80d4caa4, rsp = 0xfffffe0084131c00, rbp = 0xfffffe0084131c40 --- turnstile_wait() at turnstile_wait+0x134/frame 0xfffffe0084131c40 __mtx_lock_sleep() at __mtx_lock_sleep+0x171/frame 0xfffffe0084131cd0 crfree() at crfree+0xaf/frame 0xfffffe0084131cf0 in_pcbfree() at in_pcbfree+0x280/frame 0xfffffe0084131d20 sorele_locked() at sorele_locked+0x89/frame 0xfffffe0084131d40 tcp_close() at tcp_close+0x159/frame 0xfffffe0084131d80 tcp_timer_2msl() at tcp_timer_2msl+0xf9/frame 0xfffffe0084131dd0 tcp_timer_enter() at tcp_timer_enter+0x101/frame 0xfffffe0084131e10 softclock_call_cc() at softclock_call_cc+0x134/frame 0xfffffe0084131ec0 softclock_thread() at softclock_thread+0xe9/frame 0xfffffe0084131ef0 fork_exit() at fork_exit+0x7f/frame 0xfffffe0084131f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084131f30 --- trap 0xa42be40b, rip = 0x8ba58ba52e552c55, rsp = 0xb48fb48f3dac3dac, rbp = 0x2bce2bca8e5e8e7e --- -
Upvoted a bunch of your posts, you should be good to avoid the spam filters now.
That looks like a completely different crash though. What, if anything, has changed since the last one?
I've seen that one time before and it seemed to be openvpn related.
-
@stephenw10
The change is the telegraf config file. I thought I saw some more stability in the package. When changing it over 3 other devices, some it caused that crash. I have had OpenVPN in the past, so it might linger in my config, but it is not currently installed as I moved over to WG exclusively. -
Hmm, might need to wait for another crash and see if it's identical. The only previous time we've seen this it was a one time incidents and we never found a cause.
