2.2.4 daily page faults - fresh install with reloaded config.
-
Hi folks, I have recently started getting crashes with pfsense (lasts from 5-24 hours then will crash/reboot).
I started getting the issue some weeks ago and could not track it down on 2.2.3 so moved to 2.2.4.
I am routing one subnet (192.168.20.x) out the openvpn but otherwise it's pretty standard stuff.
I have disables this gatweay and openvpn service (as well as uninstalled all but arping, cron and nrpe packages) but still get the crashes.Any help would be appreciated inl. pointers on additional logs etc.
end of the dump is as follows, which seems to point to inetd?
<118>Bootup complete <5>ovpnc1: link state changed to UP <118>Aug 26 19:09:50 ipsec_starter[48511]: shunt policy 'bypasslan' uninstalled <118>Aug 26 22:16:13 miniupnpd[54471]: remove port mapping 40413 TCP because it has expired Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xffffffff00000050 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80cf2820 stack pointer = 0x28:0xfffffe00002ff850 frame pointer = 0x28:0xfffffe00002ff880 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 23997 (inetd) version.txt06000024712567332332 7624 ustarrootwheelFreeBSD 10.1-RELEASE-p15 #0 c5ab052(releng/10.1)-dirty: Sat Jul 25 20:20:58 CDT 2015 root@pfs22-amd64-builder:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.10inetd.conf has the following
$ cat /var/etc/inetd.conf tftp-proxy dgram udp wait root /usr/libexec/tftp-proxy tftp-proxy -v 19000 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2300 19000 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2300 19001 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2301 19001 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2301 19002 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2302 19002 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2302 19003 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2303 19003 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2303 19004 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2304 19004 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2304 19005 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2305 19005 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2305 19006 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2306 19006 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2306 19007 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2307 19007 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2307 19008 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2308 19008 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2308 19009 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2309 19009 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2309 19010 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2310 19010 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2310 19011 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2010 19011 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2010 19012 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2011 19012 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2011 19013 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2012 19013 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2012 19014 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2013 19014 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2013 19015 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2014 19015 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2014 19016 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2015 19016 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2015 19017 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2016 19017 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2016 19018 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2017 19018 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2017 19019 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2018 19019 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2018 19020 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2019 19020 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2019 19021 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 2020 19021 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 2020 19022 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 27016 19022 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 27016 19023 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 24000 19023 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 24000 19024 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.225 80 19024 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.225 80 19025 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 20040 19025 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 20040 19026 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 9987 19026 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 9987 19027 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 10011 19027 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 10011 19028 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 30033 19028 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 30033 19029 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.10.3 3389 19029 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.10.3 3389 19030 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 20030 19030 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 20030 19031 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 20031 19031 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 20031 19032 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32450 19032 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32450 19033 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32451 19033 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32451 19034 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32452 19034 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32452 19035 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32453 19035 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32453 19036 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32454 19036 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32454 19037 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32455 19037 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32455 19038 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32456 19038 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32456 19039 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32457 19039 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32457 19040 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32458 19040 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32458 19041 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32459 19041 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32459 19042 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32460 19042 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32460 19043 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32461 19043 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32461 19044 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32462 19044 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32462 19045 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32463 19045 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32463 19046 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32464 19046 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32464 19047 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32465 19047 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32465 19048 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32466 19048 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32466 19049 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32467 19049 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32467 19050 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32468 19050 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32468 19051 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32469 19051 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32469 19052 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32470 19052 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32470 19053 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32471 19053 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32471 19054 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32472 19054 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32472 19055 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32473 19055 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32473 19056 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32474 19056 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32474 19057 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32475 19057 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32475 19058 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32476 19058 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32476 19059 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32477 19059 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32477 19060 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32478 19060 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32478 19061 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32479 19061 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32479 19062 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.6 32480 19062 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.6 32480 19063 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.2 25 19063 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.2 25 19064 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.2 110 19064 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.2 110 19065 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.2 143 19065 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.2 143 19066 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.2 587 19066 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.2 587 19067 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.2 3000 19067 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.2 3000 19068 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.2 3001 19068 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.2 3001 19069 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.2 3002 19069 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.2 3002 19070 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.2 3003 19070 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.2 3003 19071 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 192.168.0.11 52199 19071 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.11 52199 19072 dgram udp wait nobody /usr/bin/nc nc -u -w 2000 192.168.0.31 123 ```    -
I doubt it's directly related to the reflection, but does it stop if you switch to pure NAT mode reflection? That's a better option most of the time anyway.
-
Thanks for the suggestion.
I just made the change and will wait and see :) 24 hours to tell…
-
Well, that did not take long…
Different crash this time...
<118>Aug 28 18:44:05 ipsec_starter[47323]: shunt policy 'bypasslan' uninstalled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xa40c050150 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80cf0d26 stack pointer = 0x28:0xfffffe001abfa710 frame pointer = 0x28:0xfffffe001abfa7a0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi1: netisr 0) version.txt06000024712570030643 7616 ustarrootwheelFreeBSD 10.1-RELEASE-p15 #0 c5ab052(releng/10.1)-dirty: Sat Jul 25 20:20:58 CDT 2015 root@pfs22-amd64-builder:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.10 -
Something strange going on as I just posted a suggestion but it had dissappeared.
Anyway disable OpenVPN and see if that resolves the problem, as I suspect OpenVPN is being used to crash your system.
-
There is not enough information in the small portion of the crash dump posted to determine anything. Please post the entire crash dump, or submit it and let us know what IP address it was submitted from along with the approximate time.
Typically a crash that changes (different areas each time) tends to be more likely a hardware issue than a software issue, but without seeing the backtraces and other info it's impossible to determine or even make a proper educated guess.
-
Thanks Guys. Was not sure what was needed re. info.
Latest full dump is attached.
firewalluser, the crash happens even when OpenVPN is disabled (both gateway and service) - so I do not think it's openvpn causing it.
Thanks again folks!
ps. wrt hardware, it's running under esxi along with a number of other vm's. It had been running fine with no issues for months. None of the other vm's have issues. Incase it was a disk issue, I have tried cloning and also a fresh install (with reloaded config).
I am considering doing a fresh install and slowly adding in config. to see where it breaks but that is a REAL pain in the but! So hopefully the dump can narrow it down.
-
All of the backtraces are different but they all end in the same place, IPsec. Might be the same as one of the other IPsec crashes we've been tracking. Can you elaborate on your IPsec config (number of tunnels, ciphers used, etc)
-
Thanks for the quick response.
It is a very simple IPsec setup to enable remote VPN from an iPhone.
Let me know if you need more info. than the below screen caps.
I'll disable the IPsec VPN and see if the crashes stop.
Current Uptime 03 Hours 37 Minutes 37 Seconds
-
Uptime 1 Day 01 Hour 09 Minutes 00 Seconds
IPsec is looking like the culprit.
-
Uptime 2 Days 14 Hours 13 Minutes 21 Seconds
i think we have a winner!!
IPSec.
-
It's a bit strange, nothing on there would seem to be out of the ordinary… was the mobile IPsec device connected at all times? Or was it connected at all?
Curious if maybe the device was on at all times if it might have been timed such that the phone roamed from tower to tower or went to sleep/woke up, etc.
-
Hi Jimp.
The mobile was an occasional connection and had no correlation with the crashes.
I have since rebooted due to some isp issues. Once I get a chance, I'll turnthe ipsec back on and see how it behaves.
Thanks.
-
In android, you can down load free apps which will force your phone to use a particular cell tower, this will remove one variable namely you phone switching between cell towers as phone companies have software running on these towers to bunk users around to load balance the connection, but it can be overridden with a simple free app which also happens to make it harden to triangulate your position.
Dont know if similar apps exist on iphone or others, androids a bit of a free for all.
-
ok, so it's been running sold with no crash for a few days now with IPSec turned off…
Now, I'll turn it on and see what happens
-
Turned it on and it lasted 12 hours before crashing :(
-
So it's definitely IPsec then. We've seen some other IPsec crashes but I'm not sure we've seen anything that regular, especially for a mobile only tunnel.
We are bringing back a bunch of IPsec updates from FreeBSD as soon as we can, might be in a 2.2.5 release, though I don't think it's there yet.
-
You mentioned an iPhone, is that just a VPN for a single iPhone? If not, knowing which other devices and how many might help.
-
Thanks guys.
Yes, it's just a single iphone that accesses the IPSec VPN (mine). Note is will crash when the vpn is enabled but there has been no access via the iphone/vpn. So just being enabled will cause a crash…not accessing it.
Happy to provide whatever config., logs and do whatever tests you guys want to help narrow it down.
I guess I can also look at moving over to openvpn client on the iphone.
-
If you could get me a backup of your config, that would definitely help. Can PM it to me here, or email to cmb at pfsense dot org, or email me to arrange other means of transfer. I don't see a means of replicating from that, so that should help.
