Internet access fail, can't ping pfSense
-
Hi,
We started having an annoying issue lately, recurring every few days:
Office are loosing access to the internet, remote home users can't reach the office.- pinging the pfSense from LAN (where it's default gateway) gives request timeout
- pinging pfSense from the world gives request timeout (it's usually pingable, of course)
I can't reach the WebGUI from LAN
When I try ping (both a LAN address as well as an internet address) from the console, I get ping: sendto: operation not permitted
Restarting the pfSense machine solves the problem, untill next time.We're running 2.0.1-RELEASE (i386) on a Vmware virtual machine
I've just enabled the remote syslog server so I don't have a log yet untill next failure (the 2000 entries in system log are taken by the reboot messages)Any help will be appreciated!
Let me know of any more details you might need for troubleshooting.Thanks
-
Well you running it as VM, can you look at the console of pfsense on your VM server to see what has happened? What does it show?
-
well, here's a screenshot of the console on the time of failure
(attached as on failure.png)
the message shows:
WARNING: pseudo-random number generator used for IPsec processingI also attach a screenshot of the ping attempt to LAN from console - (pinging to world gave same result (ping on failure.png)
And a ping from my workstation (192.168.0.1) to the pfSense (192.168.0.254) (on reset.png)while resetting the pfSense host.Hope this can this point to somewhere useful
Thanks!![ping on failure.PNG](/public/imported_attachments/1/ping on failure.PNG)
![ping on failure.PNG_thumb](/public/imported_attachments/1/ping on failure.PNG_thumb)
![on failure.PNG](/public/imported_attachments/1/on failure.PNG)
![on failure.PNG_thumb](/public/imported_attachments/1/on failure.PNG_thumb)
![on reset.PNG](/public/imported_attachments/1/on reset.PNG)
![on reset.PNG_thumb](/public/imported_attachments/1/on reset.PNG_thumb) -
from the console access the shell and look at the logs. Are your interfaces up? They seem to be from the status? You can view logs with clog command at the shell.
-
When I try ping (both a LAN address as well as an internet address) from the console, I get ping: sendto: operation not permitted
Restarting the pfSense machine solves the problem, untill next time.I suspect you have run out of network resources, most likely mbufs. Use pfSense shell command```
netstat -m
-
Hi all,
johnpoz, excuse my ignorance.
I'll learn how to use the clog and see what's in there - I have to look at the log portion on time of failure… if you can point me to some tutorial it will be great.wallabybob, I attached the first netstat -m results, after ~10 hours uptime.
I'll post more later. do you see anything?tnx, much appreciated!
![netstat 20121001-2250.PNG](/public/imported_attachments/1/netstat 20121001-2250.PNG)
![netstat 20121001-2250.PNG_thumb](/public/imported_attachments/1/netstat 20121001-2250.PNG_thumb) -
from shell type clog and the file you want to look at, clog /var/etc/system.log
you could pipe with more if you want
clog /var/etc/system.log | moreYou could look end and have it pop up new entries with -f
clog -f /var/etc/system.log
you could send it to file that you could copy off and look at or post
clog /var/etc/system.log >/tmp log.txtetc.. etc. look in your var/etc dir for what log files you might want to look at
[2.1-BETA0][admin@pfsense.local.lan]/var/log(8): ls apinger.log l2tps.log poes.log system.log dhcpd.log lastlog portalauth.log userlog dmesg.boot lighttpd.error.log ppp.log vpn.log filter.log lighttpd.log pptps.log wireless.log gateways.log ntp relayd.log installer.log ntpd.log resolver.log ipsec.log openvpn.log routing.log
http://doc.pfsense.org/index.php/Why_can%27t_I_view_view_log_files_with_cat/grep/etc%3F_%28clog%29
-
Got it!
Strange thing is that log entries start only from the reboot onward, so I can't see the ones on time of failure.
Is this expected? Are logs in RAM only?
Might be because I forced the reboot from the VMware (and not from the pfSense console option)?
attached result of clog /var/etc/system.log | more
???
![Clog system log .PNG](/public/imported_attachments/1/Clog system log .PNG)
![Clog system log .PNG_thumb](/public/imported_attachments/1/Clog system log .PNG_thumb) -
did you disable logging to local disk? Under system log settings?
"Local Logging Disable writing log files to the local disk"
Either way, next time it fails - access it via console on your esxi box and look at the logs. Do they show anything?
I just looked at first entry in my system.log and it goes back to
[2.1-BETA0][admin@pfsense.local.lan]/var/log(7): clog /var/log/system.log | more
ing dynamic DNS entry.
Aug 23 21:42:04 pfsense php: : DynDns: updatedns() startingThat is before I did my last snap upgrade even which was like 9 days ago. So the log entries should be there going back for a while - depending on how much is logged, etc.
-
I didn't…
Sure, next time I'll look at the logs and netstat -m before rebooting.
If there is anything else I should be looking at on time of failure, or some other setting I should prepare myself with, pls let me know.
Thanks :-)
![sys log settings.PNG](/public/imported_attachments/1/sys log settings.PNG)
![sys log settings.PNG_thumb](/public/imported_attachments/1/sys log settings.PNG_thumb) -
So I would verify if you can ping other VMs on the same vswitch - when your connections fail, can pfsense even ping the devices on the same vswitch?
That .7 address you were trying to ping is that on your physical network, same vswitch as pfsense, different vswitch?
-
on failure, I could reach other machines on the same physical host.
.7 is the actual ESXi host where pfSense is gueststill, the physical host has 4 NICs
1 for the ESXi management, + other guests
2 for pfSense LAN
3 for pfSense WAN
4 for pfSense ADSL (not in use)so while NIC 1 was reachable from the LAN, NIC 2, NIC3 were not reachable from LAN, WAN respectively, neither I could ping out of either from pfSense console.
BTW. we managed to get the log entries for time of failure from the syslog server - see attached log on faiulure.png
The failure occurred 22:33
We dont see anything related to the failure in the logs (but we see it in the RSS graphs)but we see that, !!after!! failure the log entries were being sent to the syslog server on the LAN (.152), and ipSec was being established on the WAN :o
I get from it that the NICs were not completely "dead"In the mean time, we rolled back some changes we did before the problem started, mainly uninstalling bandwithD package.
![log on failure.PNG](/public/imported_attachments/1/log on failure.PNG)
![log on failure.PNG_thumb](/public/imported_attachments/1/log on failure.PNG_thumb) -
An update:
After everything was working fine for a few weeks after rolling back the changes,
we installed packages:- ntop
- Zabbix Agent
in 48 hours, we had the issue occurring twice.
I suspect it's one of the two.
Uninstalled ntop
Waiting to see what happens -
no re occurance since uninstalling ntop
previous cycle solution was to uninstall bandwithdso it's something to do with bandwith management packages together with our configuration.
hope this helps someone :-)
and thank you all for your assistance