Internet access fail, can't ping pfSense

[YossiH]

Hi,

We started having an annoying issue lately, recurring every few days:
Office are loosing access to the internet, remote home users can't reach the office.

• pinging the pfSense from LAN (where it's default gateway) gives request timeout
• pinging pfSense from the world gives request timeout (it's usually pingable, of course)

I can't reach the WebGUI from LAN
When I try ping (both a LAN address as well as an internet address) from the console, I get ping: sendto: operation not permitted
Restarting the pfSense machine solves the problem, untill next time.

We're running 2.0.1-RELEASE (i386) on a Vmware virtual machine
I've just enabled the remote syslog server so I don't have a log yet untill next failure (the 2000 entries in system log are taken by the reboot messages)

Any help will be appreciated!
Let me know of any more details you might need for troubleshooting.

Thanks

[johnpoz]

Well you running it as VM, can you look at the console of pfsense on your VM server to see what has happened?  What does it show?

[YossiH]

well, here's a screenshot of the console on the time of failure
(attached as on failure.png)
the message shows:
WARNING: pseudo-random number generator used for IPsec processing

I also attach a screenshot of the ping attempt to LAN from console - (pinging to world gave same result (ping on failure.png)
And a ping from my workstation (192.168.0.1) to the pfSense (192.168.0.254) (on reset.png)while resetting the pfSense host.

Hope this can this point to somewhere useful
Thanks!


[johnpoz]

from the console access the shell and look at the logs.  Are your interfaces up?  They seem to be from the status? You can view logs with clog command at the shell.

[wallabybob]

@YossiH:

When I try ping (both a LAN address as well as an internet address) from the console, I get ping: sendto: operation not permitted
Restarting the pfSense machine solves the problem, untill next time.

I suspect you have run out of network resources, most likely mbufs. Use pfSense shell command```

netstat -m

[YossiH]

Hi all,
johnpoz, excuse my ignorance.
I'll learn how to use the clog and see what's in there - I have to look at the log portion on time of failure… if you can point me to some tutorial it will be great.

wallabybob, I attached the first netstat -m results, after ~10 hours uptime.  
I'll post more later. do you see anything?

tnx, much appreciated!


[johnpoz]

from shell type clog and the file you want to look at, clog /var/etc/system.log

you could pipe with more if you want
clog /var/etc/system.log | more

You could look end and have it pop up new entries with -f

clog -f /var/etc/system.log

you could send it to file that you could copy off and look at or post
clog /var/etc/system.log >/tmp log.txt

etc.. etc.  look in your var/etc dir for what log files you might want to look at

[2.1-BETA0][admin@pfsense.local.lan]/var/log(8): ls
apinger.log        l2tps.log          poes.log           system.log
dhcpd.log          lastlog            portalauth.log     userlog
dmesg.boot         lighttpd.error.log ppp.log            vpn.log
filter.log         lighttpd.log       pptps.log          wireless.log
gateways.log       ntp                relayd.log
installer.log      ntpd.log           resolver.log
ipsec.log          openvpn.log        routing.log

http://doc.pfsense.org/index.php/Why_can%27t_I_view_view_log_files_with_cat/grep/etc%3F_%28clog%29

[YossiH]

Got it!

Strange thing is that log entries start only from the reboot onward, so I can't see the ones on time of failure.

Is this expected? Are logs in RAM only?

Might be because I forced the reboot from the VMware (and not from the pfSense console option)?

attached result of clog /var/etc/system.log | more

???


[johnpoz]

did you disable logging to local disk?  Under system log settings?

"Local Logging Disable writing log files to the local disk"

Either way, next time it fails - access it via console on your esxi box and look at the logs.  Do they show anything?

I just looked at first entry in my system.log and it goes back to

[2.1-BETA0][admin@pfsense.local.lan]/var/log(7): clog /var/log/system.log | more
ing dynamic DNS entry.
Aug 23 21:42:04 pfsense php: : DynDns: updatedns() starting

That is before I did my last snap upgrade even which was like 9 days ago.  So the log entries should be there going back for a while - depending on how much is logged, etc.

[YossiH]

I didn't…

Sure, next time I'll look at the logs and netstat -m before rebooting.

If there is anything else I should be looking at on time of failure, or some other setting I should prepare myself with, pls let me know.

Thanks :-)


[johnpoz]

So I would verify if you can ping other VMs on the same vswitch - when your connections fail, can pfsense even ping the devices on the same vswitch?

That .7 address you were trying to ping is that on your physical network, same vswitch as pfsense, different vswitch?

[YossiH]

on failure, I could reach other machines on the same physical host.
.7 is the actual ESXi host where pfSense is guest

still, the physical host has 4 NICs
1 for the ESXi management, + other guests
2 for pfSense LAN
3 for pfSense WAN
4 for pfSense ADSL (not in use)

so while NIC 1 was reachable from the LAN, NIC 2, NIC3 were not reachable from LAN, WAN respectively, neither I could ping out of either from pfSense console.

BTW. we managed to get the log entries for time of failure from the syslog server - see attached log on faiulure.png
The failure occurred 22:33
We dont see anything related to the failure in the logs (but we see it in the RSS graphs)

but we see that, !!after!! failure the log entries were being sent to the syslog server on the LAN (.152), and ipSec was being established on the WAN  :o
I get from it that the NICs were not completely "dead"

In the mean time, we rolled back some changes we did before the problem started, mainly uninstalling bandwithD package.


[YossiH]

An update:
After everything was working fine for a few weeks after rolling back the changes,
we installed packages:

• ntop
• Zabbix Agent
  in 48 hours, we had the issue occurring twice.

I suspect it's one of the two.
Uninstalled ntop
Waiting to see what happens

[YossiH]

no re occurance since uninstalling ntop
previous cycle solution was to uninstall bandwithd

so it's something to do with bandwith management packages together with our configuration.

hope this helps someone :-)
and thank you all for your assistance