Internet access fail, can't ping pfSense



  • Hi,

    We started having an annoying issue lately, recurring every few days:
    Office are loosing access to the internet, remote home users can't reach the office.

    • pinging the pfSense from LAN (where it's default gateway) gives request timeout
    • pinging pfSense from the world gives request timeout (it's usually pingable, of course)

    I can't reach the WebGUI from LAN
    When I try ping (both a LAN address as well as an internet address) from the console, I get ping: sendto: operation not permitted
    Restarting the pfSense machine solves the problem, untill next time.

    We're running 2.0.1-RELEASE (i386) on a Vmware virtual machine
    I've just enabled the remote syslog server so I don't have a log yet untill next failure (the 2000 entries in system log are taken by the reboot messages)

    Any help will be appreciated!
    Let me know of any more details you might need for troubleshooting.

    Thanks


  • LAYER 8 Global Moderator

    Well you running it as VM, can you look at the console of pfsense on your VM server to see what has happened?  What does it show?



  • well, here's a screenshot of the console on the time of failure
    (attached as on failure.png)
    the message shows:
    WARNING: pseudo-random number generator used for IPsec processing

    I also attach a screenshot of the ping attempt to LAN from console - (pinging to world gave same result (ping on failure.png)
    And a ping from my workstation (192.168.0.1) to the pfSense (192.168.0.254) (on reset.png)while resetting the pfSense host.

    Hope this can this point to somewhere useful
    Thanks!

    ![ping on failure.PNG](/public/imported_attachments/1/ping on failure.PNG)
    ![ping on failure.PNG_thumb](/public/imported_attachments/1/ping on failure.PNG_thumb)
    ![on failure.PNG](/public/imported_attachments/1/on failure.PNG)
    ![on failure.PNG_thumb](/public/imported_attachments/1/on failure.PNG_thumb)
    ![on reset.PNG](/public/imported_attachments/1/on reset.PNG)
    ![on reset.PNG_thumb](/public/imported_attachments/1/on reset.PNG_thumb)


  • LAYER 8 Global Moderator

    from the console access the shell and look at the logs.  Are your interfaces up?  They seem to be from the status? You can view logs with clog command at the shell.



  • @YossiH:

    When I try ping (both a LAN address as well as an internet address) from the console, I get ping: sendto: operation not permitted
    Restarting the pfSense machine solves the problem, untill next time.

    I suspect you have run out of network resources, most likely mbufs. Use pfSense shell command```

    netstat -m



  • Hi all,
    johnpoz, excuse my ignorance.
    I'll learn how to use the clog and see what's in there - I have to look at the log portion on time of failure… if you can point me to some tutorial it will be great.

    wallabybob, I attached the first netstat -m results, after ~10 hours uptime.  
    I'll post more later. do you see anything?

    tnx, much appreciated!

    ![netstat 20121001-2250.PNG](/public/imported_attachments/1/netstat 20121001-2250.PNG)
    ![netstat 20121001-2250.PNG_thumb](/public/imported_attachments/1/netstat 20121001-2250.PNG_thumb)


  • LAYER 8 Global Moderator

    from shell type clog and the file you want to look at, clog /var/etc/system.log

    you could pipe with more if you want
    clog /var/etc/system.log | more

    You could look end and have it pop up new entries with -f

    clog -f /var/etc/system.log

    you could send it to file that you could copy off and look at or post
    clog /var/etc/system.log >/tmp log.txt

    etc.. etc.  look in your var/etc dir for what log files you might want to look at

    
    [2.1-BETA0][admin@pfsense.local.lan]/var/log(8): ls
    apinger.log        l2tps.log          poes.log           system.log
    dhcpd.log          lastlog            portalauth.log     userlog
    dmesg.boot         lighttpd.error.log ppp.log            vpn.log
    filter.log         lighttpd.log       pptps.log          wireless.log
    gateways.log       ntp                relayd.log
    installer.log      ntpd.log           resolver.log
    ipsec.log          openvpn.log        routing.log
    
    

    http://doc.pfsense.org/index.php/Why_can't_I_view_view_log_files_with_cat/grep/etc%3F_(clog)



  • Got it!

    Strange thing is that log entries start only from the reboot onward, so I can't see the ones on time of failure.

    Is this expected? Are logs in RAM only?

    Might be because I forced the reboot from the VMware (and not from the pfSense console option)?

    attached result of clog /var/etc/system.log | more

    ???

    ![Clog system log .PNG](/public/imported_attachments/1/Clog system log .PNG)
    ![Clog system log .PNG_thumb](/public/imported_attachments/1/Clog system log .PNG_thumb)


  • LAYER 8 Global Moderator

    did you disable logging to local disk?  Under system log settings?

    "Local Logging Disable writing log files to the local disk"

    Either way, next time it fails - access it via console on your esxi box and look at the logs.  Do they show anything?

    I just looked at first entry in my system.log and it goes back to

    [2.1-BETA0][admin@pfsense.local.lan]/var/log(7): clog /var/log/system.log | more
    ing dynamic DNS entry.
    Aug 23 21:42:04 pfsense php: : DynDns: updatedns() starting

    That is before I did my last snap upgrade even which was like 9 days ago.  So the log entries should be there going back for a while - depending on how much is logged, etc.



  • I didn't…

    Sure, next time I'll look at the logs and netstat -m before rebooting.

    If there is anything else I should be looking at on time of failure, or some other setting I should prepare myself with, pls let me know.

    Thanks :-)

    ![sys log settings.PNG](/public/imported_attachments/1/sys log settings.PNG)
    ![sys log settings.PNG_thumb](/public/imported_attachments/1/sys log settings.PNG_thumb)


  • LAYER 8 Global Moderator

    So I would verify if you can ping other VMs on the same vswitch - when your connections fail, can pfsense even ping the devices on the same vswitch?

    That .7 address you were trying to ping is that on your physical network, same vswitch as pfsense, different vswitch?



  • on failure, I could reach other machines on the same physical host.
    .7 is the actual ESXi host where pfSense is guest

    still, the physical host has 4 NICs
    1 for the ESXi management, + other guests
    2 for pfSense LAN
    3 for pfSense WAN
    4 for pfSense ADSL (not in use)

    so while NIC 1 was reachable from the LAN, NIC 2, NIC3 were not reachable from LAN, WAN respectively, neither I could ping out of either from pfSense console.

    BTW. we managed to get the log entries for time of failure from the syslog server - see attached log on faiulure.png
    The failure occurred 22:33
    We dont see anything related to the failure in the logs (but we see it in the RSS graphs)

    but we see that, !!after!! failure the log entries were being sent to the syslog server on the LAN (.152), and ipSec was being established on the WAN  :o
    I get from it that the NICs were not completely "dead"

    In the mean time, we rolled back some changes we did before the problem started, mainly uninstalling bandwithD package.

    ![log on failure.PNG](/public/imported_attachments/1/log on failure.PNG)
    ![log on failure.PNG_thumb](/public/imported_attachments/1/log on failure.PNG_thumb)



  • An update:
    After everything was working fine for a few weeks after rolling back the changes,
    we installed packages:

    • ntop
    • Zabbix Agent
      in 48 hours, we had the issue occurring twice.

    I suspect it's one of the two.
    Uninstalled ntop
    Waiting to see what happens



  • no re occurance since uninstalling ntop
    previous cycle solution was to uninstall bandwithd

    so it's something to do with bandwith management packages together with our configuration.

    hope this helps someone :-)
    and thank you all for your assistance


Log in to reply