4. Pfsense IPSec tunnel going down and up repeatedly

Pfsense IPSec tunnel going down and up repeatedly

  • S

    i just took over a new client who already has pfsense (2.1.5-i386) installed.

    as typical, i created an IPsec tunnel from my pfsense (2.1.4-amd64) to this new one.  my nagios server sees the internal IP of this firewall going up and down every few minutes.  i also monitor their default gateway, and it is not flapping.  also, when the ipsec tunnel is down, i can log into the admin-gui at the external interface (thus this issue seems to be affecting only the ipsec tunnel, the client is not complaining that the internet is up and down all day).

    i have 18 other tunnels, and none of them go up and down unexpectedly.  i dont see events on this particular firewalls system logs that match why this tunnel would be accessable/unaccessable.  this is typical events i see in nagios:

    December 04, 2014 19:00

    Host Up[12-04-2014 19:09:12] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 60.41 ms
    Host Down[12-04-2014 19:08:08] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 19:06:38] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 19:01:08] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 57.55 ms
    Host Down[12-04-2014 19:00:04] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%

    December 04, 2014 18:00

    Host Down[12-04-2014 18:58:34] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 18:53:04] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 62.33 ms
    Host Down[12-04-2014 18:52:00] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 18:50:30] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 18:39:56] HOST ALERT: pfsense_IPSec;UP;SOFT;2;PING OK - Packet loss = 0%, RTA = 58.54 ms
    Host Down[12-04-2014 18:38:52] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 18:28:18] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 66.22 ms
    Host Down[12-04-2014 18:27:14] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 18:25:44] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 18:15:10] HOST ALERT: pfsense_IPSec;UP;SOFT;2;PING WARNING - Packet loss = 90%, RTA = 61.11 ms
    Host Down[12-04-2014 18:13:40] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 18:03:06] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 67.03 ms
    Host Down[12-04-2014 18:02:02] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 18:00:32] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;(Host check timed out after 30.00 seconds)

    December 04, 2014 17:00

    Host Up[12-04-2014 17:55:02] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 55.79 ms
    Host Down[12-04-2014 17:53:58] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 17:52:28] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 17:41:54] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 57%, RTA = 2761.17 ms
    Host Down[12-04-2014 17:40:41] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 17:39:11] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 17:23:17] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING WARNING - Packet loss = 93%, RTA = 1143.25 ms
    Host Down[12-04-2014 17:21:47] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 17:20:17] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 17:09:37] HOST ALERT: pfsense_IPSec;UP;SOFT;2;PING WARNING - Packet loss = 83%, RTA = 56.88 ms
    Host Down[12-04-2014 17:08:08] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%

    December 04, 2014 16:00

    Host Up[12-04-2014 16:57:34] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 61.62 ms
    Host Down[12-04-2014 16:56:30] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 16:55:00] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 16:49:30] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 63.67 ms
    Host Down[12-04-2014 16:48:26] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 16:46:56] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 16:41:26] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 57.36 ms
    Host Down[12-04-2014 16:40:22] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 16:38:52] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 16:33:22] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 61.83 ms
    Host Down[12-04-2014 16:32:18] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 16:30:48] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 16:25:18] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 81.23 ms
    Host Down[12-04-2014 16:24:14] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 16:22:44] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 16:12:10] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 72.42 ms
    Host Down[12-04-2014 16:11:06] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 16:09:36] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 16:04:06] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 62.22 ms
    Host Down[12-04-2014 16:03:02] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 16:01:32] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%

    December 04, 2014 15:00

    Host Up[12-04-2014 15:56:02] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 59.25 ms
    Host Down[12-04-2014 15:54:58] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 15:53:28] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 15:47:58] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 769.69 ms
    Host Down[12-04-2014 15:46:54] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 15:45:24] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 15:39:54] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 129.40 ms
    Host Down[12-04-2014 15:38:49] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 15:37:19] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 15:31:49] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 150.47 ms
    Host Down[12-04-2014 15:30:45] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 15:29:15] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 15:18:41] HOST ALERT: pfsense_IPSec;UP;SOFT;2;PING OK - Packet loss = 0%, RTA = 305.24 ms
    Host Down[12-04-2014 15:17:37] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%
    Host Up[12-04-2014 15:07:03] HOST ALERT: pfsense_IPSec;UP;SOFT;3;PING OK - Packet loss = 0%, RTA = 62.38 ms
    Host Down[12-04-2014 15:05:59] HOST ALERT: pfsense_IPSec;DOWN;SOFT;2;PING CRITICAL - Packet loss = 100%
    Host Down[12-04-2014 15:04:29] HOST ALERT: pfsense_IPSec;DOWN;SOFT;1;PING CRITICAL - Packet loss = 100%

    the above, its like that every day, all day.  the pfsense system logs do show that openvpn interfaces are restarting frequently (openvpn is not in use), but they dont match the up/downs 100%.

    can anyone give me any ideas where i can start to troubleshoot this issue?

    0
  • Z

    I would first start to check connection stability on the 19th site since as you say, 18 others sites connected with tunnels works fine.

    Maby we have a dns server problem or a bad dsl line on the 19th site.

    Try to ping google a hundreed times with the console and check if there are lost packet.

    windows command line:    ping google.ca -n 100

    Does the pfsense box on this site is connected directly to a cable or dsl modem configured in BRIDGE MODE? (PfSense obtaining directly the public ip adress)?

    Does something else drain or sature the bandwidth on that specific site?

    Zikmen

    0
  • S

    when i took over, it did directly connect to the modem.  now the cable goes to the main switch and the DMZ is in its own separate VLAN.  also, the previous firewall god-knows-how-old DL380G1 was changed out for a VM running on an R710.

    with with a brand-new VM, the issue still persists.

    i called the internet provider, and they ran tests from their end to the modem, and everything seemed to check out.

    i suppose i could replace the cable from the modem to the switch and see if the cable is actually the issue, but im running out of hairs to pull out.

    0
  • C

    What do your IPsec logs show for that connection?

    0
Log in to reply
 

Products

Services

Support

News

Resources

Company

Our Mission

We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

Subscribe to our Newsletter

Product information, software announcements, and special offers. See our newsletter archive for past announcements.

© Copyright 2019 Rubicon Communications, LLC | Privacy Policy