Constant: One or more OpenVPN tunnel endpoints may have changed its IP.



  • I have Pfsense 2.1 running on a Supermicro XSPA7-HF atom with 4 GB RAM fine.  Using snort, pfblocker and several packages with relatively low CPU utilization (4%) and RAM (17%) however the gateway has begun showing a yellow PACKETLOSS status (62%).  While I have several pfblocker lists from iblocklist.com and have raised my max table size to 800,000 entries, the pfsense home router doesn't appear to be breaking a sweat.  I am uploading at 5 Mbit to a cloud backup service though it's always has a healthy (green) Gateway status.  What would cause this?

    I've gone into System > Routing > Gateway and made the following changes, but OpenVPN keeps "flapping" and there aren't any active OpenVPN connections.

    Probe Interval = 10 seconds
    Down = 30 seconds

    Oct 18 09:44:46 php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Oct 18 09:44:43 check_reload_status: Reloading filter
    Oct 18 09:44:43 check_reload_status: Restarting OpenVPN tunnels/interfaces
    Oct 18 09:44:43 check_reload_status: Restarting ipsec tunnels
    Oct 18 09:44:43 check_reload_status: updating dyndns WANGW
    Oct 18 09:44:30 php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Oct 18 09:44:27 check_reload_status: Reloading filter
    Oct 18 09:44:27 check_reload_status: Restarting OpenVPN tunnels/interfaces
    Oct 18 09:44:27 check_reload_status: Restarting ipsec tunnels
    Oct 18 09:44:27 check_reload_status: updating dyndns WANGW
    Oct 18 09:44:16 php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Oct 18 09:44:13 check_reload_status: Reloading filter
    Oct 18 09:44:13 check_reload_status: Restarting OpenVPN tunnels/interfaces
    Oct 18 09:44:13 check_reload_status: Restarting ipsec tunnels
    Oct 18 09:44:13 check_reload_status: updating dyndns WANGW
    Oct 18 09:44:11 bandwidthd: DNS timeout for 192.168.0.5: This problem reduces graphing performance
    Oct 18 09:44:06 php: rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Oct 18 09:44:03 check_reload_status: Reloading filter
    Oct 18 09:44:03 check_reload_status: Restarting OpenVPN tunnels/interfaces
    Oct 18 09:44:03 check_reload_status: Restarting ipsec tunnels
    Oct 18 09:44:03 check_reload_status: updating dyndns WANGW
    Oct 18 09:43:31 syslogd: kernel boot file is /boot/kernel/kernel



  • If you really do have significant packet loss then you need to work out what the problem is on the WAN side and ISP. As the packet loss varies "randomly" apinger is going to declare WANGW down and up, resulting in stuff that might depend on WANGW being restarted.
    If you do only have a single WAN, then you can disable gateway monitoring, since there is nothing to failover to anyway. The downside of that is that you don't get to see latency, packet loss stats. I did put a feature request in RedMine to allow apinger to do its thing, reporting the stats, but not actually take any action. That might be useful for single-WAN installs so that the WAN can be monitored without triggering useless "failover" attempts. Just need some time to think about how to implement it…



  • Thanks.  I've been using pfsense successfully at home on a single WAN config (50 mbit down/5 mbit up) for several years.  Kept gateway monitor enable by default.  No issues.  I then recently configured pfsense traffic shaper to prioritize my VOIP, cloud backup and default traffic accordingly.  No issues including packet loss or latency.  However my upstream is 100% saturated by my cloud backup upload, that's when gateway monitor reports high latency and proceeds to check status on OpenVPN, etc.  Even if it's not OpenVPN being used (even if OpenVPN interface is disabled).

    Nevertheless, if I scale back the max upstream of my cloud backup on WAN to 80-90% (down from 100%) the latency monitor reports green/OK status.  I've since disabled the gateway monitor altogether and removed the 80-90% upstream WAN limit on the cloud backup traffic and the issue has gone away.

    I guess if my connection is now working and pfsense isn't restarting the interface driving up CPU utilization, etc. I'm fine.



  • You can also adjust the gateway monitoring parameters - expand the advanced section when editing the gateway parameters. You can make the latency and packet loss limits much higher so that apinger does not react when you flood the link as you describe with the cloud backup traffic. That's what I do on low speed links that are easily saturated.



  • @phil.davis:

    You can also adjust the gateway monitoring parameters - expand the advanced section when editing the gateway parameters. You can make the latency and packet loss limits much higher so that apinger does not react when you flood the link as you describe with the cloud backup traffic. That's what I do on low speed links that are easily saturated.

    Good suggestion.  Would you mind sharing the gateway parameters you used on lower speed links so that I can test with them?  I tried more relaxed settings, but perhaps they weren't as relaxed for a saturated highspeed connection as I had thought.  Thanks!



  • Have you tried prioritizing ICMP packets?  That should prevent the pings to the gateway from being dropped when the connection is being saturated and triggering apinger.



  • @dreamslacker:

    Have you tried prioritizing ICMP packets?  That should prevent the pings to the gateway from being dropped when the connection is being saturated and triggering apinger.

    No, but that's a great idea.  Would I prioritize ICMP to go into my highest priority queue alongside my VOIP traffic?  Is it just a single floating rule for my WAN interface for ANY ICMP traffic?  Or does there need to be both a TCP and UDP pair?



  • @miles267:

    No, but that's a great idea.  Would I prioritize ICMP to go into my highest priority queue alongside my VOIP traffic?  Is it just a single floating rule for my WAN interface for ANY ICMP traffic?  Or does there need to be both a TCP and UDP pair?

    ICMP is a protocol on its own.  Just select Out on WAN for the floating rule for any ICMP traffic.  Depending on your WAN connection type, you might also need to adjust the default Apinger parameters.  Nevertheless, it is still a good idea to scale back on the backup upload slightly.



  • @dreamslacker:

    @miles267:

    No, but that's a great idea.  Would I prioritize ICMP to go into my highest priority queue alongside my VOIP traffic?  Is it just a single floating rule for my WAN interface for ANY ICMP traffic?  Or does there need to be both a TCP and UDP pair?

    ICMP is a protocol on its own.  Just select Out on WAN for the floating rule for any ICMP traffic.  Depending on your WAN connection type, you might also need to adjust the default Apinger parameters.  Nevertheless, it is still a good idea to scale back on the backup upload slightly.

    So far, this single WAN-out (floating) rules for any ICMP traffic appears to have addressed my issue.  Will continue to monitor and confirm.



  • Would you mind sharing the gateway parameters you used on lower speed links so that I can test with them?

    It looks like you have found a better solution by giving ICMP priority - I should do the same!
    I have some very slow links (e.g. 256kbps, 384kbps) that are easily swamped by downloads. So I increase the latency to 3000-4000ms and packet loss to 30-40%, ping every 2 seconds and wait 30 seconds to declare a gateway down. The latency and packet loss parameters are a bit extreme! But it does mean that a gateway is only declared down if it really is struggling badly or is actually down. These parameters are something that really have to be tuned to what happens in real life - for modern-speed links much much lower values should be needed.



  • I am having the same issue with "One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints" constantly reloading. Causing the internet to go out for 3-5 minuets at a time (which can get really frustrating). I am on 2.2.4-RELEASE (amd64). I tried disabling gateway monitoring but that does not fix the issue. I would like to try this

    So far, this single WAN-out (floating) rules for any ICMP traffic appears to have addressed my issue.

    Would you mind posting a screenshot of that rule so I can mimic it? I am not really sure what that rule should look like.



  • Anyone have any thoughts? This seems to happen the most when I access my home network through openVPN from work and then remote desktop into my home computer, but I have also had it happen when I try and log into pfSense web GUI… seems pretty random. Here are some logs of it happening.

    9/22/2015 12:44	php-fpm[20222]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use PIA_VPN_VPNV4.
    9/22/2015 12:44	php-fpm[20222]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use PIA_VPN2_VPNV4.
    9/22/2015 12:44	check_reload_status: Reloading filter
    9/22/2015 12:44	check_reload_status: Restarting OpenVPN tunnels/interfaces
    9/22/2015 12:44	check_reload_status: Restarting ipsec tunnels
    9/22/2015 12:44	check_reload_status: updating dyndns PIA_VPN_VPNV4
    9/22/2015 12:44	check_reload_status: Reloading filter
    9/22/2015 12:44	check_reload_status: Restarting OpenVPN tunnels/interfaces
    9/22/2015 12:44	check_reload_status: Restarting ipsec tunnels
    9/22/2015 12:44	check_reload_status: updating dyndns PIA_VPN2_VPNV4
    9/22/2015 12:44	php-fpm: /rc.start_packages: Restarting/Starting all packages.
    9/22/2015 12:44	php-fpm: /rc.start_packages: Restarting/Starting all packages.
    9/22/2015 12:44	check_reload_status: Starting packages
    9/22/2015 12:44	php-fpm: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 10.XXX.XXX.XXX -> 10.XXX.XXX.XXX - Restarting packages.
    9/22/2015 12:44	php-fpm: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 10.XXX.XXX.XXX -> 10.XXX.XXX.XXX - Restarting packages.
    9/22/2015 12:44	php-fpm: /rc.start_packages: Restarting/Starting all packages.
    9/22/2015 12:44	php-fpm: /rc.start_packages: Restarting/Starting all packages.
    9/22/2015 12:44	php-fpm: /rc.newwanip: Creating rrd update script
    9/22/2015 12:44	php-fpm: /rc.newwanip: Creating rrd update script
    9/22/2015 12:44	php-fpm: /rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface opt3
    9/22/2015 12:44	php-fpm: /rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface opt3
    9/22/2015 12:44	check_reload_status: Starting packages
    9/22/2015 12:44	php-fpm[37931]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 10.XXX.XXX.XXX -> 10.XXX.XXX.XXX - Restarting packages.
    9/22/2015 12:44	php-fpm: /rc.newwanip: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.8 Copyright 2004-2015 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Wrote 0 deleted host decls to leases file. Wrote 0 new dynamic host decls to leases file. Wrote 21 leases to leases file. Listening on BPF/em1/0X:3X:XX:XX:XX:XX/192.XXX.XXX.0/24 Sending on BPF/em1/0X:3X:XX:XX:XX:XX/192.XXX.XXX.0/24 Can't bind to dhcp address: Address already in use Please make sure there is no other dhcp server running and that there's no entry for dhcp or bootp in /etc/inetd.conf. Also make sure you are not running HP JetAdmin software, which includes a bootp server. If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get
    9/22/2015 12:44	php-fpm: /rc.newwanip: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.8 Copyright 2004-2015 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Wrote 0 deleted host decls to leases file. Wrote 0 new dynamic host decls to leases file. Wrote 21 leases to leases file. Listening on BPF/em1/0X:3X:XX:XX:XX:XX/192.XXX.XXX.0/24 Sending on BPF/em1/0X:3X:XX:XX:XX:XX/192.XXX.XXX.0/24 Can't bind to dhcp address: Address already in use Please make sure there is no other dhcp server running and that there's no entry for dhcp or bootp in /etc/inetd.conf. Also make sure you are not running HP JetAdmin software, which includes a bootp server. If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get
    9/22/2015 12:44	php-fpm[37931]: /rc.newwanip: Creating rrd update script
    9/22/2015 12:44	php-fpm[37931]: /rc.newwanip: Ignoring IPsec reload since there are no tunnels on interface opt2
    9/22/2015 12:44	php-fpm: /rc.newwanip: IP has changed, killing states on former IP 10.XXX.XXX.XXX.
    9/22/2015 12:44	php-fpm: /rc.newwanip: IP has changed, killing states on former IP 10.XXX.XXX.XXX.
    9/22/2015 12:44	php-fpm: /rc.newwanip: rc.newwanip: on (IP address: 10.XXX.XXX.XXX) (interface: PIA_VPN2[opt3]) (real interface: ovpnc1).
    9/22/2015 12:44	php-fpm: /rc.newwanip: rc.newwanip: on (IP address: 10.XXX.XXX.XXX) (interface: PIA_VPN2[opt3]) (real interface: ovpnc1).
    9/22/2015 12:44	php-fpm: /rc.newwanip: rc.newwanip: Info: starting on ovpnc1.
    9/22/2015 12:44	php-fpm: /rc.newwanip: rc.newwanip: Info: starting on ovpnc1.
    9/22/2015 12:44	php-fpm[37931]: /rc.newwanip: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.8 Copyright 2004-2015 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Wrote 0 deleted host decls to leases file. Wrote 0 new dynamic host decls to leases file. Wrote 21 leases to leases file. Listening on BPF/em1/0X:3X:XX:XX:XX:XX/192.XXX.XXX.0/24 Sending on BPF/em1/0X:3X:XX:XX:XX:XX/192.XXX.XXX.0/24 Can't bind to dhcp address: Address already in use Please make sure there is no other dhcp server running and that there's no entry for dhcp or bootp in /etc/inetd.conf. Also make sure you are not running HP JetAdmin software, which includes a bootp server. If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you d
    9/22/2015 12:44	check_reload_status: rc.newwanip starting ovpnc1
    9/22/2015 12:44	kernel: ovpnc1: link state changed to UP
    9/22/2015 12:44	check_reload_status: Reloading filter
    9/22/2015 12:44	kernel: ovpnc1: link state changed to DOWN
    9/22/2015 12:44	php-fpm[37931]: /rc.newwanip: IP has changed, killing states on former IP 10.XXX.XXX.XXX.
    9/22/2015 12:44	php-fpm[37931]: /rc.newwanip: rc.newwanip: on (IP address: 10.XXX.XXX.XXX) (interface: PIA_VPN[opt2]) (real interface: ovpnc2).
    9/22/2015 12:44	php-fpm[37931]: /rc.newwanip: rc.newwanip: Info: starting on ovpnc2.
    9/22/2015 12:44	check_reload_status: rc.newwanip starting ovpnc2
    9/22/2015 12:44	kernel: ovpnc2: link state changed to UP
    9/22/2015 12:44	php-fpm: /rc.filter_configure_sync: Could not find IPv4 gateway for interface (opt2).
    9/22/2015 12:44	php-fpm: /rc.filter_configure_sync: Could not find IPv4 gateway for interface (opt2).
    9/22/2015 12:44	php-fpm: /rc.filter_configure_sync: Could not find IPv4 gateway for interface (opt2).
    9/22/2015 12:44	php-fpm: /rc.filter_configure_sync: Could not find IPv4 gateway for interface (opt2).
    9/22/2015 12:44	check_reload_status: Reloading filter
    9/22/2015 12:44	check_reload_status: Reloading filter
    9/22/2015 12:44	kernel: ovpnc2: link state changed to DOWN
    9/22/2015 12:39	php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
    9/22/2015 12:39	php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.
    9/22/2015 12:39	check_reload_status: Reloading filter
    9/22/2015 12:39	check_reload_status: Restarting OpenVPN tunnels/interfaces
    9/22/2015 12:39	check_reload_status: Restarting ipsec tunnels
    9/22/2015 12:39	check_reload_status: updating dyndns WAN_DHCP
    
    


  • About the ICMP rule, it's easier if you run the traffic shaper wizard and select hight priority for ICMP there; rule will be created automatically.

    The solutions are discussed above. I had the same issue as first post, and my main fix was more relaxed settings for gateway monitoring for WAN interface, and disabling gateway monitoring for the VPN interface(s). Since then it's been completely stable for months now. The ping priority setting didn't matter as much for me.

    I set the WAN gateway monitor to look out for disconnections only rather than a "poor quality" connection. Mine looks like this (attachment)




  • @pf3000:

    I set the WAN gateway monitor to look out for disconnections only rather than a "poor quality" connection. Mine looks like this (attachment)

    THANK YOU! I modified my settings to look very similar to yours and also turned of VPN monitoring (I didn't realize I had that turned on). I am hopeful that this will solve the issue. I'll report back after a few days.

    EDIT:
    I have made it a full 24 hours without any issues! First time in months that has happened.


Log in to reply