Another person with apinger problems



  • A few months ago, I changed our internal IP scope from 192.168.2.x to 10.20.1.x.  Around that time, we started having intermittent internet outages (2-10 times per day for about 30 seconds each) where I could ping the internal IP of the pfSense box (10.20.1.1) but could not ping our external internet gateway.  Either the internet gateway is going down or the pfSense box is breaking.  When the problem happens, pfSense shows stuff like this in the system log:

    Note that I installed the latest alpha of 2.2 (built on Sun Jul 13) but that did not seem to help.  Our ISP says everything looks fine on their end, even when our internet is down.  pfSense runs on Hyper-V and we use OpenVPN lightly.  Everything ran fine for many months until the intermittent issues started happening.

    
    Jul 28 16:06:57	php-fpm[98656]: /rc.start_packages: Restarting/Starting all packages.
    Jul 28 16:06:56	check_reload_status: Starting packages
    Jul 28 16:06:56	php-fpm[98656]: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 16:06:56	check_reload_status: Reloading filter
    Jul 28 16:06:56	php-fpm[98656]: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 16:06:56	php-fpm[98656]: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 16:06:55	check_reload_status: rc.newwanip starting ovpns1
    Jul 28 16:06:54	kernel: ovpns1: link state changed to UP
    Jul 28 16:06:54	check_reload_status: Reloading filter
    Jul 28 16:06:54	kernel: ovpns1: link state changed to DOWN
    Jul 28 16:06:54	php-fpm[81793]: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 16:06:54	php-fpm[81793]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 16:06:53	check_reload_status: Reloading filter
    Jul 28 16:06:53	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Jul 28 16:06:53	check_reload_status: Restarting ipsec tunnels
    Jul 28 16:06:53	check_reload_status: updating dyndns WANGW
    Jul 28 16:06:44	php-fpm[81793]: /rc.start_packages: Restarting/Starting all packages.
    Jul 28 16:06:43	check_reload_status: Starting packages
    Jul 28 16:06:43	php-fpm[66875]: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 16:06:43	check_reload_status: Reloading filter
    Jul 28 16:06:43	php-fpm[66875]: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 16:06:43	php-fpm[66875]: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 16:06:41	check_reload_status: rc.newwanip starting ovpns1
    Jul 28 16:06:41	php-fpm[66875]: /rc.start_packages: Restarting/Starting all packages.
    Jul 28 16:06:41	kernel: ovpns1: link state changed to UP
    Jul 28 16:06:41	check_reload_status: Reloading filter
    Jul 28 16:06:40	check_reload_status: Reloading filter
    Jul 28 16:06:40	kernel: ovpns1: link state changed to DOWN
    Jul 28 16:06:40	php-fpm[71745]: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 16:06:40	php-fpm[71745]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 16:06:40	check_reload_status: Starting packages
    Jul 28 16:06:40	php-fpm[66875]: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 16:06:40	check_reload_status: Reloading filter
    Jul 28 16:06:40	php-fpm[66875]: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 16:06:40	php-fpm[66875]: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 16:06:39	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Jul 28 16:06:39	check_reload_status: Restarting ipsec tunnels
    Jul 28 16:06:39	check_reload_status: updating dyndns WANGW
    Jul 28 16:06:38	check_reload_status: rc.newwanip starting ovpns1
    Jul 28 16:06:38	kernel: ovpns1: link state changed to UP
    Jul 28 16:06:38	check_reload_status: Reloading filter
    Jul 28 16:06:38	kernel: ovpns1: link state changed to DOWN
    Jul 28 16:06:37	php-fpm[52298]: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 16:06:37	php-fpm[52298]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 16:06:36	check_reload_status: Reloading filter
    Jul 28 16:06:36	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Jul 28 16:06:36	check_reload_status: Restarting ipsec tunnels
    Jul 28 16:06:36	check_reload_status: updating dyndns WANGW
    Jul 28 16:06:24	php-fpm[52298]: /rc.start_packages: Restarting/Starting all packages.
    Jul 28 16:06:23	check_reload_status: Starting packages
    Jul 28 16:06:23	php-fpm[49445]: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 16:06:23	check_reload_status: Reloading filter
    Jul 28 16:06:23	php-fpm[49445]: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 16:06:23	php-fpm[49445]: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 16:06:22	check_reload_status: rc.newwanip starting ovpns1
    Jul 28 16:06:21	kernel: ovpns1: link state changed to UP
    Jul 28 16:06:21	check_reload_status: Reloading filter
    Jul 28 16:06:21	kernel: ovpns1: link state changed to DOWN
    Jul 28 16:06:20	php-fpm[40031]: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 16:06:20	php-fpm[40031]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 16:06:19	check_reload_status: Reloading filter
    Jul 28 16:06:19	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Jul 28 16:06:19	check_reload_status: Restarting ipsec tunnels
    Jul 28 16:06:19	check_reload_status: updating dyndns WANGW
    Jul 28 16:06:19	php-fpm[40031]: /rc.start_packages: Restarting/Starting all packages.
    Jul 28 16:06:17	check_reload_status: Starting packages
    Jul 28 16:06:17	php-fpm: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 16:06:17	php-fpm: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 16:06:17	check_reload_status: Reloading filter
    Jul 28 16:06:17	php-fpm: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 16:06:17	php-fpm: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 16:06:17	php-fpm: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 16:06:17	php-fpm: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 16:06:16	check_reload_status: rc.newwanip starting ovpns1
    Jul 28 16:06:16	kernel: ovpns1: link state changed to UP
    Jul 28 16:06:16	check_reload_status: Reloading filter
    Jul 28 16:06:15	kernel: ovpns1: link state changed to DOWN
    Jul 28 16:06:15	php-fpm: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 16:06:15	php-fpm: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 16:06:15	php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 16:06:15	php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 16:06:14	check_reload_status: Reloading filter
    Jul 28 16:06:14	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Jul 28 16:06:14	check_reload_status: Restarting ipsec tunnels
    Jul 28 16:06:14	check_reload_status: updating dyndns WANGW
    Jul 28 15:51:15	php-fpm[75060]: /rc.start_packages: Restarting/Starting all packages.
    Jul 28 15:51:14	check_reload_status: Starting packages
    Jul 28 15:51:13	php-fpm[75060]: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 15:51:13	check_reload_status: Reloading filter
    Jul 28 15:51:13	php-fpm[75060]: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 15:51:13	php-fpm[75060]: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 15:51:12	check_reload_status: rc.newwanip starting ovpns1
    Jul 28 15:51:11	kernel: ovpns1: link state changed to UP
    Jul 28 15:51:11	check_reload_status: Reloading filter
    Jul 28 15:51:11	kernel: ovpns1: link state changed to DOWN
    Jul 28 15:51:11	php-fpm[65238]: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 15:51:11	php-fpm[65238]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 15:51:10	check_reload_status: Reloading filter
    Jul 28 15:51:10	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Jul 28 15:51:10	check_reload_status: Restarting ipsec tunnels
    Jul 28 15:51:10	check_reload_status: updating dyndns WANGW
    Jul 28 15:51:07	php-fpm[65238]: /rc.start_packages: Restarting/Starting all packages.
    Jul 28 15:51:06	check_reload_status: Starting packages
    Jul 28 15:51:05	php-fpm[57618]: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 15:51:05	check_reload_status: Reloading filter
    Jul 28 15:51:05	php-fpm[57618]: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 15:51:05	php-fpm[57618]: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 15:51:04	check_reload_status: rc.newwanip starting ovpns1
    Jul 28 15:51:03	kernel: ovpns1: link state changed to UP
    Jul 28 15:51:03	check_reload_status: Reloading filter
    Jul 28 15:51:03	check_reload_status: Reloading filter
    Jul 28 15:51:03	kernel: ovpns1: link state changed to DOWN
    Jul 28 15:51:02	php-fpm[57618]: /rc.start_packages: Restarting/Starting all packages.
    Jul 28 15:51:02	php-fpm[49675]: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 15:51:02	php-fpm[49675]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 15:51:01	check_reload_status: Starting packages
    Jul 28 15:51:01	php-fpm[21842]: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 15:51:01	php-fpm[21842]: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 15:51:01	php-fpm[21842]: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 15:51:01	check_reload_status: Reloading filter
    Jul 28 15:51:01	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Jul 28 15:51:01	check_reload_status: Restarting ipsec tunnels
    Jul 28 15:51:01	check_reload_status: updating dyndns WANGW
    Jul 28 15:51:00	check_reload_status: rc.newwanip starting ovpns1
    Jul 28 15:50:59	kernel: ovpns1: link state changed to UP
    Jul 28 15:50:59	check_reload_status: Reloading filter
    Jul 28 15:50:59	kernel: ovpns1: link state changed to DOWN
    Jul 28 15:50:59	php-fpm[21842]: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 15:50:59	php-fpm[21842]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 15:50:58	check_reload_status: Reloading filter
    Jul 28 15:50:58	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Jul 28 15:50:58	check_reload_status: Restarting ipsec tunnels
    Jul 28 15:50:58	check_reload_status: updating dyndns WANGW
    Jul 28 15:50:27	php-fpm[21842]: /rc.start_packages: Restarting/Starting all packages.
    Jul 28 15:50:26	check_reload_status: Starting packages
    Jul 28 15:50:26	php-fpm: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 15:50:26	php-fpm: /rc.newwanip: pfSense package system has detected an ip change -> 10.20.2.1 ... Restarting packages.
    Jul 28 15:50:26	check_reload_status: Reloading filter
    Jul 28 15:50:26	php-fpm: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 15:50:26	php-fpm: /rc.newwanip: rc.newwanip: on (IP address: 10.20.2.1) (interface: []) (real interface: ovpns1).
    Jul 28 15:50:26	php-fpm: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 15:50:26	php-fpm: /rc.newwanip: rc.newwanip: Informational is starting ovpns1.
    Jul 28 15:50:25	check_reload_status: rc.newwanip starting ovpns1
    Jul 28 15:50:25	kernel: ovpns1: link state changed to UP
    Jul 28 15:50:24	check_reload_status: Reloading filter
    Jul 28 15:50:24	kernel: ovpns1: link state changed to DOWN
    Jul 28 15:50:24	php-fpm: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 15:50:24	php-fpm: /rc.openvpn: OpenVPN: Resync server1 Carden VPN
    Jul 28 15:50:24	php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 15:50:24	php-fpm: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WANGW.
    Jul 28 15:50:23	check_reload_status: Reloading filter
    Jul 28 15:50:23	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Jul 28 15:50:23	check_reload_status: Restarting ipsec tunnels
    Jul 28 15:50:23	check_reload_status: updating dyndns WANGW
    
    


  • More information please.  A list of your interfaces, especially WAN, details about your VPN etc etc etc etc.

    You can see from the log that it's complaining about your WAN IP address changing.  What network is 10.20.2.x?  Is that your DHCP pool or something?



  • Thank you.  I didn't want to make my first post too long and annoying to read :)

    10.20.2.0/24 is the OpenVPN "IPv4 Tunnel Network"

    10.20.1.0/24 is the OpenVPN "IPv4 Local Network/s"

    pfSense LAN NIC is 10.20.1.1 /24

    pfSense WAN gateway is 24.?.?.121 (censored with question marks) (this is the IP address on the Cable Modem, which is plugged via ethernet directly into the server that hosts pfSense)

    pfSense WAN NIC has multiple Static IPs: 24.?.?.122 /29

    As far as I know, there's no reason for my external IP to change.  We pay for a static IP range.

    I forgot to mention, I see these messages in the Gateways log:

    
    Jul 28 17:09:14	apinger: Starting Alarm Pinger, apinger(18353)
    Jul 28 16:07:11	apinger: alarm canceled: WANGW(24.?.?.121) *** delay ***
    Jul 28 16:07:02	apinger: alarm canceled: WANGW(24.?.?.121) *** down ***
    Jul 28 16:06:56	apinger: ALARM: WANGW(24.?.?.121) *** down ***
    Jul 28 16:06:43	apinger: ALARM: WANGW(24.?.?.121) *** delay ***
    Jul 28 16:06:29	apinger: alarm canceled: WANGW(24.?.?.121) *** down ***
    Jul 28 16:06:26	apinger: ALARM: WANGW(24.?.?.121) *** down ***
    Jul 28 16:06:09	apinger: alarm canceled: WANGW(24.?.?.121) *** down ***
    Jul 28 16:06:04	apinger: ALARM: WANGW(24.?.?.121) *** down ***
    Jul 28 15:51:00	apinger: alarm canceled: WANGW(24.?.?.121) *** delay ***
    Jul 28 15:50:51	apinger: alarm canceled: WANGW(24.?.?.121) *** down ***
    Jul 28 15:50:48	apinger: ALARM: WANGW(24.?.?.121) *** down ***
    Jul 28 15:50:13	apinger: ALARM: WANGW(24.?.?.121) *** delay ***
    
    


  • I see you have the apinger problem that some people have, where it thinks your link is going up & down like a yo-yo, and it may be the root of your problems.  I don't know about this scenario to be able to point you in the right direction.  Perhaps do a forum search for php-fpm /rc.newwanip: pfSense package system has detected an ip change and follow the breadcrumbs.



  • FYI I just updated to the latest alpha 2.2 built on Mon Jul 28 12:22:20 CDT 2014 and I still have the problem.

    I remember reading something about the apinger in the forums a while back.  I just did a search for "apinger" and see that lots of people are having problems with this. I'll disable apinger for now and watch the forums for a fix.

    Thank you very much for your time, KOM!  And thank you, developers, for pfSense!

    –EDIT I renamed this forum thread to attract less attention now that I believe we've found the problem.