An earnest appeal - please do fix APINGER in 2.2
-
+1
This is a quite annoying issue.and the outcome in my case is a nonfunctional apinger with miles of logentries like: apinger: No usable targets found, exiting
-
I'm having better luck with the 9/15 build. I am not sure if something changed between 9/13 and 9/15 that would have affected apinger, but the status seems to be working better. Still not 100% correct though. Before, the connection would show insanely high RTT in the console; now, that isn't the case. However, when I look at the logs I am getting a lot of apinger "down" log messages.
-
I'm having better luck with the 9/15 build.
Nevermind. I've been checking it off and on and haven't seen it go "Latency" or "Offline" today. It finally did. Typical stuck at really high RTT and jumps between latency and offline. I got too excited considering it hadn't gotten "stuck" for nearly a day. :(
-
Still no luck with Sep 14 09:09:38 CDT 2014 build, two pppoe links are online before my isp resets my connections, after the reset, default gw gets online normally, but the opt1 gets stuck although the link reconnected successfully.
And the OPT1 gets online again after I restart apinger.
-
Ok can you please try next snapshots.
I put a fix for recovering from down state. -
It definitely appears to recover better now than it ever did before. As you can tell from times on screenshots below, I did successfuly recover from a "down" state. You do occasionally catch it in a down state in the GUI when it isn't really down. Also, still getting the APINGER down log messages, but I assume that is normal?
-
Does your interface flap a lot or what is the reason you get all those down events?
Just to understand that you say it marks down occasionally.
-
I'm pretty confident those are false down notifications. I use nagios/centreon to monitor the same gateway host, and it never sends me a down alert. Of course maybe one packet dropped from pfSense but it didn't from my monitoring system, but even if it did I wouldn't expect it to take that long to recover.
In the screenshot I posted where it was down, I was able to ping the monitoring IP from the console successfully while it was showing the gateway down.
Just to give you a better idea of my setup, I am using the following:
Gateway IP is google DNS (8.8.8.8) - I know some suggest using first hop…don't know if that is contributing but I wouldn't think so.
I am also a hyper-v user (Server 08 R2). I've been building my own kernels for 2.1, but I am just using stock release since I've been running 2.2 alpha snapshots.Since I am using a non-traditional setup, perhaps it would be better to see if someone else is also having the problems I am before digging much deeper into it.
-
I do not understand how to try the new apinger.
Do I need to update to pfsense 2.2 beta? Can I just install the apinger package?
Can you provide a short explanation for me?Thanks,
Mario -
Just upgrade to latest snapshots from the system->Firmware.
-
Just got "image file is corrupt error" :-(
-
Just Successfully recovered from ISP reset on bulid Sep 19 23:21:59 CDT 2014, all online now, will keep an eye on …
Someone may want to know this.
-
And the Sep 23‘s new build don't work well for me, rebooted twice, and opt1 is always at offline status until apinger being restarted manually.
-
What kind of interface is OPT1?
Can you share some more details from your system log in what happened with this interface? -
WAN and OPT1 are both pppoe from same ISP.
The last two logs are created after apinger was restarted manually.
-
I meant logs from system logs.
Also do you have the same gateway ip for monitoring?
On new snapshots i have added a toggle under system->advanced->monitoring where you can enable debugging messages.
Can you enable that and see what you get in your apinger logs when this occurs? -
GW of WAN and OPT1 are same, but monitoring IPs are different, and the connection of OPT1 is OK, i can ping any hosts from webgui with source address of opt1.
And bellow is a list of logs under System–>Gerneral:
Sep 24 19:59:51 syslogd: kernel boot file is /boot/kernel/kernel
Sep 24 19:59:51 kernel: Copyright (c) 1992-2014 The FreeBSD Project.
Sep 24 19:59:51 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Sep 24 19:59:51 kernel: The Regents of the University of California. All rights reserved.
Sep 24 19:59:51 kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Sep 24 19:59:51 kernel: FreeBSD 10.1-PRERELEASE #32 30e366f(HEAD)-dirty: Wed Sep 24 05:03:13 CDT 2014
Sep 24 19:59:51 kernel: root@pf22-amd64-snap:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.10 amd64
Sep 24 19:59:51 kernel: FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
Sep 24 19:59:51 kernel: CPU: Intel(R) Pentium(R) CPU G3420 @ 3.20GHz (1803.48-MHz K8-class CPU)
Sep 24 19:59:51 kernel: Origin = "GenuineIntel" Id = 0x306c3 Family = 0x6 Model = 0x3c Stepping = 3
Sep 24 19:59:51 kernel: Features=0x1f83fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt>Sep 24 19:59:51 kernel: Features2=0xccd82203 <sse3,pclmulqdq,ssse3,cx16,sse4.1,sse4.2,movbe,popcnt,xsave,osxsave,rdrand,hv>Sep 24 19:59:51 kernel: AMD Features=0x20100800 <syscall,nx,lm>Sep 24 19:59:51 kernel: AMD Features2=0x21 <lahf,abm>Sep 24 19:59:51 kernel: Structured Extended Features=0x2200 <erms>Sep 24 19:59:51 kernel: real memory = 671088640 (640 MB)
Sep 24 19:59:51 kernel: avail memory = 608530432 (580 MB)
Sep 24 19:59:51 kernel: Event timer "LAPIC" quality 400
Sep 24 19:59:51 kernel: ACPI APIC Table: <vrtual microsft="">Sep 24 19:59:51 kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Sep 24 19:59:51 kernel: FreeBSD/SMP: 1 package(s) x 2 core(s)
Sep 24 19:59:51 kernel: cpu0 (BSP): APIC ID: 0
Sep 24 19:59:51 kernel: cpu1 (AP): APIC ID: 1
Sep 24 19:59:51 kernel: ioapic0: Changing APIC ID to 0
Sep 24 19:59:51 kernel: ioapic0 <version 1.1="">irqs 0-23 on motherboard
Sep 24 19:59:51 kernel: ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
Sep 24 19:59:51 kernel: ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Sep 24 19:59:51 kernel: module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff805f6300, 0) error 1
Sep 24 19:59:51 kernel: ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
Sep 24 19:59:51 kernel: ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Sep 24 19:59:51 kernel: module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff805f63b0, 0) error 1
Sep 24 19:59:51 kernel: ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
Sep 24 19:59:51 kernel: ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
Sep 24 19:59:51 kernel: module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff805f6460, 0) error 1
Sep 24 19:59:51 kernel: iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
Sep 24 19:59:51 kernel: iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Sep 24 19:59:51 kernel: module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff8061c640, 0) error 1
Sep 24 19:59:51 kernel: wlan: mac acl policy registered
Sep 24 19:59:51 kernel: iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
Sep 24 19:59:51 kernel: iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Sep 24 19:59:51 kernel: module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff8061c6f0, 0) error 1
Sep 24 19:59:51 kernel: iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
Sep 24 19:59:51 kernel: iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
Sep 24 19:59:51 kernel: module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff8061c7a0, 0) error 1
Sep 24 19:59:51 kernel: random: <software, yarrow="">initialized
Sep 24 19:59:51 kernel: kbd1 at kbdmux0
Sep 24 19:59:51 kernel: vmbus0: <vmbus devices="">on motherboard
Sep 24 19:59:51 kernel: cryptosoft0: <software crypto="">on motherboard
Sep 24 19:59:51 kernel: padlock0: No ACE support.
Sep 24 19:59:51 kernel: acpi0: <vrtual microsft="">on motherboard
Sep 24 19:59:51 kernel: acpi0: Power Button (fixed)
Sep 24 19:59:51 kernel: acpi0: reservation of 0, a0000 (3) failed
Sep 24 19:59:51 kernel: acpi0: reservation of 100000, f7f00000 (3) failed
Sep 24 19:59:51 kernel: cpu0: <acpi cpu="">on acpi0
Sep 24 19:59:51 kernel: cpu1: <acpi cpu="">on acpi0
Sep 24 19:59:51 kernel: attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
Sep 24 19:59:51 kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Sep 24 19:59:51 kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Sep 24 19:59:51 kernel: atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
Sep 24 19:59:51 kernel: Event timer "RTC" frequency 32768 Hz quality 0
Sep 24 19:59:51 kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Sep 24 19:59:51 kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
Sep 24 19:59:51 kernel: pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
Sep 24 19:59:51 kernel: pci0: <acpi pci="" bus="">on pcib0
Sep 24 19:59:51 kernel: isab0: <pci-isa bridge="">at device 7.0 on pci0
Sep 24 19:59:51 kernel: isa0: <isa bus="">on isab0
Sep 24 19:59:51 kernel: atapci0: <intel piix4="" udma33="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0
Sep 24 19:59:51 kernel: ata0: <hyper-v ata="" storage="" disengage="" driver="">at channel 0 on atapci0
Sep 24 19:59:51 kernel: ata1: <ata channel="">at channel 1 on atapci0
Sep 24 19:59:51 kernel: pci0: <bridge>at device 7.3 (no driver attached)
Sep 24 19:59:51 kernel: vgapci0: <vga-compatible display="">mem 0xf8000000-0xfbffffff irq 11 at device 8.0 on pci0
Sep 24 19:59:51 kernel: vgapci0: Boot video device
Sep 24 19:59:51 kernel: atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
Sep 24 19:59:51 kernel: atkbd0: <at keyboard="">irq 1 on atkbdc0
Sep 24 19:59:51 kernel: kbd0 at atkbd0
Sep 24 19:59:51 kernel: atkbd0: [GIANT-LOCKED]
Sep 24 19:59:51 kernel: psm0: <ps 2="" mouse="">irq 12 on atkbdc0
Sep 24 19:59:51 kernel: psm0: [GIANT-LOCKED]
Sep 24 19:59:51 kernel: psm0: model IntelliMouse Explorer, device ID 4
Sep 24 19:59:51 kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Sep 24 19:59:51 kernel: uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
Sep 24 19:59:51 kernel: fdc0: <floppy drive="" controller="" (fde)="">port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Sep 24 19:59:51 kernel: fd0: <1440-KB 3.5" drive> on fdc0 drive 0
Sep 24 19:59:51 kernel: orm0: <isa option="" rom="">at iomem 0xc0000-0xcbfff on isa0
Sep 24 19:59:51 kernel: sc0: <system console="">at flags 0x100 on isa0
Sep 24 19:59:51 kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Sep 24 19:59:51 kernel: vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Sep 24 19:59:51 kernel: ppc0: cannot reserve I/O port range
Sep 24 19:59:51 kernel: Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
Sep 24 19:59:51 kernel: Timecounters tick every 10.000 msec
Sep 24 19:59:51 kernel: IPsec: Initialized Security Association Processing.
Sep 24 19:59:51 kernel: storvsc0 on vmbus0
Sep 24 19:59:51 kernel: storvsc1 on vmbus0
Sep 24 19:59:51 kernel: hyperv-utils0 on vmbus0
Sep 24 19:59:51 kernel: hyperv-utils0: Hyper-V Service attaching: Hyper-V Heartbeat Service
Sep 24 19:59:51 kernel:
Sep 24 19:59:51 kernel: hyperv-utils1 on vmbus0
Sep 24 19:59:51 kernel: hyperv-utils1: Hyper-V Service attaching: Hyper-V Shutdown Service
Sep 24 19:59:51 kernel:
Sep 24 19:59:51 kernel: hyperv-utils2 on vmbus0
Sep 24 19:59:51 kernel: hyperv-utils2: Hyper-V Service attaching: Hyper-V Time Synch Service
Sep 24 19:59:51 kernel:
Sep 24 19:59:51 kernel: hn0: <synthetic network="" interface="">on vmbus0
Sep 24 19:59:51 kernel: hn1: <synthetic network="" interface="">on vmbus0
Sep 24 19:59:51 kernel: hn2: <synthetic network="" interface="">on vmbus0
Sep 24 19:59:51 kernel: random: unblocking device.
Sep 24 19:59:51 kernel: cd0 at ata1 bus 0 scbus0 target 0 lun 0
Sep 24 19:59:51 kernel: cd0: <msft virtual="" cd="" rom="" 1.0="">Removable CD-ROM SCSI-5 device
Sep 24 19:59:51 kernel: cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
Sep 24 19:59:51 kernel: cd0: Attempt to query device size failed: NOT READY, Medium not present
Sep 24 19:59:51 kernel: da0 at blkvsc0 bus 0 scbus1 target 0 lun 0
Sep 24 19:59:51 kernel: da0: <msft virtual="" disk="" 1.0="">Fixed Direct Access SCSI-4 device
Sep 24 19:59:51 kernel: da0: 300.000MB/s transfers
Sep 24 19:59:51 kernel: da0: Command Queueing enabled
Sep 24 19:59:51 kernel: da0: 1024MB (2097152 512 byte sectors: 64H 32S/T 1024C)
Sep 24 19:59:51 kernel: SMP: AP CPU #1 Launched!
Sep 24 19:59:51 kernel: Trying to mount root from ufs:/dev/ufs/pfsense0 [ro,sync,noatime]…
Sep 24 19:59:51 kernel:
Sep 24 19:59:52 php: rc.bootup: Starting 3gstats.php on device '' for interface 'wan'
Sep 24 19:59:52 kernel:
Sep 24 19:59:52 php: rc.bootup: Starting 3gstats.php on device '' for interface 'opt1'
Sep 24 19:59:52 php: rc.bootup: Resyncing OpenVPN instances.
Sep 24 19:59:52 kernel: done.
Sep 24 19:59:52 kernel: pflog0: promiscuous mode enabled
Sep 24 19:59:52 php: rc.bootup: Could not find IPv4 gateway for interface (opt1).
Sep 24 19:59:52 php: rc.bootup: RRD create failed exited with 1, the error is: ERROR: creating '/var/db/rrd/WAN_PPPOE-quality.rrd': No such file or directory
Sep 24 19:59:52 kernel: ...done.
Sep 24 19:59:52 php: rc.bootup: The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/WAN_PPPOE-quality.rrd N:U:U' returned exit code '1', the output was 'ERROR: opening '/var/db/rrd/WAN_PPPOE-quality.rrd': No such file or directory'
Sep 24 19:59:53 php-fpm[287]: /rc.newwanip: ROUTING: setting default route to 222.92.225.13
Sep 24 19:59:53 php-fpm[287]: /rc.newwanip: Removing static route for monitor 221.224.226.82 and adding a new route through 222.92.225.13
Sep 24 19:59:54 php-fpm[286]: /rc.newwanip: Removing static route for monitor 221.224.226.82 and adding a new route through 222.92.225.13
Sep 24 19:59:54 kernel: done.
Sep 24 19:59:55 kernel: done.
Sep 24 19:59:56 php: rc.bootup: ROUTING: setting default route to 222.92.225.13
Sep 24 19:59:56 kernel: done.
Sep 24 19:59:56 check_reload_status: Updating all dyndns
Sep 24 19:59:56 kernel: …..done.
Sep 24 20:00:00 php-fpm[286]: /rc.newwanip: phpDynDNS: updating cache file /conf/dyndns_opt1noip-free'pftest.ddns.net'2.cache: 221.225.73.206
Sep 24 20:00:00 php-fpm[286]: /rc.newwanip: phpDynDNS (pftest.ddns.net): (Success) DNS hostname update successful.
Sep 24 20:00:01 php: rc.bootup: Creating rrd update script
Sep 24 20:00:01 kernel: done.
Sep 24 20:00:01 php-fpm[286]: /rc.newwanip: Resyncing OpenVPN instances for interface OPT1.
Sep 24 20:00:01 kernel: done.
Sep 24 20:00:01 php-fpm[286]: /rc.newwanip: Creating rrd update script
Sep 24 20:00:02 php-fpm[287]: /rc.newwanip: phpDynDNS: updating cache file /conf/dyndns_wannoip-free'huiguang.no-ip.org'0.cache: 221.225.73.201
Sep 24 20:00:02 php-fpm[287]: /rc.newwanip: phpDynDNS (huiguang.no-ip.org): (Success) DNS hostname update successful.
Sep 24 20:00:02 php-fpm[285]: /rc.dyndns.update: phpDynDNS (huiguang.no-ip.org): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 24 20:00:03 php-fpm[286]: /rc.newwanip: pfSense package system has detected an ip change 0.0.0.0 -> 221.225.73.206 … Restarting packages.
Sep 24 20:00:03 check_reload_status: Starting packages
Sep 24 20:00:03 check_reload_status: Reloading filter
Sep 24 20:00:03 php-fpm[286]: /rc.start_packages: Restarting/Starting all packages.
Sep 24 20:00:04 php-fpm[286]: /rc.start_packages: Restarting/Starting all packages.
Sep 24 20:00:04 php-fpm[287]: /rc.newwanip: phpDynDNS: updating cache file /conf/dyndns_wanhe-net'cloud.cutebox.net'1.cache: 221.225.73.201
Sep 24 20:00:05 login: login on ttyv0 as root
Sep 24 20:00:05 php-fpm[287]: /rc.newwanip: phpDynDNS: (Success) IP Address Updated Successfully!
Sep 24 20:00:05 php-fpm[285]: /rc.dyndns.update: phpDynDNS (cloud.cutebox.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 24 20:00:05 sshlockout[83735]: sshlockout/webConfigurator v3.0 starting up
Sep 24 20:00:06 php-fpm[287]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Sep 24 20:00:06 php-fpm[287]: /rc.newwanip: Creating rrd update script
Sep 24 20:00:06 php-fpm[285]: /rc.dyndns.update: phpDynDNS (pftest.ddns.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 24 20:00:08 php-fpm[287]: /rc.newwanip: pfSense package system has detected an ip change 0.0.0.0 -> 221.225.73.201 … Restarting packages.
Sep 24 20:00:08 check_reload_status: Starting packages
Sep 24 20:00:08 check_reload_status: Reloading filter
Sep 24 20:00:09 php-fpm[285]: /rc.start_packages: Restarting/Starting all packages.
Sep 24 20:00:14 check_reload_status: updating dyndns OPT1_PPPOE
Sep 24 20:00:14 check_reload_status: Restarting ipsec tunnels
Sep 24 20:00:14 check_reload_status: Restarting OpenVPN tunnels/interfaces
Sep 24 20:00:14 check_reload_status: Reloading filter
Sep 24 20:00:15 php-fpm[285]: /rc.dyndns.update: phpDynDNS (pftest.ddns.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 24 20:00:16 kernel: calcru: runtime went backwards from 522624 usec to 295023 usec for pid 285 (php-fpm)
…........(many lines same as above)
....................
Sep 24 20:01:08 kernel: calcru: runtime went backwards from 405510687 usec to 228805402 usec for pid 0 (kernel)
Sep 24 20:05:54 login: login on ttyv0 as root
Sep 24 20:06:20 php-fpm[285]: /index.php: Successful login for user 'admin' from: 58.214.197.206
Sep 24 20:06:20 php-fpm[285]: /index.php: Successful login for user 'admin' from: 58.214.197.206
Sep 24 20:09:10 login: login on ttyv0 as root
Sep 24 20:11:34 check_reload_status: Syncing firewallI could provide other info if needed.</msft></msft></synthetic></synthetic></synthetic></generic></system></isa></floppy></ps></at></keyboard></vga-compatible></bridge></ata></hyper-v></intel></isa></pci-isa></acpi></acpi></at></at></acpi></acpi></vrtual></software></vmbus></software,></version></vrtual></erms></lahf,abm></syscall,nx,lm></sse3,pclmulqdq,ssse3,cx16,sse4.1,sse4.2,movbe,popcnt,xsave,osxsave,rdrand,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt>
-
I'm still not having much luck. I'm on Tuesdays beta build. I added two additional gateways for testing. I have my normal default gateway with monitor IP set to 8.8.8.8., another gateway with monitor IP set to my pfsense lan facing adapters IP, and the other gateway set to the default monitoring option (wan gateway IP, first hop outside my network)
All three show downs in the gateway log at the same time, multiple times. Latency is thru the roof on the gateway status area.
I will update to latest snap and turn on the additional logging and post my results. I'm almost wondering if something is crapping out with hyperv kernel modules, is anyone else having issues like this on their physical box?
See attached screens.
-
…and 45 minutes in on the Wednesday build I have "offline" for all gateways...even the fake one I setup that is the LAN adapter's IP address. I didn't get a chance to set one of the monitor IP's to an external IP, so there are two monitors that are my WAN default gateway and the other monitor is my LAN adapter IP address.
I had verbose logging on at the time. See attachments. I can provide other logs or info if needed.
-
I need the gateway logs if you have the new setting turned on.
Though from your description its not a good thing to do to setup a monitoring IP to your lan in general!