Apinger **delay** how to interpret

brcisna

Hello All,

pfSense-1.2.3-RELEASE
squid
squidGaurd
lightsquid
Multi-WAN -load balancing– failover
IPsec vpn x2

Hardware-- castoff 1u servers of about p4 2.4 ghz with 1gig ram 40 gig HD.

Have set up three pfSense boxes at three different school buildings and each are working great.!
Almost a year running now!
I have tried searching here and can not really find a definitive answer.

On each of the three pfSense machines in the load balancing log I see the following contstantly. I am guessing this is nothing to worry about as they each are working fantastic. I am just trying to narrow down some things in regards to understanding why the 'delay' items constantly happens.

load balancer log:

Jan 14 08:14:50 apinger: ALARM: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 08:14:59 apinger: alarm canceled: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 08:15:41 apinger: ALARM: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 08:15:58 apinger: alarm canceled: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 08:16:12 apinger: ALARM: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 08:16:24 apinger: alarm canceled: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 08:16:34 apinger: ALARM: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 08:16:42 apinger: alarm canceled: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 09:33:36 apinger: ALARM: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 09:35:47 apinger: alarm canceled: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 13:30:56 apinger: ALARM: x.x.x.x(x.x.x.x) *** delay ***
Jan 14 13:31:04 apinger: alarm canceled: x.x.x.x(x.x.x.x) *** delay ***

end log.

Sometimes I do see loss as well as down* very rarely.

This will be mostly to the primary /default gateways dns server,but does happen to the second wan's dns server sometimes . I also tried switching to each wan's gateway as the pinger and the results were almost identical.

I have read here were it is possible in pfSense 2.0 to change the apinger threshold times.
Just curious if others here see's these types on log entries routinely.
Also sometimes throughout the day if I manually look at the load balancer in the webui one of the two wan's have gone 'yellow' but very seldom do I see this.

Each pfSense box has varying nics in each so it appears this is not nic specific.
Default gateway is a ethernet/wireless bridge that provides internet access to several schools through a consortium at 3MB down/up. The second Wan is Frontier business dsl and amounts to 3MB down and 750 up.

thank You,
Barry

indesman

I don't know if this has been happening on a regular basis or not but apinger was going wide open on my box last night. It went down and up two or three times a minute for almost forty minutes. I called Comcast and they ran some tests on my modem which seemed to make it stop for about ten minutes but then it started back up. I restarted my system and things seemed to be okay until late this morning.

While this seems to be happening predominately on OPT1 (Cable) I have seen it on WAN (DSL) as well. PfSense is running failover in addition to load balancing using each ISPs' gateway as the monitor IP and Google DNS servers.

I'm running a dual WAN setup as follows:

em0: WAN - MoBo NIC - DSL (provided by AT&T, formerly SBC, formerly PacBell)
em1: LAN - MoBo NIC
em2: OPT1 - Intel PCI NIC - Cable Internet (provided by Comcast)

Hardware setup as follows:

Mother Board: MSI Industrial IM-945GSE-A Atom Mini-ITX Mainboard (MS-9830)
2 x Intel 82574L GbE LAN on MoBo
IDE to CF Adaptor: Syba Ultra Ide To Cf Adapter Dual IDE 40/44PIN To Compact Flash
CF-Card: SanDisk SDCFX3-004G-A31 4 GB Extreme III CompactFlash Card
RAM: Kingston 1GB DDR2 PC2-4200 SODIMM 533MHz
NIC: Intel PWLA8391GT PRO/1000 GT PCI Network Adapter

Here's the end of my loadbalance log file:
First
Jan 28 11:55:23 apinger: ALARM: 76.103..(76.103..) *** down ***
Jan 28 11:55:31 apinger: alarm canceled: 76.103..(76.103..) *** down ***
Second
Jan 28 11:56:32 apinger: ALARM: 76.103..(76.103..) *** down ***
Jan 28 11:56:32 apinger: alarm canceled: 76.103..(76.103..) *** down ***
Third
Jan 28 12:12:44 apinger: ALARM: 76.103..(76.103..) *** delay ***
Jan 28 12:12:52 apinger: alarm canceled: 76.103..(76.103..) *** delay ***
Fourth
Jan 28 12:13:31 apinger: ALARM: 76.103..(76.103..) *** down ***
Jan 28 12:14:32 apinger: alarm canceled: 76.103..(76.103..) *** down ***

And here's the end of my syslog:
First
Jan 28 11:55:23 apinger: ALARM: 76.103..(76.103..) *** down ***
Jan 28 11:55:31 apinger: alarm canceled: 76.103..(76.103..) *** down ***
Second
Jan 28 11:55:35 check_reload_status: reloading filter
Jan 28 11:55:42 check_reload_status: reloading filter
Jan 28 11:56:32 apinger: ALARM: 76.103..(76.103..) *** down ***
Jan 28 11:56:32 apinger: alarm canceled: 76.103..(76.103..) *** down ***
Jan 28 11:56:44 check_reload_status: reloading filter
Third
Jan 28 12:12:44 apinger: ALARM: 76.103..(76.103..) *** delay ***
Jan 28 12:12:52 apinger: alarm canceled: 76.103..(76.103..) *** delay ***
Jan 28 12:12:57 check_reload_status: reloading filter
Jan 28 12:13:03 check_reload_status: reloading filter
Fourth
Jan 28 12:13:31 apinger: ALARM: 76.103..(76.103..) *** down ***
Jan 28 12:13:45 check_reload_status: reloading filter
Jan 28 12:13:53 kernel: em2: link state changed to DOWN
Jan 28 12:13:54 kernel: em2: link state changed to UP
Jan 28 12:13:56 check_reload_status: rc.linkup starting
Jan 28 12:13:57 php: : Processing em2 - start
Jan 28 12:13:57 php: : DEVD Ethernet attached event for em2
Jan 28 12:13:57 php: : HOTPLUG: Configuring optional interface - opt
Jan 28 12:13:57 php: : Processing start -
Jan 28 12:13:57 php: : Processing -
Jan 28 12:14:32 apinger: alarm canceled: 76.103..(76.103..) *** down ***
Jan 28 12:14:42 check_reload_status: reloading filter

I read about apinger issues in the forums and saw that lsoltero had posted a patch. In my case, after initial setup I've been administering the system via webGIU so I'm unable to install said patch without plugging in KVM. I just ordered a nullmodem and 15' serial extension so I can run in console mode when that arrives. As a noob I'd appreciate any pro tips that could keep me out of trouble.
Michael

brcisna

indesman,

Your configuration in regards to network looks almost identical to our setup. The only difference in hardware is we are using P4 and HDD ,were you are using Atom and CF card. Your loadbalance.log looks identical to ours.
Your syslog has an odd entry in the ;
Jan 28 12:13:57 php: : Processing em2 - start
Jan 28 12:13:57 php: : DEVD Ethernet attached event for em2
Jan 28 12:13:57 php: : HOTPLUG: Configuring optional interface - opt
Jan 28 12:13:57 php: : Processing start -
Jan 28 12:13:57 php: : Processing -

Do you see this entry very often?

Not much help, just trying to compare setups.

Barry

SeventhSon

You should be able to configure apinger from

Diagnostic : Edit file

/var/etc/apinger.conf

Standard values:

## "Down" alarm definition. 
## This alarm will be fired when target doesn't respond for 30 seconds.
alarm down "down" {
	time 10s
}

## "Delay" alarm definition. 
## This alarm will be fired when responses are delayed more than 200ms
## it will be canceled, when the delay drops below 100ms
alarm delay "delay" {
	delay_low 200ms
	delay_high 500ms
}

## "Loss" alarm definition. 
## This alarm will be fired when packet loss goes over 20%
## it will be canceled, when the loss drops below 10%
alarm loss "loss" {
	percent_low 10
	percent_high 20
}

In 2.0 these settings are in the GUI:

System: Gateways: Edit gateway