An earnest appeal - please do fix APINGER in 2.2

jimp

The problem is that none of us can reproduce this on demand in an environment we control and can debug. Yes, we know some people have problems, but they don't affect the majority of users.

Multi-WAN works fine here, and for many others. Ermal has been working on fixing this up but it's been a long process since the exact parameters to reproduce the problem have never been clearly identified or replicated.

The notifications are a separate issue entirely, but that's not slated to be cleaned up on 2.2 but sometime afterward.

pubmsu

Thanks for your update jimp - we at least now know the difficulty in fixing it.

If you want, I can help you with access to an otherwise good multi-WAN test environment that has this issue, which is an exact replication of our production environment. You can let me know in PM if this will help.

We have been using pfSense for last 7 years I guess, and really need this to be resolved.

jimp

Getting access wouldn't help as much as definitively identifying the specific condition leading to the problem if possible (e.g. a latency over X for Y amount of time, or Z gateways with Q latency, etc)

Depending on how long it takes for the problem to repeat it could still be difficult for us to find time to watch it closely enough to find when the problem starts specifically.

pubmsu

Got it Jim. In our case though the problem starts within 5 to 10 minutes.

jimp

Seeing the quality graph for your gateways may help as well, with notes about where apinger was restarted and where the problem was first noticed.

I tried to artificially induce latency using one firewall in front of another and increasing the delay on a limiter for ICMP traffic. Each time I let it run for 15+ minutes at various latencies and then lifted the limiter. Each time it always bounced back to close to 0 for me, I never saw it get stuck, so there must be a few different factors at work making it get stuck over time for others.

athurdent

Maybe this is something to consider: I never had any problems with my setup running NanoBSD for the last year. I switched to a full install recently (CF died, bought an SSD) and now I am seeing Packetloss steadily increasing for my HENet tunnel. 120% packetloss ATM, uptime of the firewall is 4 days. I am pinging the same IPv6 Host via Smokeping from a Linux host behind the pfSense GW and the graphs look a little different. This is on 2.1.4, not on 2.2.

Screenshot_1.png_thumb

status_rrd_graph_img.png_thumb

tunnel-endpoint_last_864000.png_thumb

Jeremy11one

I have this problem too. Apinger reports that my WAN connection keeps going up and down several times every hour. It started a few months ago. I have not switched ISPs or anything. I installed the latest snapshot (built on Mon Jul 28 12:22:20 CDT 2014) and still have the problem.

I do not use multiple WANs. Just one.

ridnhard19

I had this same issue as well and ended up coding in the local/private cable modems IP address into the config (192.168.100.1) and that was the workaround I used. Doesnt do anything for monitoring the connection but it's not always bouncing the connection up and down.

Jeremy11one

When you say "the config," do you mean the "Monitor IP"? My config was monitoring the default gateway IP, which is on the cable modem and I still had the problem.

georgeman

I don't think this is related to the main issue described here, but I have observed a similar behavior under high network load and while using the traffic shaper, because the ping probes are put on the default queue instead of the one specified by the floating rule on WAN that is supposed to handle the situation. Probably this happens because apinger starts before the firewall itself, since killing the related states makes them go into the correct queue immediately

naras

Issue still exsits in recent bulids in my testing enviroments.

…and frequently results in tunnels (IPsec or openVPN) going down for no obvious reasons, except for apinger freakin' out.

I increased the times for apinger alarm significantly, that helps at least a little...

Supermule

I dont have these problems at all running 40+ pfsenses….

I use traceroute to monitor the wanted IP upstream to decide if the GW is down.

All are stable currently running 0% packetloss..... No change from 2.0.X

I dont like the idea of monitoring other external hosts not in your upstream environment. That way you dont get a real picture of your GW status.

Capture.PNG_thumb

Raul Ramos

@Supermule:

I dont have these problems at all running 40+ pfsenses….

What's your config for WAN interfaces? I see allot of people write that have problems but doesn't put configs to help troubleshoot the problem.

Some times i have the problem in my multi-wan interface (PPPoE only config user and pass and a ppp (LTE) WAN only config default number). Don't see the problem when i disconnect my ppp.

Supermule

More or less the same for all 40+….

Capture.PNG_thumb

naras

@Supermule:

I use traceroute to monitor the wanted IP upstream to decide if the GW is down.

All are stable currently running 0% packetloss….. No change from 2.0.X

I dont like the idea of monitoring other external hosts not in your upstream environment. That way you dont get a real picture of your GW status.

Could you please tell us how to use traceroute to monitor the wanted IP upstream to decide if the GW is down?

Multi-wan with static IPs and different gateways within each wan subnets are stable at least in my tests, but I use pppoe connections and we get the same gateway IP allmost all the times, so we have to set at least one monitor IP outside the wan subnet, and this line with outside monitor ip allways gets offline as the apinger reported, but the connection functional as normal.

If there is another to monitor the gatwway, that really helps.

Supermule

http://ping.eu/traceroute/

Use the first one thats not in your WAN subnet.

naras

@Supermule:

http://ping.eu/traceroute/

Use the first one thats not in your WAN subnet.

It's not within pfsense, and not done automaticly either?

Supermule

I understand why you are confused…

I use traceroute to monitor the wanted IP upstream to decide if the GW is down.

I use traceroute to locate the IP to monitor and then use the built in GW monitor tool in PFSense.

Works fine here.

naras

@Supermule:

I understand why you are confused…

I use traceroute to monitor the wanted IP upstream to decide if the GW is down.

I use traceroute to locate the IP to monitor and then use the built in GW monitor tool in PFSense.

Works fine here.

OK, I did that several months ago, and with no use.
The next hop routers are always outside my wan subnet:(

Thanks anyway.