Multi WAN redundancy problem

Roots0

I have two ADSL2+ lines each one running though a TP-Link TD-8840T In bridging mode. Even though they are PPPoA I have each pfsense set to PPPoE and this works.
One line is Cable and wireless(OPT1), the other its British Telecom (WAN).

I have a gateway group set and firewall rule and load balancing works well.

The problem I have is if the British Telecom line is unplugged the Cable and Wireless interface drops off as well as the British Telecom line. Leaving me with no connection.
If I unplug the Cable and Wireless line the BT line works fine.

The primary reason for having two lines was to have the redundancy if unplugging the BT lines renders the C&W line useless it defeats the point! I have "Allow default gateway switching" ticked. Am i missing some configuration? I don't see any related errors in systemlog.

Nachtfalke

Do both WAN connections have different gateways ?
Did you set different monitor IPs for both gateways ?
Are you using squid on this pfsense ?

What do you mean when you say the link is down ? Is the gateway status down or do you just not get connection to the internet ?

Roots0

Both WAN connections have different gateways. Same first three octets then one is .72 other .73
For monitor IP I have left it as default presuming it just uses the gateway or should I change this?
Not using squid.

When I say link is down it comes up as red on the dashboard and the gateway appears as offline.

Nachtfalke

@Roots0:

Both WAN connections have different gateways. Same first three octets then one is .72 other .73
For monitor IP I have left it as default presuming it just uses the gateway or should I change this?
Not using squid.

When I say link is down it comes up as red on the dashboard and the gateway appears as offline.

Is there packet loss or high latency !?
Set a monitor IP - some Gateways stop responding on ping after a certain time.
Try with google-DNS (8.8.8.8 and 8.8.4.4).

In Gateway Groups - did you set "High latency or packet loss" - try this if not already set.

Roots0

Put googles DNS servers in as monitor IP's will test if C&W connection drops with the BT tonight or not. As links are in use atm.
No high packet loss or latency normaly.
I have set "High latency or packet loss" I only have one gateway group with both gateway's on at Tier 1 is this ok?

Got this in the log when i changed the monitor IP's:


check_reload_status: Syncing firewall
Jul 30 14:44:21	php: /system_gateways.php: ROUTING: setting default route to 94.30.127.72
Jul 30 14:44:21	check_reload_status: Reloading filter
Jul 30 14:44:21	php: /system_gateways.php: Removing static route for monitor 8.8.8.8 and adding a new route through 94.30.127.72
Jul 30 14:44:23	php: : Gateways status could not be determined, considering all as up/active.
Jul 30 14:44:26	dhcp6c[11845]: client6_send: transmit failed: Network is down
Jul 30 14:44:26	dhcp6c[11845]: client6_send: transmit failed: Network is down
Jul 30 14:44:35	check_reload_status: Syncing firewall
Jul 30 14:44:48	php: /system_gateways.php: ROUTING: setting default route to 94.30.127.72
Jul 30 14:44:48	check_reload_status: Reloading filter
Jul 30 14:44:48	php: /system_gateways.php: Removing static route for monitor 8.8.8.8 and adding a new route through 94.30.127.72
Jul 30 14:44:48	php: /system_gateways.php: Removing static route for monitor 8.8.4.4 and adding a new route through 94.30.127.73
Jul 30 14:44:50	php: : Gateways status could not be determined, considering all as up/active.
Jul 30 14:46:30	dhcp6c[11845]: client6_send: transmit failed: Network is down
Jul 30 14:46:30	dhcp6c[11845]: client6_send: transmit failed: Network is down

Nachtfalke

Your gateway group with same tier is ok.

Do you use pfsense 2.1 ? There is some IPv6 output on the syslog.

Can you post screenshots of your firewall rules on LAN, your Gateways, your gateway group ?
Want to make sure that loadbalancing is configured properly and it will do its job after one GW is down.

Roots0

Yup 2.1 should't really be, it was doing the same thing in 2.0 though.

Untitled.jpg_thumb

Nachtfalke

Ok, your configuration looks correct. You RTT is near to 100ms which is the default low latency threashold. The high threashold is 200ms.
So try to increase the limits to lets say 300ms and 500ms in SYSTEM -> Rounting -> Gateways.

But I am really not sure why you have problems when one line goes down but no problems if the other one goes down.
Did you try with disabled "Allow gateway switching" ? Not sure how this works on pfsense 2.1 but on 2.0.x it sometimes switched to my LAN interface if my WAN gateways were down.

Roots0

I think I have found the cause of the problem:
The BT line has the option of changing the line profile as it’s a more expensive Annex M connection the C&W line does not have this. Originally I had been using Interleaving on the BT line. But when I turned it off and switched between non-interleaving profiles, the C&W line stayed up.

These two lines are run in the same cable as I guess the engineer who installed them was being lazy or its standard policy either way I think there is some crosstalk on the lines which was causing the problems. Either that or its my dodgy Cat5e telephone extension cable!

I should really test it locally but I can’t do that at the moment.

Thanks for the help Nachtfalke, in this very odd case its much appreciated! Wish they would hurry up and deploy fibre + VDSL!