LoadBalancing 3WAN - one comes not up after 24h reconnect

Nachtfalke

Hi together,

I am using pfsense with LoadBalancing of 2 ADSL WANs since over one year without any major problems. Some weeks ago we got a third ADSL connection. The loadbalancing is working with all three lines as it should but every night at ~04:30 there is the 24h reconnect of my provider. After this reconnect my WAN3 does not come up again automatically. To get it up again I go to SYSTEM -> ROUTING -> Edit the WAN3 gateway -> save -> apply settings.

This is in system log:

Jun 16 17:22:52 	php: /index.php: Successful webConfigurator login for user 'admin' from 10.0.32.6
Jun 16 17:22:52 	php: /index.php: Successful webConfigurator login for user 'admin' from 10.0.32.6
Jun 16 12:08:37 	root: rc.update_bogons.sh is sleeping for 54838
Jun 16 12:08:37 	root: rc.update_bogons.sh is starting up.
Jun 16 12:08:37 	root: rc.update_bogons.sh is ending the update cycle.
Jun 16 12:08:37 	root: Could not download http://files.pfsense.org/mirrors/bogon-bn-nonagg.txt.md5 (md5 mismatch)
Jun 16 12:08:36 	root: rc.update_bogons.sh is beginning the update cycle.
Jun 16 04:33:24 	php: : MONITOR: WAN3 is down, removing from routing group
Jun 16 04:33:24 	php: : MONITOR: WAN3 is down, removing from routing group
Jun 16 04:33:24 	php: : Message sent to xxxxxx@.xxx OK
Jun 16 04:33:23 	php: : MONITOR: WAN3 is down, removing from routing group
Jun 16 04:33:18 	check_reload_status: Reloading filter
Jun 16 04:33:08 	apinger: ALARM: WAN3(8.8.8.8) *** WAN3down ***
Jun 16 01:05:34 	check_reload_status: Reloading filter
Jun 16 01:05:24 	apinger: alarm canceled: WAN1(217.0.43.113) *** loss ***
Jun 16 01:04:51 	php: : MONITOR: WAN1 has packet loss, removing from routing group
Jun 16 01:04:51 	php: : MONITOR: WAN1 has packet loss, removing from routing group
Jun 16 01:04:51 	php: : Message sent to xxxxxx@.xxx OK
Jun 16 01:04:51 	php: : MONITOR: WAN1 has packet loss, removing from routing group
Jun 16 01:04:45 	check_reload_status: Reloading filter
Jun 16 01:04:35 	apinger: ALARM: WAN1(217.0.43.113) *** loss ***
Jun 15 23:23:45 	root: rc.update_bogons.sh is sleeping for 45323
Jun 15 23:23:45 	root: rc.update_bogons.sh is starting up.
Jun 15 23:23:45 	root: rc.update_bogons.sh is ending the update cycle.
Jun 15 23:23:45 	root: Could not download http://files.pfsense.org/mirrors/bogon-bn-nonagg.txt.md5 (md5 mismatch)
Jun 15 23:23:45 	root: rc.update_bogons.sh is beginning the update cycle.

I tried with different monitor IPs but without luck.
All three gateways are copnfigured equal - just other gateway IP, interface and monitor IP.
In front of all three lines there is a modem/router which is doing NAT - not more - pfsense interface is the exposed host.

I am using pfsense 2.0.1 x86
Additional packages on this machine are:
cron
freeradius2
openvpn-export utility

Thanks for your help in advance!

Edit_Gateway.jpg_thumb

Gateways.jpg_thumb

Gateways_Groups.jpg_thumb

heper

did you check the routing table if the 8.8.8.8 route is still there for wan3 and not some other interface ?
since you are an experienced pf user i would think you have checked all the common things like overlapping subnet of vpn's and stuff….
but i recently overlooked that i had a unused ovpn server running but the tunnel was not online. When i added a new interface with the same subnet, it worked fine cuss it overwrote the routes of the ovpn, it started giving me trouble with gateway going down and routes that got mixed up between the actual interface and the ovpn server.

i don't have any systems with more then 2 wans, so can't confirm or deny if there might be an issue or not.

Nachtfalke

Thanks heper,

I will check the routing table tomorrow when WAN3 will be down again.

But there are no overlapping subnets. Not on VPN, not on any other interface. My LAN iterfaces are on 172.16.0.0/16 and 172.17.0.0/16 (a second pfsense behind my first one). OpenVPN is on 10.0.x.x. I do not have "switch default gateway" enabled. Could it be a problem with the "default" gateway or something link that ? But the other 2 WANs are working.

Or could it be a problem with the router before my WAN3 interface ? What could I check there ? I use the latest firmware.

Thanks

heper

you could see if you were able to ping the router from pfsense when its supposedly down … i've had a dsl modem/router before that started to "ignore" all pfsense packets on random occasions. Other devices connected to it would keep functioning (only rebooting the modem solved it). In the end i swapped the modem and it never occured again

i don't have any experience with changing MTU settings but it might be worth a try. Also forcing to speed/duplex seems like a shot to nothing.

hopefully someone else can help ya out

Nachtfalke

Hi heper,

tonight no problems - didn't change anything.

About the duplex speed and so on. The modem is able to do 10/100/1000 Fullduplex, my pfsense NIC can do the same but I tried it with 100full.

Further you are right - I found out that pfsense showed the gateway as down but I could access the router/modem…perhaps it will help if I set a priority for my pfsense traffic on the router/modem.

Nachtfalke

Nothing changed - I can see that there are no errors on WAN3 OUT but there is a high number of 188856 ERRORS IN.

Any ideas on that ?

dhatz

@Nachtfalke:

In front of all three lines there is a modem/router which is doing NAT - not more - pfsense interface is the exposed host.

Although it's probably unrelated to your WAN3 issue, why did you put another router in front of pfsense to handle NAT ? …

Nachtfalke

@dhatz:

@Nachtfalke:

In front of all three lines there is a modem/router which is doing NAT - not more - pfsense interface is the exposed host.

Although it's probably unrelated to your WAN3 issue, why did you put another router in front of pfsense to handle NAT ? …

I had trouble with PPPoE connections/reconnect and because all my three ADSL connections do have the same gateway.