One gateway down after upgrade to 2.3, issue doesn't seem to be dpinger related

andy.max

After upgrading pfSense from 2.0 to 2.3 I've encountered an issue with one interface/gateway. I've followed the advice on https://forum.pfsense.org/index.php?topic=110043.0 without fixing the issue so I thought I'd make a dedicated thread.

I currently run pfSense with 8 network interfaces.

WAN 1 - PPPoE DSL ISP A - 212.159.X.A - GW: 62.24.X.X
WAN 2 - PPPoE DSL ISP A - 92.27.X.A - GW: 62.24.X.X
WAN 3 - PPPoE DSL ISP A - 212.159.X.B - GW: 62.24.X.X
WAN 4 - PPPoE DSL ISP A - 212.159.X.C - GW: 62.24.X.X
WAN 5 - PPPoE DSL ISP B - 81.149.X.A - GW: 81.139.X.X
LAN 1 - 10.0.1.1/24
LAN 2 - 10.0.2.1/24
LAN 3 - 10.0.3.1/24

WAN 1 - 4 are fibre DSL connections from the same IP. All IP addresses are static and assigned when establishing the PPPoE connection. WAN2's static IP was changed at some point by the ISP to a range different from the other interfaces from the same ISP.

WAN 5 is also a fibre DSL connection from a different ISP with again a static IP address but assigned by the ISP.

All interfaces are up and the DSL looks like it is connected. The WANs are in a gateway group. All connections use the same BT openreach modems. However after upgrading from 2.0 –-> 2.3, WAN 2 gateway's status always shows as down.

I've sanity checked the gateway monitoring configuration and added a data payload as suggested in the above linked thread. It's had no effect and I don't think its related to dpinger, as I cannot manually ping from WAN2 using the web gui:

From WAN 2:

PING 8.8.8.8 (8.8.8.8) from 92.27.X.A: 56 data bytes
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

From WAN1:

PING 8.8.8.8 (8.8.8.8) from 212.159.X.A: 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=10.271 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=10.764 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=10.399 ms
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 10.271/10.478/10.764/0.209 ms

Also interesting is tracerouting. If I run a traceroute via WAN2 I get the following:

From WAN2:

1  81.139.X.X  10.935 ms  11.389 ms  10.708 ms
2  * * *
3  * * *

From WAN1:

1  62.24.X.X  5.102 ms  4.897 ms  4.539 ms
 2  78.151.230.145  5.407 ms  5.432 ms  5.204 ms
 3  78.151.230.128  5.431 ms  5.668 ms  5.668 ms

The first hop looks very strange. in fact…

From WAN5:

 1  81.139.X.X  14.068 ms  14.197 ms  13.859 ms
 2  213.120.178.141  14.056 ms  14.311 ms  13.643 ms
 3  213.120.177.98  14.972 ms  14.960 ms  14.556 ms

.. it looks like when running a traceroute from WAN 2 the first hop is the same as a traceroute from WAN 5 which uses a different ISP and different gateway!

Out of interest, incoming connections to WAN2 are functioning and I've tried deleting the interface and gateway and setting up again. I've power cycled the router and the WAN2 vDSL modem. This setup has been running for 3 years using an older version of pfSense and has been a stable working setup for our office.

Any advice would be greatly appreciated and please let me know if I can provide any extra information.

Many thanks,
Andy