Routing appears broken

Phil

I have been attempting to set up a multi-wan (2 tier failover) setup for a few hours now. It's not going well. The problem seems to be that something is inherently broken with the way my pfsense install is handling routing. It appears that it is in fact not handling routing. Here's the routing table:

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            31.24.0.194        UGS         0    13420    sk0
31.24.0.192/26     link#5             U           0        2    sk0
31.24.0.195        link#5             UHS         0        0    lo0
31.24.0.198        link#5             UHS         0        0    lo0 =>
31.24.0.198/32     link#5             U           0        0    sk0
37.77.176.177      31.24.0.194        UGHS        0     1538    sk0
83.218.143.225     31.24.0.193        UGHS        0     1540    sk0
127.0.0.1          link#10            UH          0      364    lo0
192.168.1.0/24     link#6             U           0    36372    sk1
192.168.1.1        link#6             UHS         0        0    lo0

You'll see that the IP 83.218.143.225 should be routed via the gateway 31.24.0.193. This route is one added by pfsense because it is the monitor IP of the gateway 31.24.0.193.

When I traceroute this, I get:

[2.0.1-RELEASE][root@prop-router-rugby.local]/root(59): traceroute -n 83.218.143.225
traceroute to 83.218.143.225 (83.218.143.225), 64 hops max, 40 byte packets
 1  31.24.0.194  0.974 ms  0.933 ms  0.965 ms
 2  37.77.176.177  3.841 ms  3.741 ms  3.758 ms
 3  83.218.143.140  4.039 ms  4.053 ms  3.944 ms
 4  83.218.143.154  4.747 ms *  4.543 ms

You'll see that it's actually using the default route. This is an issue because the secondary gateway is always considered to be down.

This problem also seems to occur of I add static routes via the web interface - i.e. the default gateway is used, regardless.

I have packet logged a ping to 83.218.143.225 and imported the cap file into Wireshark. This shows the frames routing via the default router at 31.24.0.194 and not 31.24.0.193.

I see that the use counters for the routes increment when I ping, however I am certain that the route is not being obeyed.

Any thoughts..?

heper

what you can try:

-remove all static routes
-set the correct gateways for each wan
-go to firewalling LAN tab, create a rule: destination 83.218.143.225, at the bottom select the gateway you wish to use

Phil

Fraid not :/

Image below.

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            31.24.0.194        UGS         0  1336316    sk0
31.24.0.192/26     link#5             U           0      683    sk0
31.24.0.195        link#5             UHS         0        0    lo0
31.24.0.198        link#5             UHS         0        0    lo0 =>
31.24.0.198/32     link#5             U           0        0    sk0
37.77.176.177      31.24.0.194        UGHS        0    32205    sk0
127.0.0.1          link#10            UH          0    13779    lo0
192.168.1.0/24     link#6             U           0  1773871    sk1
192.168.1.1        link#6             UHS         0        0    lo0

[2.0.1-RELEASE][root@prop-router-rugby.local]/root(12): traceroute -n 83.218.143.225
traceroute to 83.218.143.225 (83.218.143.225), 64 hops max, 40 byte packets
 1  31.24.0.194  1.020 ms  0.940 ms  1.062 ms
 2  37.77.176.177  3.772 ms  3.852 ms  3.871 ms

lan-rules.png_thumb

TheGuyver

Heya,

Under Status, Gateways, does it show the gateway status as Online?

(I actually have a similar issue with pfsense ignoring routing :/ appears to be related to if the interface is set to the same as the DG and the gateway is on the same network)

R

Phil

Thanks for your help.

Both show as being online. See the attached.

gateway-status.png_thumb

TheGuyver

Mine was fixed by going to System: Adavanced - Firewall/Nat and ticking Disable reply-to (this is under version 2.01)

Phil

That doesn't seem to work either, sadly.

I have had luck when removing the default route from the routing table. In this case, traffic is directed to the gateway group by the firewall. This does however break VPNs etc. as pfsense itself cannot route packets. Is it possible to define a firewall rule to force pfsense itself to use this gateway group?

Thanks

cmb

why do you have two gateways on the same WAN? The OS's policy routing will break that scenario for complicated reasons about how the policy routing has to function and the fact that particular scenario is almost never seen.

You can work around that with floating rules, which can override the default "pass out" policy routing. You'll also have to disable reply-to in that scenario and may have issues with return routing because of it. Lot of complications caused by having two gateways on one WAN, generally not something I would recommend and that exact scenario isn't going to be easy to accommodate perfectly.

Phil

Thanks for the help, cmb. The situation is thus…

There's a primary (fiber) connection and a backup (ADSL) connection. Both have different WAN IPs but sit in a /30 subnet of the router on our side and the router on the ISP's side. A subnet is routed over both of these connections but the ADSL cannot be dialled when the primary connection is up, else packet loss occurs. The setup is such that 2 routers (a Cisco router and a low end ADSL router) sit in front of pfsense, each assigned an IP in the routed subnet. The ADSL router is set to dial on demand and disconnect when idle. The intended setup was that pfsense could monitor traffic over the primary link and switch to routing traffic over the backup, in the case of failure.

Can you suggest an alternative setup, given the above?

Thanks :)

cmb

That scenario would be easier to accommodate entirely on the Cisco and having a single WAN and gateway on the firewall. It's in a better position to determine connectivity status too. It would just have a higher metric default route pointing to the ADSL router.

Phil

Good plan. Configured using ip sla and track on the Cisco. Thanks for the help :)