Multi Gateway same interface

fadygh · Mar 11, 2019, 1:03 PM

Hello
I'm trying to build load balancing gateway I have 4 ADSL connection all of them on the same subnet example 192.168.0.1 to 192.168.0.4
I added them in the gateways and interface is WAN, and I created a gateway group, I tried to tested it and noticed that if I turn off the default gateway it will mark it as down and but failover not working but if I turned off the other routers it will mark them down but the internet is still working, do I need to have seperat interface for each gateway and do they need to be on a different subnets

Best

Grimson · Mar 11, 2019, 11:35 AM

RTFM: https://docs.netgate.com/pfsense/en/latest/book/multiwan/multi-wan-caveats-and-considerations.html

fadygh · Mar 11, 2019, 11:40 AM

Just to confirm my problem isn't because I'm using same interface but because they are on the same subnet. is that right

stephenw10 · Mar 11, 2019, 12:38 PM

It's not ideal but you can have 4 gateways on the same interface and in the same subnet. As long as the gateways themselves are different IPs then pfSense can route to them independently.

As long as the interface they are on can carry the traffic for all 4 WANs that is. It probably can if it's Gigabit Ethernet and the WANs are ADSL though.

Because they are on one interface it makes things like traffic shaping and firewall rule more complex as they are not separated.

You would likely also have issues with port forwards on anything but the default gateway.

Steve

fadygh · Mar 11, 2019, 12:50 PM

Can anyone guide me to best practice for multiwan:

Does it work if multi WAN are all on the same subnet
Do I need separate Network interface for each WAN

Best

NogBadTheBad · Mar 11, 2019, 1:03 PM

@fadygh said in Multi Gateway same interface:

Hello
I'm trying to build load balancing gateway I have 4 ADSL connection all of them on the same subnet example 192.168.0.1 to 192.168.0.4
I added them in the gateways and interface is WAN, and I created a gateway group, I tried to tested it and noticed that if I turn off the default gateway it will mark it as down and but failover not working but if I turned off the other routers it will mark them down but the internet is still working, do I need to have seperat interface for each gateway and do they need to be on a different subnets

Best

Can you not put the 4 ADSL connections into modem mode ?

stephenw10 · Mar 11, 2019, 4:09 PM

Yeah, I just told you above you don't need to.

However best practice here is to use 4 separate interfaces and connect them to devices acting as a modem so that you have public IP addresses on those interfaces.

Steve

fadygh · Mar 12, 2019, 11:20 AM

ok now I followed your suggestion but I'm now using two gateways with two WAN interfaces each on a different subnet I can ping using both wans, I also configured firewall rule in LAN interface and selected gateway the group-wan but still not working I unplugged WAN1 and I lost internet connection on laptop, but if I unplug WAN2 I still receive reply from 8.8.8.8
any suggestions where should I search, thanks in advance

NogBadTheBad · Mar 12, 2019, 12:38 PM

This post is deleted!

stephenw10 · Mar 12, 2019, 1:25 PM

Do you have DNS servers on both WANs and the service in forwarding mode?:
https://docs.netgate.com/pfsense/en/latest/routing/multi-wan.html#dns-considerations

Or the failover group set as the default gateway which will allow it work in resolving mode?

Edit: Ok I see you have 'group_wan' set as the default gateway. Is that the load-balancing group? If so that's invalid, you can only use individual gateways or failover groups there.
Set up an additions group as failover and use that.

Steve

fadygh · Mar 12, 2019, 2:22 PM

I'm able to ping booth 8.8.8.8 and www.google.com from both interfaces but in the dashboard gateways status it shows me one of them is offline

stephenw10 · Mar 12, 2019, 6:01 PM

Well what is 192.168.5.253? It's not responding to ping.

fadygh · Mar 13, 2019, 7:16 AM

@stephenw10 I finally managed to fix the gateway marked down I followed the below thread
https://forum.netgate.com/topic/98151/2-3-gateway-monitor-not-working/2

now I tested load balancing by marking the gateways as down and load balancing is working but when I manually unplug the cables internet will go down when I unplug wan1 which is the default but not wan2

stephenw10 · Mar 13, 2019, 3:29 PM

You configured DNS to use both as I outlined above?

How are you testing to see "internet will go down"?

Steve

fadygh · Mar 14, 2019, 11:04 AM

@stephenw10 yes I configured DNS for both gateways and I set the same DNS for the monitoring IP, but I think that I have a routing problem I created the firewall rule and linked it to the WAN-group but I'm still having the same problem only one interface is working even though they both have inernet and I can verify that by doing traceroute command I see from pfsense I tested it from two wans and I can see that each wan has different hops IP addresses but I still unable to do load balancing I also tried to force the firewall rule to pass only from the gateway that have problem with it but still no internet on computer, the computer is connected directly to pfsense machine LAN port, I can only get internet from one gateway even thouh they both have internet and the status of both gateways in online except when I unplug any any cable it can detect that it's offline
any suggestion would be appreciated

stephenw10 · Mar 14, 2019, 4:11 PM

Ok so when you disconnect the main WAN what exactly does and doesn't work?

I assume you are still able to ping out and do dns lookups from pfSense itself? Without specifiying a source IP?

Can you do dns lookups from a client on LAN?

Can you ping an external IP (by IP) from a client?

Can you ping the WAN2 gateway or DNS server on WAN2 from the client?

If you traceroute from the client where does it fail?

Check /tmp/rules.debug. When WAN1 is down it should be removed from the gateway group.

Steve

fadygh · Mar 18, 2019, 10:24 AM

I did two continuous pings from computer one ping to www.google.com and another ping to 8.8.8.8 if two WAN cable are connected they both get reply. but if I unplugged WAN1 I get request time out on www.google.com and if I unplug WAN2 I get request time out on 8.8.8.8. any suggestions for this situation

stephenw10 · Mar 18, 2019, 12:24 PM

Continuous pings is not a good test. The firewall states are not removed when the gateway goes down unless you have set Flush all states when a gateway goes down in Sys > Adv > Misc. As long as the ping is still running the state will not timeout. If you stop the ping and restart it after some time it should go out over the good gateway.

Are you using 8.8.8.8 as a DNS server for the firewall? If so that may have a static route via WAN2 which means it can never work over WAN1.

Steve

fadygh · Mar 19, 2019, 8:54 AM

I'm sure that there is something missing in the manual I followed all he instructions with no success. now I did factor default reset, I have three NIC interface I configured them as follow WAN1 WAN2 and LAN
LAN is 192.168.1.1
WAN1 static IP address 192.168.0.171 Gateway 192.168.0.239 DNS is 8.8.8.8
WAN2 static IP address 192.168.5.254 Gateway 192.168.2.253 DNS is 8.8.4.4 (I put a NAT device in order to change the range of the network as mentioned in the manual)
in routing I set monitoring IP address same as DNS for each interface
I created a wangroup and set them both tier1 and trigger level is member down
I modified the internet rule and in the gateway I selected the wangroup

is there anything else that I have to do in order to make it work
I want to make load balancing by making users to get internet from both gateways and if one gateway fails the users that are on failed gateway will failover to the other gateway
is there any specific log that I can check to to post it may be it can help
please note I'm facing problem that sometimes one of gateways appears down even though it's not down

fadygh · Mar 19, 2019, 11:28 AM

finally it worked I used DNS forwarding instead of DNS resolver and it's working now
thanks everyone for help