Howto: simple multiWAN (VDSL + cable) for the noobs like me :-)
-
Create a group 'Failover1' which would mean failing over from WAN1 to WAN2. Set the priority of the backup WAN (WAN2 in this example) to the highest, hence 'Tier1', and set the priority of the main WAN (WAN1) to 1 lower, hence 'Tier2'. Do exactly the opposite for the next group, 'Failover2'. This is needed so traffic can be directed back to WAN1 when WAN2 fails or when WAN1, which is the default gateway, comes back online. I have set the 'trigger', so when to jump from one WAN to the other, to 'package loss'. I have a question about that below, because you could also do 'member down', but I don't know which one is the best. It currently works, but it might even work better other wise.
You do not use a 2nd group to make anything "fail back". The Failover1 description is around the wrong way - WAN should be Tier1 and WAN2 Tier2. Then traffic that you put into Failover1 will go out WAN normally, and when WAN is down it will go out WAN2.
Packet loss is good - member down really just looks at if the cable has fallen out.
If you normally want some traffic to go on WAN and some to go on WAN2 then you have to have rule/s that match some traffic and specify Failover1 and then other traffic and specify Failover2, and then maybe have a LoadBalance group where you feed the general "pass everything else" rule.
Services/Load Balancing is for when you have a few (often web) servers that people can connect to from the internet (e.g. serving your web site). That balances incoming connections to those servers. So you can ignore that.
For easy-to-match traffic you can design rules that match it and send to Failover1 or Failover2 as you wish. For Bit-torrent and other "nasties" like that, they will jump all over the place using various port numbers… trying to "evade capture". The failover and load balance software works fine if you can just dream up a way to reliably match the traffic :'(Thanks Phil ;D
So I managed to get something to work but actually it shouldn't be working ;D
Because: I test this setup; I disconnect WAN1 (VDSL) and pfSense switched to WAN2.
Are you also saying Failover2 is not neccessary, or did I misunderstand that?
-
If you put all your traffic into Failover1 then it will go on the main WAN normally, then on the backup WAN when main WAN is down. When main WAN comes back up again it will go back to main WAN. Failover2 is not needed for the main-to-backup scenario.
If you want to do "manual load balancing" then you have Failover1 (WAN1 normally, failing to WAN2) and Failover2 (WAN2 normally, failing to WAN1). Then put rules to match some traffic and direct it to Failover1, and rules to match other traffic and put it to Failover2. That way, when both WAN are up, you get some traffic on each. That sort of scheme actually can work quite well if you know your traffic and can design some rules that usually select half the traffic into Failover1 and half into Failover2.
Otherwise, you make a LoadBalance gateway group with WAN1 and WAN2 both on tier 1 and pump all the traffic into that. Then the system does the load balancing for you - round robin allocation of client connects to WANs. -
If you put all your traffic into Failover1 then it will go on the main WAN normally, then on the backup WAN when main WAN is down. When main WAN comes back up again it will go back to main WAN. Failover2 is not needed for the main-to-backup scenario.
If you want to do "manual load balancing" then you have Failover1 (WAN1 normally, failing to WAN2) and Failover2 (WAN2 normally, failing to WAN1). Then put rules to match some traffic and direct it to Failover1, and rules to match other traffic and put it to Failover2. That way, when both WAN are up, you get some traffic on each. That sort of scheme actually can work quite well if you know your traffic and can design some rules that usually select half the traffic into Failover1 and half into Failover2.
Otherwise, you make a LoadBalance gateway group with WAN1 and WAN2 both on tier 1 and pump all the traffic into that. Then the system does the load balancing for you - round robin allocation of client connects to WANs.Thank you Phil. You can explain things very clearly, you would make a very good teacher (or perhaps you even are one already :P).
I did as you told me and removed zFailover2. Tested it, and (of course) you were right. However, I do have one follow up problem:
After disconnecting WAN1 the boss in the house, WIFE (I have been thinking, perhaps I should also have a failover, WIFE2 ;D), started complaining she had no internet. At first I thought it was Squid which is proxy for LAN, but I didn't see anything there to tell about failover. Squid complained as in the attached screenshot.
After an hour of wondering I did a reset of the states, and now she could go on the internet again.
Shouldn't states automatically been taken care off? As in: killing them when WAN1 drops? And, even weirder: she couldn't browse any website, so not even the ones she had open in tabs, but just completely weird websites also weren't available.
Would you perhaps know how to solve this?
Thank you again Phil, I appreciate your help very much ;D
 -
If you put all your traffic into Failover1 then it will go on the main WAN normally, then on the backup WAN when main WAN is down. When main WAN comes back up again it will go back to main WAN. Failover2 is not needed for the main-to-backup scenario.
If you want to do "manual load balancing" then you have Failover1 (WAN1 normally, failing to WAN2) and Failover2 (WAN2 normally, failing to WAN1). Then put rules to match some traffic and direct it to Failover1, and rules to match other traffic and put it to Failover2. That way, when both WAN are up, you get some traffic on each. That sort of scheme actually can work quite well if you know your traffic and can design some rules that usually select half the traffic into Failover1 and half into Failover2.
Otherwise, you make a LoadBalance gateway group with WAN1 and WAN2 both on tier 1 and pump all the traffic into that. Then the system does the load balancing for you - round robin allocation of client connects to WANs.Hi Phil ;D
If I may bother you one more time :-[
I also read your thoroughly this post of you that I just found:
https://forum.pfsense.org/index.php?topic=64612.0
I had to read and compare it a couple of times since my limited brain at first thought it confliced with my thread here. But I think it doesn't ;D
Because, if I understand you correctly, you are saying:
1. If you have 1 default gateway and 1 backup WAN, [b]only 1 failover rule is sufficient.
2. That is even the case if you add 'automatic' load balancing to the scenario (automatic = pfSense does round Robin).
3. The case changes however if you want to do 'manual' load balancing ('policy based routing', as in: you manually direct traffic to specific gateway groups), in that case you do need a second fail over rule. Because as you are forcing traffic to WAN2 (which under failover rule 1 is the backup WAN for WAN1), you need a rule to have that forced traffic in WAN2 to go to WAN1 should WAN2 go down.Did I understand this correctly?
Because I think I want to do the manual load balancing, and something strange happened. I'll explain:
1. WAN1 is an unmetered plan. So I want all my downloading (torrents and stuff, which my Synology takes care off) to happen there.
2. WAN2 is a metered plan. So I want the normal browsing and email to go here. When WAN2 goes down I want to be moved to WAN1 (where traffic shaper is running which will then still give decent browsing speeds).So, from the perspective of WAN2, I then would need to have the second failover rule, right? But: I noticed that the Synology downloading on WAN1 (default gateway) so which I explicitly forced into WAN1, when WAN1 went down the other day, was pushed into WAN2 (which, per the current single Failover1, is the backup WAN). So even explicitly assigning it to WAN1 will not prevent it being sent to WAN2. I found a setting 'allow gateway switching' which probably takes care of this, and so I set a block rule for the Synology to ever go to WAN2. That worked; the downloading simply stopped.
So, now I am confused again (not the first time ;D): Could I ask, in the scenario I want:
A. Do I need a Failover2 rule for browsing to be transferred to WAN1, or not?
B. And in this Failover2 (so WAN2 ->WAN1) is WAN2 then Tier1 and WAN1 Tier2 (I think so?).
C. I do need to put the DNS-lookups on Failover1 (or Failover2, but on a failover, right?)
D. Where do I need to put these rules? Some people say at the top, others say at the bottom. It is being complex because of my Synology rules (I now put these in a different gateway group, with WAN1 = Tier1, and WAN2 = never, to prevent the Synology from draining the limited monthly traffic I have on WAN2).I created a screenshot of my LAN-rules, I will have to upload them externally as the attachment is too big.
Thank you once again very much Phil ;D,
Bye,
-
A. Do I need a Failover2 rule for browsing to be transferred to WAN1, or not?
Yes, you need Failover2 gateway group, and then as many rules as you need to specify what traffic goes into that gateway group.
B. And in this Failover2 (so WAN2 ->WAN1) is WAN2 then Tier1 and WAN1 Tier2 (I think so?).
Yes
C. I do need to put the DNS-lookups on Failover1 (or Failover2, but on a failover, right?)
In System->General Setup, have 2 DNS server IPs (e.g. Google 8.8.8.8 and 8.8.4.4 or whatever are your favourites) and select the first to WAN1, the 2nd to WAN2. Then whichever WAN is up, there will be a DNS server available.
D. Where do I need to put these rules? Some people say at the top, others say at the bottom. It is being complex because of my Synology rules (I now put these in a different gateway group, with WAN1 = Tier1, and WAN2 = never, to prevent the Synology from draining the limited monthly traffic I have on WAN2).
Synology - yes, that is right, you want to just put that traffic to WAN1 and not failover to anywhere. Your solution works. You can also just specify WAN1 as the gateway in the rule, the gateway group is not actually needed if the traffic must only go to 1 WAN.
The other rules, put the most specific rules first, then down the end is the general rule (that probably pushes "everything else" into failover2, so that general traffic not matched by any of the specifics rules goes to failover2=WAN2 priority) -
Thanks for this extremely fast reply, Phil ;D
(I will go to the site in your sig, btw).
I finally found a site that would let me upload my screenshot:
If you look at it, it currently is a mess (please ignore the disabled rules; my logs are flooded with all kinds of LAN-nonsense, and I can't get them to go away), but you will see I use DNS-forwarder.
And I can't just send all LAN over Failover1 as the first rule, as I want to restrict where my Synology goes. So it may DDNS, it may torrent (usenet), and then it may not go anywhere. So that is why I am messing with the order of the rules.
Now I will go and visit your site ;D
Thank you Phil :P
-
Yes, you have accumulated a bit of a list of rules there while playing/testing!
The gateway group rules are going to work OK in that list. There is a group of 4 rules towards the bottom that have source z_nas.- pass to zddns_synology - looks good
- pass to z_usenet goes into gateway group zSynology - looks good
- reject to z_usenet goes into gateway group WAN2_DHCP - that has no real effect, because (2) has already passed all this traffic to zSynology. Also there is no point feeding a block/reject rule into a gateway group, since there will be no packets coming out of it to actually feed into the GWG (hmmm - perhaps the GUI should complain about selecting a GWG on a block/reject rule?) - anyway, this rule could be removed with no harm.
- reject z_nas to anywhere -a good thing to do, any other traffic from z_nas not already passed by (1) or (2) gets rejected.
-
Yes, you have accumulated a bit of a list of rules there while playing/testing!
The gateway group rules are going to work OK in that list. There is a group of 4 rules towards the bottom that have source z_nas.- pass to zddns_synology - looks good
- pass to z_usenet goes into gateway group zSynology - looks good
- reject to z_usenet goes into gateway group WAN2_DHCP - that has no real effect, because (2) has already passed all this traffic to zSynology. Also there is no point feeding a block/reject rule into a gateway group, since there will be no packets coming out of it to actually feed into the GWG (hmmm - perhaps the GUI should complain about selecting a GWG on a block/reject rule?) - anyway, this rule could be removed with no harm.
- reject z_nas to anywhere -a good thing to do, any other traffic from z_nas not already passed by (1) or (2) gets rejected.
Thank you once again Sir Phil ;D ;D ;D
As to the bold, I would have expected it to be what you say. But in my strange situation the case was: I first indeed assigned the alias z_nas directly to WAN1 as the gateway (and later on even to the separate gateway group zSynology (in itself only containing WAN1) to see if this would help), but in testing my experience was: if I do not put the blockrule to prevent synology going on WAN2 (WAN2_DHCP, I am not allowed to change the rule into simply WAN2), it still failed over to WAN2. So even 'though I told it to stay on WAN1, and not be part of a gateway group for failover, it still did failover.
I think this is because I enabled 'allow default gateway switching'. Which I did because I've read this in the forum as a solution to a problem I thought I was having. I am not joking, but: I have no clue anymore why I flagged it, and what problem I was trying to solve (yes, I know, I am vague now ::)).
Or perhaps this other setting on the same page: 'Skip rules when gateway is down' , which I had not selected might have done something (from reading the description)? But then again, that description talks about a specific gateway being switched to the default gateway. But in my case this was not the case: WAN1 was the default gateway on which zSynology was. When WAN1 went down a failover to WAN2, not the default gateway, occurred and Synology travelled to WAN2 too. So the description of that setting says: 'if WAN1 down, a specific zSynology will be transferred to the default, WAN1. So not to WAN2. But still it did.
(Vague, I know ;D).
Do you think I'd better disable/enable the settings mentioned (in System/Advanced/Misc)?
Thank you again very much Phil ;D
-
C. I do need to put the DNS-lookups on Failover1 (or Failover2, but on a failover, right?)
In System->General Setup, have 2 DNS server IPs (e.g. Google 8.8.8.8 and 8.8.4.4 or whatever are your favourites) and select the first to WAN1, the 2nd to WAN2. Then whichever WAN is up, there will be a DNS server available.
One more question if I may, Phil?
I had already done what you wrote in the above whilst setting up dual WAN and following a tutorial. Both WAN's have their own DNS server. But: I am also running DNS Forwarder. As I understand it, DNS-forwarder takes any of the DNS-servers in System/General Setup to query for dns lookups, and then acts as its own private little cosy ( ;D) DNS-server on each LAN/VLAN.
So, this leaves me with:
1. These 4 external DNS-servers I entered in System/General, of which two of them are assigned to respectively WAN1 and WAN2 (and so two are not assigned), which ones are now used in this fail over scenario? Only the two that are assigned to WAN1 and WAN2, or all four? I mean, given that DNS Forwarder acts as my own internal DNS-server, why do WAN1 and WAN2 need to be assigned specifically to an external DNS? Doesn't DNS-forwarder simply take care of this? My assumption was that it is or/or: or you don't use DNS-forwarder and then you need to explicitly assign an external DNS-server to WAN1 and WAN2, or you do use DNS-forwarder and then you don't need to specifically assign external DNS-servers to WAN1 and WAN2, as DNS Forwarder automatically takes care of it.
2. Given I use DNS-forwarder on LAN and VLAN, do I explicitly need to have a rule on LAN: allow, src LAN-NET, dest LAN-ADDRESS, port DNS, gateway group zFailover1, or does pfSense take care of this automatically without this rule?I am indebted for all your help Phil, thank you very much ;D
-
I think this is because I enabled 'allow default gateway switching'.
That will allow "regular" traffic not directed to a gateway (group) to go out another WAN when the default is down - it is usually switched on if you want pfSense itself to find a way out when the default gateway is down (for update checking, getting packages…)
Traffic directed to a specific gateway should be forced to the gateway even if the gateway is down, but I will have a quick check of that tomorrow when I am doing some testing of 2.1.1-prerelease stuff.Or perhaps this other setting on the same page: 'Skip rules when gateway is down' , which I had not selected
Yes, if this had been selected, then when the gateway went down, the special rules for feeding traffic into the gateway would be removed from the running "pf" rule set. And so the traffic would fall through to the "regular" rules and go out the default gateway. In combination with 'default gateway switching' the traffic would end up going out WAN2. But you say this was not selected.
The DNS questions:
The DNS servers in General Setup are for the DNS Forwarder to use. The clients all talk to DNS Forwarder, but DNS Forwarder needs to know how to actually get the DNS information from the public internet. DNS servers specified in General Setup will all be used by DNS Forwarder. If gateway "none" is selected, then the route to them will be via the default route - usually WANGW. If WANGW is down then DNS queries cannot get out.
The ways around that are:- default gateway switching - after turning that on, then the default route will switch over when WAN goes down, and all the DNS queries should go out WAN2. In simple 2-WAN configurations with ordinary LANs that should nowadays work fine - there is only WAN2 to switch to. In 3 WAN configs you don't know if it will pick WAN2 or WAN3, and you might care if WAN3 is really slow. Other complicated configs might also have less-than-desirable/predictable outcomes with default gateway switching.
- specify the WAN for each DNS server - then specific route is made for traffic to that DNS server IP to go out that WAN. Then when DNS Forwarder makes queries, they go to the various defined DNS servers out the various WANs. As long as at least 1 of the WANs is still up then there will be a path to a DNS server. (this is the preferred/recommended solution)
On LAN you just want to let the clients reach the DNS Forwarder listening on LANaddress. Do not push that traffic into a gateway (group). So, on LAN:
allow, src LAN-NET, dest LAN-ADDRESS, port DNS
The default allow all on LAN rule allows the above anyway. But if you are removing the "allow all on LAN" rule and just specifying exactly what you want to allow, then you need to allow the above access to DNS Forwarder. -
Sorry for not responding sooner Phil :'(
The thing is, i was trying 1001 things to solve my problem myself and didn't want to bother you with it because you've already devoted so much time in helping me - for which thank you once again ;D
But I keep on fighting with the Synology going to WAN2 when WAN1 has connection problems, which I don't want it to do.
As a recap:
1. WAN1 is VDSL with unlimited traffic; Synology downloads here.
2. WAN2 is Cable, metered 100 GB monthly traffic, so purely meant as fall back. Synology shall never go there.
3. Given your remarks before I put z_nas (= synology) not on the failover group but on the Gateway 'WAN1'.
4. Occassionally, I wake up in the morning discovering WAN1 was down and Synology hopped over on WAN2 and downloaded too much there (metered).I know you said this shouldn't be possible, but it does do it( :-). I double checked to make sure the Synology always on WAN1 rule is before the 'LAN to any' which goes through Failover1. So, if I understand correctly the Synology traffic is covered by the more specific rule, and therefor should never be hit by the more general 'LAN to any' rule that comes later.
Would you happen to know how I might fix this mess?
Also, I don't quite understand this (screenshots). I have a rule that says Synology should never go to WAN2 (you wrote in the above that won't work, but I am still playing with it to see). But now in the firewall log there are blocks caused by that rule, preventing the Synology to go DNS on the WAN1-interface.
I suspected before already that copying a rule and adjusting it turns out buggy (I get the wrong descriptions in the logs, for example a block on LAN shows up in the logs as a block triggered by a rule for OPT3, which is a VLAN).
As ever I am in big debt towards you Phil; thank you for your help ;D