Netflix buffering with 3 WANs
-
I have 3 WANs setup with load-balancing configuration. The bandwidth and latency varies greatly between them. They are all US ISPs.
One is using cable, another fixed wireless, and the other 5G.The setup seems to work well enough when I do things like browse the web and download files from various wired and wireless devices. With enough streams, I'm able to fully utilized the combined download bandwidth of about 1.7 Gbps. Once in a while there is a bit of lag when the slower ISP is hit.
However, there is a problem with Netflix streaming. I have seen buffering happen, when none should occur, even with the slowest of all 3 ISPs, the 5G one. I did test them all individually for several days / weeks / years before to verify that.
Earlier tonight, buffering got stuck at 0% and there was no progress for over a minute. I checked the pfSense interface and it showed that all 3 gateways were up with 0 packet loss on any of them, and normal latency. I disabled 2 of the ISPs, but that did not fix the problem. Netflix just wouldn't continue streaming. A speedtest on my phone showed 200 Mbps, a bit on the low side, but it may have been routed to the middle ISP. I had to reboot pfSense to resolve the streaming issue. There was no more buffering for the rest of the night - several hours worth of streaming. Since one of the main reasons I am experimenting with multiple ISPs is for reliability, particularly cable outages while streaming Netflix late at night, this behavior is obviously problematic. I also want to be able to VPN remotely with redundancy - that part I got working fine since all 3 ISPs have public IP addresses, and 3 FreeDNS names did the trick.
I could experiment with failover in place of load balancing, and see if the problem happens again. But I would still like to understand the root cause of this issue. Is the load balancing expected to break Netflix ?
Here is a screenshot of the gateway & routing configuration :
The last screen, showing the firewall rule for IPv4 LAN with a gateway of v4LB, is what enabled the load balancing for me. Reverting it to default disables it - all outgoing connections from all devices are routed to Comcast.
The reason I'm not doing load balancing for IPv6 is that only Comcast supports it properly. Sail is IPv4 only. Verizon may support it, but not without additional configuration.
I have the weights of the v4 DHCP gateways set to 12 for Comcast, 2 for Sail, and 1 for Verizon, based on the advertised download rates of 1.2 Gbps, 200 Mbps, and 100 Mbps.
Both Comcast and Sail can go a lot higher at times. Verizon is hard capped. -
@madbrain If they are all different bandwidth, I would set different priorities, so they load-balance evenly. Might help but not sure.
Netflix servers probably don't like streaming connections flipping between 3 different public IP addresses. It might also trigger their account sharing blockers ? Try setting sticky connections in System / Advanced / Miscellaneous.
-
@pwood999 Thanks. I just set the sticky connections option.
What do you mean by "priorities" ? Do you mean the tier in the group ? If so, they are all at tier 1, otherwise there would be no load balancing.
I already set different weights for each gateway.
-
@madbrain Thats ok then. Yes all tier-1 but different priorities should use them in proportion.
The best way I found to test it is with Speedtest App rather than a browser. Have the PfSense dashboard traffic graphs running & then do thge test. I should be obvious if all 3 are being used correctly.
-
@pwood999 i think you are referring to weight, not priorities. The later is for the tier
And yes, speedrest definitely shows all 3 ISPs being used. And I have monitored the traffic graphs in pFSense as well.
That's not the issue. Netflix buffering is. -
@madbrain create a failover gateway with the 3 wan, and then use an alias to force Netflix clients to use that rather than load balance.
-
@pwood999 said in Netflix buffering with 3 WANs:
@madbrain create a failover gateway with the 3 wan, and then use an alias to force Netflix clients to use that rather than load balance.
Besides sticky connections, I think this might be the way to go, as the most likely cause here is issues accessing Netflix servers.
At least Comcast and Verizon probably host Netflix servers which are only accessible from inside the respective networks. Unclear to me when the buffering happens but I'm guessing that a stream that has successfully started, will work fine as long as you don't stop it??
Starting to watch something else, or even reconnecting and continuing to watch, might initiate a new connection/session. But the server which the Netflix client is trying to reach is no longer accessible. if that session ended up going out via a different WAN because of load balancing.Are you using pfBlockerNG? If so, you can use that to create an Alias for Netflix ASN's. Then create a rule that you place above your current v4LB rule, targeting that Alias and directing all traffic for Netflix servers to a new Gateway group that you create as a failover group instead (different Tiers).
Simplest way to do that is to copy v4LB, and just change the name of the group and the Tiers to 1, 2 and 3. At least make sure Comcast is Tier 1 if that is what you want for Netflix. Also set the weight to 1 on all of them in that group I suppose.
[A note on the Gatway in the rule, referring back to the other thread about load balancing. I had it set to Default to make it work, and you need to specify it in the rule... The difference here is that I have specified it in the System / Routing / Gateways section where you have Automatic instead... ]
-
M madbrain referenced this topic on
-
@madbrain said in Netflix buffering with 3 WANs:
sticky
Unfortunately, using sticky connections effecttively disables load balancing.
Here is a traffic graph without sticky connections :
And here is the graph with sticky connections enabled :
As you can see, only the first WAN, Comcast gets used. The overall speed is also much lower.
Even more surprising, the bandwidth on Comcast is only about half what it was without sticky connections - 700 Mbps vs 1400 Mbps.
This may be due additional overhead in pfSense for sticky connections. Although this is hard to believe since my pfSense is running an AMD 5700g APU.
Indeed, when I repeated the test, I got different results each time. Sometimes hitting 1.4 Gbps, others not.The interesting part is that I tried to run speedtest from 3 different hosts - 2 on LAN, plus my smartphone, and still only the Comcast WAN got used. So it seems "use sticky connections" really disables load balancing effectively.
-
@pwood999 Thanks. I'm not familiar with aliases. What do they accomplish ? I will RTFM.
My goal is to keep the load balancing in place, and also have things work seamlessly if one ISP goes down.
The problems with Netflix & load balancing have not been consistent, though they happened several days. Not sure if there is a way to see the history of which ISP(s) I was using by time period as I have been switching things around a bit (disabling some, enabling others).
-
@Gblenn No, actually on several occasions, the streaming started playing fine for a while, and then buffered in the middle a few times. I tried to restart stop and restart the stream, but at that point it got stuck at 0%. Very strange behavior.
Not sure about which ISPs host Netflix servers.
I believe Netflix is supposed to work with multiple ISPs and load balancing based on the following article : https://wifiranger.com/how-load-balancing-makes-netflix-and-chill-better/ . However, pfSense is not mentioned, even in the link, and the devil is in the details.
I'm not using pfBlockerNG.. It's a pretty vanilla configuration other than the load balancing, having 3 VPNs configured (one for each ISP), and a large number of hosts on the LAN/WLAN, and using 10 Gbps Ethernet on the LAN side.
Changing the tiers in the way you suggested would change it from a load-balancing configuration to a failover configuration, right ?
Netflix can work on any of the 3 ISPs, even at 4K. I don't have a preference on where to run it.I have seen some guides that configure multiple WANs with both load-balancing, by creating additional failover gateways. I have not done that. Right now, all my ISPs are up and there is no need for failover. I can simulate failover for Comcast by unplugging the cable. Harder to go on the roof and repoint the antenna for Sail. Unplugging the Ethernet from pfSense to the PoE that powers the modem/antenna would disable the NIC and cause a different error condition. For Verizon, I would need a Faraday cage for that 5G gateway. I tried before with another device, and failed because the device inside overheated. I can't recall what it was. Pretty surprised that cell phones still work in a microwave oven (turned off, obviously). The only thing that I have seen really block wireless signals is water, when my (waterproof) phone drops in the hot tub . Wifi drops instantly. Can't remember if cell does. The cell signal is unusable in the first place, even without being underwater.
As far as the rules, I do have the "Default Gateway IPv4" set to automatic. If I change it to my "v4LB" gateway group, load balancing stops working, strangely. I had to set it in "Firewall / Rules / LAN" for load balancing to work. It's a bit strange that we are seeing different behavior.
-
@madbrain said in Netflix buffering with 3 WANs:
@Gblenn No, actually on several occasions, the streaming started playing fine for a while, and then buffered in the middle a few times. I tried to restart stop and restart the stream, but at that point it got stuck at 0%. Very strange behavior.
Not sure about which ISPs host Netflix servers.
I believe Netflix is supposed to work with multiple ISPs and load balancing based on the following article : https://wifiranger.com/how-load-balancing-makes-netflix-and-chill-better/ . However, pfSense is not mentioned, even in the link, and the devil is in the details.
Perhaps this article actually explains why it is in buffering also in the middle. If Netflix is in fact using multiple TCP streams to fill your buffer, there is a fair chance that at lest one of them end up on a different gateway. And then if the service Netflix is connecting to is hosted within one or the other ISP, it can't reach it with that stream and ends up retrying until it fails and decides perhaps to retry.
Routing the Netflix traffic over a faileover gateway instead, will solve this problem, and sticky connections should also solve it.I'm not using pfBlockerNG.. It's a pretty vanilla configuration other than the load balancing, having 3 VPNs configured (one for each ISP), and a large number of hosts on the LAN/WLAN, and using 10 Gbps Ethernet on the LAN side.
The point of using pfBlockerNG is to automate the discovery of Netflix IP's, based on Alias and Netflix ASN (2906). You can then create a policy rule that routs only traffic targeting those IP's via the failover group, instead of the loadbalancer. This rule needs to sit above your current allow all rule(s).
Changing the tiers in the way you suggested would change it from a load-balancing configuration to a failover configuration, right ?
I am not suggesting changing the group you have created for loadbalancing. The idea is to create another group, with a failover setup, so you end up with two different groups. One which is used for most traffic, and another that is only used for Netflix (and perhaps other streaming services).
Netflix can work on any of the 3 ISPs, even at 4K. I don't have a preference on where to run it.
I have seen some guides that configure multiple WANs with both load-balancing, by creating additional failover gateways. I have not done that. Right now, all my ISPs are up and there is no need for failover. I can simulate failover for Comcast by unplugging the cable. Harder to go on the roof and repoint the antenna for Sail. Unplugging the Ethernet from pfSense to the PoE that powers the modem/antenna would disable the NIC and cause a different error condition. For Verizon, I would need a Faraday cage for that 5G gateway. I tried before with another device, and failed because the device inside overheated. I can't recall what it was. Pretty surprised that cell phones still work in a microwave oven (turned off, obviously). The only thing that I have seen really block wireless signals is water, when my (waterproof) phone drops in the hot tub . Wifi drops instantly. Can't remember if cell does. The cell signal is unusable in the first place, even without being underwater.
You can simulate failover simply by disabling the desired interface from within pfsense. Or create a rule that blocks the IP you use to monitor the gateway, which makes it think it is failing.
As far as the rules, I do have the "Default Gateway IPv4" set to automatic. If I change it to my "v4LB" gateway group, load balancing stops working, strangely. I had to set it in "Firewall / Rules / LAN" for load balancing to work. It's a bit strange that we are seeing different behavior.
Are you chaning in both places? Under System / Routing / Gateways Setting AND then in the firewall rule? So that IPv4 is like you have it defined for IPv6...
It should work since your "system routing" has been set under exactly that section...
-
@madbrain said in Netflix buffering with 3 WANs:
Unfortunately, using sticky connections effecttively disables load balancing.
I don't think sticky connections breaks your loadbalancing. It just changes the way your test works, where all connections end up going out one interface since they are in fact targeting the same server. Which is what you want when watching Netflix as well.
If you can test from multiple PC's simultaneously, selecting different speedtest servers, you will likely see them ending up on more than one interface. Or try Speedtest at the same time as some of the other tests you can use, speedof.me and fast.com for example. Netflix has it's own "speedtest" that you can do from within the App, under Profile > App-settings > Diagnostics.
Or you can start a Netflix stream and a couple of YouTube sessions, perhaps from different PC's, so you can "see" in the graphs where they end up. -
@madbrain I have a simple alias containing the URL's that need to bypass load-balance.
Then in LAN rules, for connections to those site to use one GW.
If that GW is down, then it's manual intervention to flip the rule to another GW
-
@pwood999 said in Netflix buffering with 3 WANs:
@madbrain I have a simple alias containing the URL's that need to bypass load-balance.
Then in LAN rules, for connections to those site to use one GW.
If that GW is down, then it's manual intervention to flip the rule to another GW
Why do you need to manually change to a different GW? Why not use a failover group instead?
-
@Gblenn You are correct, hence why I suggested it above !!
Doing on my setup got delayed due to other work & family stuff. Since then I have ditched the 2nd broadband due to rising costs !!
-
@Gblenn Each instance of speedtest is not going to the same server, actually. The sticky connections really do break the load balancing for me.
I'm down to 2 ISPs now (cancelled Comcast yesterday).Example from my Windows machine alone without sticky connections :
Server: WiLine Networks - San Francisco, CA (id: 17587) ISP: Verizon Wireless Idle Latency: 9.99 ms (jitter: 0.01ms, low: 9.97ms, high: 9.99ms) Download: 324.46 Mbps (data used: 448.5 MB) 103.45 ms (jitter: 34.71ms, low: 17.02ms, high: 350.77ms) Upload: 30.59 Mbps (data used: 30.3 MB) 27.33 ms (jitter: 8.53ms, low: 10.49ms, high: 63.04ms) Packet Loss: Not available. Result URL: https://www.speedtest.net/result/c/d9138c04-a261-4a97-99f9-337aeaafbb5aFrom my Linux box alone : ``` Server: NETGEAR Inc. - San Jose, CA (id: 43447) ISP: Sail Internet Idle Latency: 9.94 ms (jitter: 1.82ms, low: 9.94ms, high: 17.22ms) Download: 324.81 Mbps (data used: 579.1 MB) 99.00 ms (jitter: 25.55ms, low: 19.52ms, high: 186.42ms) Upload: 40.70 Mbps (data used: 55.4 MB) 238.00 ms (jitter: 65.47ms, low: 14.98ms, high: 543.64ms)In each of these tests, the server is different, and both WANs were used, which increased the throughput. I could see it in the traffic graphs.
Now, if I turn on sticky connections, here is what happens :
Server: NETGEAR Inc. - San Jose, CA (id: 43447) ISP: Sail Internet Idle Latency: 9.94 ms (jitter: 1.82ms, low: 9.94ms, high: 17.22ms) Download: 324.81 Mbps (data used: 579.1 MB) 99.00 ms (jitter: 25.55ms, low: 19.52ms, high: 186.42ms) Upload: 40.70 Mbps (data used: 55.4 MB) 238.00 ms (jitter: 65.47ms, low: 14.98ms, high: 543.64ms)In this case, the traffic graph showed everything went to Sail Internet, and nothing to Verizon.
Now, from my Windows box :
Server: Primcast - Oakland, CA (id: 62644) ISP: Verizon Wireless Idle Latency: 32.38 ms (jitter: 10.87ms, low: 26.48ms, high: 47.46ms) Download: 45.05 Mbps (data used: 57.4 MB) 166.92 ms (jitter: 52.56ms, low: 38.98ms, high: 422.89ms) Upload: 12.04 Mbps (data used: 18.1 MB) 351.14 ms (jitter: 73.21ms, low: 19.75ms, high: 1257.77ms)Everything went to Verizon in this case.
Now, both simultaneously :
Server: eero - San Jose, CA (id: 41818) ISP: Sail Internet Idle Latency: 9.95 ms (jitter: 0.06ms, low: 9.91ms, high: 10.09ms) Download: 66.31 Mbps (data used: 39.9 MB) 25.81 ms (jitter: 9.77ms, low: 12.38ms, high: 91.09ms) Upload: 32.49 Mbps (data used: 43.9 MB) 249.48 ms (jitter: 67.59ms, low: 77.49ms, high: 663.97ms) Server: Verizon - San Francisco, CA (id: 29623) ISP: Verizon Wireless Idle Latency: 19.43 ms (jitter: 9.60ms, low: 15.29ms, high: 29.71ms) Download: 51.55 Mbps (data used: 66.5 MB) 190.94 ms (jitter: 58.95ms, low: 13.09ms, high: 427.55ms) Upload: 11.74 Mbps (data used: 12.2 MB) 284.39 ms (jitter: 71.24ms, low: 15.91ms, high: 811.42ms)Traffic graph shows some traffic went to both ISPs. This is different from the results I had before, when I ran 3 of these on different hosts against different servers, and all the traffic went to a single WAN. The combined bandwidth is much lower than without sticky connections, though. Perhaps the change is due to the weight of my Gateways. Sail is set to weight 3 and Verizon set to 1. Comcast was previously set to 12, and everything was going to Comcast.
-
@madbrain said in Netflix buffering with 3 WANs:
Each instance of speedtest is not going to the same server, actually. The sticky connections really do break the load balancing for me. I'm down to 2 ISPs now (cancelled Comcast yesterday).
Well, what you are showing is that each instance is in fact going to one and only one server, and that using sticky connections does exactly what is expected... and that it's still working.
What I mean with server is the target machine which speedtest uses in the test.
Here you are running one test from the Windows machine which goes to one server : WiLine Networks - San Francisco, CA (id: 17587)Example from my Windows machine alone without sticky connections :
Server: WiLine Networks - San Francisco, CA (id: 17587) ISP: Verizon Wireless Idle Latency: 9.99 ms (jitter: 0.01ms, low: 9.97ms, high: 9.99ms) Download: 324.46 Mbps (data used: 448.5 MB) 103.45 ms (jitter: 34.71ms, low: 17.02ms, high: 350.77ms) Upload: 30.59 Mbps (data used: 30.3 MB) 27.33 ms (jitter: 8.53ms, low: 10.49ms, high: 63.04ms) Packet Loss: Not available. Result URL: https://www.speedtest.net/result/c/d9138c04-a261-4a97-99f9-337aeaafbb5aYour Linux box ended up using a different server : NETGEAR Inc. - San Jose, CA (id: 43447)
So two different sessions and different servers but only one server per session, agree?
Now, without sticky connections, the "streams" for each one of these tests, are going via both your ISP connections (WAN). Which proves to you that load balancing is working.
However great it is to see that you get better results in speedtest, the way this is working is not really the idea behind load balancing. Rather you are after better overall performance for the company as a whole, or household in your case. So even though the shared bandwidth is now the aggregate, you still expect each application or session to use only one connection at a time.
Splitting them up across two or more WAN's may even "break the connection"... Like you are experiencing with Netflix.And to be honest, I don't know if a household could really benefit from load balancing all that much, unless you have quite low bandwidth per ISP. And you do a lot of downloading of application updates, game updates etc.
Now if you have two ISP's for the purpose of failover, you can still use them for loadbalancing of course, so there is absolutely no harm in doing it.Anyway, when you turn on sticky connections, all traffic from a single session is expected to go out via one and only one of the WAN interfaces. Which you also tested and showed...
Now, if I turn on sticky connections, here is what happens :
Server: NETGEAR Inc. - San Jose, CA (id: 43447) ISP: Sail Internet Idle Latency: 9.94 ms (jitter: 1.82ms, low: 9.94ms, high: 17.22ms) Download: 324.81 Mbps (data used: 579.1 MB) 99.00 ms (jitter: 25.55ms, low: 19.52ms, high: 186.42ms) Upload: 40.70 Mbps (data used: 55.4 MB) 238.00 ms (jitter: 65.47ms, low: 14.98ms, high: 543.64ms)In this case, the traffic graph showed everything went to Sail Internet, and nothing to Verizon.
And this is the expected outcome, since the whole idea with sticky connections is to keep each application or session going out only one of the WAN's, to avoid problems.
When you test both of them simultaneously, you say that you still see the traffic going out both WAN's, meaning that load balancing is in fact still working.
Now, both simultaneously :
Server: eero - San Jose, CA (id: 41818) ISP: Sail Internet Idle Latency: 9.95 ms (jitter: 0.06ms, low: 9.91ms, high: 10.09ms) Download: 66.31 Mbps (data used: 39.9 MB) 25.81 ms (jitter: 9.77ms, low: 12.38ms, high: 91.09ms) Upload: 32.49 Mbps (data used: 43.9 MB) 249.48 ms (jitter: 67.59ms, low: 77.49ms, high: 663.97ms) Server: Verizon - San Francisco, CA (id: 29623) ISP: Verizon Wireless Idle Latency: 19.43 ms (jitter: 9.60ms, low: 15.29ms, high: 29.71ms) Download: 51.55 Mbps (data used: 66.5 MB) 190.94 ms (jitter: 58.95ms, low: 13.09ms, high: 427.55ms) Upload: 11.74 Mbps (data used: 12.2 MB) 284.39 ms (jitter: 71.24ms, low: 15.91ms, high: 811.42ms)Traffic graph shows some traffic went to both ISPs.
This might not happen every time though...
Without sticky connections, the large number of "streams" that are set up by speedtest end up being distributed (round-robin) based on the interface weights that you set.
With sticky connections turned on, they are all getting "stuck" together per application/session and all of the streams from each individual speedtest session go out only one interface. And running only two sessions simultaneously, you will now and then end up with both of them going out the same interface (3:1 it's Verizon).
Which sort of shows the reduced benefit of loadbalancing in a household scenario with only a few simultaneous users... Since it's only with a large number of parallell sessions that you make full use of both WAN connections.Why the combined bandwidth is much lower when using sticky connections is a bit of a mystery though. And perhaps someone else can explain that fenomenon, if it is always the case...
-
Why the combined bandwidth is much lower when using sticky connections is a bit of a mystery though. And perhaps someone else can explain that fenomenon, if it is always the case...
Yes, it is strange. However, I did not run the 2 simultaneous speedtests with sticky connections disabled. I just did it now and here are the results :
Server: Sonic.net, Inc. - San Jose, CA (id: 17846) ISP: Sail Internet Idle Latency: 14.92 ms (jitter: 6.14ms, low: 9.38ms, high: 20.02ms) Download: 255.09 Mbps (data used: 344.9 MB) 18.34 ms (jitter: 6.05ms, low: 9.15ms, high: 57.91ms) Upload: 30.02 Mbps (data used: 54.5 MB) 199.27 ms (jitter: 64.95ms, low: 23.46ms, high: 311.32ms) Server: Macarne LLC - Santa Clara, CA (id: 58605) ISP: Sail Internet Idle Latency: 9.98 ms (jitter: 0.07ms, low: 9.88ms, high: 10.08ms) Download: 152.55 Mbps (data used: 204.4 MB) 117.95 ms (jitter: 35.04ms, low: 13.50ms, high: 353.08ms) Upload: 26.62 Mbps (data used: 48.2 MB) 198.95 ms (jitter: 60.69ms, low: 66.69ms, high: 331.88ms) Packet Loss: 0.0%
For the first test with Sail internet, it seems to have hit the limits of available bandwidth for that ISP.
For the second test, it clearly used both ISPs, as Verizon maxes out at 110 Mbps down / 12 Mbps up.
Both ISPs have a lot of variability due to being wireless and thus sharing bandwidth with other customers.
I ran the 2 speedtests again, still without sticky connections, and got this traffic graph. I believe both ISPs were maxed out in this case.
Now, I turned sticky connections back on. Interestingly, I got a very similar result : both ISPs maxed out. Quite different from what I tested earlier yesterday. It looks like this is really unpredictable. Doing it in the middle of night as I am right now, might make it more stable - fewer customers using the bandwidth at the same time.
I ran it again and got another similar traffic graph - both ISPs maxed out. :
What's interesting is the results of individual speedtests.
Server: EGI Hosting - Santa Clara, CA (id: 32408) ISP: Sail Internet Idle Latency: 10.57 ms (jitter: 1.59ms, low: 9.91ms, high: 13.12ms) Download: 306.95 Mbps (data used: 498.5 MB) 43.92 ms (jitter: 20.13ms, low: 12.18ms, high: 351.24ms) Upload: 39.24 Mbps (data used: 52.8 MB) 131.21 ms (jitter: 39.09ms, low: 9.05ms, high: 420.47ms) Packet Loss: 0.0% Server: WiLine Networks - San Francisco, CA (id: 17587) ISP: Verizon Wireless Idle Latency: 15.34 ms (jitter: 3.59ms, low: 14.51ms, high: 28.80ms) Download: 102.18 Mbps (data used: 154.3 MB) 100.57 ms (jitter: 35.16ms, low: 16.95ms, high: 486.20ms) Upload: 11.13 Mbps (data used: 15.2 MB) 32.04 ms (jitter: 16.00ms, low: 12.92ms, high: 292.68ms)The second speedtest bandwidth of 102 Mbps down / 11 Mbps up is pretty close to the max capabilities of Verizon.
The first test is pretty close to the max I can get from Sail, though that service has more variability.
I will leave it on for now.
I still would like to make sure both the fail-over and the load-balancing work..
Streaming not being interrupted is one of the key reasons for having 2 ISPs :) Getting rid of Comcast yesterday should help that, though.
Nevertheless, if sticky connections are on, and the ISP that these connections are going to goes down, what can I expect to happen to the streaming ? Disconnect, brief interruption, forced to manually resume ? I can of course try to disable an interface, or unplug one Ethernet cable, but I'm 100% sure this will be cause the same error condition as if one ISP link fails, but the 2 wireless modems stay connected to pfSense..
-
@madbrain said in Netflix buffering with 3 WANs:
Why the combined bandwidth is much lower when using sticky connections is a bit of a mystery though. And perhaps someone else can explain that fenomenon, if it is always the case...
Yes, it is strange. However, I did not run the 2 simultaneous speedtests with sticky connections disabled. I just did it now and here are the results :
Now, I turned sticky connections back on. Interestingly, I got a very similar result : both ISPs maxed out. Quite different from what I tested earlier yesterday. It looks like this is really unpredictable. Doing it in the middle of night as I am right now, might make it more stable - fewer customers using the bandwidth at the same time. I ran it again and got another similar traffic graph - both ISPs maxed out. :  What's interesting is the results of individual speedtests.Server: EGI Hosting - Santa Clara, CA (id: 32408)
ISP: Sail Internet
Idle Latency: 10.57 ms (jitter: 1.59ms, low: 9.91ms, high: 13.12ms)
Download: 306.95 Mbps (data used: 498.5 MB)
43.92 ms (jitter: 20.13ms, low: 12.18ms, high: 351.24ms)
Upload: 39.24 Mbps (data used: 52.8 MB)
131.21 ms (jitter: 39.09ms, low: 9.05ms, high: 420.47ms)
Packet Loss: 0.0%Server: WiLine Networks - San Francisco, CA (id: 17587) ISP: Verizon WirelessIdle Latency: 15.34 ms (jitter: 3.59ms, low: 14.51ms, high: 28.80ms)
Download: 102.18 Mbps (data used: 154.3 MB)
100.57 ms (jitter: 35.16ms, low: 16.95ms, high: 486.20ms)
Upload: 11.13 Mbps (data used: 15.2 MB)
32.04 ms (jitter: 16.00ms, low: 12.92ms, high: 292.68ms)The second speedtest bandwidth of 102 Mbps down / 11 Mbps up is pretty close to the max capabilities of Verizon. The first test is pretty close to the max I can get from Sail, though that service has more variability. I will leave it on for now. I still would like to make sure both the fail-over and the load-balancing work.. Streaming not being interrupted is one of the key reasons for having 2 ISPs :) Getting rid of Comcast yesterday should help that, though. Nevertheless, if sticky connections are on, and the ISP that these connections are going to goes down, what can I expect to happen to the streaming ? Disconnect, brief interruption, forced to manually resume ? I can of course try to disable an interface, or unplug one Ethernet cable, but I'm 100% sure this will be cause the same error condition as if one ISP link fails, but the 2 wireless modems stay connected to pfSense..Ok, so then it seems everything is actually working as advertized and there isn't a problem with capacity or throughput when using sticky connections...
And I agree, wireless connections will render a bit more varying results than would a fiber or cable connections. So testing at different time of day will generate sligthly different results. But your new tests seem to show that you are actually getting the same results also when using sticky connections.
The difference between failover and loadbalancing is only in the Tiers you set on the interfaces. I have a metered wireless connection which I obviously don't want to use too much, since I would run out of "data". Therefore I have it as Tier 2 and my fiber connection as Tier 1. This means pfsense is routing all traffic via the fiber, unless that connection is deemed down.
In your case, pfsense is routing traffic from your clients/applications through both your ISP connections, set at the same Tier (1). The additional setting with "weight" is just to make sure the round-robin distribution between the two interfaces is done with the right balance, according to their respecitve bandwidth.
But as a user in your network will never know which one you will end up on, unless you create separate rules per user or application. And this is of course fine as long as the connections provide "equal performance", or performance good enough for everything you are doing.
By activating sticky connections you have now made sure that all streams/connections from any one single application, are routed through the same interface, so nothing risks breaking.
In the case of an actual failover, where one or the other ISP goes down, you will have some connections that happen to be on the failing interface and some that are already on the working interface. But whether or not you or anyone in your network will actually notice anything depends on the application they happen to use.
Real time applications like gaming or some form of communication, like Teams, Zoom etc will notice a brief (5-10 seconds) interruption, before reconnecting on the second interface. They will then stay on that interface until the session is over.
Other applications like web browsing, or anything that is buffering like YouTube, Netflix etc will most likely not notice anything at all. Since from the time the interface goes down, until pfsense decides it is decidedly down and it switches over, you will be watching the movie or clip from the buffer already on the PC or device.
-
@Gblenn Thanks. I'm wondering where you got a metered wireless connection. I'm interested in one too. The one I have from Verizon is unlimited, but the price is a little too high for failover.