Netflix buffering with 3 WANs

[pwood999]

@Gblenn You are correct, hence why I suggested it above !!

Doing on my setup got delayed due to other work & family stuff. Since then I have ditched the 2nd broadband due to rising costs !!

[madbrain]

@Gblenn Each instance of speedtest is not going to the same server, actually. The sticky connections really do break the load balancing for me.
I'm down to 2 ISPs now (cancelled Comcast yesterday).

Example from my Windows machine alone without sticky connections :

Server: WiLine Networks - San Francisco, CA (id: 17587)
         ISP: Verizon Wireless
Idle Latency:     9.99 ms   (jitter: 0.01ms, low: 9.97ms, high: 9.99ms)
    Download:   324.46 Mbps (data used: 448.5 MB)
                103.45 ms   (jitter: 34.71ms, low: 17.02ms, high: 350.77ms)
      Upload:    30.59 Mbps (data used: 30.3 MB)
                 27.33 ms   (jitter: 8.53ms, low: 10.49ms, high: 63.04ms)
 Packet Loss: Not available.
  Result URL: https://www.speedtest.net/result/c/d9138c04-a261-4a97-99f9-337aeaafbb5a

From my Linux box alone :

      ```
Server: NETGEAR Inc. - San Jose, CA (id: 43447)
         ISP: Sail Internet
Idle Latency:     9.94 ms   (jitter: 1.82ms, low: 9.94ms, high: 17.22ms)
    Download:   324.81 Mbps (data used: 579.1 MB)                                                   
                 99.00 ms   (jitter: 25.55ms, low: 19.52ms, high: 186.42ms)
      Upload:    40.70 Mbps (data used: 55.4 MB)                                                   
                238.00 ms   (jitter: 65.47ms, low: 14.98ms, high: 543.64ms)

In each of these tests, the server is different, and both WANs were used, which increased the throughput. I could see it in the traffic graphs.

Now, if I turn on sticky connections, here is what happens :

Server: NETGEAR Inc. - San Jose, CA (id: 43447)
         ISP: Sail Internet
Idle Latency:     9.94 ms   (jitter: 1.82ms, low: 9.94ms, high: 17.22ms)
    Download:   324.81 Mbps (data used: 579.1 MB)                                                   
                 99.00 ms   (jitter: 25.55ms, low: 19.52ms, high: 186.42ms)
      Upload:    40.70 Mbps (data used: 55.4 MB)                                                   
                238.00 ms   (jitter: 65.47ms, low: 14.98ms, high: 543.64ms)

In this case, the traffic graph showed everything went to Sail Internet, and nothing to Verizon.

Now, from my Windows box :

Server: Primcast - Oakland, CA (id: 62644)
         ISP: Verizon Wireless
Idle Latency:    32.38 ms   (jitter: 10.87ms, low: 26.48ms, high: 47.46ms)
    Download:    45.05 Mbps (data used: 57.4 MB)
                166.92 ms   (jitter: 52.56ms, low: 38.98ms, high: 422.89ms)
      Upload:    12.04 Mbps (data used: 18.1 MB)
                351.14 ms   (jitter: 73.21ms, low: 19.75ms, high: 1257.77ms)

Everything went to Verizon in this case.

Now, both simultaneously :

Server: eero - San Jose, CA (id: 41818)
         ISP: Sail Internet
Idle Latency:     9.95 ms   (jitter: 0.06ms, low: 9.91ms, high: 10.09ms)
    Download:    66.31 Mbps (data used: 39.9 MB)                                                   
                 25.81 ms   (jitter: 9.77ms, low: 12.38ms, high: 91.09ms)
      Upload:    32.49 Mbps (data used: 43.9 MB)                                                   
                249.48 ms   (jitter: 67.59ms, low: 77.49ms, high: 663.97ms)

      Server: Verizon - San Francisco, CA (id: 29623)
         ISP: Verizon Wireless
Idle Latency:    19.43 ms   (jitter: 9.60ms, low: 15.29ms, high: 29.71ms)
    Download:    51.55 Mbps (data used: 66.5 MB)
                190.94 ms   (jitter: 58.95ms, low: 13.09ms, high: 427.55ms)
      Upload:    11.74 Mbps (data used: 12.2 MB)
                284.39 ms   (jitter: 71.24ms, low: 15.91ms, high: 811.42ms)

Traffic graph shows some traffic went to both ISPs. This is different from the results I had before, when I ran 3 of these on different hosts against different servers, and all the traffic went to a single WAN. The combined bandwidth is much lower than without sticky connections, though. Perhaps the change is due to the weight of my Gateways. Sail is set to weight 3 and Verizon set to 1. Comcast was previously set to 12, and everything was going to Comcast.

[Gblenn]

@madbrain said in Netflix buffering with 3 WANs:

Each instance of speedtest is not going to the same server, actually. The sticky connections really do break the load balancing for me. I'm down to 2 ISPs now (cancelled Comcast yesterday).

Well, what you are showing is that each instance is in fact going to one and only one server, and that using sticky connections does exactly what is expected... and that it's still working.

What I mean with server is the target machine which speedtest uses in the test.
Here you are running one test from the Windows machine which goes to one server : WiLine Networks - San Francisco, CA (id: 17587)

Example from my Windows machine alone without sticky connections :

Server: WiLine Networks - San Francisco, CA (id: 17587)
         ISP: Verizon Wireless
Idle Latency:     9.99 ms   (jitter: 0.01ms, low: 9.97ms, high: 9.99ms)
    Download:   324.46 Mbps (data used: 448.5 MB)
                103.45 ms   (jitter: 34.71ms, low: 17.02ms, high: 350.77ms)
      Upload:    30.59 Mbps (data used: 30.3 MB)
                 27.33 ms   (jitter: 8.53ms, low: 10.49ms, high: 63.04ms)
 Packet Loss: Not available.
  Result URL: https://www.speedtest.net/result/c/d9138c04-a261-4a97-99f9-337aeaafbb5a

Your Linux box ended up using a different server : NETGEAR Inc. - San Jose, CA (id: 43447)

So two different sessions and different servers but only one server per session, agree?

Now, without sticky connections, the "streams" for each one of these tests, are going via both your ISP connections (WAN). Which proves to you that load balancing is working.

However great it is to see that you get better results in speedtest, the way this is working is not really the idea behind load balancing. Rather you are after better overall performance for the company as a whole, or household in your case. So even though the shared bandwidth is now the aggregate, you still expect each application or session to use only one connection at a time.
Splitting them up across two or more WAN's may even "break the connection"... Like you are experiencing with Netflix.

And to be honest, I don't know if a household could really benefit from load balancing all that much, unless you have quite low bandwidth per ISP. And you do a lot of downloading of application updates, game updates etc.
Now if you have two ISP's for the purpose of failover, you can still use them for loadbalancing of course, so there is absolutely no harm in doing it.

Anyway, when you turn on sticky connections, all traffic from a single session is expected to go out via one and only one of the WAN interfaces. Which you also tested and showed...

Now, if I turn on sticky connections, here is what happens :

Server: NETGEAR Inc. - San Jose, CA (id: 43447)
         ISP: Sail Internet
Idle Latency:     9.94 ms   (jitter: 1.82ms, low: 9.94ms, high: 17.22ms)
    Download:   324.81 Mbps (data used: 579.1 MB)                                                   
                 99.00 ms   (jitter: 25.55ms, low: 19.52ms, high: 186.42ms)
      Upload:    40.70 Mbps (data used: 55.4 MB)                                                   
                238.00 ms   (jitter: 65.47ms, low: 14.98ms, high: 543.64ms)

In this case, the traffic graph showed everything went to Sail Internet, and nothing to Verizon.

And this is the expected outcome, since the whole idea with sticky connections is to keep each application or session going out only one of the WAN's, to avoid problems.

When you test both of them simultaneously, you say that you still see the traffic going out both WAN's, meaning that load balancing is in fact still working.

Now, both simultaneously :

Server: eero - San Jose, CA (id: 41818)
         ISP: Sail Internet
Idle Latency:     9.95 ms   (jitter: 0.06ms, low: 9.91ms, high: 10.09ms)
    Download:    66.31 Mbps (data used: 39.9 MB)                                                   
                 25.81 ms   (jitter: 9.77ms, low: 12.38ms, high: 91.09ms)
      Upload:    32.49 Mbps (data used: 43.9 MB)                                                   
                249.48 ms   (jitter: 67.59ms, low: 77.49ms, high: 663.97ms)

      Server: Verizon - San Francisco, CA (id: 29623)
         ISP: Verizon Wireless
Idle Latency:    19.43 ms   (jitter: 9.60ms, low: 15.29ms, high: 29.71ms)
    Download:    51.55 Mbps (data used: 66.5 MB)
                190.94 ms   (jitter: 58.95ms, low: 13.09ms, high: 427.55ms)
      Upload:    11.74 Mbps (data used: 12.2 MB)
                284.39 ms   (jitter: 71.24ms, low: 15.91ms, high: 811.42ms)

Traffic graph shows some traffic went to both ISPs.

This might not happen every time though...

Without sticky connections, the large number of "streams" that are set up by speedtest end up being distributed (round-robin) based on the interface weights that you set.
With sticky connections turned on, they are all getting "stuck" together per application/session and all of the streams from each individual speedtest session go out only one interface. And running only two sessions simultaneously, you will now and then end up with both of them going out the same interface (3:1 it's Verizon).
Which sort of shows the reduced benefit of loadbalancing in a household scenario with only a few simultaneous users... Since it's only with a large number of parallell sessions that you make full use of both WAN connections.

Why the combined bandwidth is much lower when using sticky connections is a bit of a mystery though. And perhaps someone else can explain that fenomenon, if it is always the case...

[madbrain]

@Gblenn

Why the combined bandwidth is much lower when using sticky connections is a bit of a mystery though. And perhaps someone else can explain that fenomenon, if it is always the case...

Yes, it is strange. However, I did not run the 2 simultaneous speedtests with sticky connections disabled. I just did it now and here are the results :

Server: Sonic.net, Inc. - San Jose, CA (id: 17846)
         ISP: Sail Internet
Idle Latency:    14.92 ms   (jitter: 6.14ms, low: 9.38ms, high: 20.02ms)
    Download:   255.09 Mbps (data used: 344.9 MB)
                 18.34 ms   (jitter: 6.05ms, low: 9.15ms, high: 57.91ms)
      Upload:    30.02 Mbps (data used: 54.5 MB)
                199.27 ms   (jitter: 64.95ms, low: 23.46ms, high: 311.32ms)

      Server: Macarne LLC - Santa Clara, CA (id: 58605)
         ISP: Sail Internet
Idle Latency:     9.98 ms   (jitter: 0.07ms, low: 9.88ms, high: 10.08ms)
    Download:   152.55 Mbps (data used: 204.4 MB)                                                   
                117.95 ms   (jitter: 35.04ms, low: 13.50ms, high: 353.08ms)
      Upload:    26.62 Mbps (data used: 48.2 MB)                                                   
                198.95 ms   (jitter: 60.69ms, low: 66.69ms, high: 331.88ms)
 Packet Loss:     0.0%

For the first test with Sail internet, it seems to have hit the limits of available bandwidth for that ISP.

For the second test, it clearly used both ISPs, as Verizon maxes out at 110 Mbps down / 12 Mbps up.

Both ISPs have a lot of variability due to being wireless and thus sharing bandwidth with other customers.

I ran the 2 speedtests again, still without sticky connections, and got this traffic graph. I believe both ISPs were maxed out in this case.

Now, I turned sticky connections back on. Interestingly, I got a very similar result : both ISPs maxed out. Quite different from what I tested earlier yesterday. It looks like this is really unpredictable. Doing it in the middle of night as I am right now, might make it more stable - fewer customers using the bandwidth at the same time.

I ran it again and got another similar traffic graph - both ISPs maxed out. :

What's interesting is the results of individual speedtests.

Server: EGI Hosting - Santa Clara, CA (id: 32408)
         ISP: Sail Internet
Idle Latency:    10.57 ms   (jitter: 1.59ms, low: 9.91ms, high: 13.12ms)
    Download:   306.95 Mbps (data used: 498.5 MB)
                 43.92 ms   (jitter: 20.13ms, low: 12.18ms, high: 351.24ms)
      Upload:    39.24 Mbps (data used: 52.8 MB)
                131.21 ms   (jitter: 39.09ms, low: 9.05ms, high: 420.47ms)
 Packet Loss:     0.0%

      Server: WiLine Networks - San Francisco, CA (id: 17587)
         ISP: Verizon Wireless
Idle Latency:    15.34 ms   (jitter: 3.59ms, low: 14.51ms, high: 28.80ms)
    Download:   102.18 Mbps (data used: 154.3 MB)                                                   
                100.57 ms   (jitter: 35.16ms, low: 16.95ms, high: 486.20ms)
      Upload:    11.13 Mbps (data used: 15.2 MB)                                                   
                 32.04 ms   (jitter: 16.00ms, low: 12.92ms, high: 292.68ms)

The second speedtest bandwidth of 102 Mbps down / 11 Mbps up is pretty close to the max capabilities of Verizon.

The first test is pretty close to the max I can get from Sail, though that service has more variability.

I will leave it on for now.

I still would like to make sure both the fail-over and the load-balancing work..

Streaming not being interrupted is one of the key reasons for having 2 ISPs :) Getting rid of Comcast yesterday should help that, though.

Nevertheless, if sticky connections are on, and the ISP that these connections are going to goes down, what can I expect to happen to the streaming ? Disconnect, brief interruption, forced to manually resume ? I can of course try to disable an interface, or unplug one Ethernet cable, but I'm 100% sure this will be cause the same error condition as if one ISP link fails, but the 2 wireless modems stay connected to pfSense..

[Gblenn]

@madbrain said in Netflix buffering with 3 WANs:

@Gblenn

Why the combined bandwidth is much lower when using sticky connections is a bit of a mystery though. And perhaps someone else can explain that fenomenon, if it is always the case...

Yes, it is strange. However, I did not run the 2 simultaneous speedtests with sticky connections disabled. I just did it now and here are the results :

Now, I turned sticky connections back on. Interestingly, I got a very similar result : both ISPs maxed out. Quite different from what I tested earlier yesterday. It looks like this is really unpredictable. Doing it in the middle of night as I am right now, might make it more stable - fewer customers using the bandwidth at the same time.

I ran it again and got another similar traffic graph - both ISPs maxed out. :

![97068218-cb1b-48ae-8c40-604e6a19bb42-image.png](/assets/uploads/files/1715418041082-97068218-cb1b-48ae-8c40-604e6a19bb42-image.png) 

What's interesting is the results of individual speedtests.

Server: EGI Hosting - Santa Clara, CA (id: 32408)
ISP: Sail Internet
Idle Latency: 10.57 ms (jitter: 1.59ms, low: 9.91ms, high: 13.12ms)
Download: 306.95 Mbps (data used: 498.5 MB)
43.92 ms (jitter: 20.13ms, low: 12.18ms, high: 351.24ms)
Upload: 39.24 Mbps (data used: 52.8 MB)
131.21 ms (jitter: 39.09ms, low: 9.05ms, high: 420.47ms)
Packet Loss: 0.0%

  Server: WiLine Networks - San Francisco, CA (id: 17587)
     ISP: Verizon Wireless

Idle Latency: 15.34 ms (jitter: 3.59ms, low: 14.51ms, high: 28.80ms)
Download: 102.18 Mbps (data used: 154.3 MB)
100.57 ms (jitter: 35.16ms, low: 16.95ms, high: 486.20ms)
Upload: 11.13 Mbps (data used: 15.2 MB)
32.04 ms (jitter: 16.00ms, low: 12.92ms, high: 292.68ms)

The second speedtest bandwidth of 102 Mbps down / 11 Mbps up is pretty close to the max capabilities of Verizon.

The first test is pretty close to the max I can get from Sail, though that service has more variability.

I will leave it on for now.

I still would like to make sure both the fail-over and the load-balancing work..

Streaming not being interrupted is one of the key reasons for having 2 ISPs :) Getting rid of Comcast yesterday should help that, though.

Nevertheless, if sticky connections are on, and the ISP that these connections are going to goes down, what can I expect to happen to the streaming ? Disconnect, brief interruption, forced to manually resume ? I can of course try to disable an interface, or unplug one Ethernet cable, but I'm 100% sure this will be cause the same error condition as if one ISP link fails, but the 2 wireless modems stay connected to pfSense..

Ok, so then it seems everything is actually working as advertized and there isn't a problem with capacity or throughput when using sticky connections...

And I agree, wireless connections will render a bit more varying results than would a fiber or cable connections. So testing at different time of day will generate sligthly different results. But your new tests seem to show that you are actually getting the same results also when using sticky connections.

The difference between failover and loadbalancing is only in the Tiers you set on the interfaces. I have a metered wireless connection which I obviously don't want to use too much, since I would run out of "data". Therefore I have it as Tier 2 and my fiber connection as Tier 1. This means pfsense is routing all traffic via the fiber, unless that connection is deemed down.

In your case, pfsense is routing traffic from your clients/applications through both your ISP connections, set at the same Tier (1). The additional setting with "weight" is just to make sure the round-robin distribution between the two interfaces is done with the right balance, according to their respecitve bandwidth.

But as a user in your network will never know which one you will end up on, unless you create separate rules per user or application. And this is of course fine as long as the connections provide "equal performance", or performance good enough for everything you are doing.

By activating sticky connections you have now made sure that all streams/connections from any one single application, are routed through the same interface, so nothing risks breaking.

In the case of an actual failover, where one or the other ISP goes down, you will have some connections that happen to be on the failing interface and some that are already on the working interface. But whether or not you or anyone in your network will actually notice anything depends on the application they happen to use.

Real time applications like gaming or some form of communication, like Teams, Zoom etc will notice a brief (5-10 seconds) interruption, before reconnecting on the second interface. They will then stay on that interface until the session is over.

Other applications like web browsing, or anything that is buffering like YouTube, Netflix etc will most likely not notice anything at all. Since from the time the interface goes down, until pfsense decides it is decidedly down and it switches over, you will be watching the movie or clip from the buffer already on the PC or device.

[madbrain]

@Gblenn Thanks. I'm wondering where you got a metered wireless connection. I'm interested in one too. The one I have from Verizon is unlimited, but the price is a little too high for failover.

[Gblenn]

@madbrain Well, as I'm located in Sweden, I'm afraid my selection of carriers is not going to help you much...

But there are alternatives of course, like US Cellular who have a few metered options and should have good coverage in the Bay Area. There are others as well, like mymobilex.com which is a newcomer running off of Verizons network. And you could also shop around for prepaid subscriptions where there is often a metered alternative.

[madbrain]

@Gblenn thanks. My home is in a bit of a dead zone for cell signals. Not even voice calls get through, let alone data. I rely on WiFi calling. That's why I was so surprised the Verizon 5G service worked. They are using one of the UWB bands and that lone band seems to cover my house when literally nothing else does.
The other carriers signal is better in the rear part of my lot, uphill. But my office is down below in the front. And that's where the router is.
Will look into US cellular.

[Gblenn]

@madbrain If you are to rely on wireless, it's a good idea to look into wiring your house so that you can be more flexible wrt positioning of the FWA device.

On Android phones there are apps that let you see all the signals that the phone can handle and you could figure out the best spot for the device. Ideally you would try to use an outdoor unit and pull an ethernet cable directly from that to pfsense WAN port. Such a device could even be placed on a pole if needed and some of them can use PoE so you only need the Ethernet cable.

And if you haven't done that already, you could make sure to put AP's at strategic locations to improve wifi coverage.

[madbrain]

@Gblenn My house is unfortunately not wired for Ethernet, fiber, or anything else. I have had cabling contractors come and tell me it would likely be impossible to run cable it indoors without essentially destroying the house.
Running cabling outside would be very ugly as there are a lot of rooms and 2.5 levels. I'm not saying I will never do it but even outside would be quite pricey. We are talking hundreds of feet of cabling and conduit if not thousands.

The closest indoor location to a usable cell signal for T-Mobile is the upstairs master bathroom. Not sure about AT&T as it's been 14 years since It tried them. BTW US Cellular doesn't have coverage in San Jose. Verizon is the only one currently willing to sell a home Internet plan at my address, and it actually works. All 3 main operators (AT&T, T-Mobile, Verizon) will sell a cell phone plan with data, even though it essentially doesn't work. The coverage maps are simply inaccurate.

As far as Wifi, I have been using mesh systems for a while.I used to have Orbi. Now I have some Unifi APs, 6 of them. 2 or wired, the other 4 meshed. Phones gets a stable Wifi connection everywhere indoors, not as good (slower) outdoors. Unfortunately the Wifi calling is still dicey, even when the phone is in a direct line of sight and 10ft away from the Wifi AP. I don't know why the cell phone often still initiates calls on cell network, despite the priority being set to Wifi calling. I have to set it to airplane mode to really force it to make the call on Wifi. Otherwise on cell I will get long silent pauses, of 1 second or more every 10 seconds. Incoming calls are usually received on cell network also and not Wifi. If the phone is in airplane mode, I just miss the call - it goes straight to voicemail. Which is not that different than when it's not in airplane. Many times calls go straight to voicemail too.

What does work very reliably is my wired VoIP with voip.ms and a Grandstream HT802 ATA, connected to Panasonic DECT 6.0 wireless base with a total of 6 handsets. Those perfectly cover my entire property for voice, inside and out, unlike either the cell or Wifi signal, which are both not reliable for making or receiving calls. I should probably just forward the calls from my cell phone to that VoIP line when I'm home. It's a bit of a pain, though. My MVNO can do that with an HTTPS request. Maybe an app could get the GPS location and initiate that request depending on whether I'm home or not ... And turn airplane mode on/off while we're at it, though, though that would probably require rooting the phone.

[Gblenn]

@madbrain I hear you, it's not without challenges to get your house "up to date" with new tech. I went through the process of cabling most of our house about 20 years ago. Fortunately I used Cat6 cable which now allows me to get 10Gig in most of the outlets around the house, which is quite an upgrade compared to when it was built in 1928...

But I get the pont the contractor was making although you don't have to wire every single room though. And if you start looking, you may find ways to actually make it work...

The challenge I found was moving from one end to the other without visible cabling, as well as between floors.

So do you have an attic, or at least some such space which you have access to? Or perhaps there is a basement, or both even?
They are typically good places to pull cables from one end to the other without anything being visible in the rooms. And from both such areas, you can likely get to any of the rooms on the adjacent level (up/down).

I used built in closets in some rooms to get a cable down from the attic into the rooms on the floor. And the same can be done going up from the basement perhaps. Then you might even be so lucky that you find closets in the "same location" in two rooms on top of each other. A perfect place to cross from bottom to the top...

Otherwise I sort of assume there is a vent drain for the plumbing in the house? Which then should go all the way from the foundation to the roof. It might be built into a small "duct" of sorts, which could be a good place to drop cables all the way through from attic to basement/bottom... Not the pipe itself but the space it uses...

Anyway, if you start looking around, you might find a way to get at least one or two cables to where you want them. Even if you rely on wifi for most of the house, at least then you can have one or two AP's that are wired, and of course the FWA device in the right location can make a big difference.

[madbrain]

@Gblenn I have looked, and there really is no way to run the wiring inside. There is no basement (not common in California), no attic, and the house is built on slab, meaning no crawl space either. And it's not just one cabling contractor telling me wiring cannot be added inside. I would have done it a long time ago if it was practical. Not sure what you mean by a vent drain. There are vents in the garage and in a utility room for the 2 gas furnaces and 2 gas water heaters. I'm not aware of any other vent.
I do have a 10Gbe LAN as well between two rooms - my office and home theater. CAT6A might be required for the distances involved in my home. It would help get faster speeds for the Wifi APs to wire them, instead of meshing them wirelessly. But it still wouldn't solve Wifi calling issues. Those seem to be software/firmware related problems between the Samsung phone and/or the Ubiquiti APs. And they happen even with a wired AP in my office, not mesh. There is actually another option which is to use Bluetooth between the phone and Panasonic DECT 6.0 phone base. Bluetooth has a very limited range, unfortunately, so that implies leaving the smartphone near the base. Not practical.

[Gblenn]

@madbrain Well, with no attic or any other type of crawl space, I suppose you don't have many options. I guess one could dig a "trench" close to the wall to hide a cable that goes around the house, unless there are concrete patios or similar, blocking that option... Another possibility would be to use the gutters to hide cables behind them...

Perhaps vent drain is not the correct translation, but what I meant was the vent for your plumbing. When flushing for example, air needs to come in from somewhere. But with no attic, that is a moot point anyway...