For content streaming services, Netflix is a reality that’s impossible to ignore. It’s the world’s largest streaming media provider, with 158 million subscribers in 2019. It’s almost always present at customer sites either because it is bundled by the provider itself or because many users -- 46% of US households, according to Investors’ Business Daily -- subscribe to more than a single streaming service. But Netflix doesn’t play well with others.
We’ve talked to numerous OTT providers, who complain mightily that Netflix takes over the network bandwidth and chokes out other content types. You may have experienced this yourself in your home network: when one or more people are watching Netflix, other Internet activities tend to go more slowly.
Why is Netflix killing my streaming media?
The source of the problem is Netflix’s use of BBR, which we already discussed in detail in our previous post. BBR was developed by Google, and while it’s effective at getting its own streams where they need to go effectively, it’s known to be very aggressive towards standard TCP traffic.
It’s not something that’s remaining under the radar any more. Recently, the UK’s The Telegraph newspaper ran an article reporting that “a Google algorithm can annex almost half of an internet network’s capacity, even when there are multiple online services vying for connections.”
How Netflix claims more than its fair share of bandwidth
It’s difficult to predict exactly what happens when BBR-based streaming media encounters other traffic on the network, but research carried out by academics at Carnegie Mellon University (on which the Telegraph article is based) measured the impact of BBR on other streaming media. In the words of the researchers, “as one BBR flow competes with ever increasing number of Cubic flows, BBR’s fraction of the bandwidth remains the same.”
Their research revealed that when one connection using BBR competes with 16 connections using TCP Cubic, the TCP Cubic streams end up sharing slightly over 50% of the bandwidth, while the BBR stream takes over around 40%, even though it’s joining other connections!
BBR’s “probing” phase, when it tries to ascertain the narrowest part of the network, causes additional problems in shallow-buffered networks. The researchers discovered that this “causes buffer overflows and bursty loss for competing flows; these bursts can lead to [TCP] Cubic and Reno, nearly starving for bandwidth.”
Dramatic Network Ups & Downs Lead to Poor Quality of Experience
Our own labs confirmed these results in a series of experiments. Let’s take a look: initially, if there’s only one BBR-based connection and one TCP-based connection alongside each other, there is a very good chance that BBR will take over the major part of the available bandwidth. For example, in Figure 1 below, we looked at how one BBR connection and one TCP cubic connection share a 40 Mbps link. BBR on average takes about 30 Mbps out of this link and the Cubic is left with approximately 10 Mbps. Also, as you can see, the throughput is very jumpy, with the TCP Cubic link going down to nearly zero when the BBR link is at its peak.
Figure 1: How one TCP Cubic stream and one BBR stream “share” a 40 Mbps link
The situation worsens when one BBR connection shares a link with several TCP Cubic connections. Intuitively one might expect five separate streams to somewhat equally share the allocated bandwidth. However the BBR connection still grabs the vast majority of the available bandwidth, leaving the four TCP Cubic streams to share the crumbs.
In Figure 2 below, we show how one BBR flow and four TCP Cubic flows share the same 40 Mbps link. Observing the two BBR experiments we can also see that every few seconds BBR reduces its sending rate dramatically (this is when BBR tries to measure the minimum round-trip time).
Figure 2: How BBR and 4 TCP Cubic streams interact on a 40 Mbps link
These dramatic ups and downs in bandwidth allocation are terrible for video streaming, leading to instability in picture quality and even rebuffering. The reason for this is that the ABR streaming client measures the performance on the network, and then requests the next segment in the appropriate resolution for that level of performance. So if the network is performing slowly, it will request a low resolution segment; conversely, if there is a bandwidth peak, it will request a high resolution segment. And as you can see from the obvious up/down of the graph, there will almost certainly be a mismatch between when the high resolution segment arrives and the actual bandwidth allocated to it, so you end up with segments getting stuck in network congestion, leading to poor quality of experience.
Can streaming services resolve the problem?
Fortunately, other content streaming services and OTT providers don’t have to give up and leave it to Netflix to rule the bandwidth. There are a number of ways to approach this issue.
1. If you can’t beat them, join them
Since BBR unfairly hogs so much of the bandwidth, perhaps other media streaming services should abandon good-old TCP and adopt Google’s BBR protocol. BBR v1 is even part of the Linux distribution now.
It sounds like an easy solution, but it’s not that simple. Primarily, this is because many deployed Linux systems still use old kernels, so they can’t always take advantage of BBR. Additionally, adopting BBR won’t necessarily bring media streaming services their fair share of bandwidth, because of BBR v1’s known fairness issues. The same third-party research mentioned above also found that many BBR flows competing against each other still won’t share their link capacity equally.
2. Wait it out for BBR v2
Google announced that they are working on BBR v2 in order to resolve these issues. v2 promises to be less aggressive towards TCP. So you could sit back and wait for BBR v2 to be released and hope that Netflix chooses to adopt it, and then your problems will be over.
However, this passive approach is very risky. There’s no guarantee that v2 will solve all the bandwidth grabbing tactics of v1, and no further guarantee that Netflix and any other media-streaming services that use BBR will decide to use the new version.
3. Use PCC to stand up to BBR
In an earlier blog post, we discussed in depth the benefits of Compira Labs’ PCC protocol for streaming media services. Compira Labs makes PCC available to older Linux releases, so it’s equally open to all OTT providers.
PCC can play well with TCP Cubic, sharing available network bandwidth nearly equitably; see, for example, Figure 3, that shows one PCC flow sharing the same 40 Mbps link with four TCP cubic flows. We can see that the bandwidth allocation is much more equitable, and the bandwidth partition on the link is much more stable than in the BBR case. Increased stability leads to better quality of experience for every stream.
Figure 3: 1 PCC stream shares a 40 Mbps link with 4 TCP Cubic streams
However the focus of this post is how to avoid getting “killed” by Netflix, so let’s see how PCC manages one-on-one with BBR. As we can see in Figure 4, PCC is far more resilient than TCP Cubic to the dominance of BBR. The fairness police might say that PCC is actually taking too large a bite out of the available bandwidth when competing with BBR, however one of the unique advantages of the PCC framework is that it is configurable, so the graph in Figure 4 demonstrates one aggressive configuration of PCC. Depending on the network conditions, the number and types of streams, PCC can be tuned to behave in the most optimized way for the desired QoE goals. We will discuss this configurability in a later post.
Figure 4: 1 PCC Stream and 1 BBR stream share a 40 Mbps link
The light at the end of the tunnel
While Netflix still poses a serious threat to Quality of Experience for other streaming media content due to its use of BBR, there’s a light at the end of the tunnel. There’s no need to try to shift your media streaming to BBR or to wait for BBR v2 to resolve these challenges. Compira Labs’ PCC can stand up to BBR to make bandwidth sharing more equitable for all media streaming content services.