Opinion: Ookla's Network Performance Data Is Far From Perfect


Ookla, the company behind Speedtest.net, has been sourced more and more lately for data that different companies are using to tout their networks being better, or more recently, used to show that their modems are better than the competition. However, Ookla's data shouldn't be the "golden standard" when trying to showcase how much better your network or modem is when compared to the competition. And it should be taken with a grain of salt, much like these other emerging reports based on it. Let's explain why.

Crowdsourced Data Is More Accurate

Ookla's data is actually crowdsourced. Each time you do a speed test on its app or website, that data goes into the reports that Ookla puts out. Which ideally is better than what RootMetrics uses for its reports – they have people driving around the country testing all four networks on the same smartphone – especially for some of those smaller places that are not near large metro areas like Houston, Dallas, Los Angeles, Denver and New York City. The reason for this is because people do speed tests at random times, and not all at the same time. So it gives a more accurate representation of its network, versus RootMetrics or even Nielsen. This is important seeing as network speeds will differ from minute to minute, as there are fewer or more people connected to any particular tower.


Everyone loves crowdsourced data because it delivers more data points and also shows the real-life experience on these wireless networks. Instead of getting speeds from tests that were ran only once. Instead, you get many more variables here, which is also why you'll see bigger differences between Ookla, RootMetric, and Nielsen's reports when those come out quarterly.

However, Crowdsourced Data Can Be Skewed

On the surface, this sounds great, and like it would be the most accurate way to see which network is the best. But that's not entirely true. This is because these results can be skewed. About 39-percent of American's live on the east coast. So if everyone in the US does a speed test, the speeds on the east coast will hold more weight than those in the middle of the US or even on the west coast. Which means that it's not quite accurate. That's not really Ookla's fault, it's just the way things are here but it's still something to keep in mind when going through its data. Especially when looking at the average speeds for the entire US, when it comes down to a state-by-state or city-by-city report, it's going to be a bit more accurate.


A good example of this is the announcement that Qualcomm just put out. Showcasing the fact that smartphones running its Snapdragon 845 chipset with its latest modem, are around 50-percent faster than those running on Intel's newer modems (these are found almost exclusively in iPhones). However, Qualcomm couldn't say if the tests were split 50-50, or if these were all done in the same area, on the same network. There are a lot of variables at play in that scenario. As a test on T-Mobile might be 50Mbps, but five miles down the road, it might be 75Mbps or even 10Mbps. Not to mention the time of day when these tests were done. Obviously, a test between 7 AM and 8 PM are going to be slower than those done the rest of the day, seeing as there are fewer people on the network at that time. That's why it's important to remain somewhat skeptical about such estimates.

It's Not Ookla's Fault

Ookla's data is still good – it's not faked or anything of that sort. It's just the fact that there are so many different variables at play here, and that's not Ookla's fault. There are many different modems out there in smartphones, as well as different wireless networks and wireless networks don't have the same speed everywhere in the country it serves. So there's not much that Ookla can do here, to make its data more representative. This is why many prefer the way RootMetrics does testing of wireless networks, since it uses the same phone on each network, and drives around the country doing speed tests all day long (for weeks). While some carriers will complain about RootMetrics because the phones being used don't sport the latest technology to really take advantage of the network, it does work better to get more representative numbers for data.



The skinny here is that Ookla's data is pretty good, but like all reports, it's hard to get things as accurate as possible. So, when you see headlines from companies saying that their network is the fastest and sourcing Ookla's data, such a claim has to be taken with a grain of salt. While its numbers are likely pretty similar to real-life speeds and performance, when you look at a country as a whole, the figures may not be as representative as they should be. This data is kind of like benchmarks being run on a smartphone. These should be taken with a lot of reservations, as they won't completely translate to real-life experience on the network. Which is the same thing with benchmarks like AnTuTu. Where your phone might score over 200,000, but that doesn't mean it's going to remain speedy in the long term. So, the bottom line here is that data sourced from Ookla is relatively accurate but far from perfect and there's room for a better solution to come further down the road.