Measurement, benchmarking, and data analysis are underrated
A question I get asked with some frequency is: why bother measuring X, why not build something instead? More bluntly, in a recent conversation with a newsletter author, his comment on some future measurement projects I wanted to do (in the same vein as other projects like keyboard vs. mouse , keyboard , terminal and end-to-end latency measurements), delivered with a smug look and a bit contempt in the tone, was "so you just want to get to the top of Hacker News?" The implication for the former is that measuring is less valuable than building and for the latter that measuring isn't valuable at all (perhaps other than for fame), but I don't see measuring as lesser let alone worthless. If anything, because measurement is, like writing , not generally valued, it's much easier to find high ROI measurement projects than high ROI building projects. Let's start by looking at a few examples of high impact measurement projects. My go-to example for this is Kyle Kingsbury's work with Jepsen . Before Jepsen, a handful of huge companies (the now $1T+ companies that people are calling "hyperscalers") had decently tested distributed systems. They mostly didn't talk about testing methods in a way that really caused the knowledge to spread to the broader industry. Outside of those companies, most distributed systems were, by my standards , not particularly well tested. At the time, a common pattern in online discussions of distributed correctness was: Person A : Database X corrupted my data.
Measurement, benchmarking, and data analysis are often underrated in the tech industry, with many people dismissing them as less valuable than building new features or products. This perspective is not only misguided but also short-sighted. Measurement projects can have a significant impact, not just for the companies that implement them but also for the broader industry. In this article, we'll explore why measurement is crucial and examine some high-impact examples of measurement projects that have shaped the tech landscape.
One of the most compelling arguments against measurement is the assumption that it's a less valuable endeavor compared to building something new. This mindset often stems from a belief that measurement is trivial or even a waste of time. However, this view overlooks the profound value that measurement can bring. For instance, measurement allows companies to identify bottlenecks, optimize performance, and ensure the reliability of their systems. It also enables them to make informed decisions about where to allocate resources and prioritize improvements.
A prime example of the importance of measurement is Kyle Kingsbury's work with Jepsen. Before Jepsen, large tech companies, often referred to as "hyperscalers," had developed distributed systems that were reasonably well-tested. However, they rarely shared the details of their testing methods, which limited the spread of knowledge within the industry. Outside of these companies, most distributed systems were not thoroughly tested, leading to a lack of trust and reliability.
At the time, discussions about distributed correctness often highlighted the problem. A common exchange between developers went like this:
Person A: Database X corrupted my data.
Person B: It works for me. It's never corrupted my data.
A: How do you know? Do you ever check for data corruption?
B: What do you mean? I'd know if we had data corruption (Alternate answer: sure, we sometimes have data corruption, but it's probably a hardware problem and therefore not our fault).
This exchange underscored the critical need for rigorous testing and measurement in distributed systems. Kingsbury's early work with Jepsen found significant flaws in nearly everything he tested, despite the tool being less sophisticated than it is today. For example, he uncovered issues in Redis Cluster and Redis Sentinel, which had not been adequately addressed by the industry.
Jepsen's impact extended beyond the companies that used it. By providing a standardized framework for testing distributed systems, it encouraged other organizations to improve their testing practices. This, in turn, led to a more robust and reliable ecosystem for distributed systems.
Another notable example of the value of measurement is the keyboard vs. mouse debate. While this may seem trivial, it had a profound impact on user experience and productivity. Measurement projects like these often serve as a catalyst for innovation, driving companies to reevaluate their approaches and invest in new technologies.
In the realm of end-to-end latency measurements, companies like Google and Facebook have used measurement to optimize their systems and improve user experience. By measuring and analyzing latency, these companies have identified areas for improvement and implemented changes that significantly enhance performance.
Measurement is also essential for benchmarking. By setting clear benchmarks, companies can compare their performance against industry standards and identify areas where they can excel. This not only drives internal improvements but also fosters a culture of continuous innovation.
Despite the undeniable value of measurement, it is often overlooked or dismissed as less important than building new features. This is a mistake. Measurement is foundational to understanding system performance, identifying bottlenecks, and making data-driven decisions. It is a crucial component of the tech industry's evolution and should be given the attention and respect it deserves.
In conclusion, measurement, benchmarking, and data analysis are underrated but essential components of the tech industry. They provide valuable insights into system performance, enable companies to make informed decisions, and drive innovation. By recognizing the importance of measurement, organizations can unlock new opportunities for growth and improvement. The examples of Jepsen, the keyboard vs. mouse debate, and end-to-end latency measurements demonstrate the significant impact that measurement can have. As the industry continues to evolve, it is crucial to embrace measurement as a vital tool for progress and success.
