LinkedIn Cut Data Job Failures By 97 Percent
TL;DR: LinkedIn dramatically improved its big data systems by adopting Apache Celeborn. The change slashed daily processing failures from 750 to 20, cut compute costs by 10%, and made applications run 14% faster.
Key facts
- Category
- Infrastructure
- Impact
- High
- Published
- Source
- InfoQ
Full summary
LinkedIn cut daily data processing failures by 97% while also lowering compute costs and speeding up jobs with a new open-source tool.
LinkedIn has successfully tackled a major challenge in its large-scale data processing by implementing Apache Celeborn. According to a report from InfoQ, this move has yielded impressive results across the company's Spark and Flink workloads, which are used for massive data analysis. The engineering team reported a dramatic 97% reduction in daily "shuffle" failures, with the number of incidents dropping from around 750 to just 20. Beyond stability, the deployment also delivered significant efficiency gains, including a 10% reduction in overall compute costs and a 14% improvement in application runtimes. This case study highlights a powerful solution to a common and costly problem in the world of big data.
To understand the impact, it's important to know what a "shuffle" is. In distributed data processing systems like Spark, data is spread across many computers, or nodes. When performing complex operations like sorting or grouping, data needs to be exchanged and reorganized between these nodes. This process, known as the shuffle, is one of the most resource-intensive and failure-prone stages. If a single node becomes slow or fails during a shuffle, the entire job can be delayed or crash. Apache Celeborn solves this by creating a separate, dedicated service just for managing the shuffle process. Instead of each compute node handling its own shuffle data, it sends it to Celeborn. This approach, called a remote shuffle service, decouples the shuffle from the computation, making the system more resilient. Celeborn uses dedicated storage and data replication to ensure that even if some hardware fails, the data is safe and the job can continue.
LinkedIn's success is part of a broader industry trend toward disaggregated architectures in data infrastructure. For years, the standard model was to have compute and storage tightly coupled on the same machines. While simple to start with, this model creates bottlenecks and inefficiencies at scale. The move to separate, specialized services for different functions—like compute, storage, and now shuffling—allows teams to optimize each component independently. This leads to better performance, higher reliability, and more efficient resource utilization. By open-sourcing Celeborn through the Apache Software Foundation, the technology is not just a proprietary solution for one tech giant. It represents a community-driven effort to build a more robust foundation for the next generation of data processing, making these advanced capabilities accessible to any organization running Spark or Flink.
For CTOs, infrastructure managers, and data engineers, this story provides a clear, actionable insight. If your organization struggles with the reliability, cost, or speed of its big data pipelines, the shuffle phase is a prime suspect. The metrics from LinkedIn—a 97% drop in failures and double-digit improvements in cost and speed—offer a compelling business case for evaluating a remote shuffle service. Adopting a tool like Apache Celeborn is not a minor tweak; it's a significant architectural change that can fundamentally improve the stability and economics of your data platform. As more companies face growing data volumes and demand for faster analytics, we can expect to see wider adoption of Celeborn and similar disaggregated shuffle services, turning them from a niche optimization into a standard best practice for large-scale data engineering.
Why it matters
For engineers, this shows a concrete solution to the notoriously fragile and expensive shuffle phase in distributed data systems. LinkedIn's success with Apache Celeborn provides a strong open-source option for improving the reliability and performance of Spark and Flink workloads at scale.
Business impact
This demonstrates a direct path to reducing big data infrastructure costs by over 10% while making critical data pipelines 14% faster and significantly more reliable. These improvements translate directly to lower operational expenses and faster time-to-insight for any data-driven company.
Related on Notifire
Related stories
Primary source: InfoQ
