Pinterest Tames Its Massive, Never-Ending Data Streams

Pinterest built a new system to know when its massive, constantly changing data sets are finally ready and complete for processing.

Pinterest is tackling a problem familiar to any company working with massive, continuously flowing data: knowing when a set of data is actually “done” and ready to be used. According to a report from InfoQ, the company developed a new system for its database ingestion framework to provide a clear, reliable signal of data completeness. The solution is designed to manage petabyte-scale data streams, particularly from Change Data Capture (CDC) systems, which track every modification in a database. At its core, the system uses two key open-source technologies, Apache Flink for processing the streams and Apache Iceberg for managing the data's metadata. This combination allows Pinterest to create what it calls an “event time watermark,” a timestamp that effectively tells all downstream applications that the data up to that point is complete and safe to process for analytics, machine learning, or other business tasks.

This system works by carefully tracking the time at which events actually occurred, not just when they were processed. In complex data systems, events can arrive out of order or with significant delays. Pinterest’s framework uses Apache Flink, a powerful stream-processing engine, to monitor the timestamps of incoming data events. Flink calculates a “watermark,” which is a timestamp that advances as the system becomes confident that it has seen all events prior to that time. The crucial innovation is how this watermark is published. Instead of just being an internal signal within Flink, it is written to the metadata of an Apache Iceberg table. Iceberg is a table format designed for huge analytic datasets, and its robust metadata capabilities ensure that this watermark is stored durably and consistently. This creates a reliable, non-regressing signal—meaning the watermark only ever moves forward—that any other system can check to determine if a specific partition of data is ready, solving a major coordination problem in distributed data architecture.

This development fits into a broader industry trend of moving away from traditional, slow batch processing toward more responsive, real-time data systems. As companies increasingly rely on fresh data for operational decisions and product features, the challenges of streaming data have become more acute. The problem of determining data completeness, often called the “closed partition problem,” is a classic headache for data engineers. Processing data too early leads to incorrect results and costly re-calculations, while waiting too long introduces unacceptable delays. Major tech companies have been building solutions to this for years, with projects like Apache Hudi and Delta Lake offering similar capabilities. Pinterest’s approach is a significant endorsement of using Flink and Iceberg together as a modern, open-source stack for solving this fundamental challenge at an extreme scale, providing a valuable reference for other organizations facing similar issues.

The practical takeaway for CTOs and data teams is that Pinterest has validated a powerful pattern for building more reliable and timely data platforms. This case study demonstrates that the combination of a stateful stream processor like Flink and a modern data table format like Iceberg can provide the foundation for trustworthy, large-scale analytics. For organizations struggling with the unreliability of ad-hoc timing mechanisms or complex inter-service dependencies, this watermark-based approach offers a cleaner, more robust alternative. As more companies adopt real-time data ingestion, we can expect to see this pattern of using durable, metadata-driven signals for process coordination become more widespread. The focus will now shift to how the broader community adopts and potentially standardizes these techniques, making it easier for everyone to build dependable systems on top of ever-flowing streams of data.

For data engineers working with streaming data, determining partition completeness is a critical challenge. Pinterest's watermark-based approach offers a robust pattern using Flink and Iceberg that prevents incorrect downstream processing due to incomplete data, improving reliability for petabyte-scale analytics and machine learning pipelines.

This system allows companies to make faster, more reliable decisions based on fresh data. By ensuring data completeness automatically, it reduces the risk of flawed analytics, improves machine learning model accuracy, and lowers the operational cost of managing massive, real-time data pipelines.