17 verified briefings on Data Engineering. Each story includes a plain-English summary, why it matters, and the concrete action engineering teams should take.
DuckDB now lets developers query datasets on the Hugging Face Hub directly using SQL. This integration removes the need to download large files, simplifying and speeding up data exploration for machine learning projects.
ClickHouse has released replica-aware routing in public beta. The new feature ensures that all queries within a single session are sent to the same server, solving a common problem with temporary data in distributed clusters.
Amazon Web Services (AWS) has acquired DuckLabs, the company behind the popular open-source database DuckDB. Despite the acquisition, DuckDB's projects will remain free and open-source under the MIT license, managed by the independent DuckDB Foundation.
When real-time AI fails in production, the data pipeline is often the real culprit, not the model. This means teams are often looking in the wrong place to fix issues like latency and degrading accuracy.
Google Cloud's serverless Apache Spark service now uses AI to automatically diagnose and suggest fixes for failed data jobs. This helps teams spend less time debugging infrastructure and more time building data pipelines.
DuckDB, a popular analytics database, now fetches data from the cloud asynchronously. This architectural change significantly speeds up queries on large datasets by keeping the processor busy instead of waiting for data.
A developer treated an espresso machine like a distributed system, using OpenTelemetry and ClickHouse to analyze every shot. The project is a creative case study on applying observability principles to real-world, unconventional systems.
AI coding tools quickly generate code from simple prompts, a practice called "vibe coding." While this speeds up development, it creates code that is difficult to understand, maintain, or fix months down the line.
Managing numerous schemas in data pipelines like Kafka and Flink can become complex and costly. An InfoQ article explores this "schema proliferation" and suggests a consolidation strategy to simplify queries, reduce maintenance, and make systems more scalable and resilient.
ClickHouse has significantly improved its data join performance, achieving a 26x speed increase on a standard benchmark. This was the result of a two-year engineering effort focused on parallel hash joins, runtime filters, lazy column replication, and smarter query planning for complex analytical queries.
ClickHouse Cloud has launched executable user-defined functions (UDFs) in public beta. This feature allows developers and data engineers to write functions in Python, upload them to a cluster, and call them directly from SQL queries, simplifying complex data processing and machine learning workflows.
Java Champion Gunnar Morling shares insights on building high-performance Java applications for data engineering. He discusses experiments with durable execution engines and the development of Apache Hardwood, a new, minimal-dependency Java parser for the Apache Parquet file format, offering lessons for developers and engineering leaders.
ClickHouse announced several major updates at its Open House 2026 event. Key developments include deeper integration with Postgres, new data ingestion tools called ClickPipes and ClickHouse Agents, and a partnership with Langfuse for LLM observability. The updates aim to simplify real-time data analytics.
DuckDB, the popular in-process analytical database, has introduced a new remote protocol called Quack. It uses HTTP to enable a client-server model, allowing multiple users and applications to connect to and query the same database instance over a network, a significant shift from its embedded-only origins.
Meta's engineering team successfully migrated its petabyte-scale data ingestion platform, which processes social graph data from MySQL. The team used advanced techniques like reverse shadowing and continuous checksum monitoring to ensure a seamless transition with zero downtime, improving both reliability and operational efficiency.
dbt Labs has launched dbt Agent Skills, a new feature in dbt Cloud. It allows developers to package data logic into reusable "skills" for AI agents. This helps agents answer data-related questions more reliably and accurately by using pre-defined logic instead of generating SQL from scratch.
NASDAQ handles up to a trillion messages daily across its 26 business lines. To manage this massive scale, the company built a governed intelligence layer using dbt and Databricks. This modern data stack enables them to ensure data quality, security, and accessibility for decision-making.