One post tagged with "duckdb"

This blog post summarizes my Google Summer of Code 2026 project for Apache Wayang. The original goal, "Make Wayang More Datalake-Friendly", was to integrate modern analytical SQL engines and Parquet-based data sources into Wayang's optimization and execution framework.

Apache Wayang lets users describe a data-processing pipeline once and then either choose an execution platform or let Wayang's optimizer decide where the pipeline should run. This matters for modern analytical workloads because data is often no longer stored inside a single database. It may live as Parquet files in object storage, in federated catalogs queried by Trino or Presto, in a serverless warehouse such as BigQuery, or in local analytical files processed by DuckDB. The project focused on making these engines visible to Wayang as execution platforms rather than treating them as separate external connectors.