Skip to content

Add DuckDB platform support with Parquet source and cost model - #794

Open
JunWang222 wants to merge 7 commits into
apache:mainfrom
JunWang222:feature/duckdb-platform
Open

Add DuckDB platform support with Parquet source and cost model#794
JunWang222 wants to merge 7 commits into
apache:mainfrom
JunWang222:feature/duckdb-platform

Conversation

@JunWang222

Copy link
Copy Markdown
Contributor

This PR adds DuckDB as a new Wayang platform, following the existing JDBC-based platform structure used by Trino and Presto.

Main Changes

  • Add wayang-duckdb platform module
  • Add DuckDB platform, plugin, mappings, and execution operators
  • Add DuckDB JDBC configuration and default properties
  • Add DuckDB operator integration tests aligned with the Trino/Presto operator tests
  • Add DuckDB Parquet source support
  • Add local Parquet test coverage for DuckDB
  • Add DuckDB cost profiling utilities and update DuckDB cost parameters
  • Add README/setup documentation for DuckDB usage, testing, Parquet, and profiling

…form

# Conflicts:
#	wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/execution/JdbcExecutor.java
…form

# Conflicts:
#	wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/execution/JdbcExecutor.java
@juripetersen

Copy link
Copy Markdown
Contributor

Hi @JunWang222,
As mentioned in Issue #796, I think we need to address the setup and platform specific files like docker-compose.yml in this PR as well.

Can you take a look at the issue and the problems I pointed out there and also see where this PR can be adapted to fit my description.

Thank you!

@JunWang222

Copy link
Copy Markdown
Contributor Author

Hi @JunWang222, As mentioned in Issue #796, I think we need to address the setup and platform specific files like docker-compose.yml in this PR as well.

Can you take a look at the issue and the problems I pointed out there and also see where this PR can be adapted to fit my description.

Thank you!

Thanks for clarifying this in #796. I agree that having demos and setup projects in several different locations makes the conventions harder to follow.
For this PR, I propose moving the DuckDB demo into wayang-applications, removing the standalone Docker-based setup project, and updating the documentation to run the example directly through JDBC with a configurable database path. DuckDB is embedded, so Docker is not necessary for this workflow. Platform integration tests would remain in the DuckDB module.
I’ll also review the profiling scripts and configuration currently under the setup directory and move the relevant pieces into the appropriate existing module.
For the existing Presto, Trino, and BigQuery setup files and OS-specific scripts mentioned in #796, I suggest handling that cleanup in a separate PR so this one stays focused on DuckDB. Would that scope work for you?

…duckdb-platform

# Conflicts:
#	wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/execution/JdbcExecutor.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants