DuckDB engine materializes final results with fetchdf(), defeating out-of-core execution

[JohnnyWilson16]

Why it matters

The DuckDB backend is presented as the spill-to-disk out-of-core execution path, but every successful run materializes the final relation as a pandas DataFrame. A cleaned result larger than available RAM can therefore still fail at the final step, even when DuckDB successfully processed the input under its configured memory limit.

Steps to reproduce

1. Create a Parquet dataset whose cleaned result is larger than available RAM, or run with a deliberately small EngineConfig.memory_limit_gb.
2. Run a native-only clean, for example:

from freshdata import clean
from freshdata.config import CleanConfig
from freshdata.execution import EngineConfig

clean(
    "large.parquet",
    config=CleanConfig(strategy="conservative", fix_dtypes=False),
    engine="duckdb",
    engine_config=EngineConfig(engine="duckdb", memory_limit_gb=0.5),
)

3. Inspect DuckDBEngine._run_sql_pipeline.

Expected behavior

The DuckDB execution path should either stream the final result to a requested sink or clearly enforce and document that the returned result must fit in memory.

Actual behavior

_run_sql_pipeline ends with conn.execute(cur).fetchdf(). fetchdf() materializes the entire relation into pandas before output_format conversion, so the backend cannot return a result larger than RAM.

Likely root cause

The current backend API requires a materialized frame before run_with_engine can convert it. The DuckDB connection is also closed immediately after execution, so returning a relation is not currently possible.

Suggested first investigation path

Split SQL-plan execution from result delivery. Add an explicit output-path or sink option that uses DuckDB COPY (query) TO ... for Parquet/Arrow output, or document and enforce an in-memory result budget for frame-returning calls. Add a regression test that verifies a large DuckDB result can be written without fetchdf().