Skip to content

Raincloud oracle: Parquet duplicate column names cause header mismatch vs Vortex [n] suffix #256

Description

@dfa1

Problem

Two Raincloud corpus slugs fail with a header mismatch at line 1:

uci-parkinsons — two columns both named mdvp_jitter in Parquet; Vortex stores them as mdvp_jitter and mdvp_jitter [1].
uci-spambase — six columns all named char_freq in Parquet; Vortex stores them as char_freq, char_freq [1], …, char_freq [5].

The oracle uses cols.stream().map(ColumnSchema::name) for the header — emitting the raw Parquet duplicate names — while Vortex/CsvExporter emits the de-duplicated names. The comparison fails at line 1.

Data rows are also at risk: rows.getInt(name) is name-based, so for duplicate column names hardwood may return the same column for every occurrence — even for genuinely distinct data.

Root cause

The Rust Vortex writer de-duplicates duplicate Arrow column names by appending [N] (space-bracket-N-bracket, N starting at 1 for the second occurrence). The Parquet sibling keeps the original duplicate names. The oracle must match the Vortex naming.

Fix

In RaincloudConformanceIntegrationTest.writeOracleCsv:

  1. Header: de-duplicate Parquet column names using the same [N] algorithm Vortex uses before writing the CSV header row.
  2. Data: switch oracleCell from name-based access (rows.getInt(col.name())) to index-based access (rows.getInt(col.columnIndex())). Hardwood's StructAccessor supports getInt(int), getLong(int), getString(int), isNull(int), etc.

After fixing, flip both uci-parkinsons and uci-spambase from gap:256 to ok in expected-status.csv.

Files

  • integration/src/test/java/io/github/dfa1/vortex/integration/RaincloudConformanceIntegrationTest.java
    • writeOracleCsv: de-duplicate column names for header
    • oracleCell: switch to col.columnIndex() for all rows.getXxx() and rows.isNull() calls
  • integration/src/test/resources/raincloud/expected-status.csv — flip uci-parkinsons and uci-spambase

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions