Skip to content

RaincloudConformanceIntegrationTest OOM on large corpus files #254

Description

@dfa1

Summary

The conformance test crashes with OOM when testing large corpus files (≥157 MB) because it materializes both the vortex-java CSV and the Parquet oracle CSV as Java heap Strings.

Root cause

exportVortex() called CsvExporter.exportCsv(vortex, StringWriter, ...) then stringWriter.toString().lines().toList() — materializing potentially GB-sized CSVs in heap as String objects (UTF-16 internally, so 2× the text size).

For the 157 MB bi-euro2016.vortex file, the CSV output runs ~200 MB+ text, which as a String takes ~400 MB+ heap. Two such strings (oracle + result) plus JVM overhead exhausted available heap before reaching the corrupted line (line 262146), so the data corruption in that file was silently masked.

Fix

Replaced StringWriter with temp files using CsvExporter.exportCsv(Path, Path, ExportOptions) for the vortex export and Files.newBufferedWriter() for the oracle. Comparison now streams both files via BufferedReader.readLine() — constant heap usage regardless of file size.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions