Summary
The conformance test crashes with OOM when testing large corpus files (≥157 MB) because it materializes both the vortex-java CSV and the Parquet oracle CSV as Java heap Strings.
Root cause
exportVortex() called CsvExporter.exportCsv(vortex, StringWriter, ...) then stringWriter.toString().lines().toList() — materializing potentially GB-sized CSVs in heap as String objects (UTF-16 internally, so 2× the text size).
For the 157 MB bi-euro2016.vortex file, the CSV output runs ~200 MB+ text, which as a String takes ~400 MB+ heap. Two such strings (oracle + result) plus JVM overhead exhausted available heap before reaching the corrupted line (line 262146), so the data corruption in that file was silently masked.
Fix
Replaced StringWriter with temp files using CsvExporter.exportCsv(Path, Path, ExportOptions) for the vortex export and Files.newBufferedWriter() for the oracle. Comparison now streams both files via BufferedReader.readLine() — constant heap usage regardless of file size.
Summary
The conformance test crashes with OOM when testing large corpus files (≥157 MB) because it materializes both the vortex-java CSV and the Parquet oracle CSV as Java heap Strings.
Root cause
exportVortex()calledCsvExporter.exportCsv(vortex, StringWriter, ...)thenstringWriter.toString().lines().toList()— materializing potentially GB-sized CSVs in heap asStringobjects (UTF-16 internally, so 2× the text size).For the 157 MB
bi-euro2016.vortexfile, the CSV output runs ~200 MB+ text, which as a String takes ~400 MB+ heap. Two such strings (oracle + result) plus JVM overhead exhausted available heap before reaching the corrupted line (line 262146), so the data corruption in that file was silently masked.Fix
Replaced
StringWriterwith temp files usingCsvExporter.exportCsv(Path, Path, ExportOptions)for the vortex export andFiles.newBufferedWriter()for the oracle. Comparison now streams both files viaBufferedReader.readLine()— constant heap usage regardless of file size.