Skip to content

fix(import-export): keep binary values, booleans, all-NULL rows and zoned times through copy, import and export - #4221

Merged
datlechin merged 8 commits into
mainfrom
fix/audit-w6-io
Nov 12, 2026
Merged

datlechin merged 8 commits into
mainfrom
fix/audit-w6-io

Conversation

@datlechin

@datlechin datlechin commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Copying a binary cell copies every byte as 0x hex, not the first 64. Pasting 0x hex into a binary column stages the bytes, in a cell or in pasted rows; a 0x value that is not hex is refused. (IX-1)
  • Row import reads the target table's column types once. A binary column decodes CSV and XLSX 0x hex and JSON Base64 into bytes. Other text is written as text, as before, so PostgreSQL's own \x bytea JSON still imports; a 0x value that is not hex fails its row. (IX-3)
  • On engines whose dialect writes booleans as 1 and 0 (MySQL, MariaDB, SQL Server, SQLite, Oracle, ClickHouse and others), a boolean column takes true, false, yes, no, on, off, t, f, y, n as 1/0. A new SQLite table types a boolean field BOOLEAN (NUMERIC affinity) so it reads as one. (IX-5)
  • CSV import keeps a record of empty fields (,,) as a row, and an empty line in a one-column file as a NULL. Empty records after the last value are still skipped: a spreadsheet saves its empty rows that way. A JSON {} row writes an all-defaults INSERT instead of being dropped and counted; an engine with no such INSERT fails the row and says why. (IX-2)
  • XLSX export splits a batch across as many sheets as it needs, so no sheet passes 1,048,576 rows. Zero bytes are written 0x, apart from NULL. (IX-6, IX-3)
  • Parquet export writes timestamp with time zone, timestamptz, Oracle with local time zone (the driver writes its offset) and timetz as TIMESTAMPTZ and TIMETZ, adjusted to UTC, instead of dropping the offset. Checked on the shipped DuckDB with the probe. (IX-4)
  • Copy as IN Clause writes 1/0 on numeric-boolean engines and binary values as the engine's binary literal instead of dropping them. (IX-7)
  • The AI results summary keeps the grid's 64-byte binary form, since copy no longer stops there.

Tests changed on purpose:

  • CSVImportPluginTests.testIsBlank: ["", ""] is no longer blank.
  • ExportImportRoundTripTests FormatLimits.xlsx: zero bytes now round-trip as 0x.
  • XLSXWriterStreamingTests.workbookDigest: the fixture's zero-byte cell is now 0x.
  • ImportDataSinkAdapterBitColumnTests.anEngineThatReadsBitsAsDigitsIsNotAsked, now anEngineThatReadsBitsAsDigitsGetsNoCast: a PostgreSQL row import reads the columns once.
  • ImportDataSinkAdapterMappingTests.emptyRowIsNotAnError, emptyRowInBatchIsNotAnError: still no error; the empty row is now written as a row of defaults.

Needs the maintainer:

  • All-NULL rows at the very end of a CSV export, and all-NULL rows in an XLSX export, still import as nothing: they are the same text or cells as a spreadsheet's empty rows. Kept as today.
  • JSON text in a binary column that happens to be valid Base64 (SQLite can keep 'abcd' in a BLOB) is decoded. JSON has no binary type, and Base64 is what the export writes.

Docs: TableProApp/docs#212

Test plan

  • New: ImportDataSinkAdapterTargetValueTests, ImportBinaryTextTests, XLSXSheetRowsTests, BlobFormattingServiceCopyTests, cases in InClauseConverterTests, DataGridPasteIntoSelectionTests, RowOperationsManagerPasteTests, CSVImportRowShapeTests, CSVImportPluginTests, ImportTypeMapperTests.
  • Local: 30 suites touching the change, 377 tests passed (TableProTests and FormatPluginTests); verify.sh plugins, lint and l10n pass.
  • apple/scripts/probes/check-parquet-export-values.sh passes against the shipped libduckdb.a (v2.0.0), with zoned timestamps, a zoned time and Oracle's local time zone added.

@datlechin
datlechin enabled auto-merge (squash) November 12, 2026 05:27
# Conflicts:
#	apple/TablePro/Core/Services/Export/ImportService.swift
#	apple/TablePro/Core/Utilities/SQL/RowValueCopyFormatter.swift
#	apple/TablePro/Extensions/String+HexDump.swift
#	apple/TablePro/Resources/Localizable.xcstrings
# Conflicts:
#	apple/TablePro/Core/Plugins/ImportDataSinkAdapter.swift
#	apple/TableProTests/Core/Plugins/ImportDataSinkAdapterBitColumnTests.swift
@datlechin
datlechin merged commit 9a9c935 into main Nov 12, 2026
2 checks passed
@datlechin
datlechin deleted the fix/audit-w6-io branch November 12, 2026 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant