Skip to content

Raise the CSV field limit, and set VERSION to 0.2.0 - #16

Merged
steven-hubbard merged 2 commits into
mainfrom
csv-field-limit-and-v0.2.0
Sep 4, 2026
Merged

steven-hubbard merged 2 commits into
mainfrom
csv-field-limit-and-v0.2.0

Conversation

@steven-hubbard

Copy link
Copy Markdown
Contributor

Summary

Two changes, both needed before a release can go out.

Raise the CSV field limit. dbprofiler.py never called csv.field_size_limit(), so it ran on the module default of 128 KiB per field. Now that statistics values are published as PostgreSQL computed them rather than as fixed-width digests, that cap is reachable at ordinary settings: ANALYZE caps an individual value at about 1 KiB (WIDTH_THRESHOLD) but collects statistics_target of them, so a wide text column at the default target of 100 lands near the cap and a raised target goes past it.

statistics_target digest (64 ch) verbatim (≤1 KiB)
100 (default) ~6 KB ~100 KB — at the edge
250 ~16 KB ~256 KB — over
10000 (max) ~650 KB ~10 MB

A run against such a column failed with _csv.Error: field larger than field limit (131072). The bug was latent before and only reachable at extreme targets; publishing values verbatim moved it into normal use.

Uses 2**31 - 1 rather than the more common sys.maxsize. The limit is stored in a C long, which is 32 bits on 64-bit Windows, so csv.field_size_limit(sys.maxsize) raises OverflowError there. 2 GB is still ~200x the worst realistic field.

Set VERSION to 0.2.0. The released v0.1.0 tag is taken, and the release workflow refuses to build when the tag disagrees with VERSION. Minor rather than patch because the bundle contract is now 2.0 — payload set, column set, and field names all changed.

Why this is urgent

The releases page still serves v0.1.0, which is the tokenizing build and also carries this bug. Anyone following the install instructions in the README today gets it, and hits a prompt for a DBPROFILER_TOKEN_KEY environment variable that no longer exists in the project.

Test plan

  • python3 -m unittest — 394 tests, OK
  • python3 dbprofiler.py --check-safety — OK (17 SQL constants checked)
  • Two tests added, written failing first: one parses a field past the default cap through run_psql, one asserts the limit is raised to a value that fits a 32-bit long
  • python3 dbprofiler.py --version reports 0.2.0
  • All three modules parse under a 3.9 grammar check
  • Integration suite against a live PostgreSQL 16 (not run here — no server configured)
  • Tag v0.2.0 after merge so the releases page stops serving the tokenizing build

steven-hubbard and others added 2 commits September 3, 2026 20:40
Statistics values are now published as PostgreSQL computed them rather than
as fixed-width digests, which made the csv module's default 128 KiB per-field
cap reachable at ordinary settings. ANALYZE caps an individual value at about
1 KiB but collects statistics_target of them, so a wide text column with the
default target of 100 lands near the cap and a raised target goes well past
it. A run against such a column died with "field larger than field limit".

Uses 2**31-1 rather than sys.maxsize: the limit is stored in a C long, which
is 32 bits on 64-bit Windows, where sys.maxsize raises OverflowError.

VERSION goes to 0.2.0 because the released 0.1.0 tag is taken and the bundle
contract is now 2.0.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
The UUID-shaped token tests were the only users, and they went with
tokenization. ruff has been failing on main since.

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
@steven-hubbard
steven-hubbard merged commit 8474d53 into main Sep 4, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant