feat: add data_migration action for schema-neutral DML

[owjs3901]

Why

MigrationAction::RawSql is skipped during baseline replay because its effect is unknown. That is harmless when the SQL is pure DML ??it changes no schema, so there is nothing for replay to reflect. It is fatal when the SQL is DDL: replay silently loses the schema change, the reconstructed baseline is permanently wrong, and vespertide diff reports the same already-applied changes on every run.

We hit exactly this in a production repository. Four migrations used raw_sql for DDL, so vespertide diff permanently reported 14 unrelated pending changes, and vespertide revision then generated a 15-action migration when only ONE new table was needed ??it would have tried to re-create existing tables and failed.

Today nothing in the type system lets an author say "this SQL is data-only". A reviewer has to read every raw_sql body and judge it by hand. That contract should be expressible as a type.

What changed

1. MigrationAction::DataMigration

{ "type": "data_migration",
  "description": "set tier+kind on legacy rows only",
  "sql": "UPDATE \"user\" SET tier = 'PRO', kind = 'X' WHERE kind = 'legacy'" }

sql also accepts a per-backend object for statements that cannot be portable:

{ "type": "data_migration",
  "description": "text column -> JSON object keyed by locale",
  "sql": {
    "postgres": "UPDATE t SET j = jsonb_build_object('ko', c)",
    "mysql":    "UPDATE t SET j = JSON_OBJECT('ko', c)",
    "sqlite":   "UPDATE t SET j = json_object('ko', c)"
  } }

All three backend keys are required in the per-backend form ??a missing key would silently emit nothing for that backend, which is the same class of silent loss this action exists to prevent. description is optional but encouraged; it is what vespertide diff and vespertide log display for the action.

This fills a real gap. The existing backfill facilities are all coupled to a schema change and heavily constrained:

Facility	Limitation
add_column.fill_with	Only fires when the column is NOT NULL and has no default and fill_with is set (add_column.rs:122). If the column has a default, fill_with is silently ignored. Only applies to a newly added column.
modify_column_default.backfill	One column, one fixed value, every row, no WHERE ??and coupled to changing the default.
modify_column_type.fill_with	Only remaps enum labels.

So a conditional backfill of existing columns, a correlated-subquery backfill, or data reshaping during a format change all required raw_sql before this PR.

2. Verbatim emission

build_data_migration passes the statement through untouched ??no case folding, no cast rewriting, no reformatting. It deliberately bypasses every backend-normalising helper (convert_default_for_backend, normalize_fill_with, ??, and the module doc says so, so nobody "fixes the inconsistency" later. Snapshot evidence (identical on all three backends):

UpDaTe "User"
  SET meta = '{"a": 1}'::jsonb, n = N + 1
  WHERE Kind = 'Legacy';

Mixed-case keywords, quoted identifiers, :: casts, and multi-line formatting all survive.

3. Replay contract

apply_action gets its own explicit DataMigration arm calling apply/data_migration.rs, kept separate from apply/raw_sql.rs precisely so the reason is documented and cannot be collapsed:

RawSql is skipped because its effect is unknown.
DataMigration is skipped because changing no schema is its contract.

4. DDL guard enforces the contract

A statement whose first token ??after trimming leading whitespace, -- line comments and /* */ block comments ??is CREATE, ALTER, DROP, or TRUNCATE (case-insensitive) is rejected with PlannerError::DataMigrationContainsDdl. Matching requires a token boundary, so CREATED_AT_FIXUP() is fine and DROPLET is not DROP. Every branch of a per-backend form is checked, so DDL hidden in only the sqlite key is still caught.

The guard runs inside validate_migration_plan, which the loader calls per file, so it fires at load time and therefore at plan time too. Real CLI output:

$ vespertide diff
Error: validate migration: migrations\0004_sneaky.json

Caused by:
    data_migration contains DDL: the statement starts with `ALTER`
    (ALTER TABLE "user" ADD COLUMN sneaky int;). `data_migration` promises to
    change data only ??baseline replay skips it on that basis, so schema changes
    hidden here are lost forever and `vespertide diff` will report phantom
    pending changes. Express the schema change with a typed action, or use
    `raw_sql` if you genuinely need the escape hatch.
$ echo $?
1

(That input was "-- just a tidy-up\n /* honest */ ALTER TABLE ..." ??the comments are stripped before the check and before the quoted preview.)

raw_sql keeps its DDL freedom; only data_migration is constrained.

5. The replay hazard is now visible

Previously vespertide diff output was silently wrong with no hint at all ??which is how the 14 phantom changes went unnoticed. diff and status now warn when the applied history contains raw_sql, naming the affected versions. In diff it prints before the action list, since the list itself is what may be untrustworthy:

$ vespertide diff
??2 applied migration(s) use raw_sql ??baseline replay may be incomplete:
  versions: 4 (1 action), 5 (2 actions)
  why: replay cannot interpret raw SQL, so any schema change those actions made is
       missing from the reconstructed baseline ??`vespertide diff` may report
       changes that are already applied
  fix: re-express schema changes as typed actions; use `data_migration` for
       data-only SQL so replay can skip it safely
Found 1 change(s) to apply:

1. Add column: user.nickname

That reproduction is the production bug in miniature: nickname was already added by a raw_sql DDL action, so the reported change is a phantom. A history using data_migration instead produces no warning and No differences found.

6. Docs point users at the new action

The rustdoc on MigrationAction::RawSql (and therefore its description in schemas/migration.schema.json, which is generated from it) now explains the difference and points at data_migration for data-only changes.

Test evidence

cargo test --workspace --all-features ??4433 passed, 0 failed
cargo clippy --workspace --all-targets --all-features -- -D warnings ??clean
cargo fmt --all --check ??clean
sh scripts/check-line-budget.sh ??clean
Schema drift (git diff --no-index schemas _tmp_schemas) ??empty

Covering each stated requirement:

• Verbatim SQL ??uniform_sql_is_emitted_byte_for_byte asserts byte equality against deliberately hostile SQL (mixed case, :: cast, quoted identifier, newlines) across all 3 backends, plus snapshots.
• Per-backend selection ??per_backend_sql_selects_the_matching_statement across all 3 backends.
• DDL rejected ??data_migration_ddl.rs: 6 DML forms accepted (incl. correlated subquery and CTE), 7 DDL forms rejected with the offending keyword, DDL hidden in a single backend branch rejected, raw_sql unaffected, and all offending actions reported rather than just the first. Plus a 22-case rstest on leading_ddl_keyword covering comments, case, token boundaries and unterminated comments.
• Replay is schema-neutral ??apply_data_migration_leaves_schema_untouched asserts the schema is unchanged for both wire forms.
• diff/status warn ??raw_sql_warning.rs unit tests on the rendered text plus cmd_diff_warns_when_history_contains_raw_sql, cmd_diff_stays_quiet_when_history_uses_data_migration, and cmd_status_warns_when_history_contains_raw_sql.
• Wire format stability ??round-trip tests assert byte-identical JSON for all three shapes; description uses skip_serializing_if so omitting it round-trips unchanged.

All of the CLI output quoted above was captured by running the built vespertide binary against a scratch project, not reasoned about.

Note for the two sibling PRs

Two PRs are in flight on this repo touching add_column.fill_with lowercasing and modify_column_type.fill_with double-quoting. This PR touches schemas/migration.schema.json and the MigrationAction enum, which they may also touch, so those edits were kept deliberately minimal:

• MigrationAction: the new variant is appended after RawSql, at the end of the enum. No existing variant's fields were changed. The only edits to existing lines are the two RawSql doc-comment additions (requirement 5).
• schemas/migration.schema.json: regenerated, +59/??. The new DataMigrationSql $def and the data_migration entry are appended; the 2 changed lines are the two RawSql descriptions. The add_column.fill_with and modify_column_type.fill_with schema regions are untouched, so conflicts should be trivial or absent.

Also worth flagging for the fill_with lowercasing PR: this action deliberately does not route through the normalising helpers, and build_data_migration's doc records that as intentional ??so it should not reintroduce that bug class.

One unrelated line was required to compile: vespertide-lsp's ErrorLocation::from_planner_error matches PlannerError exhaustively, so the new variant needed an arm. It returns None (a data_migration DDL violation lives in a migration file, which the model-file locator cannot anchor to).

Release / semver

This PR ships a changepack (.changepacks/changepack_log_fIoUZOkWt-518L5MIjOim.json):

Crate	Level	Why
vespertide-core	Minor	New DataMigration variant (on a #[non_exhaustive] enum, so additive) + new public DataMigrationSql / leading_ddl_keyword / sql_preview
vespertide-planner	Minor	Breaking: PlannerError is not #[non_exhaustive], so the new DataMigrationContainsDdl variant breaks exhaustive downstream matches. Also adds find_raw_sql_replay_hazards
vespertide-query	Minor	New sql::data_migration module + build_data_migration
vespertide-cli	Minor	New raw_sql replay warning in diff / status
vespertide-lsp	Patch	Internal only - one match arm added to ErrorLocation::from_planner_error

For 0.x crates Minor is the breaking bump (0.2.1 -> 0.3.0), which is declared honestly here rather than merely conveniently: the PlannerError variant addition genuinely is breaking.

Worth a follow-up: PlannerError is the odd one out - MigrationAction, TableConstraint, QueryError, ColumnType and friends are all #[non_exhaustive]. Marking PlannerError #[non_exhaustive] would make every future error variant additive, but it is itself a breaking change and touches a shared surface the two sibling PRs may also hit, so it is deliberately left out of this PR.

Note the cargo-semver-checks gate derives its release-type from the changepack this PR introduces (a descriptor already on main must not relax the gate). Without one it assumed minor and failed on breaking changes that were already merged to main - notably the pub -> pub(crate) narrowing of sql::helpers in bca0bf6. Those are pre-existing and untouched here; crates/vespertide-query/src/sql/helpers.rs is byte-identical to main on this branch.

[github-actions]

Changepacks

vespertide@0.2.1 → 0.2.2 - crates/vespertide/Cargo.toml

Patch

• Auto-update: depends on 'vespertide-core' via a local workspace dependency

vespertide-cli@0.2.1 → 0.3.0 - crates/vespertide-cli/Cargo.toml

Minor

• modify_column_type.fill_with 를 bare enum 라벨로 확정 (스키마 문서와 구현 불일치로 enum 축소 마이그레이션이 invalid input value for enum 으로 실패하던 문제). 따옴표가 이미 붙은 기존 값은 한 겹 제거 + 1회 경고로 하위호환 유지
• data_migration 액션 추가: MigrationAction::DataMigration(#[non_exhaustive]라 additive), DataMigrationSql/leading_ddl_keyword/sql_preview 공개 API 신설, DDL 가드용 PlannerError::DataMigrationContainsDdl 변형 추가(PlannerError는 exhaustive라 0.x 기준 breaking), find_raw_sql_replay_hazards 신설, diff/status의 raw_sql 리플레이 경고
• Prisma ORM exporter 추가

vespertide-config@0.2.1 → 0.3.0 - crates/vespertide-config/Cargo.toml

Minor

• 성능 최적화 웨이브: 미사용 public API 정리(NameCase::is_*, EnumValues::variant_names/to_sql_values, sql::helpers 일부, find_primary_key_removals 시그니처)와 Orm::Prisma 변형 추가로 0.x 기준 breaking

vespertide-core@0.2.1 → 0.3.0 - crates/vespertide-core/Cargo.toml

Minor

• modify_column_type.fill_with 를 bare enum 라벨로 확정 (스키마 문서와 구현 불일치로 enum 축소 마이그레이션이 invalid input value for enum 으로 실패하던 문제). 따옴표가 이미 붙은 기존 값은 한 겹 제거 + 1회 경고로 하위호환 유지
• data_migration 액션 추가: MigrationAction::DataMigration(#[non_exhaustive]라 additive), DataMigrationSql/leading_ddl_keyword/sql_preview 공개 API 신설, DDL 가드용 PlannerError::DataMigrationContainsDdl 변형 추가(PlannerError는 exhaustive라 0.x 기준 breaking), find_raw_sql_replay_hazards 신설, diff/status의 raw_sql 리플레이 경고
• 성능 최적화 웨이브: 미사용 public API 정리(NameCase::is_*, EnumValues::variant_names/to_sql_values, sql::helpers 일부, find_primary_key_removals 시그니처)와 Orm::Prisma 변형 추가로 0.x 기준 breaking

vespertide-exporter@0.2.1 → 0.3.0 - crates/vespertide-exporter/Cargo.toml

Minor

• 성능 최적화 웨이브: 미사용 public API 정리(NameCase::is_*, EnumValues::variant_names/to_sql_values, sql::helpers 일부, find_primary_key_removals 시그니처)와 Orm::Prisma 변형 추가로 0.x 기준 breaking
• Prisma ORM exporter 추가

vespertide-loader@0.2.1 → 0.2.2 - crates/vespertide-loader/Cargo.toml

Patch

• Auto-update: depends on 'vespertide-config' via a local workspace dependency

vespertide-lsp@0.2.1 → 0.2.2 - crates/vespertide-lsp/Cargo.toml

Patch

• data_migration 액션 추가: MigrationAction::DataMigration(#[non_exhaustive]라 additive), DataMigrationSql/leading_ddl_keyword/sql_preview 공개 API 신설, DDL 가드용 PlannerError::DataMigrationContainsDdl 변형 추가(PlannerError는 exhaustive라 0.x 기준 breaking), find_raw_sql_replay_hazards 신설, diff/status의 raw_sql 리플레이 경고

vespertide-macro@0.2.1 → 0.2.2 - crates/vespertide-macro/Cargo.toml

Patch

• Auto-update: depends on 'vespertide-config' via a local workspace dependency

vespertide-naming@0.2.1 → 0.3.0 - crates/vespertide-naming/Cargo.toml

Minor

• Prisma ORM exporter 추가

vespertide-planner@0.2.1 → 0.3.0 - crates/vespertide-planner/Cargo.toml

Minor

• data_migration 액션 추가: MigrationAction::DataMigration(#[non_exhaustive]라 additive), DataMigrationSql/leading_ddl_keyword/sql_preview 공개 API 신설, DDL 가드용 PlannerError::DataMigrationContainsDdl 변형 추가(PlannerError는 exhaustive라 0.x 기준 breaking), find_raw_sql_replay_hazards 신설, diff/status의 raw_sql 리플레이 경고
• 성능 최적화 웨이브: 미사용 public API 정리(NameCase::is_*, EnumValues::variant_names/to_sql_values, sql::helpers 일부, find_primary_key_removals 시그니처)와 Orm::Prisma 변형 추가로 0.x 기준 breaking

vespertide-query@0.2.1 → 0.3.0 - crates/vespertide-query/Cargo.toml

Minor

• modify_column_type.fill_with 를 bare enum 라벨로 확정 (스키마 문서와 구현 불일치로 enum 축소 마이그레이션이 invalid input value for enum 으로 실패하던 문제). 따옴표가 이미 붙은 기존 값은 한 겹 제거 + 1회 경고로 하위호환 유지
• add_column.fill_with가 사용자 SQL 표현식을 훼손하던 버그 수정: fill_with는 raw SQL 표현식 슬롯이므로 DEFAULT 정규화(convert_default_for_backend)를 태우지 않고 그대로 방출한다. parse_pg_type_cast도 첫 번째 :: 대신 따옴표/괄호를 건너뛴 마지막 top-level :: 에서 분리하도록 수정. 동작 변경: PostgreSQL은 fill_with를 항상 원문 그대로 방출하므로 NOW()가 더 이상 CURRENT_TIMESTAMP로 치환되지 않는다(MySQL/SQLite는 유지)
• data_migration 액션 추가: MigrationAction::DataMigration(#[non_exhaustive]라 additive), DataMigrationSql/leading_ddl_keyword/sql_preview 공개 API 신설, DDL 가드용 PlannerError::DataMigrationContainsDdl 변형 추가(PlannerError는 exhaustive라 0.x 기준 breaking), find_raw_sql_replay_hazards 신설, diff/status의 raw_sql 리플레이 경고
• 성능 최적화 웨이브: 미사용 public API 정리(NameCase::is_*, EnumValues::variant_names/to_sql_values, sql::helpers 일부, find_primary_key_removals 시그니처)와 Orm::Prisma 변형 추가로 0.x 기준 breaking