fix(build): a pull no longer deletes definition data silently - #810
fix(build): a pull no longer deletes definition data silently#810guill wants to merge 2 commits into
Conversation
The builder stores a Build's definition verbatim -- `api.Definition` is a
`map[string]interface{}` that `UpdateBuild` writes straight into jsonb -- so the
typed struct's `omitempty` tags never touch it, and a field the local spec sets
but the fetched Build omits is a field that never round-tripped. Until now
`merge_pull_definition` dropped those on the floor: `merged` starts as a copy of
the server definition and only carries back local keys outside
`_DEFINITION_KNOWN_FIELDS`, so a known-but-absent field left the spec without a
word.
`merge_pull_definition` now diffs the local key set against the merged one and
raises `UnsyncedDefinitionError` when anything would disappear, naming the
fields in `details.fields`. Turning silent data loss into a refusal costs a pull
that used to "succeed": a spec written by `comfy build init` carries
`environment`, and pulling a Build that never received it now exits 1 rather
than deleting it. That is the intended trade -- the recovery is one `comfy build
push`, and the alternative is losing the field without ever being told.
Three details fall out of the same round-trip premise:
- `definition.schema` is exempt. It names the spec's own file format, not build
state, so the builder has no concept of it and its absence server-side says
nothing. It is restored from local rather than refused.
- `description` is not a definition field and gets the opposite treatment. It is
`*string` + `omitempty` on the builder, which collapses `""` to a nil pointer,
so an empty description genuinely arrives absent. Defaulting it to `""` is the
faithful read of the wire, where refusing would be a false alarm -- and the
`isinstance(description, str)` check rejected the `None` outright, so pulling
any Build with no description failed before reaching the merge at all.
- `customNodePolicy` joins `_DEFINITION_KNOWN_FIELDS`, which already held
`modelPolicy` and `partnerNodePolicy`. Missing from the set, it was carried
back from local unconditionally, so a policy the server cleared came straight
back on the next pull -- and it alone among the three would have escaped the
new check.
A Build that carries a field as an empty value is an intentional clear and is
applied normally; only an absent key is refused.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
skishore23
left a comment
There was a problem hiding this comment.
Re-reviewed against the builder source and I need to walk back the earlier approval — the guard's premise only holds for one of the server's write paths.
1. (high) build_pull.py:44,243-245 — the guard refuses every adopt/rebind pull from a scan-initialized spec, and the hint's recovery duplicates or 409s. comfy build init (scan path, build.py:552-558) always writes pipDependencies and environment. The builder has no concept of environment at all (zero non-comment references in comfy-builder Go). And "omitempty never touches the definition" is true for UpdateBuild/create (verbatim map[string]interface{}, builds_update.go:64, create.go:104) but not for the four other doors — workflows.go:56, snapshots.go:62, from_snapshot.go:64, from_workflow.go:60 all store via definition.ToMap, which is json.Marshal of the typed struct with omitempty, so an empty pipDependencies (definition.go:70) or policy is stored absent. Scenario: comfy build init → comfy build pull --id X -y to adopt a build created from the web/snapshot/workflow (the cross-id flow test_cross_id_pull_rebinds... pins) → exit 1, fields: [environment, pipDependencies]. Following the hint: comfy build push with id: null creates a new build instead of adopting X; push --id X sends expectedUpdatedAt: null → Go zero time → UpdateTimeEQ matches nothing → 409 stale, whose hint says pull again. The only non-clobbering exit is hand-deleting the keys. Fix direction: exempt environment alongside schema (both authoring-only from the server's view), and don't treat absence as "never synced" for a definition that can come from a ToMap door — or gate the refusal on the build having been last written by the CLI.
2. (medium) build_pull.py:239-240 — an absent models/customNodes still silently wipes every local entry, the same data-loss class this PR fixes. Reproduced: local spec with one local model + one local node, server definition {"baseComfyVersion": "v0.3.0"} (what a from-workflow build with no models looks like after ToMap) → pull -y exits 0, written: true, models: [], customNodes: [], and the local localPath/sha256/localDigest authoring data is gone with no warning. You flagged this as a follow-up, but the registry text for build_pull_unsynced_definition tells agents the check refuses "definition fields the local spec sets and the fetched Build omits" without the collections caveat — at minimum the text should say so.
Everything else verified: description really does arrive absent (*string,omitempty, toBuild sets it only when non-empty), so the "" default is right; all three policies are consumed at freeze (freeze.go:247-249) so customNodePolicy belongs in _DEFINITION_KNOWN_FIELDS; the refusal happens before any write; details.fields is a list; the push/pull bounce is not a deadlock (push --force or deleting the field exits it).
Tests on the branch (typer < 0.22 venv): 521 passed, 1 skipped; ruff clean.
The round-trip check the previous commit added can only ever see top-level
scalars. `merge_pull_definition` assigns `merged["models"]` and
`merged["customNodes"]` unconditionally, and only then diffs the local key set
against the merged one -- so the two collections are structurally incapable of
reaching `dropped`, and a fetched Build whose definition omits them still wipes
every local entry. That is not hypothetical: `definition.ToMap` is
`json.Marshal` of the typed struct, every field `omitempty`, so a build created
from a workflow import stores as little as `{"customNodes":[...]}` with no
`models` key at all. Pulling it drops the local `localPath`, `source`, `sha256`
and `localDigest` rows and exits 0 without a word.
`-y` is not what makes that silent. The guard runs thirty lines before
`confirm()` is consulted, so the scalar half already refuses regardless of
`--yes` -- and the interactive half asked only "Pull build X and overwrite the
local spec at PATH?", naming no consequence at all. There was no warning for
`-y` to suppress.
`pull` now reports, the way `update` already does. `diff_definitions` runs
against the definition on disk, `render_definition_diff` prints the table in
pretty mode, the summary is echoed inside the confirmation, and `summary` plus
`diff` ride in the `--json` payload. The baseline is deliberately the file on
disk rather than `prepare_push`'s reconciled copy: that copy recomputes model
`sha256` and node `localDigest`, values this write also lands, so diffing it
would hide them.
`--dry-run` joins it, for the reason `update` has one. With `--yes` the payload
only arrives after the write, so a non-interactive caller had no way to read the
diff while it could still act on it. It short-circuits ahead of the
confirmation, since a prompt whose only outcome is a write it will not perform
is noise, and `build_pull_needs_confirm` now names it alongside `--yes`.
`merge_pulled_spec` returns a `PulledSpec` carrying the merged definition beside
the spec. Indexing a `JsonObject` yields `JsonValue`, which is not a
`Mapping[str, JsonValue]`; the function builds the definition and knows its
type, so it hands it back typed rather than making the caller cast it.
This reports the collection drop rather than refusing it. Whether an absent
collection should also be refused is a separate question from whether it should
be visible, and the answer to the second is not "no".
Stack of 7 — review bottom-up (this is #810)
feat(build): packaging, sync and Builder client primitives — basemainfeat(build)!: replace the legacy build and distribution surface (breaking)feat(deploy): the deployment control planefeat(deploy): the asset and job data-plane clientscomfy deploy run#805 —feat(deploy):comfy deploy runfix(build): a pull no longer deletes definition data silently ← this PRfix(cli): unbreak the interactive pickers and Go timestamp parsing on 3.10Each PR is based on the branch below it, so its own diff is only its commits. Merge in order.
TL;DR:
comfy build pulldeleted definition data the fetched Build did not carry, without a word. Two commits: it now refuses the case it can prove is data loss, and reports everything else it is about to change. Stacked on #805.The bug
merge_pull_definitionstartsmergedas a copy of the server definition and only carries back local keys outside_DEFINITION_KNOWN_FIELDS. A known-but-absent field —environment,pipDependencies, a policy — was dropped from the spec silently. So weremodelsandcustomNodes, by a different route (below).1.
148199e— refuse a pull that would silently delete definition fieldsmerge_pull_definitiondiffs the local key set against the merged one and raisesUnsyncedDefinitionErrorwhen a top-level scalar would disappear, naming the fields indetails.fields. New codebuild_pull_unsynced_definition;pull_cmdrenders it and exits 1, before any write.definition.schemais exempt. It names the spec's own file format, not build state, so the builder has no concept of it and its absence server-side says nothing. Restored from local rather than refused — the one member of_UNSYNCED_DEFINITION_FIELDS.descriptiongets the opposite treatment, and this is a straight bug fix. It is*string+omitemptyon the builder, which collapses""to a nil pointer, so an empty description genuinely arrives absent. It now defaults to""; previously theisinstance(description, str)check rejected theNoneoutright, so pulling any Build with no description failed before reaching the merge at all.customNodePolicyjoins_DEFINITION_KNOWN_FIELDS, which already heldmodelPolicyandpartnerNodePolicy. Missing from the set, it was carried back from local unconditionally, so a policy the server cleared came straight back on the next pull — and it alone among the three would have escaped the new check.2.
39abbe1— say what a pull changes before it changes itThe check above can only ever see top-level scalars.
merge_pull_definitionassignsmerged["models"]andmerged["customNodes"]unconditionally, and only then computesdropped— so the two collections are structurally incapable of reaching it. A fetched Build whose definition omits them still wiped every local entry, dropping the locallocalPath,source,sha256andlocalDigestrows and exiting 0.-yis not what made that silent. The guard runs ~30 lines beforeconfirm()is consulted, so the scalar half already refuses regardless of--yes. And the interactive path asked only "Pull build X and overwrite the local spec at PATH?" — naming no consequence at all. There was no warning for-yto suppress.pullnow reports, the waybuild updatealready does:diff_definitionsruns against the definition on disk — deliberately notprepare_push's reconciled copy, which recomputes modelsha256and nodelocalDigest, values this write also lands, so diffing it would hide them.render_definition_diffprints the change table in pretty mode.Pull build X and overwrite … (models +0 -1 ~0, customNodes +0 -1 ~0, baseComfyVersion)?summaryanddiffride in the--jsonpayload;build_pull.jsongains them plusdry_run.--dry-run, for the reasonupdatehas one: with--yesthe payload only arrives after the write, so a non-interactive caller had no way to read the diff while it could still act on it. It short-circuits ahead of the confirmation, andbuild_pull_needs_confirmnow names it alongside--yes.merge_pulled_specreturns aPulledSpeccarrying the merged definition beside the spec — indexing aJsonObjectyieldsJsonValue, which is not aMapping[str, JsonValue], and the function already knows the type it built.This reports the collection drop rather than refusing it. Whether an absent collection should also be refused is a separate question from whether it should be visible, and the answer to the second is not "no".
Validation
tests/comfy_cli/command/+tests/comfy_cli/output/: 3749 passed, 3 skipped.ruff check ./ruff format --check .clean at CI's pin (0.15.15).build.py:893is pre-existing, frome4f661f).ToMap-shaped definition:--dry-runrenders the table and leaves the file byte-identical; the prompt carries the summary; declining aborts without writing; the--jsonpayload validates against the shipped schema.pulltotal — four new: the collection wipe is reported, the confirmation names the change,--dry-runwrites nothing, and--dry-rundoes not prompt a TTY caller.