Skip to content

Add extended object naming mode - #260

Closed
liubin07 wants to merge 1 commit into
pgsty:mainfrom
liubin07:feat/object-naming-extended
Closed

liubin07 wants to merge 1 commit into
pgsty:mainfrom
liubin07:feat/object-naming-extended

Conversation

@liubin07

Copy link
Copy Markdown

Summary

Adds an opt-in extended object naming mode to Silo, seeded with
--object-naming=extended / MINIO_OBJECT_NAMING=extended on a cluster's
first start. The mode removes the long-standing "conflicting objects" listing
limitation: an object and a prefix of the same name (obj1 and obj1/obj2)
can coexist, both appear in LIST results, and exact-key reads, writes and
deletes stay independent.

On disk, user keys are escaped (␣⫽◖◗◊◉◈◇◆⬡ → ␣0…␣9) and regular objects
carry a terminal marker, so obj1 is stored at obj1◊/xl.meta while obj1/
remains a plain prefix directory. An object directory can therefore never
double as another key's parent — the structural invariant that safe mode
cannot provide. Directory objects keep the legacy __XLDIR__ representation.

Design notes

  • Where the transformation lives: at the pooled object-layer boundary, where
    upstream already applies encodeDirObject. This keeps composite paths built
    by the erasure layer (object/DataDir/part.N, object/<uuid>) and every
    meta-bucket (.minio.sys) layout untouched. The storage layer stays entirely
    in the disk domain.
  • Listing order: the terminal marker (U+25CA) sorts after / in UTF-8 byte
    order, so disk order is not S3 key order. The walk decodes and re-sorts each
    directory level in the user domain and recurses inline, keeping the stream
    ascending; for the same name a directory object precedes its prefix. The
    merge layer compares exact names in this mode so obj1 and obj1/ are both
    retained, while the object-over-directory rule still applies to X/ vs X/.
  • Mode selection: recorded in every drive's format.json (objectNaming),
    verified per drive, per pool, and across nodes (existing bootstrap env
    comparison), and rejected on mismatch at startup. Fresh clusters only; the
    mode cannot be changed or migrated in place.
  • Safe mode is unchanged: default behavior, disk layout, and the
    documented conflicting-objects limitation are all preserved.

Testing

  • New unit/integration tests (cmd/object-naming_test.go): encode/decode
    round-trips and escape order preservation; the full conflict matrix
    (delimiter + root, recursive, prefix and exact-key listing, pagination and
    marker resume, parent delete/overwrite leaving the child intact, heal);
    versioned bucket with delete markers; object + directory object + child
    coexistence; names containing reserved characters; multipart completion onto
    a conflicting key; format.json persistence and mode-mismatch rejection;
    safe-mode behavior lock-in.
  • Regression: LIST/metacache, format, xl-storage, multipart, delete,
    decommission/rebalance, heal, scanner/data-usage, bucket/config,
    object-lock/tier/replication and the HTTP API suites all pass.
  • End-to-end against a real single-drive server started with
    MINIO_OBJECT_NAMING=extended, using a stock S3 client: all cases from the
    conflict analysis checklist (root delimiter, recursive, prefix=obj1/,
    GET/HEAD, delete parent, overwrite parent) pass.
  • go build ./..., go vet ./cmd, gofmt, rebrand-guard and
    verify-rebrand.sh are clean (the compatibility baseline records the new
    MINIO_OBJECT_NAMING identifier).

Note: TestXLStorageVerifyFile panics on this machine with Go 1.27.3
(ringbuffer divide-by-zero) on the unmodified baseline as well; it is
unrelated to this change.

Docs

  • docs/minio-limits.md documents the mode, its restrictions (fresh cluster
    only, no in-place migration, raw-drive tools such as mc admin inspect must
    use encoded names, encoded-name length limits, no cross-mode replication) and
    keeps the safe-mode limitation.
  • CHANGELOG.md gains an Unreleased entry. Companion-site documentation
    follows in the silo.pgsty.com repository.

Introduce an opt-in on-disk object naming mode selected at first format
with --object-naming=extended or MINIO_OBJECT_NAMING=extended. Object
keys are escaped and regular objects carry a terminal marker on disk, so
an object directory can never double as another key's parent: obj1 and
obj1/obj2 coexist, both appear in LIST results, and exact-key reads,
writes and deletes stay independent.

- The mode is persisted in every drive's format.json and verified across
  drives, pools and nodes; mixed modes are rejected at startup and the
  mode cannot be changed afterwards.
- The walk decodes and re-sorts each directory level in the user domain
  because the terminal marker sorts after '/' on disk; the merge layer
  keeps object and prefix entries with distinct names.
- Object names are transformed at the pooled object-layer boundary, so
  composite paths (object/DataDir/part.N) and meta-bucket layouts are
  untouched. Default (safe) mode keeps its existing behavior and disk
  layout.
- The scanner decodes disk names for lifecycle/replication evaluation
  and object API calls; the rebrand compatibility baseline records the
  new identifier.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: liubin <liubin07@163.com>
@liubin07 liubin07 closed this Oct 30, 2026
@liubin07
liubin07 deleted the feat/object-naming-extended branch October 30, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant