Repository navigation
Feature/tiered storage - #1049
Open
foodprocessor wants to merge 104 commits into
Open
foodprocessor wants to merge 104 commits into
foodprocessor wants to merge 104 commits into
Conversation
…t OpenFile test pass
… with no file attatched, also added some config path
…ion), added tests for OpenFile O_Create
Feature/tiered storage setup
Feature/tiered storage
foodprocessor
had a problem deploying
to
testing
September 10, 2026 23:19 — with 
GitHub Actions
Error
GitHub Actions
Error
foodprocessor
had a problem deploying
to
testing
September 10, 2026 23:19 — with
GitHub Actions
Failure
GitHub Actions
Failure
foodprocessor
had a problem deploying
to
testing
September 10, 2026 23:19 — with
GitHub Actions
Failure
GitHub Actions
Failure
…cially on Windows)
foodprocessor
marked this pull request as draft
September 11, 2026 06:53
foodprocessor
had a problem deploying
to
testing
October 1, 2026 21:49 — with
GitHub Actions
Failure
GitHub Actions
Failure
foodprocessor
had a problem deploying
to
testing
October 1, 2026 21:49 — with
GitHub Actions
Failure
GitHub Actions
Failure
foodprocessor
had a problem deploying
to
testing
October 1, 2026 22:04 — with
GitHub Actions
Failure
GitHub Actions
Failure
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Several rename, pagination, accounting, and persistence defects can cause hangs, inconsistent listings, or data loss.
16 open findings
What changed in this PR
Adds a local-primary tiered storage component with cloud overflow, LRU eviction, restart recovery, and cross-platform support.
Changes:
- Implements tiered file operations, persistence, capacity tracking, and eviction.
- Adds configuration, documentation, registration, and pipeline validation.
- Adds unit tests and an S3 mount smoke test.
| File | Description |
|---|---|
.github/workflows/unit-test.yml |
Adds tiered-storage smoke testing. |
README.md |
Documents tiered-storage usage. |
cmd/imports.go |
Registers the new component. |
cmd/mount_linux_test.go |
Verifies cleanup preserves tiered data. |
common/lock_map.go |
Adds atomic handle counts and try-locking. |
common/util.go |
Extends pipeline validation. |
common/util_test.go |
Tests tiered pipeline combinations. |
component/tiered_storage/cache_size.go |
Tracks local capacity usage. |
component/tiered_storage/lru_policy.go |
Implements parallel LRU eviction. |
component/tiered_storage/lru_policy_test.go |
Tests eviction behavior. |
component/tiered_storage/persistence.go |
Persists and recovers local state. |
component/tiered_storage/tiered_storage.go |
Implements the core component. |
component/tiered_storage/tiered_storage_linux.go |
Adds Linux filesystem support. |
component/tiered_storage/tiered_storage_linux_test.go |
Tests Linux ownership persistence. |
component/tiered_storage/tiered_storage_test.go |
Tests component operations and recovery. |
component/tiered_storage/tiered_storage_windows.go |
Adds Windows filesystem support. |
sample_configs/sampleTieredStorageConfigS3.yaml |
Provides an S3 example. |
setup/advancedConfig.yaml |
Documents advanced configuration. |
setup/baseConfig.yaml |
Adds baseline configuration options. |
testdata/config/s3_key_tiered_storage.yaml |
Adds smoke-test configuration. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+41
to
+42
| tieredStorageSnapshotPath = ".tieredStorageSnapshot.gob" | ||
| tieredStorageSnapshotVersion = 1 |
Comment on lines
+747
to
+750
| if rmErr := os.Remove(dst); rmErr != nil && !errors.Is(rmErr, os.ErrNotExist) { | ||
| return err | ||
| } | ||
| return os.Rename(src, dst) |
Comment on lines
+814
to
+817
| var node *FileNode | ||
| if val, ok := c.fileMap.Load(options.Handle.Path); ok { | ||
| node = val.(*FileNode) | ||
| } |
Comment on lines
+836
to
+850
| if err == nil { | ||
| c.setHandleDirty(options.Handle) | ||
| if node != nil { | ||
| node.isDirty.Store(true) | ||
| if info, statErr := f.Stat(); statErr == nil { | ||
| c.cacheSize.Add(info.Size() - node.size.Swap(info.Size())) | ||
| } | ||
| } | ||
| } else { | ||
| log.Err( | ||
| "TieredStorage::WriteFile : failed to write %s [%s]", | ||
| options.Handle.Path, | ||
| err.Error(), | ||
| ) | ||
| } |
| } | ||
|
|
||
| func (c *TieredStorage) RenameFile(options internal.RenameFileOptions) error { | ||
| log.Trace("TieredStorage::RenameFile : src=%s, dst=%s", options.Src, options.Dst) |
Comment on lines
+361
to
+364
| if token == "" { | ||
| for name, attr := range localAttrs { | ||
| if _, found := listed[name]; !found { | ||
| attrs = append(attrs, attr) |
Comment on lines
+1046
to
+1048
| if err := os.Rename(srcPath, dstPath); err != nil { | ||
| return err | ||
| } |
Comment on lines
+1317
to
+1322
| available := max(int64(c.maxCacheSize)-c.cacheSize.Used(), 0) | ||
| stat := &common.Statfs_t{ | ||
| Blocks: uint64(c.maxCacheSize) / blockSize, | ||
| Bavail: uint64(available) / blockSize, | ||
| Bfree: physicalFree / blockSize, | ||
| Bsize: blockSize, |
| ### Expanding Existing Local Storage Using Tiered Storage | ||
|
|
||
| When using `tiered_storage` to add cloud capacity to an existing local storage location, please ensure all access is done through the Cloudfuse mount path / virtual drive. The `tiered_storage` `path` and all its existing contents will become *internal* local storage. Accessing the `tiered_storage` `path` directly may have unpredictable results. | ||
| For example, if you want expand the capacity of your existing local drive, `D:`, you could change that local drive's letter to `E:` first, then set `tiered_storage` `path` to `E:` and mount cloudfuse to `D:`. Then any existing applications or workflows that normally use `D:` would continue as normal, now pointed at the cloudfuse mount location. |
Comment on lines
+138
to
+139
| high-threshold: <% local storage consumed which triggers eviction. Default - 80> | ||
| low-threshold: <% local storage consumed which eviction targets. Must be less than high-threshold. Default - 60> |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add tiered storage component.