Repository navigation

Feature/tiered storage - #1049

Open

foodprocessor wants to merge 104 commits into

mainfrom

feature/tiered_storage

Commits on May 27, 2026

Add tiered storage component
jfantinhardesty
committed
318a109
Remove unused functions
jfantinhardesty
committed
0d5cbf3

Commits on Jun 2, 2026

Initial structure setup(First commit)
Jacob Ho
committed
7d93b2a

Commits on Jun 3, 2026

Initial attempt at openFile using design
Jacob Ho
committed
f67f1d7
Rough Design of isOverLocalLimit and openFileHelper, logic added to OpenFile
Jacob Ho
committed
0b27908

Commits on Jun 4, 2026

More structured design flow for OpenFile and setup of test file, first OpenFile test pass
Jacob Ho
committed
554a9b0
Included function to make tests run, initial implementation of CreateFile
Jacob Ho
committed
d77b69f

Commits on Jun 5, 2026

OpenFile now actually opens the file instead of just returning handle with no file attatched, also added some config path
Jacob Ho
committed
ddd1a1b
Added WriteFile and dirty handle setters
Jacob Ho
committed
3218f91
Added tests for open, write, create
Jacob Ho
committed
de9b851
Resolved most comments from initial PR, still have to work on error control
Jacob Ho
committed
598371b

Commits on Jun 8, 2026

Added OpenFile with O_create case
Jacob Ho
committed
3ed9612
Fixed O_Create and modified CreateFile (added unlocked internal function), added tests for OpenFile O_Create
Jacob Ho
committed
55f0dfa
Added initial ReleaseFile and uploadFile components
Jacob Ho
committed
ffc3909

Commits on Jun 9, 2026

Implemented initial design of ReleaseFile and corresponding tests
Jacob Ho
committed
9cef74d

Commits on Jun 11, 2026

Implemented initial design of ReleaseFile and corresponding tests
Jacob Ho
committed
6b919a0
2nd PR Changes
Jacob Ho
committed
0b199fc
Initial ReadInBuffer component
Jacob Ho
committed
d51bfe8

Commits on Jun 15, 2026

Added initial ReadInBuffer tests
Jacob Ho
committed
d805f91
Initial implementation of DeleteFile, no tests yet
Jacob Ho
committed
432d54d

Commits on Jun 24, 2026

Resolved additional PR1 comments
Jacob Ho
committed
012bfc9

Commits on Jun 25, 2026

Merge pull request #1 from HoJacob/feature/tiered_storage_setup
HoJacob
authored
2b914dd
Merge pull request #956 from HoJacob/feature/tiered_storage
jfantinhardesty
authored
2153f1e
Initial LRU skeleton
Jacob Ho
committed
628feaa

Commits on Jun 26, 2026

Initial LRU Functions
Jacob Ho
committed
2ccb80f

Commits on Jun 29, 2026

LRU eviction initial implementation
Jacob Ho
committed
63b276c
LRU worker logic added
Jacob Ho
committed
ca744a6
Initial LRU Code w Workers/Eviction logic
Jacob Ho
committed
8e31b0c

Commits on Jul 1, 2026

LRU capacity design fix
Jacob Ho
committed
ed56450
Fixed deadlock bugs in channels
Jacob Ho
committed
ea8b42a

Commits on Jul 2, 2026

Initial test file setup
Jacob Ho
committed
3ea1864

Commits on Jul 6, 2026

Bug fixes and initial tests of lru eviction policy ready for pr draft
Jacob Ho
committed
4b15841
Feature/tiered storage/read in buffer (#959)

HoJacob
and
Jacob Ho
authored
5e54e56

Commits on Jul 9, 2026

Merge remote-tracking branch 'upstream/feature/tiered_storage' into feature/tiered_storage-lru_policy
Jacob Ho
committed
449e766

Commits on Jul 13, 2026

Handeled Dirty File State Scenario, ReleaseFile bugs, added Tests for dirty edge case
Jacob Ho
committed
3f3a1b8

Commits on Jul 14, 2026

Changed all instances of normal map to sync.map
Jacob Ho
committed
74cce48
Changed all instances of normal map to sync.map in TEST FILE NOW
Jacob Ho
committed
adef71e

Commits on Jul 17, 2026

LRU Concurrency Design Implementation
Jacob Ho
committed
e9dcb70

Commits on Jul 20, 2026

Fixed infinite loop logic and StopPolicy, old tests now work also test for Shutdown
Jacob Ho
committed
13c608c
Initial wiring of LRU Policy
Jacob Ho
committed
8ac166b

Commits on Jul 21, 2026

Wired LRU Policy with Tiered Storage, Implemented Tests against functionality
Jacob Ho
committed
d6c9a30

Commits on Jul 31, 2026

Fixed DeleteFile and added in logic for edge case in LRU Policy
Jacob Ho
committed
7a58ff6

Commits on Aug 3, 2026

Tests for DeleteFile, still working on LRU Test
Jacob Ho
committed
776d9b3

Commits on Aug 4, 2026

Potential Fix for deadlock in Enqueue and Dequeue, manually unlocked in Release and Delete
Jacob Ho
committed
5a99492
Something might be wrong with my vsCode that commit might not have worked
Jacob Ho
committed
37507f9
Initial implementation of Sync and Flush, they are identical
Jacob Ho
committed
ed6d5ed

Commits on Aug 6, 2026

WOW FINAL COMMIT MESSAGE SO COOL, added initial design of RenameFile, and finished test for FLushFile, THANK YOU GUYS SO MUCH, I hope everything goes well with this component and it gets pushed, Jacob
Jacob Ho
committed
880ab0e

Commits on Aug 14, 2026

Merge pull request #960 from HoJacob/feature/tiered_storage-lru_policy
jfantinhardesty
authored
bb28d60

Commits on Aug 26, 2026

Merge remote-tracking branch 'origin/main' into feature/tiered_storage
foodprocessor
committed
d34b4f4
Add copyright notice
foodprocessor
committed
db88ac7

Commits on Aug 27, 2026

Make handleCount atomic
foodprocessor
committed
c741105
Opus 5: TS 5-10
foodprocessor
committed
1b1631c

Commits on Aug 31, 2026

Comments
foodprocessor
committed
403a1f6
Don't retry eviction
foodprocessor
committed
6f04abc
Cache used space
foodprocessor
committed
9f7825d
Prevent path traversal exploits in local storage
foodprocessor
committed
727fe53

Commits on Sep 1, 2026

Add local tier attribute lookup
foodprocessor
committed
6776993
Persist tiered storage state
foodprocessor
committed
299536c
Honor file creation and open semantics
foodprocessor
committed
ee89ba2
Merge local and cloud directory state
foodprocessor
committed
7b4bfc1
Normalize missing file delete errors
foodprocessor
committed
b883ff8
Expand tiered storage coverage
foodprocessor
committed
0a24d4a
Don't buffer the upload channel
foodprocessor
committed
fe54f7e
Remove resize (trivial wrapper for Add)
foodprocessor
committed
2d76d42
scrub reserveSpace
foodprocessor
committed
8713572
Add comments
foodprocessor
committed
ec77960
Simplify tiered storage LRU state
foodprocessor
committed
67de5b0
Trim tiered storage implementation
foodprocessor
committed
7a9a1ba
Stream tiered storage snapshots
foodprocessor
committed
564cd8a
Simplify tiered storage shutdown
foodprocessor
committed
c793782
Trim tiered storage LRU tests
foodprocessor
committed
ba0a0c2
Combine tiered storage startup scans
foodprocessor
committed
bec401a
Configure tiered storage eviction
foodprocessor
committed
2c2bc8b
Validate tiered storage pipelines
foodprocessor
committed
51202ce
Document tiered storage configuration
foodprocessor
committed
6f34181
Implement tiered storage truncate
foodprocessor
committed
41f6832
Delegate tiered storage symlinks
foodprocessor
committed
9dd752f
Implement tiered storage metadata updates
foodprocessor
committed
1d45ac9
Implement tiered storage directory rename
foodprocessor
committed
ac933b0
Report tiered storage capacity
foodprocessor
committed
7f19ead
Complete tiered storage interfaces
foodprocessor
committed
c228602
Test tiered storage recovery boundaries
foodprocessor
committed
9c877b7
Add tiered storage mount smoke test
foodprocessor
committed
acdfe8d
Address tiered storage lint
foodprocessor
committed
e5eed32

Commits on Sep 4, 2026

Reword readme and change sample config to S3.
foodprocessor
committed
fc31563

Commits on Sep 8, 2026

Raise default thresholds to 80 90
foodprocessor
committed
afe3922
Update README
foodprocessor
committed
6e8cebf

Commits on Sep 10, 2026

Merge branch 'main' into feature/tiered_storage
foodprocessor
committed
ab27843
Fix test lint
foodprocessor
committed
88e9b92
Use subcomponent config pattern like file cache for the lru policy
foodprocessor
committed
3815de0
Use config struct in tests
foodprocessor
committed
b85b650
Spelling
foodprocessor
committed
224a187
Fix test to match new defaults
foodprocessor
committed
4f22b3e
Fix Windows error on absolute path
foodprocessor
committed
acb9dd6
Fix inexact eviction test
foodprocessor
committed
d221a90
Use tracked size to overwrite possibly stale directory listings (especially on Windows)
foodprocessor
committed
1fb38c1

Commits on Sep 22, 2026

Merge branch 'main' into feature/tiered_storage
foodprocessor
committed
1bdbb2b

Commits on Oct 1, 2026

Maintain LRU position on rename, and fix other nodeMap inconsistencies.
foodprocessor
committed
cc8136b
Comments & whitespace
foodprocessor
committed
6e87378
Merge branch 'main' into feature/tiered_storage
foodprocessor
committed
9880c46
Rename local files first, and don't revert on cloud failure.
foodprocessor
committed
af2cc31
Don't lock for GetAttr
foodprocessor
committed
282708c
Limit access to our internal cache.
foodprocessor
committed
57a89fb
Fix tests to fit Windows semantics
foodprocessor
committed
a9be77b