Repository navigation

Reduce data duplication and temp tables during transform - #72

Merged

brycekbargar merged 28 commits into

library-data-platform:release-v4.0.0from

Five-Colleges-Incorporated:fewer-temp-tables

Mar 27, 2026

Merged

Reduce data duplication and temp tables during transform#72
brycekbargar merged 28 commits into
library-data-platform:release-v4.0.0from
Five-Colleges-Incorporated:fewer-temp-tables

Commits on Mar 23, 2026
- Begin the rewrite with new Node class and implement the fixed value
  brycekbargar
  committed
  d2de294
- Implement getting the keys and types for objects
  brycekbargar
  committed
  d8dcbda
- Remove the identity node type
  brycekbargar
  committed
  25ae3e5
Commits on Mar 24, 2026
- Implement staging the array as a temp file and getting the type
  brycekbargar
  committed
  1c460d6
- Implement the new simplified transform algorithm
  brycekbargar
  committed
  c1c2356
- Track scan progress
  brycekbargar
  committed
  755d631
- Refactor sql statement generation location
  brycekbargar
  committed
  0427c60
- Share more of the json object traversal code
  brycekbargar
  committed
  a016de6
- Skeleton out the create table sql statements
  brycekbargar
  committed
  dd40621
- Build sql with output tables and columns
  brycekbargar
  committed
  154644f
- Clear the indexing time when the transform finishes
  brycekbargar
  committed
  8e00886
- Refactor to use the rewrite expansion
  brycekbargar
  committed
  b3bed23
- Use postgres sql format properly
  brycekbargar
  committed
  04b41c9
- Add a duckdb jsonb_each shim
  brycekbargar
  committed
  3cf5217
- Various syntax fixes
  brycekbargar
  committed
  60d158e
- Fix aliasing for basic datatypes column test
  brycekbargar
  committed
  df0f2f8
- Fix snake case naming
  brycekbargar
  committed
  72449e9
- Fix basic object expansion
  brycekbargar
  committed
  bc7c490
Commits on Mar 25, 2026
- WIP: Making array expansion work
  brycekbargar
  committed
  5d55799
- Infer NodeContext from Node structure
  brycekbargar
  committed
  1f521f2
- Implement json_depth
  brycekbargar
  committed
  a7dc2f2
- Replace legacy version with the rewrite
  brycekbargar
  committed
  2552364
Commits on Mar 26, 2026
- Remove delete from returning construct
  brycekbargar
  committed
  7d25e53
- Inline the ANALYZE statements necessary for the transform
  brycekbargar
  committed
  351c9fa
- Invert json_depth check and simplify unparenting empty arrays
  brycekbargar
  committed
  7955d21
- Analyze important column in raw table for postgres
  brycekbargar
  committed
  870dc1c
- Re-enable flaky source records CI test
  brycekbargar
  committed
  e2bec99
- Cleanup transform_progress
  brycekbargar
  committed
  82a7ffd

Commits

Commits on Mar 23, 2026

Commits on Mar 24, 2026

Commits on Mar 25, 2026

Commits on Mar 26, 2026