Repository navigation

Improve performance of new transform for large tables - #68

Merged

brycekbargar merged 18 commits into

library-data-platform:release-v4.0.0from

Five-Colleges-Incorporated:performance-testing

Mar 10, 2026

Merged

Improve performance of new transform for large tables#68
brycekbargar merged 18 commits into
library-data-platform:release-v4.0.0from
Five-Colleges-Incorporated:performance-testing

Commits on Mar 9, 2026
- Refactor metadata collection to be per property
  brycekbargar
  committed
  1d7d684
- Monitor progress of transformation
  brycekbargar
  committed
  1fee373
- Optimize the definitions of postgres json operators
  brycekbargar
  committed
  b37dd1f
- Simplify checks for specific data types
  brycekbargar
  committed
  0fe5986
- Allow postgres to short circuit data type checks
  brycekbargar
  committed
  f6a5978
- Sample large tables for column metadata
  brycekbargar
  committed
  4706f98
- Split progress bars between scanning and transforming
  brycekbargar
  committed
  b07ec9f
- Remove slow jextract function in favor of more native functionality
  brycekbargar
  committed
  c6081bf
- Find all possible keys in json
  brycekbargar
  committed
  8d9a61c
- Remove sampling
  brycekbargar
  committed
  b63f7e2
- Actually drop the tables in the prefix
  brycekbargar
  committed
  c23c6af
- Dont carry over jsonb column
  brycekbargar
  committed
  7d04ee3
- Simplify the query to get property names
  brycekbargar
  committed
  f619eb6
- Ignore empty source tables
  brycekbargar
  committed
  00775d1
- Better display of iterators during transform
  brycekbargar
  committed
  523bd2c
Commits on Mar 10, 2026
- Actually fix the indexing
  brycekbargar
  committed
  76d92b2
- Dont keep tqdm progress bars around
  brycekbargar
  committed
  9936f85
- Be more specific about when to include the object column name
  brycekbargar
  committed
  3fa7cc9