Persist performance source directories and display preferred paths by tomvothecoder · Pull Request #373 · E3SM-Project/simboard

tomvothecoder · 2026-10-07T21:46:17Z

Description

Closes #362.

Persist observed case/execution performance directories with staging/archive kind, ownership constraints, cascading cleanup, and concurrency-safe deduplication. Retain all distinct observed locations without changing catalog identity.
Record mappings during ingestion and visited-directory discovery, including duplicate-only and skipped-upload flows. Preserve dry runs and checkpoint pruning; report unresolved owners without fabricating records.
Expose mappings and one preferred directory through case/execution detail APIs. Display the archive-first path under Case Metadata and execution Overview → Provenance, with a copy action, temporary “Copied” feedback, accessible status announcements, and “Not recorded” when absent.
Add illustrative directories to every seeded case and execution, migration/lifecycle/API/seed coverage, and mapping documentation.
Fix host-launcher portability on macOS/Bash 3.2/BSD tools while preserving locking, credential redaction, failure statuses, and logging after console-pipe closure.

Checklist

Code follows project style guidelines
Self-reviewed code
No new warnings — existing SQLAlchemy savepoint warnings and Vite bundle-size warning remain
Tests added or updated (if needed)
All tests pass (locally and CI/CD) — local checks passed; CI pending
Documentation/comments updated (if needed)
Breaking change noted (if applicable) — additive API fields; database migration required

Validation performed

make backend-test: 1,394 passed, two SQLAlchemy warnings, on macOS with Bash 3.2. Includes migration upgrade/downgrade, concurrent persistence, ownership isolation, duplicate/skipped uploads, archive precedence, reseeding cleanup, and launcher regressions.
make frontend-lint and make frontend-build: passed (existing large-bundle warning).
pnpm run type-check and Prettier checks of changed frontend files, from frontend/: passed, including the latest copy-feedback change.
backend/.venv/bin/pre-commit run --all-files, from repository root: passed.
Bash syntax check and full branch git diff --check: passed.
Backend suite was run before the final frontend-only copy-feedback commit; frontend checks were rerun afterward.
Browser interaction/responsive verification and screenshots are unavailable. Linux execution was not independently run.

Deployment Notes (if any)

Apply make backend-upgrade before deploying updated scanners/backend. Deploy the metadata-submission API before updated runners depend on it.
make backend-seed reloads example directories for all seeded cases/executions and replaces existing seeded records and edits.
Existing records without paths remain unrecorded until observed again. Previously checkpointed snapshots remain skipped; historical backfill is outside this PR.
Paths are observations, not availability guarantees. Among multiple archive paths, stable path ordering selects the display value, not observation recency. All paths remain available through the API.
Real host runs still require flock; missing locking support fails explicitly rather than proceeding unlocked. Launcher timestamps now consistently use UTC.

…t#362)

…#362)

)

…ject#362)

…ples (E3SM-Project#362)

…ct#362)

Copilot

🟡 Changes recommended

The architecture documentation incorrectly states that the UI displays every retained mapping rather than only the preferred directory.

1 open finding

Clarify UI displays only the archive-first source directory · New

What changed in this PR

Adds persistent performance-source directory mappings across ingestion, catalog APIs, seed data, and detail views, while improving launcher portability.

Changes:

Adds archive/staging mappings with idempotent persistence and preferred-path selection.
Records mappings during ingestion and exposes them through detail APIs.
Displays preferred paths with copy feedback and expands regression coverage.

File	Description
`Makefile`	Makes site validation portable.
`frontend/src/types/catalog.ts`	Adds source-directory API types.
`frontend/src/features/catalog/components/PerformanceSourceDirectory.tsx`	Renders and copies preferred paths.
`frontend/src/features/catalog/components/ExecutionDetailsView.tsx`	Adds execution provenance path.
`frontend/src/features/catalog/CaseDetailsPage.tsx`	Adds case metadata path.
`docs/github-issues/362-persist-source-dirs/plan.md`	Documents implementation plan.
`docs/architecture/metadata-ingestion.md`	Documents mapping semantics.
`backend/tests/features/ingestion/test_source_directory_api.py`	Tests submission and ingestion APIs.
`backend/tests/features/ingestion/test_site_collection_launcher.py`	Tests launcher portability and logging.
`backend/tests/features/ingestion/test_nersc_archive_ingestor.py`	Stubs mapping persistence.
`backend/tests/features/ingestion/test_machine_ingestion.py`	Adds locking fixture support.
`backend/tests/features/ingestion/test_lcrc_v3_archive_ingestor.py`	Stubs targeted-run persistence.
`backend/tests/features/ingestion/test_hpc_upload_archive_ingestor.py`	Stubs upload-runner persistence.
`backend/tests/features/ingestion/test_diagnostics_make_targets.py`	Adds locking fixture support.
`backend/tests/features/ingestion/test_archive_source_directories.py`	Tests scanner mapping workflows.
`backend/tests/features/ingestion/source_directory_utils.py`	Provides mapping test stubs.
`backend/tests/features/catalog/test_source_directory_migration.py`	Tests migration reversibility.
`backend/tests/features/catalog/test_source_directories.py`	Tests persistence, selection, and concurrency.
`backend/tests/features/catalog/test_seed.py`	Tests seeded mappings and cleanup.
`backend/migrations/versions/20261006_000000_add_source_directories.py`	Creates mapping storage.
`backend/app/scripts/README.md`	Documents scanner behavior.
`backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh`	Improves portable logging and locking.
`backend/app/scripts/ingestion/nersc_archive_ingestor.py`	Persists visited NERSC paths.
`backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py`	Persists visited upload paths.
`backend/app/scripts/ingestion/archive_source_directories.py`	Implements scanner-side persistence.
`backend/app/scripts/ingestion/archive_discovery.py`	Collects visited execution paths.
`backend/app/scripts/db/seed.py`	Seeds source mappings.
`backend/app/scripts/db/catalog.json`	Adds illustrative paths.
`backend/app/main.py`	Registers the new router.
`backend/app/features/ingestion/source_directories.py`	Resolves observed owners.
`backend/app/features/ingestion/schemas.py`	Adds mapping payload schemas.
`backend/app/features/ingestion/api.py`	Persists explicit ingestion mappings.
`backend/app/features/catalog/source_directories.py`	Implements idempotent persistence and preference.
`backend/app/features/catalog/schemas.py`	Exposes mapping response fields.
`backend/app/features/catalog/models.py`	Adds mapping model and relationships.
`backend/app/features/catalog/enums.py`	Defines staging/archive kinds.
`backend/app/features/catalog/api.py`	Loads and serializes mappings.
`backend/app/api/source_directories.py`	Adds trusted metadata submission endpoint.

🧠 Review effort: Balanced

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

+different archive snapshots. Case and execution detail APIs return
+`sourceDirectories`; both detail pages display these paths with copy actions.


tomvothecoder added 8 commits October 7, 2026 14:25

Add performance source directory storage (E3SM-Project#362)

e65a67c

Record source directories during ingestion and discovery (E3SM-Projec…

62682ce

…t#362)

Expose source directories in case and execution details (E3SM-Project…

4f83d58

…#362)

Display performance source directories on detail pages (E3SM-Project#362

89d9f81

)

Validate and document performance source directory mappings (E3SM-Pro…

7f2ef0f

…ject#362)

Show preferred performance source directory in metadata and seed exam…

47f5b63

…ples (E3SM-Project#362)

Fix host launcher portability on macOS (E3SM-Project#362)

6e57ca7

Show feedback when copying performance source directories (E3SM-Proje…

715499d

…ct#362)

tomvothecoder requested a balanced review from Copilot October 7, 2026 21:46

Copilot started reviewing on behalf of tomvothecoder October 7, 2026 21:47 View session

Copilot AI reviewed Oct 7, 2026

View reviewed changes

Comment thread docs/architecture/metadata-ingestion.md Outdated

Comment on lines +81 to +82

different archive snapshots. Case and execution detail APIs return

`sourceDirectories`; both detail pages display these paths with copy actions.

Display all preferred case performance directories (E3SM-Project#362)

44920b9

tomvothecoder marked this pull request as ready for review October 7, 2026 22:25

tomvothecoder merged commit 9c406e1 into E3SM-Project:main Oct 7, 2026
1 check passed

tomvothecoder deleted the devops/362-persist-source-dirs branch October 7, 2026 22:25

tomvothecoder mentioned this pull request Oct 8, 2026

[DevOps]: Develop a system for backfilling new metadata for already ingested cases #360

Open

Navigation Menu

Repository navigation

Persist performance source directories and display preferred paths - #373

Persist performance source directories and display preferred paths#373
tomvothecoder merged 9 commits into
E3SM-Project:mainfrom
tomvothecoder:devops/362-persist-source-dirs

tomvothecoder commented Oct 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		different archive snapshots. Case and execution detail APIs return
		`sourceDirectories`; both detail pages display these paths with copy actions.

Conversation

tomvothecoder commented Oct 7, 2026

Description

Checklist

Validation performed

Deployment Notes (if any)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants