Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
7d83ff9
begin scaffolding
szvsw Mar 6, 2026
8a7d36e
use scythe to manage outer loop experiment
szvsw Mar 7, 2026
58058fa
enable results cache growing
szvsw Mar 7, 2026
3707ece
clean up some vestiges and update scythe
szvsw Mar 7, 2026
79f0d19
improve data management
szvsw Mar 7, 2026
44338de
begin training
szvsw Mar 8, 2026
4f9223a
enable xgboost training
szvsw Mar 8, 2026
8963690
enable recursive convergence checks
szvsw Mar 8, 2026
7db748f
reorganize some files
szvsw Mar 8, 2026
bb657b3
reorganize modules
szvsw Mar 8, 2026
31663da
enable runnable selection at surrogate level
szvsw Mar 9, 2026
3af3669
improve results collation and finalization
szvsw Mar 9, 2026
6603b70
add non linear sample size scaling
szvsw Mar 9, 2026
b3ebb8f
enabling sampling specification
szvsw Mar 9, 2026
5e277f7
vestigial cleanup
szvsw Mar 9, 2026
d7d0105
lazily import scythe to prevent docs error
szvsw Mar 9, 2026
5474f63
easier dummy fn
szvsw Mar 9, 2026
b49a701
add a bunch of log statements
szvsw Mar 9, 2026
a15f8eb
simplify dummy problem
szvsw Mar 9, 2026
49553c2
oops...
szvsw Mar 9, 2026
89f2ba4
fix insane race condition
szvsw Mar 10, 2026
cfb0aa7
enable gpu indicator for tasks
szvsw Mar 10, 2026
4cc6653
update gitignore
szvsw Mar 10, 2026
1db2fd3
add defaults
szvsw Mar 10, 2026
676a9b9
restore old env args
szvsw Mar 10, 2026
8d417db
drop torch temporarily
szvsw Mar 10, 2026
5334287
re-enable torch
szvsw Mar 10, 2026
1cbb40f
improve training with additional annual outputs, better target and fe…
szvsw Mar 11, 2026
afaf216
backup training tester script
szvsw Mar 11, 2026
7f1c8ae
add test dataset for training
szvsw Mar 11, 2026
36d3e0b
make b*9ucket prefix configurable in sst deploy
szvsw Mar 13, 2026
413691b
add better zoning control and lgb training
szvsw Mar 24, 2026
5feaa87
clean up some todos
szvsw Mar 25, 2026
b47bccc
refactor into a common pattern for backends
szvsw Mar 26, 2026
29a5c48
Refactor NN backend to support linear/quadratic regression and to imp…
szvsw Mar 27, 2026
587a9cd
Add configurable continuous feature encoding to surrogate pipeline (#32)
szvsw Mar 27, 2026
3dff6c6
Add nMBE as surrogate training metric and convergence threshold (#34)
szvsw Mar 27, 2026
e8b044c
add prefix sep
szvsw Mar 28, 2026
79dbc43
use automatic scythe comptued feature injection
szvsw Mar 28, 2026
08d552f
enable reading parquet files
szvsw Mar 28, 2026
65b81cf
add some computed features
szvsw Mar 28, 2026
08e6d4f
enable automatic calculation of recursion map factor size in simulati…
szvsw Mar 28, 2026
6bfbb55
add some tqdm logging
szvsw Mar 28, 2026
f05b27f
allow deserializing wkb
szvsw Mar 29, 2026
ddcf869
improve support for wkbs
szvsw Mar 29, 2026
4b738ec
fix broken file references in e2e test
szvsw Mar 29, 2026
76abcc9
fix serialization of Polygons
szvsw Mar 29, 2026
5fa7400
fix broken reference
szvsw Mar 29, 2026
5a35cd6
improve logging
szvsw Mar 29, 2026
e046bd0
fix fragmentation issues
szvsw Mar 30, 2026
08ce3b0
fix bug in local transformer loading
szvsw Mar 31, 2026
07ae33e
improve memory efficiency of multi-condition samplers
szvsw Apr 30, 2026
7e39bfd
change index preservation rules
szvsw Jun 8, 2026
790417f
add building area limits
szvsw Jun 8, 2026
c2d84e4
update scythe ref
szvsw Jun 8, 2026
58adc29
add scenario case matching to allocator
szvsw Jun 19, 2026
38bd6c5
allow simulations to sit in queue for much longer
szvsw Jun 20, 2026
0c7b71f
add min data in leaf flag and small package updates
daryaguettler Sep 21, 2026
f947764
move building specs into gis
daryaguettler Sep 21, 2026
31fbe3a
add local metrics
daryaguettler Sep 21, 2026
039e872
add local validation tester
daryaguettler Sep 21, 2026
873badf
update lock file
daryaguettler Sep 21, 2026
1570e56
add local test files
daryaguettler Sep 21, 2026
93550bb
make small error message updates and update training workflow registry
daryaguettler Oct 1, 2026
10b5b31
add tests and update github workflow
daryaguettler Oct 1, 2026
d520a57
update ci
daryaguettler Oct 1, 2026
eb103cb
add docs
daryaguettler Oct 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .cursor/rules/hatchet-docs.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
description: Hatchet documentation MCP server
alwaysApply: true
---

When working with Hatchet (task queues, workflows, durable execution), use the
Hatchet MCP docs server for accurate, up-to-date API reference and examples.

MCP server URL: https://docs.hatchet.run/api/mcp

Use the search_docs tool to find relevant documentation pages, or get_full_docs
for comprehensive context. Documentation covers Python, TypeScript, and Go SDKs.
9 changes: 9 additions & 0 deletions .env.scythe.training
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
SCYTHE_WORKER_SLOTS=1
SCYTHE_WORKER_DOES_FAN=False
SCYTHE_WORKER_DOES_LEAF=True
SCYTHE_WORKER_HAS_GPU=True

SCYTHE_TIMEOUT_EXPERIMENT_SCHEDULE=2h
SCYTHE_TIMEOUT_EXPERIMENT_EXECUTION=1h
SCYTHE_TIMEOUT_SCATTER_GATHER_SCHEDULE=10h
SCYTHE_TIMEOUT_SCATTER_GATHER_EXECUTION=10h
Binary file added .github/.DS_Store
Binary file not shown.
10 changes: 10 additions & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ jobs:
with:
python-version: ${{ matrix.python-version }}

- name: Generate Prisma client
run: uv run epi prisma generate

- name: Check typing
run: uv run pyright

Expand All @@ -64,5 +67,12 @@ jobs:
- name: Set up the environment
uses: ./.github/actions/setup-python-env

- name: Check out globi-data (for the generated data pages)
uses: actions/checkout@v4
continue-on-error: true
with:
repository: MITSustainableDesignLab/globi-data
path: .globi-data

- name: Check if documentation can be built
run: uv run mkdocs build -s
7 changes: 7 additions & 0 deletions .github/workflows/on-release-main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,5 +80,12 @@ jobs:
restore-keys: |
mkdocs-material-

- name: Check out globi-data (for the generated data pages)
uses: actions/checkout@v4
continue-on-error: true
with:
repository: MITSustainableDesignLab/globi-data
path: .globi-data

- name: Deploy documentation
run: uv run mkdocs gh-deploy --force
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -217,3 +217,6 @@ inputs/

.env.local.hatchet
.env.local.host.hatchet

scratch/
.globi-data/
25 changes: 25 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ docs-test: ## Test if documentation can be built without warnings or errors
docs: ## Build and serve the documentation
@uv run mkdocs serve

.PHONY: docs-internal
docs-internal: ## Serve the docs plus internal pages from GLOBI_INTERNAL_DOCS (never deploy these)
@test -n "$(GLOBI_INTERNAL_DOCS)" || (echo "Set GLOBI_INTERNAL_DOCS to the private discussion folder" && exit 1)
@uv run mkdocs serve

.PHONY: docs-deploy
docs-deploy: ## Build and serve the documentation
@uv run mkdocs gh-deploy
Expand All @@ -71,10 +76,30 @@ simulations-native: ## Run the simulations
fanouts-native: ## Run the fanouts
@uv run --env-file .env.$(AWS_ENV).aws --env-file .env.$(HATCHET_ENV).hatchet --env-file .env.scythe.storage --env-file .env.scythe.fanouts worker

.PHONY: training-native
training-native: ## Run the training
@uv run --env-file .env.$(AWS_ENV).aws --env-file .env.$(HATCHET_ENV).hatchet --env-file .env.scythe.storage --env-file .env.scythe.training worker

.PHONY: viz-native
viz-native: ## Run the visualization tool # TODO: possibly add env vars to the command
@uv run streamlit run src/globi/tools/visualization/main.py

##################### Local Surrogate Testing (no Hatchet / S3) #####################
SURROGATE_CONFIG ?= tests/data/surrogate_local/validation-dummy.yml
SURROGATE_WORKERS ?= 1

.PHONY: surrogate-local
surrogate-local: ## Train surrogate(s) locally, in-process. SURROGATE_CONFIG=... SURROGATE_WORKERS=N
@uv run globi tests surrogate-local --config $(SURROGATE_CONFIG) --workers $(SURROGATE_WORKERS)

.PHONY: surrogate-validate
surrogate-validate: ## Validate trained surrogate(s) against deterministic EnergyPlus runs
@uv run globi tests surrogate-validate --config $(SURROGATE_CONFIG) --workers $(SURROGATE_WORKERS)

.PHONY: test-energyplus
test-energyplus: ## Run the pytest suite including the EnergyPlus-backed surrogate test
@uv run pytest --run-energyplus

.PHONY: env-debug
env-debug: ## Merge env files into .env.debug for the VS Code/Cursor debugger
@uv run python scripts/merge_env_for_debug.py
Expand Down
25 changes: 25 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,31 @@ services:
- ./inputs:/code/inputs
- ./outputs:/code/outputs
- ./tests/data/e2e:/code/tests/data/e2e
training:
image: ${AWS_ACCOUNT_ID:-123456789012}.dkr.ecr.${AWS_REGION:-us-east-1}.amazonaws.com/hatchet/globi:${IMAGE_TAG:-latest}
build:
context: .
dockerfile: src/globi/worker/Dockerfile
args:
EP_VERSION: ${EP_VERSION:-25.2.0}
PYTHON_VERSION: ${PYTHON_VERSION:-3.12}
env_file:
- .env
- .env.${AWS_ENV:-local}.aws
- .env.${HATCHET_ENV:-local}.hatchet
- .env.scythe.storage
- .env.scythe.training
deploy:
mode: replicated
replicas: ${TRAINING_REPLICAS:-0}
resources:
reservations:
devices:
- capabilities: [gpu] # Requests access to all GPUs
volumes:
- ./inputs:/code/inputs
- ./outputs:/code/outputs
- ./tests/data/e2e:/code/tests/data/e2e
fanouts:
image: ${AWS_ACCOUNT_ID:-123456789012}.dkr.ecr.${AWS_REGION:-us-east-1}.amazonaws.com/hatchet/globi:${IMAGE_TAG:-latest}
build:
Expand Down
Binary file added docs/.DS_Store
Binary file not shown.
86 changes: 86 additions & 0 deletions docs/data/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Data & outputs

globi runs the simulations. Their input files and the records of their results
are kept in a separate repository,
[globi-data](https://github.com/MITSustainableDesignLab/globi-data). This
section explains how that repository is organized and shows what is in it.
Most pages here are **generated from globi-data at build time**, so they show
the data as it stands rather than a copy that can drift.

## Where everything lives

```mermaid
flowchart LR
subgraph src["Partner source data (private)"]
cfg["partner working files"]
gis[("building footprints<br/>(GIS, ~1 GB each)")]
end
subgraph gd["globi-data (git)"]
inp["inputs/partners/&lt;Partner&gt;/vNNN<br/>frozen input bundles"]
out["outputs/runs/…/vNNN<br/>output-version.yml"]
sur["outputs/surrogates/…/vNNN<br/>surrogate-version.yml"]
end
subgraph s3["S3 · ml-for-bem"]
pq[("final/*.pq<br/>~11 GB of results")]
mdl[("surrogate model")]
end
globi(["globi<br/>simulation code"])

cfg -- "globi-data inputs new" --> inp
gis -. "size + sha256 only" .-> inp
inp --> globi
globi -- "EnergyPlus runs" --> pq
pq -. "where + fingerprints" .-> out
inp -. "pinned by sha256" .-> out
globi -- "training" --> mdl
mdl -. "where + metrics" .-> sur
inp -. "pinned by sha256" .-> sur
```

Solid arrows move files. Dashed arrows record facts about files that stay
where they are.

Two things decide where a file lives: whether it may be published, and how
big it is. Building footprints are partner data and are not published.
Simulation results are too large for git. For both, globi-data records a
fingerprint (sha256), which is enough to prove later that a result came from
particular inputs without publishing the inputs themselves.

| What | Stored in | Size | Versioned by |
| ----------------------------------------------------------------------------------- | -------------------- | -------------------- | ----------------------------------------------------------------- |
| Input bundles (component library, semantic fields, component map, weather, configs) | globi-data `inputs/` | a few MB per partner | `inputs/partners/<Partner>/vNNN/` |
| Building footprints (GIS) | not published | ~1 GB per partner | size and sha256 recorded in each input version's `external_files` |
| Simulation results (parquet) | S3 | ~11 GB in total | recorded in `outputs/runs/…/vNNN/output-version.yml` |
| Trained surrogate models | S3 | varies | recorded in `outputs/surrogates/<Partner>/vNNN/` |

## Pages in this section

- [**Output structure**](outputs-structure.md) explains how results are
organized, what each output record contains, and how results are marked
stale.
- [**Run catalog**](runs/index.md) lists every recorded run, with a coverage
matrix, a timeline, and a page per partner.
- [**Results status**](status.md) shows which results still match the current
inputs.
- [**Surrogate models**](surrogates.md) lists the trained surrogates and their
feature contracts.
- [**Input bundles**](inputs.md) shows the current input version for each
partner.
- **Reference** holds globi-data's own manifest schema and how-to guides,
pulled in unchanged.

## Building these pages

The generated pages come from a local checkout of globi-data. The build looks
for one at `$GLOBI_DATA_ROOT`, then `../globi-data` and `../../globi-data`
(relative to `mkdocs.yml`), then `.globi-data`:

```bash
export GLOBI_DATA_ROOT=~/globi-data
make docs # uv run mkdocs serve
```

If no checkout is found, the build still succeeds and each generated page says
the data was missing. Set `GLOBI_DATA_REQUIRED=1` to make that an error
instead. The generator is
[`mkdocs_hooks/globi_data.py`](https://github.com/mitsustainabledesignlab/globi/blob/main/mkdocs_hooks/globi_data.py).
Loading