Skip to content

feat: add permutation feature importance to JointFMClient - #12

Open
mariusvilkas wants to merge 3 commits into
mainfrom
marius/feature_importance
Open

feat: add permutation feature importance to JointFMClient#12
mariusvilkas wants to merge 3 commits into
mainfrom
marius/feature_importance

Conversation

@mariusvilkas

@mariusvilkas mariusvilkas commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Adds JointFMClient.feature_importance(), which permutes each feature column across history rows and compares the resulting forecast against a shared baseline forecast_samples() call. Returns one dict per feature, keyed by metric ("mean" absolute shift, "distance" centered squared 2-Wasserstein) then target then horizon, so callers get both scores across every target/horizon combination without flattening rows themselves.

I validated locally with script

from jointfm_client import ColumnSpec, DataFrameSchema, JointFMClient

history_rows = [
    {"t": 0, "sales": 10.0, "promo_spend": 100.0},
    {"t": 1, "sales": 12.0, "promo_spend": 120.0},
    {"t": 2, "sales": 13.5, "promo_spend": 90.0},
    {"t": 3, "sales": 15.0, "promo_spend": 150.0},
]
schema = DataFrameSchema(
    columns=(
        ColumnSpec(name="sales", modality="numeric", role="target"),
        ColumnSpec(name="promo_spend", modality="numeric", role="feature"),
    ),
    time_index_mode="ordinal",
    time_column="t",
)

client = JointFMClient.from_env()

importance = client.feature_importance(
    history_rows,
    schema=schema,
    query_times=[4, 5, 6],           # 3 forecast steps ahead (ordinal)
    horizons=[1, 2, 3],              # labels for those same 3 steps, positionally paired
    feature_columns=["promo_spend"], # columns to permute and score
    target_columns=["sales"],        # columns to score against
    n_samples=512,
    seed=7,
)

for entry in importance:
    print(entry["feature"])
    print("mean shift:", entry["mean"]["sales"])      # {1: ..., 2: ..., 3: ...}
    print("distance:  ", entry["distance"]["sales"])  # {1: ..., 2: ..., 3: ...}

On response got raw (not normalized values)

marius@marius-le-LG6HP:~/workspace/joint-client-python$ task run -- python my_script.py
promo_spend
mean shift: {1: 0.05699951387941837, 2: 0.07691273093223572, 3: 0.06873160041868687}
distance:   {1: 0.0006692017906518421, 2: 0.0010064832108240829, 3: 0.0013303745023048718}

Note

Low Risk
Additive SDK surface on the existing forecast path with unit tests; main operational note is extra prediction calls (one baseline plus one per feature column).

Overview
Adds JointFMClient.feature_importance(...), a permutation-based importance workflow on top of sample forecasts. The client runs one shared baseline forecast(..., return_mode="samples"), then for each listed feature shuffles that column across history rows (marginal preserved) and re-forecasts with the same seed and n_samples.

Results are one dict per feature with mean (absolute shift in forecast mean) and distance (centered squared 2-Wasserstein between baseline and permuted samples), nested by target and horizon label. horizons must align positionally with query_times. DataFrame history plus an explicit schema is converted to rows before forecasting so it matches the row-payload path.

New jointfm_client.feature_importance helpers implement column permutation (row dicts or pandas), sample_w2_distance, and scoring assembly. API reference documents the method; tests cover end-to-end client behavior, validation errors, and helper edge cases.

Reviewed by Cursor Bugbot for commit 3d9e02d. Bugbot is set up for automated code reviews on this repo. Configure here.

Adds JointFMClient.feature_importance(), which permutes each feature column
across history rows and compares the resulting forecast against a shared
baseline forecast_samples() call. Returns one dict per feature, keyed by
metric ("mean" absolute shift, "distance" centered squared 2-Wasserstein)
then target then horizon, so callers get both scores across every
target/horizon combination without flattening rows themselves.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Comment thread src/jointfm_client/feature_importance.py Outdated
The presence check only looked at rows[0], then row[feature] was read
unconditionally on every row, so a sparse/nullable history row that omits
the permuted column later in the sequence raised KeyError even though
forecast() already accepts that shape (it only requires a declared column
to appear in some row, not every row). Now presence is checked across all
rows, and only rows where the key is present are shuffled; rows without it
are left untouched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 62dff02. Configure here.

Comment thread src/jointfm_client/client.py
forecast() always takes the row-payload path when schema is set, even for
DataFrame history, so passing a DataFrame together with a DataFrameSchema
(a combination permute_history_column already supports) raised "history_rows
must be a JSON array" before any permutation ran. _sample_forecast_for_importance
now converts the frame to rows itself via dataframe_to_history_rows in that
case, so callers who already have a schema and a frame don't have to convert
rows by hand.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant