Skip to content

fix(create): default the LiteLLM harness model to a keyless bedrock model - #2168

Merged
aidandaly24 merged 3 commits into
aws:refactorfrom
aidandaly24:fix/litellm-harness-transcript
Sep 2, 2026
Merged

fix(create): default the LiteLLM harness model to a keyless bedrock model#2168
aidandaly24 merged 3 commits into
aws:refactorfrom
aidandaly24:fix/litellm-harness-transcript

Conversation

@aidandaly24

@aidandaly24 aidandaly24 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #2150, which fixed the harness-first create defects the #2146 bug bash turned up. This one fixes the LiteLLM default that same bug bash surfaced.

The default LiteLLM model required a key the CLI never asks for

Before: HARNESS_DEFAULT_MODEL_IDS.lite_llm was anthropic/claude-sonnet-4-5. That prefix routes LiteLLM at Anthropic directly, which needs an Anthropic API key — but apiKeyArn is optional for LiteLLM, so nothing stopped you. A default project create --model-provider lite_llm deployed to READY and then failed its first invoke with litellm.AuthenticationError: Missing Anthropic API Key.

After: the default derives from DEFAULT_HARNESS_MODEL, giving bedrock/global.anthropic.claude-sonnet-4-6. A bedrock/ prefix routes LiteLLM at Bedrock, which it signs with SigV4 from the harness execution role — no API key, no apiKeyArn. Deriving rather than hardcoding also stops the bedrock and lite_llm defaults drifting apart.

Confirmed live in us-east-1:

model id apiKeyArn result
anthropic/claude-sonnet-4-5 omitted LiteLLM AuthenticationError
bedrock/global.anthropic.claude-sonnet-4-6 omitted authenticated via IAM, end_turn, nonzero outputTokens
bedrock/us.amazon.nova-lite-v1:0 omitted authenticated via IAM, end_turn, nonzero outputTokens

This matches the documented contract — the Harness models guide states LiteLLM models using the bedrock/ prefix use the execution role and need no API key, while direct providers require apiKeyArn. anthropic/... keeps requiring a key; only bedrock/... is keyless.

The wizard shipped its own copy of that default

Found while fixing the above, and it would have silently undone it for TUI users.

create/screen.tsx declared defaultModelId per provider, duplicating all four values including lite_llm: "anthropic/claude-sonnet-4-5". Fixing the flag path alone left the wizard pre-filling the broken id — and that value is form state, not placeholder text, so pressing enter through the wizard deployed it:

 model id
 the litellm model to use
 ╭──────────────────────────────────────────╮
 │❯ anthropic/claude-sonnet-4-5             │
 ╰──────────────────────────────────────────╯

HARNESS_DEFAULT_MODEL_IDS is now exported and supplies both the pre-filled value and the placeholder; MODEL_PROVIDERS describes only labels. The flag path and the wizard read one table, so they cannot disagree again.

Testing

  • bun test: 2735 pass, 0 fail. typecheck, lint:check, format:check clean.
  • project.test.ts already pins the flag path's litellm default, and after this change the wizard reads the same table, so that one assertion covers both paths. No new wizard test: MODEL_PROVIDERS no longer declares a per-provider default, so there is no second literal left to drift.
  • Live: project create --model-provider lite_llm writes "modelId": "bedrock/global.anthropic.claude-sonnet-4-6", and that harness authenticates and completes. All AWS test resources were deleted.

Note on the empty transcript, for anyone who tested this earlier

Until today, agentcore harness invoke rendered no assistant text for a lite_llm harness — the call
succeeded with end_turn and nonzero outputTokens, but nothing appeared. That was a harness-side
contract bug, not a CLI one: the LiteLLM provider path omitted contentBlockIndex, which
invokeHarness.smithy marks @required on all three content-block events, and the CLI correctly
rejects events that lack it.

The harness team had already fixed it (LoopyRuntimeArtifacts, "Emit contentBlockIndex as it is part
of the invoke harness contract") on 2026-08-27. It reached us-east-1 on 2026-09-02 at 16:37Z. Our
capture was taken on 2026-09-01, before the wave arrived, which is why we saw it.

No CLI change is needed for that, and this branch carries none. Mentioning it only because a reviewer
who tried a LiteLLM harness in the last few days may remember an empty transcript.

@github-actions github-actions Bot added the size/m PR size: M label Sep 2, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Sep 2, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 2, 2026

@agentcore-devx-automation agentcore-devx-automation Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentCore Harness Review

Verdict: Looks good

Two well-scoped changes, both with clear reasoning and tests:

  1. transcript.tsx — unindexed contentBlockDelta events for text/reasoning now fold into block 0 (instead of being dropped), while unindexed toolUse/toolResult deltas are still ignored. contentBlockStop falls back to 0 rather than -1. The reordering of the guard (delta === undefined first, then the kind-specific handling with an inner contentBlockIndex === undefined short-circuit before toolUse/toolResult) is correct: text/reasoning get the ?? 0 treatment while tool deltas keep requiring a real index.

  2. create/index.ts + screen.tsx — the wizard now sources per-provider defaults from the exported HARNESS_DEFAULT_MODEL_IDS, eliminating the wizard/flag-path drift. The litellm default flips from anthropic/claude-sonnet-4-5 (needs an API key the wizard never prompts for) to bedrock/${DEFAULT_HARNESS_MODEL.modelId} (keyless), which matches the wizard's UX of leaving apiKeyArn/apiBase optional. The new create.screen.test.tsx case explicitly asserts the bedrock/ prefix so a future revert can't slip through, and project.test.ts was updated to match.

One minor behavioral note (not a blocker): with the new fallback, an unindexed contentBlockStop now settles whatever occupies block 0. In a mixed-indexing stream that could target the wrong block, but real providers emit either all-indexed or all-unindexed events, so this is the pragmatic choice.

No telemetry needed — this is a rendering bugfix plus a default-value correction.

@agentcore-devx-automation agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Sep 2, 2026
@codecov-commenter

codecov-commenter commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.14%. Comparing base (6357f2e) to head (4ab1b2b).
⚠️ Report is 2 commits behind head on refactor.

Additional details and impacted files
@@             Coverage Diff              @@
##           refactor    #2168      +/-   ##
============================================
- Coverage     97.14%   97.14%   -0.01%     
============================================
  Files           519      519              
  Lines         35492    35486       -6     
============================================
- Hits          34478    34472       -6     
  Misses         1014     1014              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 2, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 2, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 2, 2026
@aidandaly24
aidandaly24 force-pushed the fix/litellm-harness-transcript branch from 4005451 to d515573 Compare September 2, 2026 14:55
@github-actions github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Sep 2, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 2, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 2, 2026
…odel

The lite_llm default was anthropic/claude-sonnet-4-5, which routes LiteLLM at
Anthropic directly and needs an Anthropic API key. Because apiKeyArn is optional
for LiteLLM, a default project deployed READY and then failed its first invoke
with a LiteLLM AuthenticationError.

A bedrock/ model ID routes LiteLLM at Bedrock, which it signs with SigV4 using
the harness execution role, so no API key is required. Derive the default from
DEFAULT_HARNESS_MODEL so the bedrock and lite_llm defaults cannot drift.
The wizard declared its own copy of every provider default, so changing the
flag path's lite_llm default left the TUI still pre-filling
anthropic/claude-sonnet-4-5. That value is form state rather than placeholder
text, so pressing through the wizard deployed a harness that reached READY and
then failed its first invoke with a LiteLLM AuthenticationError.

Export HARNESS_DEFAULT_MODEL_IDS and read it for both the pre-filled value and
the placeholder, leaving MODEL_PROVIDERS to describe only labels. The two paths
can no longer drift.
@aidandaly24
aidandaly24 force-pushed the fix/litellm-harness-transcript branch from d515573 to 0e920b4 Compare September 2, 2026 17:31
@github-actions github-actions Bot added size/s PR size: S and removed size/m PR size: M labels Sep 2, 2026
@aidandaly24 aidandaly24 changed the title fix(harness): render LiteLLM transcripts and default LiteLLM to a keyless bedrock model fix(create): default the LiteLLM harness model to a keyless bedrock model Sep 2, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 2, 2026
@github-actions github-actions Bot added size/s PR size: S and removed size/s PR size: S labels Sep 2, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 2, 2026
MODEL_PROVIDERS no longer declares a per-provider default, so the wizard and the
flag path read one table and cannot disagree. project.test.ts already pins the
value, which is the only remaining way to regress it.
@github-actions github-actions Bot added size/s PR size: S and removed size/s PR size: S labels Sep 2, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 2, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 2, 2026

@notgitika notgitika left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@aidandaly24
aidandaly24 merged commit fd7296b into aws:refactor Sep 2, 2026
33 of 37 checks passed
This was referenced Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants