Skip to content

testing.md

Latest commit

 

History

History
54 lines (43 loc) · 2.88 KB

testing.md

File metadata and controls

54 lines (43 loc) · 2.88 KB
Raw

Harness\Testing

Fakes for testing hosts, tools, middleware and providers without a model.

FakeModel plays scripted replies, one per model call: a list of deltas, or a closure fn (Prompt $prompt): iterable<Delta> that may fiber\sleep() or throw. $model->prompts records what each call was given. Running out of replies throws. The deltas are Harness\Delta\* (TextDelta, ReasoningDelta, ToolCallStart, ToolInputDelta, ToolCallEnd, Finish): what a provider emits, carrying only the payload. They are not the Harness\Event\* classes of the same names that a host receives, which carry the turn, parent and id as well.

$model = new FakeModel(
    [new ToolCallStart('c1', 'read_file'), new ToolInputDelta('c1', '{"path":"a"}'), new ToolCallEnd('c1'),
     new Finish(FinishReason::ToolCalls)],
    [new TextDelta('done'), new Usage(10, 2), new Finish(FinishReason::Stop)],
);

FakeTransport plays scripted Responses, or closures fn ($url, $headers, $body, Cancel $cancel): Response, for provider tests; $transport->requests records each request. A scripted body that waits checks $cancel as a real transport does.

TestCall::make(?ToolSpec $spec = null, mixed $context = null, string $id = 'test', ?Closure $emit = null) is a Call for calling a tool or a tool middleware directly. It belongs to an idle run, so isCancelled() is false; needsInput(), pending() and answer() behave as in a turn, and $emit receives the events the call emits (progress()).

$call   = TestCall::make(new ToolSpec('write', 'Writes', ['type' => 'object'], new Hints(destructive: true)));
$result = $policy->call(new ToolCall('test', 'write', []), $call, fn () => Result::text('ran'));
assert($call->pending()?->needs[0] instanceof Approval);
$call->answer(['approval' => Decision::allow()]);     // then call the policy again

$events = [];
$shell  = new Shell('/tmp');
$shell->call(['command' => 'echo hi'], TestCall::make($shell->spec(), emit: function ($e) use (&$events) { $events[] = $e; }));

TestPrompt::make(?string $system = null, array $messages = [], array $tools = [], Options $options = new Options(), bool $continue = false) is a Prompt with a Cancel that is never set, for calling a ModelMiddleware directly:

$model  = new FakeModel([new TextDelta('ok'), new Finish(FinishReason::Stop)]);
$deltas = [...$compact->stream(TestPrompt::make('sys', $messages), fn (Prompt $p) => $model->stream($p))];
assert($model->prompts[0]->messages !== $messages);   // what the middleware handed on

Event fields a test reads

  • Event\ToolEnded::$outcome is the bare Result (or the \Throwable it threw), not wrapped: a test asserts on it directly (assert($event->outcome instanceof Result)).
  • Event\StepStarted::$n and Event\StepEnded::$n are the step number, so a test can assert how many model calls a turn made, or that a resumed turn's first step is 0.