Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ A **WorkerPool** keeps **Workers** running and ready, and a Worker hosts the **A
The diagram forks at this snapshot, because a snapshot serves two purposes.

- **Resume** restores the same Actor onto **any Worker in the pool with room**, which is not necessarily the Worker that it ran on before. Because the snapshot captures the Actor's full state, the conversation continues from where it left off. Every idle agent takes this path.
- A **{{< gloss "Tag" >}}Tag{{< /gloss >}}** pins this snapshot, and a **New Actor** can be seeded from the tag at the moment that it is created. Resuming an existing Actor never goes through a tag.
- A **{{< gloss "Tag" >}}Tag{{< /gloss >}}** pins this snapshot so that Agent Substrate does not collect it. Seeding a **New Actor** from a tag is unavailable in this release. Resuming an existing Actor never goes through a tag.

A tag gives a snapshot a stable name, so callers do not track Agent Substrate's internal snapshot identity. The tag names one snapshot permanently, and only its visibility scope can change afterward. A tag is also a retention pin, so Agent Substrate does not delete a snapshot while a tag still names it.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ An agent spends most of its life waiting. It waits on a person to reply, and it
- **Pause**: Takes a short-term snapshot whose files stay on the node. Pausing pins the Actor to that node, because the following resume is prioritized onto the node that holds the snapshot files.
- **Resume**: Restores a suspended or paused Actor onto a Worker, from its latest snapshot. The common path restores from a snapshot rather than cold-booting the workload.

The following diagram traces an Actor through those operations, and shows the further path that opens once a snapshot is [pinned by a tag](#checkpoints).
The following diagram traces an Actor through those operations, and shows what happens once a snapshot is [pinned by a tag](#checkpoints).
</br></br>

{{< reuse "kagent-docs/snippets/snapshot-cycle-diagram.md" >}}
Expand Down Expand Up @@ -85,7 +85,7 @@ Because kagent suspends at every turn boundary, every message to a Session pays

## Checkpoints

A **checkpoint** marks one point in a conversation and keeps it. You can create a checkpoint when a Session reaches a state worth returning to, such as a long investigation that is only partway through. You can then start a second Session from that point, which is known as a {{< gloss "Fork" >}}fork{{< /gloss >}}. The fork continues the conversation from where the checkpoint marked it, new turns append only to the fork, and the fork gets its own conversation and task identifiers rather than inheriting the source's. To create a checkpoint and fork a Session from it, follow the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}).
A **checkpoint** marks one point in a conversation and keeps it. You can create a checkpoint when a Session reaches a state worth returning to, such as a long investigation that is only partway through. Starting a second Session from that point, which is known as a {{< gloss "Fork" >}}fork{{< /gloss >}}, is unavailable in this release. kagent captures process memory in every snapshot, so `ForkSession` rejects a checkpoint with `FAILED_PRECONDITION` rather than restore an Actor that still holds the source Session's identifiers. To create and inspect a checkpoint, follow the [Agent Substrate example]({{< link path="examples/agent-substrate#forking-is-temporarily-unavailable" >}}).

Create a checkpoint only after a turn finishes. The request names the task that the turn ended on, and it fails when a newer turn has already started or while that turn's snapshot is still being written.

Expand Down
2 changes: 1 addition & 1 deletion docs-site/content/kagent/1.x/about/what-is-kagent.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ kagent 1.0 separates an agent's capabilities from its runtime, then runs the two
{{< feature-card title="Agent lifecycle via CRDs" desc="Define, version, and roll out Harnesses and AgentTemplates with kubectl and GitOps, the same workflow as every other workload." >}}
{{< feature-card title="Sandboxed by default" desc="Every Session runs on a Substrate Actor, isolated from the host kernel by a gVisor sandbox. Run untrusted, model-directed code safely." >}}
{{< feature-card title="Suspend and resume" desc="Idle Sessions suspend and free their compute, then resume on demand. Run far more agents than you have capacity for at any one moment." >}}
{{< feature-card title="Checkpoint and fork" desc="Pin a snapshot of a conversation at a turn boundary, then branch a second Session from it that keeps the revision it started with." >}}
{{< feature-card title="Checkpoints" desc="Pin a snapshot of a conversation at a turn boundary so that Agent Substrate keeps the state for inspection and retention." >}}
{{< feature-card title="Pluggable agent runtimes" desc="A Harness selects the engine behind an agent: kagent's own Go and Python engines, the Codex or Claude coding agents, or any image of your own that speaks kagent's A2A contract." >}}
{{< feature-card title="Tools over MCP" desc="Bind an agent to any Model Context Protocol (MCP) server with a RemoteMCPServer resource. An installation registers two servers already, including 124 tools for Kubernetes, Helm, Istio, and Argo Rollouts." >}}
{{< feature-card title="Agent tools" desc="Bind another AgentTemplate as a tool, so an agent can hand work to a specialist. A Shared binding nests that agent inside its parent's Actor, one level deep." >}}
Expand Down
29 changes: 10 additions & 19 deletions docs-site/content/kagent/1.x/agents/agent-memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,36 +26,27 @@ Memories are scoped to the agent and to the user who created them, and carry the
## Before you begin

> [!IMPORTANT]
> Memory requires an **external** PostgreSQL database with the [pgvector](https://github.com/pgvector/pgvector) extension installed. kagent's bundled database uses the stock `postgres` image, which does not ship pgvector, so it cannot store memories. Point kagent at your own database first.
> Memory requires an **external** PostgreSQL database with the [pgvector](https://github.com/pgvector/pgvector) extension installed. The development database that `kagent install` creates uses the stock `postgres` image, which does not ship pgvector, so the database cannot store memories. Point kagent at your own database first.

1. Store the connection string for your database in a Kubernetes Secret, in the same namespace as the kagent controller. The key that you choose becomes the file name that kagent reads, so `db-url` produces the path `/var/secrets/db-url`.
1. Store the connection string for your database in a Kubernetes Secret, in the same namespace as the kagent controller.
```bash
kubectl create secret generic my-postgres-url-secret -n kagent \
--from-literal=db-url='postgres://<username>:<password>@<host>:5432/<database>?sslmode=require'
--from-literal=connectionString='postgres://<username>:<password>@<host>:5432/<database>?sslmode=require'
```

2. Save the following values, which point kagent at your database, mount the Secret into the controller, and run the migration that creates the pgvector extension.
2. Save the following values, which point kagent at your database and run the migration that creates the pgvector extension.
```yaml
cat > kagent-memory-values.yaml <<EOF
database:
postgres:
# Path to a file holding the connection string, which takes precedence over url
urlFile: /var/secrets/db-url
# Secret holding the connection string.
connectionStringSecretRef:
name: my-postgres-url-secret
key: connectionString
# Use the login user directly. To assume a role instead, create the role in PostgreSQL and set its name here.
role: ""
# Runs the vector migration, which creates the pgvector extension and the memory tables
vectorEnabled: true
bundled:
# Use your own database instead of the bundled one
enabled: false
controller:
# Mounts the Secret so that the file at urlFile exists in the controller pod
volumes:
- name: db-secret
secret:
secretName: my-postgres-url-secret
volumeMounts:
- name: db-secret
mountPath: /var/secrets
readOnly: true
EOF
```

Expand Down
2 changes: 1 addition & 1 deletion docs-site/content/kagent/1.x/examples/a2a-byo.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,5 +252,5 @@ An agent of your own takes the same path as the example image, with two differen
{{< cards >}}
{{< card link=`{{< link path="agents/bring-your-own-agent" >}}` title="Bring your own agent" subtitle="Read the full A2A contract and the limits of the byo runtime." >}}
{{< card link=`{{< link path="examples/a2a-agents" >}}` title="Call an agent over A2A" subtitle="Talk to this Agent with grpcurl instead of the CLI." >}}
{{< card link=`{{< link path="examples/agent-substrate" >}}` title="Agent Substrate" subtitle="Watch the Actor behind your image suspend, checkpoint, and fork." >}}
{{< card link=`{{< link path="examples/agent-substrate" >}}` title="Agent Substrate" subtitle="Watch the Actor behind your image suspend and checkpoint." >}}
{{< /cards >}}
84 changes: 17 additions & 67 deletions docs-site/content/kagent/1.x/examples/agent-substrate.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,30 @@
---
title: Agent Substrate
description: Watch an agent's Actor suspend between turns, pin its state with a checkpoint, and fork that checkpoint into a second agent that continues the conversation.
description: Watch an agent's Actor suspend between turns and pin its state with a checkpoint.
weight: 10
author: kagent.dev
---

[Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs every agent as an Actor: a sandboxed unit of compute that holds a {{< gloss "Worker" >}}Worker{{< /gloss >}} only while a turn is in progress, and whose state you can pin and branch. This example follows one agent through all three behaviors.
[Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}) runs every agent as an Actor: a sandboxed unit of compute that holds a {{< gloss "Worker" >}}Worker{{< /gloss >}} only while a turn is in progress. This example follows one agent through suspend, resume, and checkpoint creation.

The Actor that these steps follow is also the isolation boundary. Every Actor runs in its own {{< gloss "gVisor" >}}gVisor{{< /gloss >}} sandbox rather than sharing one with its neighbors. This isolation allows a model to safely run tools and execute commands. For what the sandbox blocks, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}).

## Before you begin

Checkpoints and {{< gloss "Fork" >}}forks{{< /gloss >}} have no kagent CLI commands yet, so this example calls `CheckpointService` with grpcurl.
Checkpoints have no kagent CLI commands yet, so this example calls `CheckpointService` with grpcurl.

{{< reuse "kagent-docs/snippets/grpcurl-prerequisites.md" >}}

5. Create a {{< gloss "Session" >}}Session{{< /gloss >}} and send it a message, then save the Session's ID and the ID of the task that the message produced. A checkpoint pins a completed turn, so the conversation needs at least one.
5. Install `kubectl-ate`, the Agent Substrate command line tool.
```bash
curl -fsSL -o kubectl-ate \
"https://github.com/kagent-dev/substrate/releases/download/v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}/kubectl-ate-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')"
chmod +x kubectl-ate
sudo mv kubectl-ate /usr/local/bin/
kubectl ate --help
```

6. Create a {{< gloss "Session" >}}Session{{< /gloss >}} and send it a message. Then save the Session's ID and the ID of the task that the message produced. A checkpoint pins a completed turn, so the conversation needs at least one.
```bash
export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id')
export TASK_ID=$(kagent agent invoke --session $SESSION_ID --task "What is 2+2?" -o json | jq -r '.task.id')
Expand Down Expand Up @@ -112,7 +121,7 @@ Each suspend writes a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}}, and Agent
> [!NOTE]
> A checkpoint captures a turn boundary, so the named task must be terminal and its snapshot must be written. A newer turn that has already started fails the call with `conversation advanced beyond the expected task`, and a snapshot that is still being written fails it with `snapshot for the expected task is not ready`. The second is safe to retry with the same values; the first needs a newer task ID.

2. Save the checkpoint's `id` to fork from it in the next section.
2. Save the checkpoint's `id` so that you can delete it during cleanup.
```bash
export CHECKPOINT_ID=<your-checkpoint-id>
```
Expand All @@ -126,65 +135,11 @@ Each suspend writes a {{< gloss "Snapshot" >}}snapshot{{< /gloss >}}, and Agent

Underneath, the checkpoint attaches a {{< gloss "Tag" >}}Tag{{< /gloss >}} named `checkpoint-<checkpoint-id>` to the snapshot, and Agent Substrate does not collect a snapshot while a tag names it. You can see the tag by running `kubectl ate get tags --atespace kagent`.

## Fork the conversation into a second agent

Forking creates a second Session from the pinned snapshot, running the revision that the checkpoint was taken on. The fork continues the conversation from the checkpoint: it inherits the {{< gloss "Transcript" >}}transcript{{< /gloss >}} up to that point, along with the Actor's durable state.

> [!NOTE]
> The two branches share everything up to the checkpoint and nothing after it. New turns append only to the Session that received them, so the original's history and the checkpoint itself stay unchanged no matter what the fork goes on to do. Each branch runs its own Actor.

1. Fork the checkpoint.
```bash
grpcurl -plaintext \
-d '{"checkpointId":"'"$CHECKPOINT_ID"'","requestId":"'"$(uuidgen)"'"}' \
localhost:8083 kagent.api.v1alpha1.CheckpointService/ForkSession
```

The response carries a new Session with its own ID. Example output:
```json
{
"session": {
"id": "0198c3e5-1d62-7f38-a904-8b3c7e2f5d16",
"agent": {
"namespace": "kagent",
"name": "my-first-agent"
},
"state": "RUNTIME_STATE_READY",
"contextId": "0198c3e5-1d62-7f38-a904-8b3c7e2f5d16",
"name": "my-first-agent turn 4"
}
}
```

A fork receives fresh conversation and task IDs rather than inheriting the source's, so `contextId` is the fork's own `id`. The transcript it replays stops at the checkpoint's boundary. The fork takes the checkpoint's `name` as its own.

2. Save the fork's ID, then ask it about a turn that happened before the checkpoint.
```bash
export FORK_ID=<your-fork-session-id>
kagent agent invoke --session $FORK_ID --task "What did I ask you first?"
```
## Forking is temporarily unavailable

The fork answers from the conversation it inherited. That is the difference between a fork and a new Session on the same Agent. Send the original a different question and the two diverge from here.
Forking creates a second Session from a checkpoint only when the pinned snapshot contains durable data without process memory. In the current kagent release, suspended Actors use `Full` snapshots so that their process memory survives suspend and resume. A `ForkSession` request for these checkpoints is rejected with `FAILED_PRECONDITION`, because a restored Actor would resume processes that still hold the source Session's identifiers.

3. List your Sessions to verify that both appear as separate conversations.
```bash
kagent agent session list
```

The two rows share an Agent, and differ in their IDs and creation times. Each one has its own Actor, named `session-<session-id>`. Example output:
```console
+--------------------------------------+----------------+-------+----------------------+
| ID | AGENT | STATE | CREATED |
+--------------------------------------+----------------+-------+----------------------+
| 0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10 | my-first-agent | READY | 2026-08-31T15:02:10Z |
| 0198c3e5-1d62-7f38-a904-8b3c7e2f5d16 | my-first-agent | READY | 2026-08-31T15:14:02Z |
+--------------------------------------+----------------+-------+----------------------+
```

A fork runs the compiled {{< gloss "Revision" >}}revision{{< /gloss >}} that its checkpoint was taken on, not whatever revision the Agent resolves to now. Editing the Agent after checkpointing does not change what a fork of that checkpoint runs. That stability is the point of a checkpoint: it holds a known-good configuration you can return to, rather than tracking the template as it moves on.

> [!NOTE]
> A checkpoint can only be forked when its snapshot captured durable data alone. kagent compiles every ActorTemplate to take a `Data`-scope snapshot on commit, so a checkpoint taken on a suspended Session is forkable. A checkpoint whose snapshot also captured process state is rejected with `Checkpoint includes process state and cannot be forked`, because process memory belongs to the one Actor that produced it.
Checkpoint creation is still useful without forking. A checkpoint pins the snapshot for retention, gives the turn boundary a stable record, and keeps the checkpoint visible in the API while the original Session continues.

## Clean up

Expand All @@ -195,11 +150,6 @@ A fork runs the compiled {{< gloss "Revision" >}}revision{{< /gloss >}} that its
localhost:8083 kagent.api.v1alpha1.CheckpointService/DeleteCheckpoint
```

2. Delete the fork. A fork is a Session in its own right, so deleting the checkpoint that it started from does not remove it.
```bash
kagent agent session delete $FORK_ID
```

## Next steps

{{< cards >}}
Expand Down
Loading
Loading