OrbPro2 MCP - Browser-based Natural Language Globe Control

A small language model (SLM) system that runs entirely in the browser using WebGPU and allows users to control a CesiumJS 3D globe using natural language commands.

Features

Browser-based LLM Inference: Runs language models directly in the browser using WebGPU via web-llm
CesiumJS Integration: Full control over a 3D globe visualization including camera, entities, layers, and time
Model Context Protocol (MCP): Structured tool definitions for the LLM to control CesiumJS
CZML Generation: Automatic CZML document generation for creating entities
Natural Language Commands: Simple commands like "Show me Paris" or "Add a marker at the Eiffel Tower"
Multiple Model Support: Choose from various small language models based on your device capabilities

Requirements

Browser: Chrome 113+ or Edge 113+ with WebGPU support
GPU: A WebGPU-compatible graphics card
Memory: 4GB+ RAM recommended (varies by model size)

Quick Start

Install dependencies:

npm install

Start the development server:

npm run dev

Open your browser to http://localhost:5173
Select a language model and wait for it to load
Start typing natural language commands!

Example Commands

Navigation

"Show me New York"
"Fly to the Eiffel Tower"
"Go to Tokyo"
"Navigate to the Grand Canyon"

Adding Markers

"Add a red marker at Paris"
"Put a point at the Statue of Liberty"
"Mark the location of Sydney Opera House"

Drawing Shapes

"Draw a line from London to Paris"
"Create a circle around Tokyo with a 100km radius"
"Draw a polygon around Manhattan"

Scene Control

"Switch to 2D mode"
"Show me the 3D globe"
"Zoom in"
"Zoom out"

Time Animation

"Play the animation"
"Pause"
"Set time to January 1, 2024"

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Browser                               │
├─────────────────────────────────────────────────────────────┤
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │   WebLLM     │  │  MCP Server  │  │    CesiumJS      │   │
│  │  (WebGPU)    │──│  (In-Browser)│──│    Viewer        │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
│         │                 │                   │              │
│         ▼                 ▼                   ▼              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │   Command    │  │    CZML      │  │    Command       │   │
│  │   Parser     │  │  Generator   │  │    Executor      │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
└─────────────────────────────────────────────────────────────┘

Components

WebLLM Engine (src/llm/web-llm-engine.ts)
- Initializes and manages the language model
- Handles inference with tool calling support
- Provides streaming response capability
MCP Server (src/mcp/cesium-mcp-server.ts)
- Implements Model Context Protocol
- Defines tools for CesiumJS control
- Handles tool calls from the LLM
Command Parser (src/llm/command-parser.ts)
- Parses LLM outputs into CesiumJS commands
- Fallback natural language parsing
- Known location database
CZML Generator (src/cesium/czml-generator.ts)
- Creates CZML documents for entities
- Supports points, labels, polylines, polygons, etc.
Command Executor (src/cesium/command-executor.ts)
- Executes commands against CesiumJS Viewer
- Camera control, entity management, scene settings

Supported Models

Small (Fast, ~500MB-2GB)

Qwen2.5-0.5B-Instruct
Qwen2.5-1.5B-Instruct
SmolLM2-360M-Instruct
SmolLM2-1.7B-Instruct

Medium (Balanced, ~2-4GB)

Qwen2.5-3B-Instruct
Phi-3.5-mini-instruct
gemma-2-2b-it

Large (Powerful, ~4-8GB)

Llama-3.2-3B-Instruct
Qwen2.5-7B-Instruct
Mistral-7B-Instruct-v0.3

API Usage

import { CesiumSLMApp } from './src/app';

const app = new CesiumSLMApp();

await app.initialize({
  cesiumContainer: 'cesium-container',
  chatContainer: 'chat-container',
  statusContainer: 'status-container',
  modelSelectorContainer: 'model-selector-container',
  cesiumToken: 'your-cesium-ion-token', // Optional
});

Using Components Independently

import { WebLLMEngine, CommandParser, CesiumCommandExecutor } from './src';

// Initialize LLM
const llm = new WebLLMEngine({
  modelId: 'Qwen2.5-1.5B-Instruct-q4f16_1-MLC',
  onProgress: (p) => console.log(`Loading: ${p.progress * 100}%`),
});
await llm.initialize();

// Generate response
const response = await llm.generate('Show me Paris');

// Parse tool calls
const parser = new CommandParser();
const commands = parser.parseToolCalls(response.toolCalls);

// Execute commands
const executor = new CesiumCommandExecutor(viewer);
for (const cmd of commands.commands) {
  await executor.execute(cmd);
}

MCP Tools

The system exposes the following MCP tools:

Tool	Description
`flyTo`	Fly the camera to a location
`lookAt`	Orient camera to look at a location
`zoom`	Zoom in or out
`addPoint`	Add a point marker
`addLabel`	Add a text label
`addPolyline`	Draw a line
`addPolygon`	Draw a polygon
`addCircle`	Draw a circle
`removeEntity`	Remove an entity
`clearAll`	Clear all entities
`setSceneMode`	Change 2D/3D mode
`setTime`	Set simulation time
`playAnimation`	Start animation
`pauseAnimation`	Pause animation
`generateCZML`	Generate CZML document

Training & Deployment Pipeline

This section describes how to train and deploy your own custom CesiumJS language model.

Prerequisites

# Python dependencies (for training)
pip install mlx mlx-lm

# Node dependencies (for data generation)
npm install

# Docker (for WebGPU compilation)
# Install from: https://www.docker.com/products/docker-desktop

Step 1: Generate Training Data

Generate synthetic training data from the tool schemas:

cd training
npx tsx generate-training-data.ts

This creates generated-training-data.jsonl with ~88,000 examples covering:

All CesiumJS tools (flyTo, addPoint, addSphere, etc.)
Known locations (cities, landmarks, airports, scientific facilities)
Various phrasings and parameter combinations

Step 2: Train the Model (MLX - Apple Silicon)

Training uses Apple's MLX framework for fast Apple Silicon training:

cd training
python3 finetune_mlx.py \
  --dataset generated-training-data.jsonl \
  --output-dir cesium-qwen-lora-mlx \
  --num-epochs 3 \
  --batch-size 4

Monitor progress:

./progress.sh

Training takes ~2 hours on Apple Silicon (M1/M2/M3).

The script automatically:

Converts data to MLX format
Trains LoRA adapters
Fuses adapters with base model
Saves merged model to cesium-qwen-lora-mlx/merged/

Step 3: Test the Model

Test the trained model locally:

cd training
python3 -c "
from mlx_lm import load, generate

model, tokenizer = load('cesium-qwen-lora-mlx/merged')

prompt = 'fly to CERN'
messages = [
    {'role': 'system', 'content': 'You are a CesiumJS controller assistant. Convert natural language commands to tool calls.'},
    {'role': 'user', 'content': prompt}
]
formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=formatted, max_tokens=256)
print(response)
"

Step 4: Compile for WebGPU (Docker)

Compile the model for browser deployment using MLC-LLM:

./scripts/compile-cesium-slm-docker.sh

This requires Docker Desktop running with at least 16GB disk space.

The script:

Builds a Docker image with Emscripten + MLC-LLM
Converts weights to quantized format (q4f16_1 = ~300MB)
Compiles to WebGPU WASM
Outputs to mlc-models/OrbPro-Cesium-SLM-0.5B-q4f16_1-MLC/

All downloads are cached in .cache/mlc-compile/ for faster subsequent builds.

Step 5: Upload to HuggingFace

Upload the compiled model:

cd mlc-models/OrbPro-Cesium-SLM-0.5B-q4f16_1-MLC

huggingface-cli login
huggingface-cli repo create OrbPro-Cesium-SLM-0.5B-q4f16_1-MLC --type model
huggingface-cli upload YOUR_USERNAME/OrbPro-Cesium-SLM-0.5B-q4f16_1-MLC .

Step 6: Update Application

Update src/llm/web-llm-engine.ts with your HuggingFace model URL.

Complete Pipeline Script

Run the entire pipeline:

# Generate data
cd training && npx tsx generate-training-data.ts

# Train (takes ~2 hours)
python3 finetune_mlx.py --dataset generated-training-data.jsonl --output-dir cesium-qwen-lora-mlx --num-epochs 3

# Compile for WebGPU (requires Docker)
cd .. && ./scripts/compile-cesium-slm-docker.sh

# Upload
cd mlc-models/OrbPro-Cesium-SLM-0.5B-q4f16_1-MLC
huggingface-cli upload YOUR_USERNAME/OrbPro-Cesium-SLM-0.5B-q4f16_1-MLC .

Caching

All large downloads are cached for fast subsequent runs:

Cache Location	Contents
`~/.cache/huggingface/`	Base model weights (~1GB)
`.cache/mlc-compile/`	Docker build cache
`training/cesium-qwen-lora-mlx/`	LoRA checkpoints

To start fresh, delete these directories.

Development

# Install dependencies
npm install

# Start dev server
npm run dev

# Type check
npx tsc --noEmit

# Build for production
npm run build

# Preview production build
npm run preview

Environment Variables

Variable	Description
`VITE_CESIUM_TOKEN`	Cesium Ion access token for terrain/imagery

Browser Support

Browser	Version	Status
Chrome	113+	Supported
Edge	113+	Supported
Firefox	Not yet	WebGPU in development
Safari	Not yet	WebGPU in development

References

web-llm - High-performance in-browser LLM inference
CesiumJS - 3D geospatial visualization
CZML Guide - CZML format specification
MCP Protocol - Model Context Protocol
mcp-wasm - MCP in WebAssembly reference

License

MIT

Name		Name	Last commit message	Last commit date
.claude		.claude
data		data
docs		docs
packages		packages
public		public
scripts		scripts
src		src
tests		tests
training		training
wasm		wasm
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Progress.md		Progress.md
README.md		README.md
TODO.md		TODO.md
agents.md		agents.md
docker-compose.yml		docker-compose.yml
index.html		index.html
nginx.conf		nginx.conf
package-lock.json		package-lock.json
package.json		package.json
tasks.md		tasks.md
test.html		test.html
tsconfig.json		tsconfig.json
vite.config.ts		vite.config.ts

Latest commit

History

Folders and files

Repository files navigation