Skip to content

About

AI-powered performance engineering platform for telemetry analysis, workload generation, load testing, bottleneck detection, root-cause analysis, and optimization recommendations.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

AI Performance Testing & Optimization Platform

Local-first performance engineering for telemetry analysis, AI-assisted workload design, runnable load-test generation, real engine execution, bottleneck detection, root-cause analysis, and grounded optimization recommendations.

This project is designed to run locally with open-source software and locally hosted AI models. No commercial AI API key, hosted database, hosted vector database, or paid SaaS service is required.

Contents

Overview

The platform supports this engineering workflow:

  1. Register and create a performance-testing project.
  2. Upload CSV, JSON, or JSONL telemetry.
  3. Normalize request, latency, resource, database, and error data.
  4. Calculate throughput, averages, weighted percentiles, error rate, and saturation indicators deterministically.
  5. Detect CPU, database, API, and connection-pool bottlenecks with evidence.
  6. Generate a validated peak-load scenario through Ollama.
  7. Generate and persist runnable k6 JavaScript or JMeter .jmx artifacts.
  8. Execute a configured local k6 or JMeter binary without shell execution.
  9. Capture status, stdout, stderr, cancellation, and normalized test metrics.
  10. Stream execution updates through Server-Sent Events.
  11. Compare baseline and candidate execution metrics.
  12. Ingest technical documents, create local embeddings, retrieve relevant chunks, and generate grounded recommendations.
  13. Run a structured root-cause workflow over deterministic evidence and retrieved knowledge.

Quantitative analysis never depends on an LLM. AI explains and extends measured evidence; it does not invent measurements.

Architecture

flowchart LR
    Browser[React dashboard] --> API[FastAPI API]
    API --> Postgres[(PostgreSQL)]
    API --> Redis[(Redis)]
    API --> Ollama[Ollama]
    API --> Chroma[(ChromaDB service)]
    API --> Engine[k6 or JMeter]
    API --> Analysis[Deterministic analysis]
    API --> Agents[Structured AI workflows]
    Agents --> Ollama
    Agents --> Knowledge[Local knowledge retrieval]
Loading

PostgreSQL is the system of record. Redis is provisioned for future worker and streaming scale-out. Ollama is accessed through the LLMProvider abstraction; agents never call a commercial provider directly.

Technology

Area Technology
Frontend React, TypeScript, Vite, React Router, CSS variables
API Python 3.12+, FastAPI, Pydantic
Persistence PostgreSQL, SQLAlchemy async, Alembic
Authentication JWT, Argon2 password hashing
Local LLM Ollama, default qwen3:8b
Local embeddings Ollama, default nomic-embed-text
Agent workflow LangGraph-compatible root-cause workflow
Load testing k6 and Apache JMeter
Infrastructure Docker Compose, Redis, ChromaDB service

Repository Layout

.
├── backend/
│   ├── app/
│   │   ├── api/              # HTTP routes and authorization boundaries
│   │   ├── core/             # settings, database, security
│   │   ├── models/           # SQLAlchemy entities
│   │   ├── providers/        # Ollama and LLM provider abstraction
│   │   ├── schemas/          # Pydantic API and AI contracts
│   │   ├── services/         # analysis, RAG, scripts, execution, comparison
│   │   └── main.py
│   ├── migrations/           # Alembic revisions
│   ├── tests/                # backend tests
│   └── requirements.txt
├── frontend/
│   ├── src/                  # React application, API client, styles
│   └── package.json
├── data/demo/                # baseline, peak, stress, and mixed telemetry
├── scripts/                  # local setup scripts
├── docker-compose.yml
├── .env.example
└── README.md

Requirements

  • Docker Desktop and Docker Compose
  • Node.js 20 or newer
  • Python 3.12 or newer
  • 8 GB RAM minimum for CPU-only local development; 16 GB or more is recommended for qwen3:8b
  • Additional storage for Ollama models
  • Optional: k6 and/or JMeter installed locally for test execution

GPU acceleration is optional. Ollama supports CPU-only hardware.

Quick Start

From the repository root:

Copy-Item .env.example .env
docker compose up --build

In a second terminal, download the default local models:

docker compose exec ollama ollama pull qwen3:8b
docker compose exec ollama ollama pull nomic-embed-text

Open:

Stop the stack:

docker compose down

Reset local database, model, and vector volumes:

docker compose down -v

Windows Setup

Copy-Item .env.example .env
.\scripts\setup.ps1

The script starts infrastructure, pulls qwen3:8b and nomic-embed-text, and builds the application containers.

For direct development without Docker:

Set-Location backend
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8001

In another terminal:

Set-Location frontend
npm install
npm run dev

PostgreSQL and Ollama still need to be available for the complete workflow.

Configuration

Copy .env.example to .env. Important settings:

Variable Purpose Default
DATABASE_URL Async PostgreSQL connection Local PostgreSQL
REDIS_URL Redis connection redis://localhost:6379/0
OLLAMA_BASE_URL Ollama HTTP endpoint http://localhost:11434
LLM_MODEL Local reasoning model qwen3:8b
EMBEDDING_MODEL Local embedding model nomic-embed-text
CHROMA_URL Local ChromaDB endpoint http://localhost:8000
JWT_SECRET_KEY JWT signing secret Change for non-local use
K6_EXECUTABLE k6 executable name/path k6
JMETER_EXECUTABLE JMeter executable name/path jmeter
CORS_ORIGINS Allowed frontend origins http://localhost:5173

Never commit .env or API credentials.

Local AI

The application communicates with Ollama over HTTP:

LLM_MODEL=qwen3:8b
EMBEDDING_MODEL=nomic-embed-text
OLLAMA_BASE_URL=http://localhost:11434

Models can be changed without source edits:

docker compose exec ollama ollama pull qwen3:14b

Set LLM_MODEL=qwen3:14b in .env and restart the backend.

If Ollama is unavailable, deterministic telemetry analysis, project management, script generation, and engine execution remain available. AI-specific endpoints return a controlled error instead of silently using a paid provider.

Demo Workflow

  1. Start Docker Compose and open http://localhost:5173.
  2. Register an account.
  3. Create a project named E-Commerce Demo.
  4. Upload data/demo/baseline-telemetry.csv and run analysis.
  5. Upload data/demo/peak-telemetry.csv and run analysis again.
  6. Upload data/demo/stress-telemetry.csv to expose stronger saturation.
  7. Review P50/P95/P99 latency, throughput, error rate, CPU, database latency, and connection-pool evidence.
  8. Generate a scenario with Ollama.
  9. Generate a k6 script or JMeter plan.
  10. Execute the artifact if the selected engine is installed.
  11. Watch execution status through the dashboard or SSE endpoint.
  12. Compare two completed executions through the comparison API.
  13. Add PostgreSQL/JVM/Redis documentation to the knowledge base.
  14. Generate grounded recommendations and root-cause analysis.

The mixed fixture, data/demo/ecommerce-telemetry.csv, combines normal traffic with CPU saturation, database latency spikes, elevated errors, and pool saturation.

API Overview

All /api routes except registration and login require a Bearer JWT.

Authentication

POST /api/auth/register
POST /api/auth/login
GET  /api/auth/me

Projects

GET    /api/projects
POST   /api/projects
GET    /api/projects/{project_id}
PATCH  /api/projects/{project_id}
DELETE /api/projects/{project_id}

Telemetry and analysis

POST /api/projects/{project_id}/telemetry/upload
GET  /api/projects/{project_id}/telemetry
POST /api/projects/{project_id}/telemetry/analysis
GET  /api/projects/{project_id}/telemetry/analysis/latest
POST /api/projects/{project_id}/telemetry/analysis/{analysis_id}/root-cause

Accepted uploads are .csv, .json, and .jsonl, limited to 10 MB. Supported normalized fields include timestamp, service, endpoint, request_count, latency_ms, status_code, cpu_percent, memory_percent, database_latency_ms, and connection_pool_percent.

Scenarios, scripts, and execution

POST /api/projects/{project_id}/scenarios/generate
POST /api/projects/{project_id}/tests/generate
POST /api/tests/{script_id}/execute
GET  /api/tests/executions/{execution_id}
POST /api/tests/executions/{execution_id}/cancel
GET  /api/tests/executions/{execution_id}/events
GET  /api/tests/executions/{candidate_id}/compare/{baseline_id}

The SSE endpoint returns text/event-stream events containing execution status, normalized metrics, and recent output. The execution service uses asyncio.create_subprocess_exec; arbitrary shell commands are never accepted.

Knowledge and recommendations

POST /api/projects/{project_id}/knowledge/documents
POST /api/projects/{project_id}/knowledge/query
POST /api/projects/{project_id}/recommendations/generate

Telemetry Analysis

Measurements are calculated in Python:

  • Average, minimum, and maximum latency
  • Weighted P50, P90, P95, and P99 latency
  • Requests per second and throughput
  • Successful and failed request counts
  • Error rate
  • CPU and memory averages
  • Database latency and connection-pool utilization
  • Deterministic bottleneck severity and confidence

Current rules include CPU saturation at 85% or higher, database latency at 150 ms or higher when API P95 is at least 500 ms, connection-pool utilization at 85% or higher, and error rate at 5% or higher.

AI and RAG

The provider boundary in backend/app/providers/ exposes structured generation and embeddings. Scenario, recommendation, and root-cause outputs are validated with Pydantic and receive one bounded correction attempt when structured output is invalid.

Knowledge documents are chunked, embedded with Ollama, and stored with metadata and embeddings in PostgreSQL JSONB records. Retrieval uses cosine similarity, with lexical fallback when embeddings are unavailable. ChromaDB is included in Compose as a local vector-service option; the current persisted retrieval path does not require a hosted vector database.

The root-cause workflow consumes persisted deterministic metrics and bottleneck evidence. It must not recalculate or invent measurements.

Test Generation and Execution

Generated k6 scripts include HTTP endpoints, methods, virtual-user stages, ramp and duration settings, P95 thresholds, and error thresholds. Generated JMeter plans include thread groups, virtual users, ramp time, duration, and HTTP samplers.

Execution output is parsed into normalized metrics. Missing k6 or JMeter binaries fail explicitly with exit code 127; successful runs are never simulated.

Database

The schema uses UUID primary keys and timestamp fields where appropriate:

users
projects
telemetry_records
performance_analyses
generated_test_scripts
test_executions
test_metrics
knowledge_documents
document_chunks
recommendations

Alembic revisions are in backend/migrations/versions/. Local startup also creates missing tables through SQLAlchemy metadata for developer convenience.

Testing

Backend tests:

Set-Location backend
python -m pytest -q

Coverage includes password hashing, JWT behavior, deterministic analysis, bottleneck rules, structured AI contracts, k6/JMeter generation, k6/JMeter result parsing, RAG ranking, recommendation fallback, and execution comparison deltas.

Frontend build:

Set-Location frontend
npm install
npm run build

Security

  • JWT-protected API routes
  • Argon2 password hashing
  • Project-level ownership checks
  • Pydantic input validation
  • 10 MB telemetry upload limit
  • Supported file-extension validation
  • No arbitrary shell command input
  • Engine executable configuration through environment variables
  • No committed secrets or required commercial API keys
  • Configurable CORS

Do not expose the local development JWT secret or engine endpoints to an untrusted network without hardening deployment settings.

Status and Limitations

Implemented:

  • Authentication, projects, authorization, PostgreSQL persistence, and migrations
  • Telemetry ingestion, deterministic analysis, and multiple demo datasets
  • Ollama provider abstraction and structured scenario generation
  • k6 and JMeter artifact generation
  • Real subprocess execution, cancellation, parsed metrics, and SSE updates
  • Execution comparison API
  • Local embeddings, knowledge retrieval, recommendations, and root-cause workflow
  • Backend regression tests and frontend production build

Known limitations:

  • The frontend is still centered around one dashboard rather than fully separated route-specific views for every product area.
  • PostgreSQL JSONB embeddings are the active retrieval store; ChromaDB is provisioned but not the active storage adapter.
  • Distributed worker execution through Redis/Celery is not enabled; local execution uses in-process asyncio tasks.
  • Live Docker, k6, JMeter, Ollama, and ChromaDB smoke tests require those tools to be installed and running on the developer machine.
  • Frontend component and browser integration coverage is smaller than backend unit coverage.

Troubleshooting

Ollama model not found

docker compose exec ollama ollama list
docker compose exec ollama ollama pull qwen3:8b
docker compose exec ollama ollama pull nomic-embed-text

Backend cannot connect to PostgreSQL

docker compose ps
docker compose logs postgres
docker compose restart postgres backend

Execution fails with exit code 127

Install k6 or JMeter locally, or set executable paths in .env:

K6_EXECUTABLE=C:\path\to\k6.exe
JMETER_EXECUTABLE=C:\path\to\jmeter.bat

Reset local state

docker compose down -v
docker compose up --build

About

AI-powered performance engineering platform for telemetry analysis, workload generation, load testing, bottleneck detection, root-cause analysis, and optimization recommendations.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages