llm-cache

Production LLM call layer for AI agents and tools: keep OpenAI/Anthropic/AI SDK/LiteLLM, hot-swap models with MDA presets, and add cache, retries, circuit breakers, key rotation, singleflight, and Python/TypeScript/Rust parity.

Drop-in OpenAI-compatible cache that replays repeated computer-use and agent workflows at near-zero cost.

Unified multi-layer caching library for AI/agent pipelines — LangChain, LangGraph, AutoGen, CrewAI, Agno, A2A

Redis Vector Library (RedisVL) -- the AI-native Java client for Redis.

Context-aware caching proxy for Anthropic and OpenAI — maximizes prompt-cache hit rates by anchoring long-lived context into stable boundaries. Drop-in for Claude Code, Codex, Cursor, and custom harnesses.

Python implementations, C++ structures, and CUDA GPU kernels that demonstrate the core algorithms, memory allocators, quantization schemes, disaggregation transports, and latency profilers used in modern LLM serving engines like vLLM, SGLang, TensorRT-LLM, llama.cpp, NVIDIA Dynamo, and LMCache.

VCAL Core — high-performance semantic cache and vector cache library for LLM applications.

Deterministic LLM caching layer with context optimization, LRU eviction, model-aware token counting, and concurrency-safe request deduplication.

Five-decimal cache-hit readout beside the chat stats line for the DeepSeek Harness Web GUI (dsh-plugin)

Adaptive semantic cache for LLMs with streaming support, ML-based thresholds, and real-time cost tracking. Built in Rust for sub-millisecond performance.

Reduce LLM API costs and speed up responses by caching completions with NeuroCache’s intelligent, provider-agnostic caching layer.

Pre-indexed code knowledge graph for Claude Code, Codex, Gemini, Cursor, OpenCode, AntiGravity, Kiro, and Hermes Agent — fewer tokens, fewer tool calls, 100% local

Correctness-aware semantic cache for LLM responses, built for regulated workloads where a wrong cache hit is a defect. Zero-dependency Go: similarity floor + verification, PII-aware keying, knowledge-epoch staleness, negative cache.

Something related to LLM caching

Semantic caching layer for LLM calls. Exact-match and embedding-similarity caching with model-version-aware invalidation, use-case segmentation, and cost-saved tracking. Adapters for Redis and DynamoDB.

CacheEval is a prompt level caching accuracy evaluation system which is intended to check the accuracy and performance of a prompt caching system like GPTCache etc.

Chengeta AI is a unified memory and caching platform for AI agents, workflows, and autonomous systems.

Single-file Python disk cache for LLM API calls. Wrap any provider client. Stop paying for the same call twice during dev/eval/demo.

Semantic cache for LLM responses, backed by Postgres and pgvector. Exact hash lookups (0.5ms) fall through to HNSW vector search only on a miss, so repeat queries never pay for an embedding call. Protocol-based stores and embedders, Alembic-owned schema, buffered hit counting off the request path, and degrades to a miss instead of raising.

Read-only performance diagnostics for DeepSeek Harness: session load (open/restore) timing, spill-hit counts, compaction count and trigger, context-injection volume (AGENTS.md/skills/tool-schema token share), and LLM cache hit rate ??surfaced via the /fast command and the fast_report tool, persisted as reconstructable