Skip to content

JossGeek/Cloud-Rag-Performance

BranchesTags

Repository files navigation

Cloud-Rag-Performance

Performance Evaluation of Neural RAG vs Internet RAG on AWS Cloud Infrastructure

FastAPI Backend

This project includes a FastAPI backend that exposes the notebook RAG workflows as HTTP endpoints. It supports:

  • Neural RAG: Retrieves passages from a local vector index, then generates an answer from the retrieved context using semantic embeddings.
  • Internet RAG: Searches the web through Tavily API when TAVILY_API_KEY is configured, scrapes page content, then generates an answer from retrieved pages.
  • RAG Comparison: Runs both methods for the same query and returns both responses plus overall resource metrics (CPU, memory).

Backend Architecture

Services:

  • EmbeddingService - Handles text embedding with SentenceTransformers (or local deterministic fallback)
  • RetrievalService - Manages vector indexing with FAISS and corpus retrieval with streaming dataset support
  • LLMService - Generates answers using Transformers (or extractive fallback for text selection)
  • TavilyService - Manages web search via Tavily API with web scraping and fallback results

Key Features:

  • ✅ Streaming Dataset Support - MS MARCO loaded with streaming to prevent memory issues (5,000 sample limit by default)
  • ✅ Graceful Fallbacks - Works without external models/datasets using local implementations
  • ✅ Resource Metrics - Tracks retrieval time, LLM time, context size, and system resource usage
  • ✅ Error Handling - Comprehensive error handling with detailed logging
  • ✅ Production Ready - Tested endpoints with proper validation and error responses

Runtime Dependencies:

fastapi>=0.136.0      - Web framework
uvicorn>=0.47.0       - ASGI server
numpy>=2.0.0          - Numerical computing
psutil>=7.0.0         - System metrics
requests>=2.32.0      - HTTP client
beautifulsoup4>=4.12.0 - Web scraping

Optional ML dependencies (when RAG_USE_REAL_MODELS=1):

sentence-transformers  - SentenceTransformers embeddings
transformers          - HuggingFace models for LLM
torch                 - PyTorch backend
faiss-cpu             - Vector indexing
bitsandbytes          - 4-bit quantization for LLMs

Run The API

Install dependencies:

For Docker/FastAPI backend only:

python -m pip install -r requirements.txt

For full local development (backend + notebooks + testing):

python -m pip install -r requirements-dev.txt

For notebooks only (without FastAPI):

python -m pip install -r requirements-notebooks.txt

Requirements Files

  • requirements.txt - Minimal FastAPI backend dependencies (for Docker builds)
  • requirements-dev.txt - Full development: backend + notebooks + testing
  • requirements-notebooks.txt - Notebook-only: ML/data analysis packages without backend

Start the development server:

python -m uvicorn backend.main:app --reload

Open the interactive API docs:

http://127.0.0.1:8000/docs

Test the health endpoint:

Invoke-WebRequest -Uri "http://127.0.0.1:8000/api/health" -UseBasicParsing | Select-Object -ExpandProperty Content

Expected response:

{"status":"ok"}

Run With Docker

The API can also run with a plain Docker image. There are no required local external services in the current backend: Tavily is called over HTTPS when TAVILY_API_KEY is configured, and the neural pipeline uses an in-process FAISS or NumPy index. The Docker image installs only the backend runtime dependencies from requirements.txt; notebook libraries remain commented out to keep the image smaller.

Build the image from the repository root:

docker build -f docker/Dockerfile -t cloud-rag-api .

Run the container:

docker run --env-file .env -p 8000:8000 cloud-rag-api

Run the container in the background:

docker run -d --env-file .env -p 8000:8000 --name cloud-rag-api cloud-rag-api

The container exposes the same docs URL:

http://127.0.0.1:8000/docs

The container reads local configuration from .env when --env-file .env is passed. Keep .env private and use .env_example as the shareable template.

Docker Compose is not required for the current backend because there are no local companion services. For AWS deployment, use AWS-native observability such as CloudWatch logs and metrics, ECS or App Runner service metrics, ALB metrics, and optionally AWS X-Ray or OpenTelemetry tracing.

Run Tests

Run the backend unit and endpoint smoke tests:

python -m pip install pytest httpx
python -m pytest backend/tests/test_services.py

The tests cover:

  • RAG service smoke checks.
  • Root, docs, OpenAPI, and health endpoints.
  • Neural RAG, Internet RAG, and compare endpoints.
  • Request validation for invalid query payloads.
  • Compare endpoint CPU and memory metrics.
  • Local Tavily fallback behavior when no API key is configured.

Note: requirements-dev.txt installs the test and notebook dependencies. requirements.txt is intentionally smaller for Docker builds.

Run Notebooks

Jupyter notebooks are available for interactive development and testing of RAG methods. To run them locally, install the development dependencies and use Jupyter:

python -m pip install -r requirements-dev.txt
jupyter notebook

Available Notebooks:

1. neural_rag.ipynb - Neural RAG Development & Testing

Tests the local neural retrieval pipeline with FAISS vector indexing and SentenceTransformers embeddings.

Features:

  • Loads MS MARCO dataset with streaming (5,000 samples by default)
  • Creates FAISS vector index from corpus
  • Retrieves top-k passages based on semantic similarity
  • Generates extractive answers using local LLM (when enabled)
  • Includes example queries and similarity scores

Environment flags:

  • RAG_USE_REAL_MODELS=1 - Load real SentenceTransformers embeddings (default: local fallback)
  • RAG_USE_REAL_DATASET=1 - Load real MS MARCO dataset (default: small fallback corpus)

2. internet_rag.ipynb - Internet RAG with Tavily Search

Tests live web search retrieval using Tavily API with fallback to local content.

Features:

  • Live internet search when TAVILY_API_KEY is configured
  • Web page scraping and text extraction
  • Generates answers from retrieved web content
  • Includes timeout handling and graceful fallbacks
  • Example queries demonstrating real-time search

Requirements:

  • Set TAVILY_API_KEY in .env to enable live search

3. experiments.ipynb - Performance Benchmarking

Comprehensive performance evaluation comparing Neural RAG vs Internet RAG across multiple dimensions.

Test Categories:

  1. Retrieval Latency Test - Measures retrieval speed for each method
  2. End-to-End Latency - Compares total request time including answer generation
  3. Context Size Impact - Analyzes how context length affects performance
  4. Memory & CPU Usage - Monitors resource consumption during operations
  5. Stability & Reliability - Tests failure handling and fallback mechanisms

Generates:

  • Performance metrics in tabular format
  • Comparative analysis with statistics (mean, std, min, max)
  • Visual charts and plots for easy comparison
  • Results saved to results/performance_summary.json

Note: Uses streaming dataset with 20-sample limit by default for faster iteration.

Running Notebooks with GPU Support (Optional)

For faster embeddings and LLM inference, enable GPU support:

# Set before running notebooks
$env:RAG_USE_REAL_MODELS=1
$env:RAG_USE_REAL_DATASET=1

# Install CUDA-enabled dependencies
python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
python -m pip install faiss-gpu

The notebook libraries are intentionally commented out in requirements.txt so they are not installed into the Docker image. To run notebooks locally, uncomment the notebook-only dependencies in requirements.txt, install them, and execute the notebooks from Jupyter.

Environment

Copy .env_example to .env and fill in the values you want to use.

TAVILY_API_KEY=your_tavily_api_key_here
HF_TOKEN=your_hugging_face_token_here
RAG_USE_REAL_MODELS=0
RAG_USE_REAL_DATASET=0

TAVILY_API_KEY enables live internet search. Without it, the internet RAG service returns local fallback content.

RAG_USE_REAL_MODELS=1 attempts to load SentenceTransformers and Transformers models. RAG_USE_REAL_DATASET=1 attempts to load MS MARCO from Hugging Face.

Dataset Streaming & Memory Optimization

When RAG_USE_REAL_DATASET=1, the MS MARCO dataset is loaded with streaming enabled to avoid memory issues on cloud deployments. Key features:

  • Streaming mode: Data is fetched on-demand without loading the entire dataset into memory
  • 5,000 sample limit: By default, only the first 5,000 samples are used (configurable via max_samples parameter in build_neural_rag_runtime())
  • Cloud-friendly: Prevents out-of-memory errors on resource-constrained cloud environments
  • Efficient batch processing: Compatible with both regular and streaming datasets

To adjust the sample limit, modify the num_samples parameter when calling build_neural_rag_runtime(num_samples=5000) in notebooks.

Endpoints

Health Check

GET /api/health

Response:

{
  "status": "ok"
}

Neural RAG

POST /api/rag/neural

Runs the local neural retrieval pipeline. It embeds the query, retrieves top-k passages from the corpus index, and generates an answer from those passages.

Request body:

{
  "query": "What is machine learning?",
  "k": 3,
  "answer_len": 50
}

Example:

curl -X POST http://127.0.0.1:8000/api/rag/neural `
  -H "Content-Type: application/json" `
  -d "{\"query\":\"What is machine learning?\",\"k\":3,\"answer_len\":50}"

Internet RAG

POST /api/rag/internet

Runs the internet retrieval pipeline. It searches Tavily when configured, scrapes page text, and generates an answer from the retrieved web content. If Tavily is not configured or unavailable, it uses local fallback content so the endpoint still runs.

Request body:

{
  "query": "What is retrieval augmented generation?",
  "k": 3,
  "answer_len": 50
}

Example:

curl -X POST http://127.0.0.1:8000/api/rag/internet `
  -H "Content-Type: application/json" `
  -d "{\"query\":\"What is retrieval augmented generation?\",\"k\":3,\"answer_len\":50}"

Compare RAG Methods

POST /api/rag/compare

Runs both Neural RAG and Internet RAG for the same query and returns both responses. The compare response also includes request-level CPU and memory metrics sampled around the combined comparison call.

Request body:

{
  "query": "How do neural networks work?",
  "k": 3,
  "answer_len": 50
}

Example:

curl -X POST http://127.0.0.1:8000/api/rag/compare `
  -H "Content-Type: application/json" `
  -d "{\"query\":\"How do neural networks work?\",\"k\":3,\"answer_len\":50}"

Response Shape

The Neural RAG and Internet RAG endpoints return:

{
  "method": "neural",
  "query": "What is machine learning?",
  "answer": "Machine learning is a subset of artificial intelligence...",
  "contexts": [
    "Retrieved passage or page content"
  ],
  "metrics": {
    "retrieval_time": 0.001,
    "llm_time": 0.001,
    "context_size": 250.0,
    "total_time": 0.002
  }
}

contexts contains strings for Neural RAG and page dictionaries for Internet RAG. metrics captures retrieval time, answer generation time, context size, and total request time.

The compare endpoint returns both RAG responses plus overall request resource metrics:

{
  "neural": {
    "method": "neural",
    "query": "How do neural networks work?",
    "answer": "A neural network maps inputs to outputs...",
    "contexts": ["Retrieved passage"],
    "metrics": {
      "retrieval_time": 0.001,
      "llm_time": 0.001,
      "context_size": 180.0,
      "total_time": 0.002
    }
  },
  "internet": {
    "method": "internet",
    "query": "How do neural networks work?",
    "answer": "Machine learning is a subset...",
    "contexts": [
      {
        "title": "Local fallback result",
        "url": "local://machine-learning",
        "content": "Machine learning is a subset..."
      }
    ],
    "metrics": {
      "retrieval_time": 0.001,
      "llm_time": 0.001,
      "context_size": 160.0,
      "total_time": 0.002
    }
  },
  "metrics": {
    "total_time": 0.004,
    "cpu_percent_before": 0.0,
    "cpu_percent_after": 12.5,
    "cpu_percent_avg": 6.25,
    "memory_rss_mb_before": 74.2,
    "memory_rss_mb_after": 75.1,
    "memory_rss_mb_delta": 0.9,
    "memory_vms_mb_before": 190.4,
    "memory_vms_mb_after": 191.0,
    "memory_vms_mb_delta": 0.6,
    "system_memory_percent_before": 61.0,
    "system_memory_percent_after": 61.1,
    "system_memory_percent_avg": 61.05
  }
}

FastAPI Architecture

backend/
|-- main.py
|-- api/
|   `-- routes.py
|-- rag/
|   |-- neural_rag.py
|   `-- internet_rag.py
|-- services/
|   |-- embedding_service.py
|   |-- retrieval_service.py
|   |-- llm_service.py
|   `-- tavily_service.py
|-- utils/
|   |-- logger.py
|   `-- metrics.py
`-- tests/
    `-- test_services.py

backend/main.py creates the FastAPI app and registers API routes.

backend/api/routes.py defines request and response models, HTTP endpoints, and cached RAG instances.

backend/rag/neural_rag.py orchestrates the neural pipeline: retrieval service plus LLM service.

backend/rag/internet_rag.py orchestrates the internet pipeline: Tavily service plus LLM service.

backend/services/embedding_service.py loads the real SentenceTransformer embedder when enabled, otherwise uses a deterministic local embedder.

backend/services/retrieval_service.py builds the corpus, creates a FAISS index when available, and falls back to a NumPy index when FAISS is not installed.

backend/services/llm_service.py loads the configured LLM when enabled, otherwise generates extractive local answers from the supplied context.

backend/services/tavily_service.py calls Tavily search, scrapes returned pages, and falls back to local content when no live search is available.

backend/utils/logger.py configures logging and loads .env. backend/utils/metrics.py provides timing helpers used by the routes and RAG classes.

About

Performance Evaluation of Neural RAG vs Internet RAG on AWS Cloud Infrastructure

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages