⚡ MailQueue-Engine

A production-grade, distributed email scheduling microservice and dashboard engine built for fault tolerance, atomic concurrency control, and distributed rate-limited email delivery.

---

📌 Executive Summary

MailQueue-Engine handles scheduled, high-volume email dispatch with absolute delivery guarantees and rate control. It decouples API scheduling from worker execution using a persistent delayed queue backed by Redis and BullMQ, enforced by PostgreSQL atomic locks and Redis Lua-scripted rate limiters.

Whether handling immediate blasts or scheduling thousands of personalized emails across custom time windows, MailQueue-Engine guarantees zero duplicate sends, automatic recovery from server crashes, and adherence to strict per-sender SMTP rate limits.

---

✨ Features & Capabilities

⚙️ Engine & Backend Architecture

• Persistent Delayed Scheduling: Enqueues thousands of emails with precise target delivery timestamps backed by BullMQ sorted sets.
• Atomic Worker Claim (Zero Duplicate Send): Optimistic locking at the database layer (UPDATE ... WHERE status = SCHEDULED) guarantees only one worker process can claim and execute any given job.
• Distributed Per-Sender Throttling: Custom Redis Lua script reserves send slots atomically per sender, maintaining minimum delay gaps without blocking worker threads.
• Hourly Sender Quotas: Enforces UTC hour-window email quotas via atomic Redis increment counters (mailqueue:hourly:{senderId}:{YYYY-MM-DDTHH}).
• Self-Healing Crash Recovery:
  • recoverStuckProcessingJobs(): Automatic recovery of stuck PROCESSING jobs on worker startup.
  • reconcileStartupJobs(): Dual-store reconciliation comparing PostgreSQL ground truth against Redis queues on reboot.
• Exponential Backoff & Retries: Multi-attempt delivery resilience with configurable backoff windows and automatic transition to FAILED status upon terminal failure.
• Observability & Metrics: Built-in Prometheus metrics at /metrics, structured JSON logging, and container health probes (/api/health, /api/ready, /api/live).

🎨 Frontend & Control Dashboard

• Next.js 16 + React 19 App: Responsive UI built with TypeScript and Tailwind CSS.
• Google OAuth Authentication: Secure user session management via Auth.js (NextAuth v5).
• Lead List Ingestion: Drag-and-drop CSV/TXT file upload with client-side column auto-detection, instant deduplication, and email syntax validation.
• Campaign Composer: Visual campaign launcher with timezone-aware start date pickers, per-sender throttle controls, and hourly sending limits.
• Real-Time Job Monitoring: Server-side paginated tables for SCHEDULED, PROCESSING, SENT, and FAILED emails with auto-polling and manual refresh capabilities.
• Job Lifecycle Controls: Single-click cancellation for scheduled emails and permanent record deletion.

---

🏗️ System Architecture

                                  ┌───────────────────────────────┐
                                  │   User Browser / Dashboard    │
                                  │    Next.js 16 (Port 3000)     │
                                  └───────────────┬───────────────┘
                                                  │
                                                  │ (Google OAuth & REST API)
                                                  ▼
                                  ┌───────────────────────────────┐
                                  │      Express REST API         │
                                  │         (Port 4000)           │
                                  └───────┬───────────────┬───────┘
                                          │               │
                        Transactional     │               │ Enqueue Delayed Jobs
                        Campaign Write    │               │ (zset epoch timestamps)
                                          ▼               ▼
                                 ┌─────────┐             ┌─────────┐
                                 │Postgres │             │ Redis 7 │
                                 │   16    │             │ BullMQ  │
                                 └────┬────┘             └────┬────┘
                                      │                       │
                                      │ Atomic Claim          │ Job Event Promotion
                                      │ (Optimistic Lock)     │
                                      ▼                       ▼
                                ┌───────────────────────────────────┐
                                │      BullMQ Worker Process        │
                                │   (Concurrency: WORKER_CONCURRENCY)│
                                └─────────────────┬─────────────────┘
                                                  │
                                                  ├─► 1. Redis Lua Throttle Check
                                                  ├─► 2. Redis Hourly Quota Check
                                                  └─► 3. Nodemailer SMTP Send
                                                          │
                                                          ▼
                                                ┌───────────────────┐
                                                │ Target SMTP Server│
                                                │(Ethereal/SES/etc.)│
                                                └───────────────────┘

---

🚀 Quick Start (Local Development)

Prerequisites

• Node.js: v18+
• Docker & Docker Compose: For local PostgreSQL 16 and Redis 7 instances.

---

Step 1: Clone & Start Infrastructure

git clone https://github.com/Yogesh10217/Email-Scheduler-Assignment.git MailQueue-Engine
cd MailQueue-Engine

# Start PostgreSQL (5432) & Redis (6379)
docker compose up -d

---

Step 2: Configure & Start Backend

cd backend
cp .env.example .env

# Install dependencies, run DB migrations, and seed initial SMTP sender
npm install
npx prisma migrate dev --name init
npm run seed:sender

# Terminal 1: Start Express API Server (Port 4000)
npm run dev

# Terminal 2: Start BullMQ Worker Process
npm run worker:dev

---

Step 3: Configure & Start Frontend

cd ../frontend
cp .env.example .env.local

# Fill in your Google OAuth credentials in .env.local (see Environment Configuration below)

npm install
npm run dev

Visit http://localhost:3000 in your browser, log in via Google OAuth, and launch your first email campaign!

---

🌐 Production & Deployment Guide

MailQueue-Engine is ready for deployment across cloud platforms (Vercel, Render, Railway, Fly.io, or AWS).

1. Production Docker Compose (Full Stack Single Machine)

For quick single-server deployments (e.g. AWS EC2, DigitalOcean Droplet, Hetzner):

# Build and launch all production containers (API, Worker, Frontend, Postgres, Redis)
docker compose -f docker-compose.prod.yml up -d --build

---

2. Multi-Platform Distributed Cloud Deployment

Component	Recommended Hosting	Steps
Frontend	Vercel / Netlify	Import frontend/ directory. Set NEXT_PUBLIC_API_URL to your production backend API URL. Add AUTH_SECRET and Google OAuth credentials.
Backend API & Worker	Render / Railway / Fly.io	Deploy backend/ as two services: one for npm start (API) and one for npm run worker:start (Worker).
Database	Neon / Supabase / AWS RDS	Provision PostgreSQL 16 instance and set DATABASE_URL in backend env.
Queue Store	Upstash / AWS ElastiCache / Redis Labs	Provision Redis 7 instance and configure REDIS_HOST, REDIS_PORT, REDIS_PASSWORD.

---

⚙️ Environment Configuration

Backend (backend/.env)

# ── Server ─────────────────────────────────────────────────────────
NODE_ENV=development
PORT=4000

# ── PostgreSQL ──────────────────────────────────────────────────────
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/mailqueue?schema=public

# ── Redis ───────────────────────────────────────────────────────────
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=

# ── BullMQ Worker ───────────────────────────────────────────────────
WORKER_CONCURRENCY=5

# ── Rate Limiting Controls ───────────────────────────────────────────
MIN_SEND_DELAY_SECONDS=2          # Minimum gap (seconds) between sends per sender
MAX_EMAILS_PER_HOUR_PER_SENDER=200 # Maximum hourly quota per sender
RATE_LIMIT_PREFIX=mailqueue

# ── SMTP Delivery Transports ───────────────────────────────────────
SMTP_HOST=smtp.ethereal.email
SMTP_PORT=587
SMTP_SECURE=false
SMTP_USER=your-ethereal-username
SMTP_PASSWORD=your-ethereal-password
SMTP_FROM_EMAIL=your-ethereal-from@ethereal.email
SMTP_FROM_NAME="MailQueue Engine"

# ── Resilience & Retries ──────────────────────────────────────────────
MAX_SEND_ATTEMPTS=3
RETRY_BACKOFF_DELAY_MS=5000
PROCESSING_TIMEOUT_SECONDS=300

# ── Internal Security ─────────────────────────────────────────────────
INTERNAL_API_SECRET=mailqueue-internal-secret

Frontend (frontend/.env.local)

# Google OAuth Configuration
GOOGLE_CLIENT_ID=your-google-client-id.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=your-google-client-secret

# Auth.js Encryption Secrets
AUTH_SECRET=your-32-character-random-secret
NEXTAUTH_SECRET=your-32-character-random-secret
NEXTAUTH_URL=http://localhost:3000

# Backend Integration URL
NEXT_PUBLIC_API_URL=http://localhost:4000

---

🧪 System Resilience & Fault Tolerance Matrix

MailQueue-Engine is engineered to survive crash scenarios at any point during campaign execution with zero mail loss:

Failure Scenario	Engine Self-Healing Mechanism
API Server Crashes	BullMQ jobs remain safe in Redis. PostgreSQL holds campaign state. Worker continues processing independently.
Worker Crashes Mid-Execution	Jobs left in PROCESSING status are auto-reclaimed on startup by recoverStuckProcessingJobs() and reset to SCHEDULED for immediate retry.
Redis Restart / Cache Flush	POST /api/internal/requeue-scheduled or reconcileStartupJobs() queries PostgreSQL ground truth for all SCHEDULED jobs and rebuilds the Redis delayed queue.
Parallel Worker Scale-Out	Workers race safely using atomic conditional SQL (UPDATE ... WHERE status = SCHEDULED). Duplicate executions are impossible.

---

📡 REST API Reference

Method	Endpoint	Description
GET	/api/health	Full system health check (PostgreSQL + Redis connectivity)
GET	/api/ready	K8s readiness probe
GET	/api/live	K8s liveness probe
GET	/metrics	Prometheus metrics exporter
POST	/api/users/sync	Upsert OAuth user into PostgreSQL
POST	/api/campaigns	Create campaign and schedule all recipient email jobs
GET	/api/campaigns	List campaigns (paginated)
GET	/api/campaigns/:id	Get campaign details and associated job list
GET	/api/emails/scheduled	List active SCHEDULED and PROCESSING jobs
GET	/api/emails/sent	List SENT and FAILED jobs (filterable by status)
PATCH	/api/emails/:id/cancel	Cancel a scheduled email job
DELETE	/api/emails/:id	Permanently delete an email record
POST	/api/internal/senders/bootstrap	Seed initial SMTP sender transport
POST	/api/internal/recover-stuck-jobs	Force recovery of timed-out PROCESSING jobs
POST	/api/internal/requeue-scheduled	Re-enqueue all PostgreSQL SCHEDULED jobs into BullMQ

---

🧪 Automated Test Suite

The engine includes 85 automated unit, integration, and load tests across 15 test suites:

cd backend
npm test

PASS src/__tests__/scheduled-at-immutability.test.ts    (6 tests)
PASS src/__tests__/delivery.test.ts                     (8 tests)
PASS src/__tests__/throttling.lifecycle.test.ts         (4 tests)
PASS src/__tests__/throttling.requeue.integration.test.ts (1 test — live Redis+PG)
PASS src/__tests__/duplicate.execution.test.ts          (2 tests)
PASS src/__tests__/recovery.test.ts                     (3 tests)
PASS src/__tests__/load.test.ts                         (8 tests)
PASS src/__tests__/campaign.service.test.ts             (14 tests)
PASS src/__tests__/campaign.validation.test.ts          (8 tests)
PASS src/__tests__/rate-limit.test.ts                   (8 tests)
PASS src/__tests__/timezone.test.ts                     (4 tests)
PASS src/__tests__/csv.parser.test.ts                   (6 tests)
PASS src/__tests__/worker.test.ts                       (5 tests)
PASS src/__tests__/users.api.test.ts                    (2 tests)
PASS src/__tests__/observability.test.ts               (8 tests)

Test Suites: 15 passed, 15 total
Tests:       85 passed, 85 total

---

🏛️ Engineering Architecture & Trade-offs

Design Area	Choice Made	Engineering Rationale
State Consistency	PostgreSQL as Ground Truth, Redis as Transitory Queue	PostgreSQL holds immutable campaign state. Redis stores ephemeral delayed jobs. If Redis loses data, PostgreSQL state allows full queue reconstruction.
Throttling Strategy	Non-blocking Re-enqueue vs Thread Sleep	When a sender hits rate limits, jobs release their worker thread immediately and re-enqueue with a Redis delay. Worker slots are never held idle.
Worker Concurrency	Atomic SQL Conditionals	Worker concurrency (WORKER_CONCURRENCY) uses UPDATE ... WHERE id = X AND status = 'SCHEDULED'. DB locks guarantee single-winner execution.
scheduledAt Integrity	Strict Immutability	Target schedule timestamps (scheduledAt) are immutable once created, ensuring UI metrics accurately reflect user intent even after retries.

---

📄 License

Distributed under the MIT License. See LICENSE for details.