Real-Time Self-Driving Car Simulation

A modular, real-time self-driving car simulation system that processes dashcam video streams or live webcam feeds to perform lane detection, obstacle identification, and rule-based driving decisions. Built using Python, OpenCV, and Ultralytics YOLOv8.

This repository is fully documented to serve as a complete system reference.

---

🚀 System Overview

The simulation operates as an offline pipeline that takes pre-recorded road videos (or live camera input), processes each frame through concurrent detection algorithms, and computes instant steering and throttle/braking commands.

                  ┌──────────────────────┐
                  │  Video / Camera Feed │
                  └──────────┬───────────┘
                             │
            ┌────────────────┴────────────────┐
            ▼                                 ▼
┌───────────────────────┐         ┌───────────────────────┐
│    Lane Detection     │         │   Obstacle Detection  │
│ (Classical CV Engine) │         │    (YOLOv8 Engine)    │
└──────────┬────────────┘         └───────────┬───────────┘
           │                                  │
           │ (Offset & Lane Status)           │ (Threat Level & Zones)
           └────────────────┬─────────────────┘
                            ▼
                ┌──────────────────────┐
                │   Decision Engine    │
                │     (Rule-Based)     │
                └──────────┬───────────┘
                           ▼
                ┌──────────────────────┐
                │      HUD & UI        │
                │  (Visual & Debug)    │
                └──────────────────────┘

The driving decisions are rendered onto a Head-Up Display (HUD) and output via terminal telemetry. A detailed 6-panel debug view is available to trace each stage of the image processing pipeline.

---

🛠️ Technology Stack

• Core Language: Python 3.8+
• Computer Vision:
  • OpenCV (opencv-python): Handles image processing, thresholding, color conversion, edge detection, morphological transformations, Hough transforms, video streaming, and HUD overlay rendering.
  • NumPy: Powers high-performance matrix and vector operations, Region of Interest (ROI) masking, and mathematical polynomial curve fitting.
• Deep Learning & Object Detection:
  • Ultralytics YOLOv8: Used for real-time object detection (yolov8n.pt) and optional instance segmentation (yolov8n-seg.pt).
  • PyTorch: Serves as the underlying deep learning tensor library, utilizing CUDA-enabled GPUs automatically for acceleration.

---

📐 System Architecture & Module Details

The application is structured into highly cohesive modules, separating the frame ingestion and user interface from the computer vision pipelines and decision logic.

1. Ingestion and Integration (main.py)

• VideoStream Class: Manages the life cycle of a single video input (file or live webcam). It processes incoming frames, updates frame rates, stores detection states, overlays the UI HUD, and produces a merged display or a 6-panel debug view.
• MultiStream Class: Pools multiple VideoStream objects to run up to 4 video feeds concurrently. It formats their HUD outputs into a unified $2 \times 2$ grid display.
• Keyboard Event Loop: Captures keystrokes to dynamically modify pipeline configurations (e.g., toggling segmentation, changing lane fitting modes, pausing, saving screenshots) in real-time.

2. Lane Detection (lane_detection.py)

Extracts boundaries of the active lane and computes the vehicle's relative lateral offset from the center. It supports two processing pipelines:

• Classical Canny Edge + Hough Transform (Default):
  1. Converts the frame to grayscale and applies a Gaussian Blur to suppress high-frequency noise.
  2. Applies Canny Edge Detection (thresholds: 50, 150) to highlight structural boundaries.
  3. Applies a trapezoidal Region of Interest (ROI) mask to restrict detection to the immediate road surface in front of the vehicle.
  4. Runs a Probabilistic Hough Line Transform to extract line segments.
  5. Computes slopes and intercepts to group lines into left and right lanes, filters out outliers based on a minimum slope parameter (0.3), averages the coordinates, and extrapolates the lines to the horizon.
• Adaptive Thresholding + Polynomial Curve Fitting (Optional, toggle with A and F keys):
  1. Applies Adaptive Gaussian Thresholding to create a binary mask, making the lane detection robust under uneven lighting or shadows.
  2. Performs a morphological Closing operation to connect broken edges and fill gaps in the markings.
  3. Splits the masked ROI horizontally. For the left and right halves, it identifies non-zero coordinate points.
  4. Fits a 2nd-degree polynomial ($x = Ay^2 + By + C$) using np.polyfit() to draw curved lanes, mapping the coordinates dynamically back onto the frame.
• Offset Calculation: Computes the deviation of the lane's midpoint from the camera's optical center. This pixel offset ($\Delta x$) dictates minor steering adjustments.

3. Object Detection & Segmentation (object_detection.py)

Identifies and tracks obstacles in front of the vehicle using YOLOv8 models.

• Spatial Grid Splitting: The frame width is divided into three vertical zones:
  • LEFT: Left 33% of the screen
  • CENTER: Middle 33% of the screen (critical path)
  • RIGHT: Right 33% of the screen
• Distance Classification: Since depth data is not available from mono-camera video, distances are estimated based on the bounding box area percentage relative to the total frame size:
  • CLOSE: Area $> 8%$ of the frame (indicates immediate collision threat)
  • NEAR: Area between $3%$ and $8%$ of the frame (indicates upcoming obstacle)
  • FAR: Area $< 3%$ of the frame (distant object, safe zone)
• Threat Level Assessment: Analyzes active detections to output an overall threat level (CLOSE, NEAR, FAR, or CLEAR) and identifies which zones are blocked.
• Instance Segmentation (Optional, toggle with G key):
  • Utilizes yolov8n-seg.pt to detect custom segment masks for vehicles, people, and objects.
  • Computes the aggregate Road Blockage Percentage based on the masked pixel area. If the road occupancy exceeds $25%$, the vehicle is instructed to halt.

4. Decision Engine (decision.py)

A priority-based, deterministic rule engine that maps current environmental states to driving actions.

• Rule Hierarchy:
  1. Immediate Threat (CLOSE obstacle in CENTER): If all zones are blocked, output REVERSE. Otherwise, output STOP.
  2. Lateral Threat (CLOSE obstacle on LEFT/RIGHT): Output TURN RIGHT or TURN LEFT to steer away from the obstacle.
  3. Approaching Threat (NEAR obstacle in CENTER): Output STOP to prevent close-range collisions.
  4. Approaching Lateral Threat (NEAR obstacle on LEFT/RIGHT): Output TURN RIGHT or TURN LEFT.
  5. Road Blockage (Road Blockage > 25%): Output STOP - BLOCKED and report the primary obstacle blocking the road.
  6. Lane Failure (No lanes detected): Output STOP - NO LANES as a safety fallback.
  7. Missing Boundaries: If only one lane marking is visible, output TURN LEFT or TURN RIGHT to drift back into the lane.
  8. Lateral Deviation (Offset $|\Delta x| > 80\text{px}$): Output STEER LEFT or STEER RIGHT to center the vehicle.
  9. Clear Path: Output FORWARD.

---

📂 Project Structure

self-driving/
├── main.py                  # Stream manager, keyboard controller, and window renderer
├── lane_detection.py        # Classical CV edge detection & polynomial/Hough lane fitting
├── object_detection.py      # YOLOv8 object detection, segmentation, and distance/zone estimation
├── decision.py              # Rule-based priority decision engine
├── requirements.txt         # Package dependencies
├── .gitignore               # Configures untracked files (dataset, model weights, pycache)
├── README.md                # System documentation (This file)
├── dataset/                 # [Local Only] Folder containing source video files (.mp4)
├── screenshots/             # Folder containing saved screenshots
├── yolov8n.pt               # [Local Only] Standard YOLOv8 nano model weights
└── yolov8n-seg.pt           # [Local Only] YOLOv8 segmentation model weights

---

💻 Setup and Installation

1. Prerequisites

• Python 3.8 or higher.
• A functional web camera (if testing live stream mode).
• A CUDA-capable NVIDIA GPU is recommended for optimal YOLOv8 frame rates, but the system will automatically fall back to CPU execution.

2. Environment Setup

Clone this repository to your local machine:

git clone https://github.com/Rahim36712/self-driving.git
cd self-driving

It is recommended to use a virtual environment:

# Create environment
python -m venv venv

# Activate environment (Windows)
.\venv\Scripts\activate

# Activate environment (macOS/Linux)
source venv/bin/activate

3. Install Dependencies

Install all required libraries using pip:

pip install -r requirements.txt

4. Fetch Pre-trained Model Weights

The YOLOv8 weights are excluded from Git version control. You must download the weights files and place them in the repository root directory:

• Standard Detection: Download yolov8n.pt
• Segmentation: Download yolov8n-seg.pt

Alternatively, you can let the Python script download them automatically on its first run (requires internet access).

5. Add Dataset Videos

Place road videos (.mp4, .mov, .avi, .ts, or .mkv) inside the dataset/ directory. If the directory does not exist, create it in the root folder.

---

🎮 Running the Simulation

Start the full system by executing the integration module:

python main.py

Keyboard Shortcuts (Runtime Controls)

Ensure the active simulation window is focused when pressing keys:

Key	Control Action	Description
Q	Quit	Gracefully closes all windows and releases streams.
P	Pause / Resume	Pauses video playback.
D	Toggle Debug View	Switches between the standard HUD merged view and a detailed 6-panel debug view showing step-by-step pipeline intermediate frames.
A	Toggle Adaptive Threshold	Switches lane boundary extraction between Canny Edges and Adaptive Gaussian Thresholding.
F	Toggle Polynomial Fit	Switches lane drawing between extrapolated linear Hough lines and 2nd-degree polynomial curves.
G	Toggle Segmentation	Activates the instance segmentation engine (yolov8n-seg.pt) to overlay object masks and calculate road blockage percentages.
N	Next Video	Cycles to the next video found in the dataset/ folder.
M	Toggle Multi-Stream	Switches the rendering display to run up to 4 dataset videos in a grid simultaneously.
W	Toggle Webcam	Activates webcam feed (Stream ID 0) as the primary input.
S	Save Screenshot	Captures the current frame (including active HUD overlays or debug panels) and saves it to the screenshots/ directory.

Module Library Notes

The files lane_detection.py, object_detection.py, and decision.py act as library files containing the logic. They do not contain executable __main__ entry blocks. They must be executed through the main application interface by running python main.py.

---

💡 System Controls & Design Adjustments

You can tune the system's runtime parameters by modifying the following constants at the top of the respective files:

• In lane_detection.py:
  • CANNY_LOW / CANNY_HIGH (default: 50, 150): Edge detection sensitivity.
  • MIN_SLOPE (default: 0.3): Filters out lines that are too horizontal (e.g. shadows, crosswalks).
  • ADAPTIVE_BLOCK (default: 11): Size of the pixel neighborhood for thresholding.
• In object_detection.py:
  • CONFIDENCE_THRESHOLD (default: 0.4): Filters out weak YOLO predictions.
  • YOLO_SKIP_FRAMES (default: 2): Runs YOLO detection once every $N$ frames to conserve processing overhead.
  • SEG_SKIP_FRAMES (default: 3): Runs segmentation mask generation once every $N$ frames.
  • CLOSE_AREA_PCT / NEAR_AREA_PCT (default: 0.08, 0.03): Bounding box thresholds for distance categorization.
• In decision.py:
  • OFFSET_THRESHOLD (default: 80): Allowable pixel offset before trigger steering corrections.
  • ROAD_BLOCK_PCT (default: 25.0): Maximum road blockage percentage allowed before invoking an emergency stop.

---

🚧 Known Issues and Future Improvements

1. Depth Accuracy: Mono-camera bounding-box area is an approximation for distance calculation. Slopes, hills, and camera angles can distort the calculation. Implementing camera calibration or using a depth-estimation model (e.g., MiDaS) would improve depth calculations.
2. Smoothing Controls: The decision engine commands can fluctuate rapidly between frames. Implementing a PID controller or temporal smoothing filter (e.g. Kalman Filter) on the lane offset and steering angle output would stabilize driving behaviors.
3. Intersection Handling: The system relies on prominent lane markings. It will currently issue a STOP - NO LANES command at wide unmarked intersections. Adding temporary heading estimation or GPS trajectory plotting would resolve this limitation.