Advanced Augmented Reality (AR) Pipeline: Tag Detection, 2D Overlay, and 3D Projection

Aditya Anand

---

1. Overview and Goal

The objective of this project is to implement a complete, real-time Augmented Reality pipeline from scratch. The system detects custom AR tags in a video or live webcam stream, decodes their 4-bit identifiers, corrects their spatial orientation, and performs two types of augmentations:

1. 2D Superimposition: Warping and overlaying a template image (e.g., iitd_logo_template.jpg) onto the tag surface.
2. 3D Model Rendering: Projecting complex 3D meshes (such as a Cow model1.obj, a Wolf model2.obj, or a Boat model3.obj) onto the tag plane using estimated camera intrinsics and a custom wireframe rendering engine with occlusion handling.

To ensure computational efficiency at interactive frame rates, critical operations like spatial filtering and contour tracing are implemented in C++ and exposed to the high-level Python pipeline using pybind11.

---

2. Pipeline Architecture & Connections

The diagram below illustrates the end-to-end flow of data and the connections between different modules:

graph TD
    A[Input Video Frame/Webcam] --> B[Grayscale Conversion]
    B --> C[Custom Gaussian Blur C++]
    C --> D[Otsu's Adaptive Thresholding]
    D --> E[Morphological Closing]
    E --> F[Suzuki-Abe Contour Extraction C++]
    F --> G[Quad Detection & Corner Ordering]
    G --> H[DLT Homography Estimation H]
    H --> I[Inverse Warp & Bilinear Sampling]
    I --> J[Tag ID Decoding & Orientation Check]
    J --> K{Overlay Mode}
    K -- 2D Overlay --> L[Inverse Homography Mapping]
    K -- 3D Projection --> M[Camera Pose Recovery R, t]
    M --> N["Projection Matrix P = K R|t"]
    N --> O[3D Mesh Vertex Projection]
    O --> P[Z-Sorting Painters Algorithm]
    L --> Q[Overlay Display Window]
    P --> Q

Loading

---

3. Detailed Component Breakdown

A. Grayscale Conversion & Spatial Filtering

1. Luminance Conversion: The color frame is converted to grayscale in colour_togray using integer-shift approximation to avoid floating-point overhead: $$\text{Gray} = (29 \times R + 150 \times G + 77 \times B) \gg 8$$
2. Separable Gaussian Blur: High-frequency noise is suppressed using gaussian_blurs implemented in C++ and exposed in bindings.cpp. It performs a 1D horizontal blur followed by a 1D vertical blur to achieve $O(K)$ complexity per pixel instead of $O(K^2)$ for kernel size $K$.

B. Adaptive Binarization & Morphology

1. Otsu's Thresholding: Implemented in threshold_otsu, this dynamically computes the optimal binarization threshold by maximizing between-class variance: $$\sigma_b^2(t) = \omega_0(t)\omega_1(t) \left[\mu_0(t) - \mu_1(t)\right]^2$$
2. Morphological Closing: Performs custom 3x3 binary dilation in dilate_input and erosion in erode_input to bridge small gaps and smooth tag borders.
3. Edge Boundary Detection: Extracted in edge_boundary via $\text{Edge} = \text{Binary} - \text{Eroded}$, leaving a precise outline for contour tracing.

C. Contour Extraction, Quad Detection & Corner Ordering

1. Contour Tracing: The Suzuki-Abe border-following algorithm is implemented in C++ get_contours to trace boundaries and construct topological parent-child relationships.
2. Quad Filtering: Polygons are simplified in simplify_polygon_closed using the Ramer-Douglas-Peucker (RDP) algorithm. We identify tags by filtering for convex, 4-sided outer shapes that contain a nested child contour representing the tag's inner border.
3. Clockwise Ordering: The four corners of the quad are sorted in clockwise order starting from the Top-Left corner in order_corners_clockwise.

D. Homography, Warping, and Orientation Correction

1. Direct Linear Transform (DLT): We estimate the homography matrix $H \in \mathbb{R}^{3 \times 3}$ in solve_homography. For four point pairs $(x_i, y_i) \leftrightarrow (u_i, v_i)$, we build the system $Ah = 0$ where each pair contributes two equations: $$\begin{pmatrix} -x_i & -y_i & -1 & 0 & 0 & 0 & x_i u_i & y_i u_i & u_i \\ 0 & 0 & 0 & -x_i & -y_i & -1 & x_i v_i & y_i v_i & v_i \end{pmatrix} h = 0$$ The system is solved using custom NumPy singular value decomposition.
2. Inverse Warping: Implemented in warp_homography using backward mapping to prevent grid holes, utilizing sample_bilinear to sample fractional coordinate intensities.
3. Tag ID & Orientation Correction: The unwarped tag is represented as an 8x8 grid. In is_valid_tag, we look for the white orientation marker in the bottom-right corner of the inner 4x4 grid. If it's elsewhere, we rotate the corner list and re-warp. Once oriented, the central 2x2 binary sub-grid is read in clockwise order starting from its top-left to decode the 4-bit Tag ID.

E. Camera Calibration & 3D Pose Reconstruction

1. Checkerboard Calibration: Using the physical pattern in caliberation_board.pdf, CameraCalibrator calculates the intrinsic camera matrix $K$: $$K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$$ The results are stored in camera_parameter.npz.
2. 3D Camera Pose Recovery: From the homography $H$ and calibration matrix $K$, the translation vector $t$ and rotation matrix columns $r_1$, $r_2$, and $r_3$ are reconstructed in projection_matrix: $$\lambda = \frac{1}{|K^{-1} h_1|}, \quad r_1 = \lambda K^{-1} h_1, \quad r_2 = \lambda K^{-1} h_2, \quad r_3 = r_1 \times r_2, \quad t = \lambda K^{-1} h_3$$ Orthogonality is enforced on $R = [r_1 \mid r_2 \mid r_3]$ using SVD to form the final projection matrix $P = K[R \mid t]$.
3. Depth Buffering (Painter's Algorithm): To render solid structures, render parses Wavefront OBJ files via OBJ. It calculates the average Z-depth of each polygon face and sorts them in descending order (furthest first). Faces are then rendered sequentially to resolve occlusion.

---

4. Qualitative Results & Visual Walkthrough

A. Camera Calibration

Webcam calibration is performed by presenting the checkerboard pattern. Sub-pixel corners are detected to extract the intrinsic focal lengths and optical centers:

Figure 1: Extracting camera corners on the checkerboard to calculate calibration parameters.

---

B. Single Tag Detection & Augmentation

With a single tag (ID 15) in the frame, the pipeline successfully isolates the borders, decodes the orientation, superimposes the template image, and projects a 3D cow model:

(a) Contour Extraction	(b) ID Decoding & Corners
(c) 2D Logo Overlay	(d) 3D Cow Model Projection

Figure 2: Pipeline steps on a single target: successfully isolating contours, calculating homography, mapping 2D templates, and projecting the 3D Cow.

---

C. Multi-Tag Detection

The system detects multiple markers simultaneously. It processes each tag in isolation to assign unique IDs and render individual 3D objects (e.g., Cow, Wolf, Bot) or overlays on each tag:

(a) Contour Detection	(b) Decoded IDs
(c) Multi 2D Overlay	(d) Multi 3D Projection

Figure 3: Simultaneous detection of multiple tags (ID 12 and ID 15) with independent homography estimation and 3D object rendering.

---

D. Live Webcam Results

The high-efficiency C++ filtering and contour extraction bindings allow the pipeline to perform smooth, real-time tracking and projection on a handheld live video stream:

(a) Live Contours	(b) Real-Time ID Display
(c) Live 2D Overlay	(d) Live 3D Model

Figure 4: Robust webcam tracking showing stable pose estimation despite frame movement and variable background illumination.

---

5. Repository File Structure

Augmented-Reality/
├── Assignment.pdf                        # Project specifications & guidelines
├── Report.pdf                            # Formal project report
├── requirements.txt                      # Project requirements (opencv-python, numpy)
├── README.md                             # Project landing page & documentation (this file)
├── assets/                               # Media files, models, and templates
│   ├── model1.obj                        # 3D Cow model
│   ├── model2.obj                        # 3D Wolf model
│   ├── model3.obj                        # 3D Bot model
│   ├── iitd_logo_template.jpg            # 2D overlay template image
│   ├── caliberation_board.pdf            # PDF pattern for checkerboard calibration
│   ├── AR Tags For Printing.pdf          # PDF of tags used for tracking
│   ├── videos_and_intrinsics/
│   │   ├── singleTag.mp4                 # Video showing a single AR tag
│   │   ├── multipleTags.mp4              # Video showing multiple AR tags
│   │   └── intrinsics.txt                # Ground-truth camera intrinsic values
│   └── [extracted_img_*.png/*.jpg]       # Figures extracted for documentation
└── code/                                 # Implementation source files
    ├── CMakeLists.txt                    # Build configuration for C++ module
    ├── build_filters.sh                  # Shell script compilation trigger
    ├── caliberate/
    │   ├── caliberate.py                 # Intrinsic checkerboard calibration utility
    │   └── camera_parameter.npz          # Saved calibration matrices (mtx, dist)
    ├── cpp/
    │   ├── filters.hpp                   # Headers for C++ filters
    │   ├── filters.cpp                   # Separation Gaussian blur & contour tracer
    │   └── bindings.cpp                  # pybind11 integration wrappers
    └── src/
        ├── utils.py                      # Core computer vision mathematics & algorithms
        └── main.py                       # Application execution entry point

---

6. Setup & Execution Instructions

1. Virtual Environment Setup

Create a virtual environment and install the required libraries:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Compile the C++ Filters Module

Build the high-performance C++ filters library using the compilation script:

cd code
./build_filters.sh
cd ..

3. Running the Pipeline on Input Videos

To run tag tracking and overlay on the provided test videos, run the main script pointing to the video source and specify an optional 3D model:

• For 2D Warped Overlay (IITD Logo):

  venv/bin/python code/src/main.py --video assets/videos_and_intrinsics/singleTag.mp4

• For 3D Model Augmentation (e.g., Cow):

  venv/bin/python code/src/main.py --video assets/videos_and_intrinsics/singleTag.mp4 --model assets/model1.obj

• For Multiple Tags:

  venv/bin/python code/src/main.py --video assets/videos_and_intrinsics/multipleTags.mp4 --model assets/model2.obj

4. Running the Pipeline on Live Webcam

Ensure you have ran camera calibration (instructions below) to obtain your camera parameters. To run using your active webcam feed, execute without a --video argument:

venv/bin/python code/src/main.py --model assets/model3.obj

5. Camera Calibration

If you need to calibrate your webcam to establish correct 3D perspective mapping:

1. Print the pattern caliberation_board.pdf.
2. Run the calibration utility:

   venv/bin/python code/caliberate/caliberate.py --checkerboard 11 8 --max 60

3. Move the board in front of the camera. When the pattern is detected, press c to capture. The utility automatically saves the intrinsic matrix to camera_parameter.npz once 60 frames are captured.

---

7. Mathematical References & Notes

• Separable Convolution: Convolving an image with a 2D Gaussian kernel of size $K \times K$ can be represented as two 1D passes: $$G_{\sigma}(x,y) = G_{\sigma}(x) * G_{\sigma}(y)$$
• Suzuki-Abe Algorithm: Suzuki, S. and Abe, K., Topological Structural Analysis of Digitized Binary Images by Border Following, CVGIP 1985.
• Direct Linear Transformation: Hartley, R. and Zisserman, A., Multiple View Geometry in Computer Vision, Cambridge University Press.
• Painter's Algorithm: Sutherland, I. E., Sproull, R. F., and Schumacker, R. A., A Characterization of Ten Raster-Graphics Algorithms, ACM Computing Surveys 1974.