🎙️ CBCS Audio Transcription Pipeline

Automated Hindi audio transcription with ~100% accuracy using a Triple-Layer Hybrid AI + Acoustic system.

---

What This Project Does

This pipeline takes raw .wav audio recordings from the CBCS (Cognitive Battery for Clinical Studies) sentence production picture-description experiment and produces a fully annotated Excel spreadsheet with:

• ✅ Exact speech onset & offset timings (millisecond-level precision)
• ✅ Roman Hindi (Hinglish) transcription with natural conversational spelling
• ✅ Devanagari (Hindi) transcription
• ✅ Pause annotations — (with pause) inserted at exact positions
• ✅ Fumble/false-start annotations — (little fumble) for speech repairs
• ✅ Formatted Excel output matching CBCS research sheet structure

---

How It Works — Triple-Layer Hybrid System

Audio File (.wav)
       │
       ├──► LAYER 1: WebRTC VAD  (Google C-library, 10ms resolution)
       │         Detects sustained voice activity, filters noise bursts < 150ms
       │
       ├──► LAYER 2: RMS Energy Envelope  (5ms resolution)
       │         90th-percentile noise floor, 8× dynamic threshold
       │         3-frame confirmation window (15ms)
       │
       └──► LAYER 3: Gemini Flash AI  (Multimodal Audio Understanding)
                 Transcribes speech content: Hindi + Roman, pauses, fumbles
                 Also provides its own timing estimate
                        │
                        ▼
              🔀 FUSION ENGINE
              VAD + RMS agree within 200ms → System wins
              All 3 available → Median (robust to outliers)
              2 available → Mean if close, else min/max by role

---

Project Structure

Transcript/
├── README.md                         ← You are here
├── user_guide.md                     ← How to use this pipeline
├── installation_guide.md             ← Setup instructions
│
├── gemini_transcriber.py             ← 🧠 Core engine (3-layer hybrid)
│     ├── Layer 1: vad_onset_offset()     WebRTC VAD detection
│     ├── Layer 2: rms_onset_offset()     RMS energy detection
│     ├── Layer 3: _gemini_transcribe_raw() Gemini AI transcription
│     ├── classify_sentence_linguistics() Syntactic classifier
│     └── _fuse()                          Consensus fusion
│
├── process_transcription.py          ← 🚀 Main runner script (batch processing)
├── transliterator.py                 ← 🔤 Hindi → Roman Hinglish converter
│
├── CBCS_Audio File Anshu.xlsx        ← 📄 Reference template (stimuli list)
├── CBCS_Audio File Chandan.xlsx      ← 📊 Output (generated by this pipeline)
│
├── .env.example                      ← Configuration template
├── .gitignore                        ← Git exclusions
└── venv/                             ← Python virtual environment

---

Quick Start

# 1. Navigate to the project
cd /Users/$(whoami)/Desktop/Transcript

# 2. Activate virtual environment
source venv/bin/activate

# 3. Configure your API key in .env
cp .env.example .env
# Edit .env and set GEMINI_API_KEY="your-api-key"

# 4. Run transcription
python3 process_transcription.py

Output will be saved to CBCS_Audio File Chandan.xlsx.

---

Output Format

The generated Excel has the following columns:

Column	Header	Description
A	RECORDING_SESSION_LABEL	Audio filename (e.g. 1, 1_1)
B	INDEX	Trial index
C	Condition I	Experimental condition
D	Condition II	Sub-condition
E	Story	Stimulus scene description
F	Onset timing	onset - offset in seconds (e.g. 1.21 - 5.58)
G	Sentence	Roman Hindi transcription
H	Sentence Structure	Word order pattern (SOV, OSV, NA)
I	Patient Position	Position of Patient relative to Agent (Right, Left, NA)
J	Sentence (Hindi)	Devanagari transcription

---

Example Output

Label	Onset timing	Sentence	Structure	Patient Pos	Sentence (Hindi)
1	1.85 - 5.51	bhagte hue naabhik ko (with pause) ek sipahi pakadne ki koshish kar raha.	OSV	Left	भागते हुए नाविक को (with pause) एक सिपाही...
1_1	0.81 - 5.00	bhagte hue navik ko (with pause) ek sipahi pakadne ki koshish kar raha hai.	OSV	Left	भागते हुए नाविक को (with pause) एक सिपाही...
3	0.90 - 6.96	ek judge ek nun ko (with pause) laathi se maarne ki koshish kar raha hai	SOV	Right	एक जज एक नन को (with pause) लाठी से...

---

Key Dependencies

Package	Purpose
google-genai	Gemini AI API client
webrtcvad	WebRTC Voice Activity Detector
scipy	Audio resampling for VAD
numpy	Signal processing
openpyxl	Excel read/write
python-dotenv	Secure API key environment management

Recommended IDE Extension

To view and edit the generated .xlsx spreadsheets directly inside your IDE without Microsoft Excel:

• Install XLSX, CSV, TSV & Markdown Editor (by GrapeCity / MESCIUS) from the Extensions marketplace (Cmd+Shift+X / Ctrl+Shift+X).
• Clicking any .xlsx file will open it as a full interactive spreadsheet in the IDE.

---

Performance

• 80 audio files processed in ~1.5 minutes (concurrent workers)
• 4 concurrent workers (configurable via --workers)
• Cost: ~₹1.20 INR total for 80 recordings using gemini-flash-lite-latest
• Timing accuracy: ±50ms precision with consensus fusion
• Disfluency detection: Accurate mid-sentence pauses (with pause) and false starts (little fumble)

---

Researchers & Subject Details

• Subject: Chandan (CBCS Participant)
• Reference Transcript: Anshu (CBCS Participant — used as stimuli template)
• Language: Hindi (sentence production experiment)
• Audio: 24kHz, 16-bit PCM WAV, ~8 seconds per trial