Propaganda — News Analysis Pipeline

A complete local-first news aggregation, analysis, and reporting pipeline.

Architecture

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│  RSS Feeds  │───▶│  MongoDB    │───▶│  NER        │
│  (main.go)  │    │  (Articles) │    │  (Flair)    │
└─────────────┘    └─────────────┘    └─────────────┘
                                              │
                   ┌──────────────────────────┘
                   ▼
              ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
              │  ChromaDB   │◀──▶│  Hybrid     │───▶│  Report     │
              │  (Vectors)  │    │  Search     │    │  Generation │
              └─────────────┘    └─────────────┘    └─────────────┘
                                              │
                   ┌──────────────────────────┘
                   ▼
              ┌─────────────┐    ┌─────────────┐
              │  T5 Bias    │    │  Video Gen  │
              │  Detection  │    │  (MGM)      │
              └─────────────┘    └─────────────┘

Quick Start

# 1. Load RSS feeds
cd rss && go run . ../config/big.tsv ../config/kill.tsv

# 2. Run NER
cd ner-hub && go run . --start-date -7 endpoints.tsv

# 3. Generate vectors
python db/mongo2chroma.py load --limit 100

# 4. Search articles
python db/hybrid.py "climate change" -n 10

# 5. Generate report
python db/report.py -7 climate_news "Climate developments" Climate

# 6. Generate video
python mgm/mgm.py article.txt output.mp4

Or use the Makefile:

make testrun    # Full pipeline
make smallthingsthatgo  # Quick test

New Capabilities

Vector Loading with Slack Backfill

Load up to 3333 articles per run, backfilling with older articles if date range has fewer:

# Via Makefile (default: 3333)
make vector

# Via Python directly
python db/mongo2chroma.py load --start-date -2 --slack 3333

See docs/slack_backfill.md for details.

Media Coverup Detection

Analyze bias data to find subjects with extreme coverage bias:

# Interactive CLI with Rich
python scripts/find_media_coverups.py --output interactive

# Output to CSV/JSON
python scripts/find_media_coverups.py --output csv,json

# Via Makefile
make analyze-bias-coverage

See docs/media_coverups.md for details.

Token Counting

Count tokens in text files:

# Fast estimation (chars/4)
python llm/tools/count_tokens.py input.txt

# Accurate count with tiktoken
python llm/tools/count_tokens.py input.txt --accurate

Title Printing in Reports

Report generation now prints all article titles for transparency:

=== [israel] Processing 40 articles (before cypher) ===
 "Israel announces new military operation in Gaza"
 "Netanyahug meets with Biden at White House"
 "Israeli forces raid West Bank refugee camp"
...

---

Components

Data Ingestion

File	Description
Makefile	Pipeline orchestration (full reference)

Database & Search

File	Description
mongo2chroma.py	MongoDB → ChromaDB vector loader
hybrid.py	Hybrid vector + BM25 search
geminize.py	LLM processing pipeline
report.py	News report generation

AI Services

File	Description
ner/main.py	Named Entity Recognition (Flair)
ollamaai.py	Ollama LLM client
mgm/mgm.py	Video generation (SD Turbo + Kokoro)

Bias Detection

File	Description
t5/bias_detector/	T5+LoRA bias detection
llm/bias_processor.py	LLM-based bias processing

Documentation (Complete)

Data Ingestion & NER

• main_go.md — RSS feed aggregator (Go)
• ner.md — NER processor (Go)

Database & Search

• mongo2chroma.md — Vector loading & semantic search
• hybrid.md — Hybrid search with BM25 reranking
• dedupe.md — Article deduplication

LLM & Bias

• geminize.md — LLM article processing pipeline
• bias_processor.md — LLM bias processing
• ollamaai.md — Ollama CLI client for text/vision
• t5_bias.md — T5 bias detection system

Training

• lora.md — LoRA training pipeline

Reporting & Generation

• report.md — News report generation with LLM failover
• mgm.md — Video generation with SD Turbo + Kokoro TTS
• tts.md — TTS utilities

Analysis & Clustering

• dashboard.md — Streamlit data visualization dashboard
• dbscan.md — News article clustering

Utilities & Reference

• utilities.md — Utility scripts reference
• makefile.md — All Makefile targets with pipeline graph

Browser Extension

• browser-extension/README.md — Chrome/Firefox extension for real-time bias detection

Shell Scripts

Shell scripts for various pipeline tasks. See individual directory READMEs for full usage documentation.

Directory	Description
db/README.md	Batch processing, vector generation, reporting scripts (37 scripts)
llm/README.md	LLM testing, SVO extraction, bias testing scripts (15 scripts)
ner/README.md	NER service management scripts
ner-hub/README.md	NER processor scripts
mgm/README.md	Video generation scripts
mp3/README.md	TTS batch processing scripts
vec/README.md	Memgraph vector database scripts
dbscan/README.md	Article clustering scripts
redist/README.md	Model training scripts
semantic/README.md	Semantic search scripts

Environment & Setup

• .env.example — All environment variables
• docs/makefile_graph.png — Visual pipeline diagram

---

Quick Reference Index

Need...	Use
Load RSS feeds	make load → main_go.md
Extract entities	make ner → ner-hub/main.go
Search articles	hybrid.md → python db/hybrid.py
Generate vectors	make vector → mongo2chroma.md
Detect bias	make t5bias → llm/bias_processor.py
Create reports	make runreport → report.md
Text-to-speech	make mp3small → mgm.md
Train custom model	make lora-full → LoRA-train/README.md
Serve model	make lora-serve → LoRA-server/server.py
Browser extension	browser-extension/README.md
Dashboard	make dashboard → dashboard.md
Full pipeline	make testrun → makefile.md

Environment Variables

Copy .env.example to .env and fill in your values:

# MongoDB
MONGO_URI=mongodb://user:pass@host:27017
MONGO_USER=root
MONGO_PASS=your_password_here

# LLM APIs
GEMINI_API_KEY=your_gemini_api_key
GROQ_API_KEY=your_groq_api_key
OLLAMA_HOST=localhost:11434

# Services
NER_URL=http://localhost:8100/extract
T5_PORT=1337
CHROMA_PATH=./chroma_db

See .env.example for the complete list.

Directory Structure

propaganda/
├── rss/               # RSS feed aggregator (Go)
├── Makefile           # Pipeline tasks
├── config/            # Feed configs
├── db/                # Database scripts
│   ├── mongo2chroma.py
│   ├── hybrid.py
│   ├── geminize.py
│   └── report.py
├── ner-hub/           # Named Entity Recognition (Go)
├── llm/               # LLM processing
├── t5/                # T5 bias detection
├── mgm/               # Video generation
├── front/             # React web UI
├── back/              # Express API
├── dashboard/         # Streamlit dashboard
└── docs/              # Documentation

Frontend

cd front && npm install && npm start

API server:

cd back && node server.js

Dashboard

cd dashboard && streamlit run app.py

License

MIT — 100% local, no API keys required (except Gemini optional).