🔍 digdig

A multi-backend web search CLI for agents
SearXNG · Ollama · DeepSeek — privacy-respecting search from your terminal

Features • Installation • Quick Start • Usage • Configuration

🌏 中文

✨ Features

🔎 Multi-Engine Search — Search across Google, Bing, DuckDuckGo, GitHub, StackOverflow, and 30+ engines simultaneously
🔌 Pluggable Backends — SearXNG is the default; select Ollama web search or DeepSeek Search with -e ollama / -e deepseek. No backend is an automatic fallback for another.
🔄 Dynamic Discovery — Auto-fetches available engines and categories from your SearXNG server
📄 Multiple Formats — Markdown (LLM-optimized) or JSON output
🧠 Deep Search — Multi-round iterative research with session accumulation and counted coverage facts
🔍 Content Extraction — Extract full article content from URLs or session results, with Obscura fallback and Agent-selected Jina Reader extraction
🗂️ Session Management — Separates search discovery from extracted bodies; stable result IDs and revisions prevent stale approvals in parallel work
🔗 External Result Fusion — Import external search or extraction output from session-scoped JSON files, without PowerShell inline-JSON escaping risks
⭐ Counted Facts, Not Verdicts — --quality reports result counts, extraction state, sources, and per-result facts; the judgement stays with the Agent
🕸️ Knowledge Graph — Structural (query→result→domain) + semantic (entity relations with approved result-ID provenance) graph layers
🔄 Query Redundancy Check — Word-level or character-bigram Jaccard detects repeated queries
💡 Agent-First Design — Outputs structured analysis data (facts, suggestions, strategy) for LLM Agent decision-making
📁 Local Document Search — Index and BM25-search local Markdown/text files with field-weighted ranking; indexed chunks remain separate extracted session results
✅ Claim—Evidence—Review Pipeline — L2/L3 only: submit atomic claims, auto-search evidence, verify with stance, then policy-aggregate by publisher-domain diversity for auto-approval or Agent review

📦 Installation

Self-host SearXNG

For WSL

WSL2 will automatically shut itself down after you exit all the connections. I suggest you use https://github.com/gardengim/keepwsl to keep it alive.

If you also want WSL to start automatically when Windows boots (so your SearXNG containers come up without manual intervention), do the following:

Press Win+R, type shell:startup, and press Enter — this opens Windows Startup folder
Right-click → New → Shortcut, set location to "C:\Program Files\WSL\wsl.exe" -d Ubuntu cd ~
Save the shortcut. Next time Windows starts, cd ~ will fail (due to the missing backslash) but WSL will have already been launched — the terminal window closes automatically and WSL keeps running in the background.

Before starting the searXNG container, you must create a settings.yml file in the ./searxng directory. You can visit https://github.com/searxng/searxng for specific configuration methods.

An example of settings.yml is just like below.

📋 Click to expand full settings.yml (30+ search engines)

use_default_settings: true

server:
  secret_key: "random string"
  limiter: false

outgoing:
  request_timeout: 10.0 # 全局默认超时
  max_request_timeout: 25.0
  pool_connections: 200
  pool_maxsize: 20
  retries: 1

search:
  safe_search: 0
  formats:
    - html
    - json
    - csv
    - rss

valkey:
  url: valkey://valkey:6379/0

engines:
  # ==================== 通用搜索 ====================
  - name: google
    engine: google
    shortcut: g

  - name: bing
    engine: bing
    shortcut: bi
    disabled: false

  - name: duckduckgo
    engine: duckduckgo
    shortcut: ddg

  - name: brave
    engine: brave
    shortcut: br

  - name: startpage
    engine: startpage
    shortcut: sp

  - name: qwant
    engine: qwant
    shortcut: qw
    disabled: false

  - name: yandex
    engine: yandex
    shortcut: yx
    disabled: false

  - name: karmasearch
    engine: karmasearch
    categories: [general, web]
    search_type: web
    shortcut: ka
    disabled: true  # 引擎文件不存在

  # ==================== 中文搜索 ====================
  - name: baidu
    engine: baidu
    shortcut: bd
    disabled: false

  - name: sogou
    engine: sogou
    shortcut: sg
    disabled: false

  - name: 360search
    engine: 360search
    shortcut: 360so
    disabled: false

  - name: quark
    engine: quark
    shortcut: qk
    disabled: false

  # ==================== 编程相关 ====================
  - name: github
    engine: github
    shortcut: gh

  - name: github code
    engine: github_code
    shortcut: ghc

  - name: gitlab
    engine: gitlab
    base_url: https://gitlab.com
    shortcut: gl
    disabled: false

  - name: codeberg
    engine: gitea
    base_url: https://codeberg.org
    shortcut: cb
    disabled: false

  - name: stackexchange
    engine: stackexchange
    shortcut: se

  - name: stackoverflow
    engine: stackexchange
    shortcut: so
    categories: q&a
    stackexchange_site: stackoverflow

  - name: npm
    engine: npm
    shortcut: npm
    disabled: false

  - name: pypi
    engine: pypi
    shortcut: py

  - name: crates.io
    engine: crates
    shortcut: crate
    disabled: false

  - name: pkg.go.dev
    engine: pkg_go_dev
    shortcut: go
    disabled: false

  - name: metacpan
    engine: metacpan
    shortcut: cpan
    disabled: false

  - name: docker hub
    engine: docker_hub
    shortcut: dh

  - name: huggingface
    engine: huggingface
    shortcut: hf
    disabled: false

  - name: huggingface datasets
    engine: huggingface
    huggingface_endpoint: datasets
    shortcut: hfd
    disabled: false

  - name: hex
    engine: hex
    shortcut: hex
    disabled: false

  - name: mdn
    engine: json_engine
    shortcut: mdn
    categories: [it]
    paging: true
    search_url: https://developer.mozilla.org/api/v1/search?q={query}&page={pageno}
    results_query: documents
    url_query: mdn_url
    url_prefix: https://developer.mozilla.org
    title_query: title
    content_query: summary

  - name: arch linux wiki
    engine: archlinux
    shortcut: al

  - name: gentoo wiki
    engine: mediawiki
    shortcut: gentoo
    categories: ["it", "software wikis"]
    base_url: "https://wiki.gentoo.org/"
    api_path: "api.php"
    search_type: text
    

  - name: lobste.rs
    engine: xpath
    search_url: https://lobste.rs/search?q={query}&what=stories&order=relevance
    results_xpath: //li[contains(@class, "story")]
    url_xpath: .//a[@class="u-url"]/@href
    title_xpath: .//a[@class="u-url"]
    content_xpath: .//a[@class="domain"]
    categories: it
    shortcut: lo
    
    disabled: false

  # ==================== IT/编程补充 ====================
  - name: superuser
    engine: superuser
    shortcut: su
    disabled: false

  - name: askubuntu
    engine: askubuntu
    shortcut: au
    disabled: false

  # ==================== 知识/问答 ====================
  - name: wikipedia
    engine: wikipedia
    shortcut: wp
    display_type: ["infobox"]
    categories: [general]

  - name: wikidata
    engine: wikidata
    shortcut: wd
    
    weight: 2
    display_type: ["infobox"]
    categories: [general]

  - name: reddit
    engine: reddit
    shortcut: re
    disabled: false

  - name: hackernews
    engine: hackernews
    shortcut: hn
    disabled: false

  # ==================== 词典 ====================
  - name: wiktionary
    engine: wiktionary
    shortcut: wkt
    disabled: false

  # ==================== 图片 ====================
  - name: google images
    engine: google_images
    shortcut: goi

  - name: bing images
    engine: bing_images
    shortcut: bii

  - name: duckduckgo images
    engine: duckduckgo_extra
    categories: [images]
    ddg_category: images
    shortcut: ddi

  - name: pinterest
    engine: pinterest
    shortcut: pin

  - name: unsplash
    engine: unsplash
    shortcut: us

  - name: pixabay
    engine: pixabay
    shortcut: pxb

  - name: deviantart
    engine: deviantart
    shortcut: da
    disabled: false

  - name: flickr
    categories: images
    shortcut: fl
    engine: flickr_noapi
    disabled: false

  - name: openverse
    engine: openverse
    categories: images
    shortcut: opv
    disabled: false

  - name: artic
    engine: artic
    shortcut: arc
    disabled: false

  - name: pexels
    engine: pexels
    shortcut: pex
    disabled: false

  # ==================== 视频 ====================
  - name: google videos
    engine: google_videos
    shortcut: gov

  - name: bing videos
    engine: bing_videos
    shortcut: biv

  - name: duckduckgo videos
    engine: duckduckgo_extra
    categories: [videos]
    ddg_category: videos
    shortcut: ddv

  - name: youtube
    engine: youtube_noapi
    shortcut: yt

  - name: bilibili
    engine: bilibili
    shortcut: bili
    disabled: false

  - name: dailymotion
    engine: dailymotion
    shortcut: dm
    disabled: false

  - name: vimeo
    engine: vimeo
    shortcut: vi
    disabled: false

  - name: odysee
    engine: odysee
    shortcut: od
    disabled: false

  - name: peertube
    engine: peertube
    shortcut: pt
    disabled: false

  # ==================== 新闻 ====================
  - name: google news
    engine: google_news
    shortcut: gon

  - name: bing news
    engine: bing_news
    shortcut: bin

  - name: duckduckgo news
    engine: duckduckgo_extra
    categories: [news]
    ddg_category: news
    shortcut: ddn

  - name: reuters
    engine: reuters
    shortcut: rtr
    disabled: false

  # ==================== 音乐 ====================
  - name: bandcamp
    engine: bandcamp
    shortcut: bc
    categories: music
    disabled: false

  - name: deezer
    engine: deezer
    shortcut: dz
    disabled: false

  - name: mixcloud
    engine: mixcloud
    shortcut: mc
    disabled: false

  - name: genius
    engine: genius
    shortcut: gen
    disabled: false

  - name: soundcloud
    engine: soundcloud
    shortcut: sc
    disabled: false

  - name: radio_browser
    engine: radio_browser
    shortcut: rb
    disabled: false

  # ==================== 学术/文档 ====================
  - name: arxiv
    engine: arxiv
    shortcut: arx

  - name: semantic scholar
    engine: semantic_scholar
    shortcut: sem

  - name: google scholar
    engine: google_scholar
    shortcut: gsch

  - name: pubmed
    engine: pubmed
    shortcut: pub

  - name: crossref
    engine: crossref
    shortcut: cr
    disabled: false

  - name: openalex
    engine: openalex
    shortcut: oa
    disabled: false

  # ==================== EU 开放研究 ====================
  - name: openaire publications
    engine: openairepublications
    shortcut: oarp
    disabled: false

  - name: openaire datasets
    engine: openairedatasets
    shortcut: oard
    disabled: false

  # ==================== 社交媒体 ====================
  - name: lemmy posts
    engine: lemmy
    lemmy_type: Posts
    shortcut: lepo
    disabled: false

  - name: mastodon users
    engine: mastodon
    mastodon_type: accounts
    base_url: https://mastodon.social
    shortcut: mau
    disabled: false

  # ==================== 文件/种子 ====================
  - name: library genesis
    engine: xpath
    search_url: https://libgen.rs/search.php?req={query}
    url_xpath: //a[contains(@href,"book/index.php?md5")]/@href
    title_xpath: //a[contains(@href,"book/")]/text()[1]
    content_xpath: //td/a[1][contains(@href,"=author")]/text()
    categories: files
    shortcut: lg
    disabled: false

  - name: kickass
    engine: kickass
    base_url:
      - https://kickasstorrents.to
      - https://kickasstorrents.cr
    shortcut: kc
    disabled: false

  - name: annas archive
    engine: annas_archive
    base_url:
      - https://annas-archive.gl
      - https://annas-archive.vg
    shortcut: aa
    disabled: false

  - name: 1337x
    engine: 1337x
    shortcut: "1337"
    disabled: false

  - name: piratebay
    engine: piratebay
    shortcut: tpb
    disabled: false

  # ==================== 翻译 ====================
  - name: lingva
    engine: lingva
    shortcut: lv
    disabled: false

  - name: currency
    engine: currency_convert
    shortcut: cc

  # ==================== 其他 ====================
  - name: imdb
    engine: imdb
    shortcut: imdb
    disabled: false

  - name: steam
    engine: steam
    shortcut: stm
    disabled: false

  - name: goodreads
    engine: goodreads
    shortcut: good
    disabled: false

  # ==================== 天气 ====================
  - name: wttr.in
    engine: wttr.in
    shortcut: wea
    disabled: false

  # ==================== 地图/位置 ====================
  - name: openstreetmap
    engine: openstreetmap
    shortcut: osm
    disabled: false

  - name: nominatim
    engine: json_engine
    search_url: https://nominatim.openstreetmap.org/search?q={query}&format=json&limit={pageno}
    results_query: "."
    url_query: display_name
    title_query: display_name
    content_query: type
    shortcut: nom
    disabled: false

  # ==================== 法律/政策 ====================
  - name: wikisource
    engine: wikisource
    shortcut: ws
    disabled: false

  # ==================== 教育/学习 ====================
  - name: wikiversity
    engine: wikiversity
    shortcut: wvy
    disabled: false

  # ==================== 旅行 ====================
  - name: wikivoyage
    engine: wikivoyage
    shortcut: wv
    disabled: false

  # ==================== 以下引擎需要付费/API Key，默认禁用 ====================
  - name: wolframalpha
    engine: wolframalpha
    shortcut: wa
    disabled: true  # 需要付费

  - name: rawg
    engine: json_engine
    search_url: https://api.rawg.io/api/games?search={query}&key=YOUR_API_KEY
    results_query: results
    url_query: slug
    title_query: name
    content_query: released
    shortcut: rawg
    disabled: true  # 需要API key

  - name: podcastindex
    engine: json_engine
    search_url: https://api.podcastindex.org/api/1.0/search/byterm?q={query}
    results_query: feeds
    url_query: link
    title_query: title
    content_query: description
    shortcut: pci
    disabled: true  # 需要API认证

An example of docker-compose.yml is just like below.

services:
   searxng:
        image: docker.io/searxng/searxng:latest
        container_name: searxng
        restart: unless-stopped
        ports:
            - "8080:8080"
        volumes:
            - ./searxng:/etc/searxng:Z
        depends_on:
            - valkey
        ulimits:
            nofile:
                soft: 10000
                hard: 65535

    valkey:
        container_name: valkey
        image: docker.io/valkey/valkey:9-alpine
        command: valkey-server --save 30 1 --loglevel warning
        restart: always
        volumes:
            - ./valkey:/data/

Start the containers:

docker compose up -d

This runs SearXNG (port 8080) and Valkey in the background. Verify with docker compose ps or dg --health.

For Python venv (no Docker)

SearXNG is not published on PyPI — there is no pip install searxng. The official method is cloning the source and installing into a venv (see https://docs.searxng.org/admin/installation-searxng.html ). Below is a one-shot script for local dev (WSL/Linux with python3 + git):

# 1) System deps (required to compile lxml/cryptography/msgspec, etc.)
sudo apt-get install -y \
  python3-dev python3-venv python-is-python3 \
  git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev

# 2) Clone + venv + install (official method)
git clone --depth 1 https://github.com/searxng/searxng.git ~/searxng-src
python3 -m venv ~/searxng-src/.venv
~/searxng-src/.venv/bin/pip install -e ~/searxng-src

# 3) Minimal settings.yml (enable json so the CLI can read it)
mkdir -p ~/.config/searxng
cat > ~/.config/searxng/settings.yml <<'EOF'
use_default_settings: true

server:
  secret_key: "ultrasecretkey"   # local dev default; replace with `openssl rand -hex 16` for anything real
  limiter: false

search:
  safe_search: 0
  formats:
    - html
    - json

valkey:
  url: false   # single-host dev doesn't need Redis/Valkey
EOF

# 4) Start (default port 8888; use uWSGI per official docs for production)
SEARXNG_SETTINGS_PATH=~/.config/searxng/settings.yml \
  ~/searxng-src/.venv/bin/python -m searx.webapp

# 5) Verify + wire into digdig
curl -s http://localhost:8888/healthz   # returns OK
dg init                                # baseUrl = http://localhost:8888
dg --health && dg "hello world"

From npm (Recommended)

npm install -g digdig

This installs three names for the same program: dg (recommended, short), digdig (explicit), and the transitional sxng alias.

⚠️ Upgrading from sxng-cli? digdig replaces sxng-cli; the version numbering starts over, so digdig@0.0.2 is newer than sxng-cli@1.3.2 even though the number is smaller. Uninstall the old package first (npm uninstall -g sxng-cli), because two installed copies keep two separate session directories and will not see each other's sessions.

For five versions (through 0.5.0) everything from the old name keeps working: the sxng command, .sxng/ sessions, sxng.config.json, and SEARXNG_* environment variables. They print a one-line [deprecation] notice to stderr (silence it with DIGDIG_NO_DEPRECATION=1) and stop working in 0.5.0. To migrate, rename .sxng → .digdig, sxng.config.json → digdig.config.json, and any SEARXNG_* variable → DIGDIG_*.

npx skills add hkwuks/digdig

⚠️ Skill Sync: After updating digdig, also update digdig skill to keep them in sync:
npx skills update hkwuks/digdig

From Source

git clone https://github.com/hkwuks/digdig.git
cd digdig/cli
npm install
npm run build
npm link

🤖 LLM One-Click Setup (WSL / Linux)

Prerequisite: a WSL or Linux environment that your LLM agent can connect to (SSH, local shell, or terminal integration). Everything below can be done by the LLM itself with the commands it is given here.

If you prefer to let an LLM agent do the install and configuration for you, point it at this repo and give it this prompt:

You have access to a WSL/Linux machine. Install digdig and set up its search backend end to end:

1. **SearXNG backend** (recommended, self-hosted, private) — two options, prefer Docker:
   - **Option A — Docker (preferred)**: Create `./searxng/settings.yml` first from the example in this README (enable `formats: html, json` so the CLI can read JSON). It must exist before the container starts, or SearXNG will not pick up your config. Then run `docker compose up -d` using this repo's example `docker-compose.yml` (SearXNG on host port 8080 + Valkey). Wait until `curl -s http://localhost:8080/healthz` returns OK.
   - **Option B — Python venv (official method, use when Docker is unavailable)**: SearXNG is **not** published on PyPI — there is **no** `pip install searxng`. Follow the official pip/venv installation (https://docs.searxng.org/admin/installation-searxng.html ):
     ```bash
     # 1) system deps (official list)
     sudo apt-get install -y \
       python3-dev python3-babel python3-venv python-is-python3 \
       git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev

     # 2) clone + venv + install (official method)
     git clone --depth 1 https://github.com/searxng/searxng.git ~/searxng-src
     python3 -m venv ~/searxng-src/searx-pyenv
     source ~/searxng-src/searx-pyenv/bin/activate
     pip install -U pip setuptools wheel pyyaml msgspec typing-extensions pybind11
     cd ~/searxng-src
     pip install --use-pep517 --no-build-isolation -e .

     # 3) minimal settings.yml (json format so the CLI can read it)
     mkdir -p ~/.config/searxng
     cat > ~/.config/searxng/settings.yml <<'EOF'
     use_default_settings: true
     server:
       secret_key: "ultrasecretkey"   # local dev; replace with `openssl rand -hex 16` for anything real
       limiter: false
     search:
       safe_search: 0
       formats:
         - html
         - json
     valkey:
       url: false   # single-host dev doesn't need Redis/Valkey
     EOF

     # 4) start (default port 8888, dev only; use uWSGI per official docs for production)
     cd ~/searxng-src
     SEARXNG_SETTINGS_PATH=~/.config/searxng/settings.yml python -m searx.webapp

     # 5) verify + wire into digdig
     curl -s http://localhost:8888/healthz   # returns OK
     ```
     Then use `baseUrl` = `http://localhost:8888` in step 4.
2. **Extra backends** (optional):
   - Ask the user for an Ollama web search API key (https://ollama.com) and/or a DeepSeek Search API key (https://api-docs.deepseek.com/api/search), or skip if they have none. Both are used only when explicitly selected with `-e ollama` / `-e deepseek`; neither is a fallback.
3. **Install the CLI**: `npm install -g digdig`.
4. **Configure**: run `dg init` and fill in `baseUrl` (`http://localhost:8080` for Docker, `http://localhost:8888` for venv), timeout, and any API keys they provided.
5. **Verify**: run `dg --health` and `dg "hello world"`. Report the result.

The LLM should run these checks before declaring success: docker compose ps (SearXNG + Valkey up), dg --health (healthy), and one real search returning non-empty results.

Obscura (Optional — for JS-heavy pages)

dg extract uses Defuddle + linkedom by default for lightweight content extraction. When a page requires JavaScript rendering (SPAs, dynamic content), enable Obscura as a fallback. If --obscura is used and the binary is absent, digdig automatically downloads the matching release asset from the official GitHub HTTPS download endpoint:

# Linux x86_64
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-linux.tar.gz
tar xzf obscura-x86_64-linux.tar.gz
cp obscura ~/.local/bin/

# macOS Apple Silicon
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-aarch64-macos.tar.gz
tar xzf obscura-aarch64-macos.tar.gz
cp obscura /usr/local/bin/

# Docker
docker run -d --name obscura -p 127.0.0.1:9222:9222 h4ckf0r0day/obscura

# Verify
obscura --version

No extra npm dependencies needed — Obscura is called via CLI. Auto-detected from PATH, ~/.local/bin/obscura, or /usr/local/bin/obscura.

🚀 Quick Start

Install the CLI:
```
npm install -g digdig
```

Configure the CLI:

dg init

Or set environment variable:

export DIGDIG_BASE_URL=http://your-searxng-instance:8080

Perform a search:
```
dg "TypeScript tutorial"
```

📖 Usage

Commands

Command	Description
`dg init`	Interactive configuration setup
`dg <query>`	Perform a web search
`dg --queries "q1,q2"`	Multi-query search with RRF fusion
`dg extract --urls <urls>`	Extract content from web pages
`dg extract --session <name>`	Extract session results and merge content
`dg extract --obscura`	JS-rendering fallback for SPA pages
`dg extract --urls <url> --jina`	Agent-selected Jina Reader extraction for explicit URLs
`dg --session new`	Create deep search session
`dg --session <name> --quality`	List pending results with what the session has gathered, as counted facts
`dg --session <name> --quality --approve-file <path>`	Approve verified pending results selected by `{id, revision}` JSON
`dg suggest-queries <session>`	Get query suggestion data for Agent
`dg strategy-info <session>`	Check current search stage
`dg session-report <session>`	Full session analysis report
`dg coverage <session>`	What the session has looked at so far, as counted facts (rounds, queries, new vs repeated, round-local terms)
`dg session-list`	List all sessions
`dg session-delete <session-name>`	Delete a session
`dg graph-preprocess <session>`	TF-IDF + co-occurrence + result provenance analysis
`dg graph-add <session> --data-file <path>`	Add semantic entities/edges backed by approved result IDs
`dg graph-search <session>`	Discover entities by keyword
`dg graph-explore <session>`	View entity relations
`dg graph-drill <session>`	Follow specific relations
`dg graph-traverse <session>`	Traverse reasoning paths
`dg graph-obfuscate <session>`	List obfuscation candidates
`dg results-add --kind <search	extracted> --data-file `
`dg doc-index <path>`	Index local documents for BM25 search
`dg doc-search <session> <query> --path <path>`	Search indexed docs and inject results into session
`dg claim-add <session> --claims-file <path>`	Submit atomic claims from UTF-8 JSON (single or batch, auto evidence-search)
`dg claim-list <session>`	List claims
`dg evidence-search <session> --claim-id <id>`	Search candidate evidence (read-only)
`dg evidence-verify <session> --claim-id <id> --evidence-file <path>`	Confirm UTF-8 JSON evidence + submit stance (+ optional auto-policy)
`dg evidence-list <session> --claim-id <id>`	List evidence for a claim
`dg verdict-list <session> --claim-id <id>`	List verdicts for a claim
`dg policy-aggregate <session>`	Run policy aggregation manually
`dg review-list <session>`	List reviews
`dg --health`	Check the SearXNG backend health (Ollama/DeepSeek have no health endpoint)
`dg --engines-list`	List available search engines
`dg --categories-list`	List available categories
`dg --help`	Show help message

Search Options

Option	Description
`-e, --engines <list>`	Comma-separated list of search engines (e.g., `google,github`); `ollama` and `deepseek` route to those providers instead of SearXNG
`-c, --categories <list>`	Comma-separated list of categories (e.g., `it,science`)
`-l, --limit <n>`	Maximum number of results (default: 10)
`-p, --page <n>`	Page number for pagination
`--lang <code>`	Language code (e.g., `en`, `zh`, `ja`)
`--time <range>`	Time range: `day`, `week`, `month`, `year`, `all`
`-f, --format <fmt>`	Output format: `md` (default) or `json`
`--queries <list>`	Multi-query with RRF fusion (e.g., `q1,q2,q3`)
`--session <session-name>`	Session directory or `new` for deep search
`--owner <session-name>`	Session owner identifier
`--desc <text>`	Session description
`--redundancy <action>`	Query redundancy check: `warn`, `adjust`, `skip`
`--quality`	List pending results and counted session facts (requires --session)
`--approve-file <path>`	Approve `{id, revision}` selections from this session's `agent-inputs` directory
`--skip-file <path>` / `--unskip-file <path>`	Skip or restore `{id, revision}` selections from this session's `agent-inputs` directory
`--merge <file>`	Merge new results with previous search JSON

Examples

# Basic search (outputs Markdown by default)
dg "machine learning"

# Output as JSON
dg --format json "machine learning"

# Search with specific engines
dg --engines google,duckduckgo "privacy tools"

# Search IT and Science categories
dg --categories it,science "kubernetes tutorial"

# Limit results and filter by time
dg --limit 5 --time week "latest AI news"

# Multi-query search with RRF fusion
dg --queries "tokio tutorial,rust async basics,async-std guide"

# List available engines (fetched from server)
dg --engines-list

# List available categories (fetched from server)
dg --categories-list

⚙️ Configuration

Configuration priority (highest to lowest):

Environment variables
Local config file (./digdig.config.json)
Global config file (~/digdig/digdig.config.json)
Default values

Environment Variables

Variable	Description	Default
`DIGDIG_BASE_URL`	SearXNG server URL	(required)
`DIGDIG_DEFAULT_ENGINE`	Default search engine	(none)
`DIGDIG_ALLOWED_ENGINES`	Comma-separated allowed engines	(all)
`DIGDIG_DEFAULT_LIMIT`	Default result limit	`10`
`DIGDIG_DEFAULT_FORMAT`	Default output format (`md` or `json`)	`md`
`DIGDIG_USE_PROXY`	Use proxy (`true`/`false`)	`false`
`DIGDIG_PROXY_URL`	Proxy URL	(none)
`DIGDIG_TIMEOUT`	Request timeout in ms	`10000`
`DIGDIG_REDUNDANCY_THRESHOLD`	Word-level Jaccard threshold for redundancy	`0.7`
`DIGDIG_REDUNDANCY_BIGRAM_THRESHOLD`	Bigram-level Jaccard threshold (short queries)	`0.5`
`OLLAMA_API_KEY`	Ollama web search API key; enable with `-e ollama` (not an automatic fallback)	(none)
`DEEPSEEK_API_KEY`	DeepSeek Search API key (optional backend; billed per successful request)	(none)

SEARXNG_* variable names are still read until 0.5.0 and emit one [deprecation] line on stderr; DIGDIG_* wins when both are set.

Config File

Config file search order (first found wins):

Local config - ./digdig.config.json (current working directory, for project-specific settings)
Global config - ~/digdig/digdig.config.json (user home directory, for global defaults)

Create digdig.config.json:

{
  "baseUrl": "http://localhost:8080",
  "defaultEngine": "",
  "allowedEngines": [],
  "defaultLimit": 10,
  "defaultFormat": "md",
  "useProxy": false,
  "proxyUrl": "",
  "timeout": 10000,
  "ollamaApiKey": "",
  "deepseekApiKey": "",
  "redundancyThreshold": 0.7,
  "redundancyBigramThreshold": 0.5
}

🧠 Deep Search

Deep search enables multi-round iterative research with session accumulation, counted coverage facts, agent-controlled graph injection, and a Query Plan that records what is still missing.

Workflow

Search → Extract → Quality Assess → Approve → Build Graph → (Loop or Explore)
                                                                         ↓
                                          (L2/L3) Claim—Evidence—Review → Final Output

Quick Example

# 1. Create a session and search
dg --session new --owner "agent-1" --desc "Rust async study" "rust async ecosystem"
# Session created: .digdig/sessions/<session-name>

# 2. Extract content from results
dg extract --session <session-name>

# 3. Check extraction output: stats.success and session.updated identify
#    results that received source text; failed URLs remain pending.

# 4. Review the pending facts and each result's verified state
dg --session <session-name> --quality

# 5. Copy selected {id, revision} objects from the --quality output into
#    .digdig/sessions/<session-name>/agent-inputs/approve.json, then approve them.
dg --session <session-name> --quality --approve-file .\.digdig\sessions\<session-name>\agent-inputs\approve.json

# 6. Preprocess session content for entity discovery and provenance
dg graph-preprocess <session-name>

# 7. Put complex graph JSON in this session's Agent scratch directory, then add semantic edges
dg graph-add <session-name> --data-file .\.digdig\sessions\<session-name>\agent-inputs\graph-data.json

# 8. Continue research with redundancy check
dg --session <session-name> --queries "tokio vs async-std,benchmark 2026" --redundancy warn

Session Coverage

A --session search shows only what that invocation found; earlier rounds still contribute to the fusion score but do not appear in the output. dg coverage <session> restores the whole-session view on demand, from results.json alone — no graph, no approval, no extraction required.

It reports counted facts, not conclusions: rounds with their queries and engines, how many results each round first introduced, how many it returned that an earlier round had already seen, which URLs recur across rounds, per round the terms confined to that round, which hosts the session actually drew from, and which of the common primary-source hosts (arXiv, GitHub, Stack Overflow, package registries) it has not touched. A round whose every result was already known is the clearest saturation signal. Terms spanning every round are the session's subject and are excluded, so each round is named by what it actually added.

Deciding which aspects are covered and which are missing is the Agent's job. dg coverage supplies the numbers.

Quality and Evidence Semantics

Extracted web content is deduplicated by normalized URL, then by full-text character 5-gram Jaccard similarity. Query redundancy uses a separate word-level or character-bigram Jaccard check.
--quality reports counted facts: how many results the session holds, how many carry an extracted body, how many distinct sources, and each pending result's id, revision, source, body length, extraction and verification state, and the round that first saw it. It does not score them.
Quality is a diagnostic, not a fact verifier. A search discovery can be approved only after extract writes a non-empty body and extraction timestamp. An explicitly imported external body must provide non-empty content and an extractor; absent extractedAt is recorded as its import time.
Claim policy treats two normalized publisher domains as two sources. It does not infer corporate ownership, editorial relationships, or syndication across domains.

Session Management

Command	Description
`dg --session new`	Create new auto-named session
`dg --session <session-name>`	Use session by name (auto-resolves to `.digdig/sessions/<session-name>`)
`dg --session <path>`	Use session by full path
`dg session-list`	List all sessions with stats
`dg session-delete <session-name>`	Delete specific session
`dg session-delete --older <hours>`	Delete old sessions

Session Path Resolution:

Pure name (e.g., my-session) → .digdig/sessions/my-session
Full path (e.g., /custom/path/session) → used as-is
new → auto-generate unique name under .digdig/sessions/

Session Data Structure

Each session stores state and Agent input files in .digdig/sessions/<session-name>/:

results.json — Accumulated search results (URL dedup, multi-round)
graph.json — Knowledge graph (structural + semantic layers)
meta.json — Session metadata (owner, description, timestamps)
agent-inputs/ — UTF-8 JSON input files for write commands; retained for inspection and retry

Knowledge Graph

Two layers:

Structural (auto-built):

Prefix	Type	Example
`q:`	Query node	`q:rust_async`
`r:`	Result node	`r:https://example.com/page`
`d:`	Domain node	`d:github_com`

Semantic (via graph-add):

Prefix	Type	Example
`e:`	Entity node	`e:tokio`
`p:`	Path node	`p:chain_001`

Graph navigation commands: graph-search (discover entities), graph-explore (view relations), graph-drill (follow specific relations), graph-traverse (traverse reasoning paths).

graph-preprocess returns resultProvenance (id, revision, url, title, approval) for every extracted body. For each semantic entity or edge, copy one or more currently approved id values into sourceResultIds. This keeps semantic facts traceable to the exact approved bodies rather than a search round.

External Search Results

Results from other search tools (Tavily, Exa, etc.) can be injected into any active session via results-add. They go through the same pipeline as native dg results:

# Import search discovery; its excerpts remain summaries and require extraction.
dg results-add <session-name> --kind search --tool exa --query "async runtime" `
  --data-file .\.digdig\sessions\<session-name>\agent-inputs\exa-search.json

Search imports become pending and require extract --session; inspect stats.success, stats.failed, and session.updated, then approve only entries whose --quality output reports verified: true. An external tool that already returned a body must instead use --kind extracted with non-empty content and extractor; it becomes pending approval without another extraction. A failed extraction remains pending and cannot enter the graph. tool and query record the discovery provenance.

Structured JSON Input

All structured writes use a UTF-8 JSON file under .digdig/sessions/<session-name>/agent-inputs/. This avoids PowerShell quoting and Windows command-line length limits, keeps concurrent sessions isolated, and gives each command one inspectable input artifact. The CLI rejects inline JSON and files outside the owning session.

Command	Required file option
External search or extracted bodies	`results-add --data-file <path>`
Graph entities/edges	`graph-add --data-file <path>`
Approval, skip, restore selections	`--approve-file` / `--skip-file` / `--unskip-file <path>`
One/batch Claim	`claim-add --claim-file` / `--claims-file <path>`
Evidence verification	`evidence-verify --evidence-file <path>`

The file must be UTF-8 (a UTF-8 BOM is accepted). Missing, malformed, cross-session, or out-of-directory input fails before state changes. The CLI never deletes Agent input files automatically; session-delete removes the entire session only when explicitly requested.

Quality output supplies {id, revision}. Save the selected objects in approve.json and pass --approve-file; never reuse a stale revision.

Local Document Search

doc-index and doc-search enable BM25 full-text search over local documents, with results flowing directly into the session pipeline:

# Index documents (auto-triggered by doc-search, no separate step needed)
dg doc-index ./docs

# Search and inject into session
dg doc-search <session-name> "search query" --path ./docs

How it works:

Auto-indexing — doc-search automatically indexes the directory on first use if no index exists. Uses Orama BM25 with field-weighted boosting: title ×3, headings ×2, content ×1.
Session injection — Each matched chunk is an extracted SessionResult with extractor: "local-index", injected into the session as pending.
Same pipeline — Results follow the exact same flow as web results: --quality → --approve-file → graph-add. Use graph-preprocess before adding new entities so their approved IDs become sourceResultIds.
Round-neutral — Local document searches do not increment the session round counter (merged with the current web round via skipRoundIncrement).

Index options:

Option	Description
`-t, --type <exts>`	File extensions to index (default: `md,txt`)

Search options:

Option	Description
`-k, --top <n>`	Top-K results (default: 10)
`--boost <field:w,...>`	Field weight overrides (e.g. `title:3,headings:2,content:1`)

When to use:

User explicitly asks to search local documents or notes
Web search results are insufficient for the topic and relevant local docs exist
Topic relates to private/internal information unlikely to be on the web

Local-source note: A file: result counts as a source under its path, so two chunks of one document are one source and a second file is a second source. A pure local session therefore reports a real source count rather than zero. Whether local-only coverage answers the question remains your judgement.

Known Limitations

Session result and graph mutations use a per-session lock plus atomic replacement; Claim, Evidence, and Review files still have no cross-file transaction.
URL extraction is not an SSRF security boundary; use only trusted public URLs.
Local document scanning has no defined symbolic-link boundary or cycle policy.
Index rebuilds overwrite the current persistence files; an interruption or disk failure can require re-indexing.

See the local, not-yet-published PRD-005 design-improvements document for the deferred remediation plan.

🏗️ Architecture

Content Extraction

dg extract uses a multi-tier extraction strategy:

Defuddle + linkedom (default, lightweight) — Parses raw HTML with linkedom, extracts readable content with Defuddle. Fast, no browser needed.
Obscura (JS rendering fallback) — When Defuddle extracts too little content (< 50 chars), Obscura renders the page with V8 JS engine and re-extracts. Use --obscura to enable.
Jina Reader (Agent-selected extraction) — Uses r.jina.ai for a specific URL only after the Agent reviews the default extraction and decides it is insufficient. It is not an automatic fallback because Jina is rate-limited.

# Default: Defuddle only (fast)
dg extract --urls "https://example.com"

# With Obscura fallback for JS-heavy pages
dg extract --urls "https://spa-site.com" --obscura

# After reviewing default extraction, use Jina for the specific URL that needs it and merge it into the session
dg extract --urls "https://complex-page.com" --session <session-name> --jina

Extraction options:

Option	Description
`--obscura`	Enable Obscura fallback for JS-rendered pages
`--obscura-path <path>`	Path to Obscura binary (auto-detected if omitted)
`--obscura-dump <format>`	Obscura output format: `html` (default) or `markdown`
`--jina`	Extract explicitly supplied URLs with Jina Reader (r.jina.ai); combine with `--session` to merge selected results

Dynamic Engine/Category Discovery

Unlike other CLI tools that hardcode supported engines and categories, this tool dynamically fetches them from your SearXNG server's /config endpoint:

Engines and categories are retrieved at runtime from the server
This ensures compatibility with any SearXNG instance configuration
Adding new engines to your SearXNG instance automatically makes them available in the CLI

Use dg --engines-list and dg --categories-list to see what's available on your server.

Output Format

The CLI supports multiple output formats:

Markdown (default) - Optimized for LLM context windows, saves ~50% tokens vs JSON
JSON - Structured envelope format for programmatic use

📝 Click to expand output format examples

Markdown Format (Default)

## Search: machine learning

**5** results
Total: 42

### 1. [Machine Learning Tutorial](https://example.com/ml) [Source score: 1]

Learn machine learning from scratch...

Rank: 1 · Engine: google · Category: general · Retrieved: 2026-09-09T00:00:00.000Z

---

### Suggestions

- deep learning tutorial
- neural networks

Result metadata keeps the source engine's score unchanged. It is not a cross-engine percentage; multi-query and session fusion uses a separate fusionScore. publishedDate is the source publication date when available, while retrievedAt records when digdig received the result.

JSON Envelope Format

{
  "status": "ok|error",
  "data": { ... },
  "error": null,
  "hint": "..."
}

💡 Projects Using digdig

digdig's deep search workflow (session accumulation, counted coverage facts, knowledge graph, claim–evidence–review) is used by projects that need persistent, multi-round research capabilities:

1052 OS — A personal AI operating system. Its search-pack integrates persistent research sessions, counted coverage facts, content extraction, and a claim–evidence–review pipeline, all powered by the same deep search concepts that drive digdig.

Open a PR to add your project here.

Name		Name	Last commit message	Last commit date
.github		.github
TaskFlowDocs		TaskFlowDocs
cli		cli
skills/digdig		skills/digdig
.gitattributes		.gitattributes
.gitignore		.gitignore
CODE_STYLE.md		CODE_STYLE.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
README.zh-CN.md		README.zh-CN.md

Latest commit

History

Folders and files

Repository files navigation