ctoken

ctoken is a cli utility to count tokens in a file or a directory and its contents — similar to how cloc is used for lines code. Useful for understanding how much context a file or directory would consume when feeding it to a coding agent.

Motivation

When developing AI agends or LLM based apps in general, sometimes it is interesting to know how much a certain data will impact context window - that is how much tokens a certain file or set of files will translate to.

There are various options - estimate it by size or word count, use various calculators on the web, or call one of LLM providers APIs - but all of them become inconvenient when this needs to be done repeadedly or on larger set of files.

Thats where ctoken utility comes in - type ctoken <dirname> and you get token count summary of its contents:

DIRECTORY  TOKENS
-----------------
.          26,091
.github     1,462
Formula       152
src        12,959
tests       3,092
-----------------
TOTAL      43,756

It also supports grouping by file type, filtering by pattern or customisable profiles, etc - see Flags for details.

Install

macOS (Apple Silicon)

brew tap RimantasZ/ctoken
brew install ctoken

Linux (Debian / Ubuntu)

Download the .deb from the latest release:

curl -LO https://github.com/RimantasZ/ctoken/releases/latest/download/ctoken_amd64.deb
sudo apt install ./ctoken_amd64.deb

Windows

Download ctoken-x86_64-windows.zip from the latest release, extract it, and add the folder to your PATH.

From source (any platform)

Requires Rust 1.70+:

cargo install --git https://github.com/RimantasZ/ctoken

Or clone and build locally:

git clone https://github.com/RimantasZ/ctoken
cd ctoken
cargo build --release
# binary at target/release/ctoken

Quick examples

# Default: token count by immediate subdirectory
ctoken .

# Group by file extension
ctoken . -t

# Use a built-in language profile
ctoken . -p rust

# Match only markdown files
ctoken . -m '**/*.md'

# Walk recursively, per-directory breakdown
ctoken . --recursive

# Just the total token count
ctoken . -s

# JSON output
ctoken . --json

# Count tokens in a single file
ctoken src/main.rs

# Show each file as it's processed
ctoken . -v

Stdin and pipe usage

ctoken can read from stdin when no path is given, making it easy to use in pipelines or with ad-hoc input.

# Pipe a file through ctoken
cat myfile.txt | ctoken

# Use as a step in a pipeline — output is a bare integer
cat myfile.txt | ctoken | xargs -I{} echo "Token count: {}"

# Count tokens from a command's output (e.g. git diff)
git diff HEAD~1 | ctoken

# Combine with other flags
cat myfile.txt | ctoken --json
cat myfile.txt | ctoken --encoding cl100k_base

# Use '-' to read stdin explicitly (useful when mixing with other flags)
ctoken - --json < myfile.txt

# Interactive mode: run with no arguments, type or paste text, press Ctrl+D when done
ctoken

When stdin is a terminal (interactive mode), ctoken prints a brief prompt to stderr and waits for input. The token count is printed to stdout once you signal end-of-input with Ctrl+D. All other flags (--json, --encoding, --verbose) work the same way in stdin mode.

Flags

Short	Long	Arg	Description
-h	--help	—	Print help and exit
	--version	—	Print version and exit
-t	--type	—	Group by file extension instead of by subdirectory
-g	--gitignore	on|off	Honor .gitignore. Default on
-m	--match	<GLOB>	Glob pattern restricting included files. Repeatable
-p	--profile	<NAME>	Use named profile from ~/.config/ctoken/profiles.toml
	--recreate-profiles	—	Rewrite built-in profile entries in profiles.toml (interactive)
	--recursive	—	Walk recursively; per-directory table grouped by file type
	--recursive-with-dir	—	Same as --recursive, but includes child directory rollups
-v	--verbose	—	Log each file processed
-s	--sum	—	Print only the grand total (single integer)
	--json	—	Emit JSON instead of a table. Incompatible with --recursive*
	--encoding	<NAME>	Tiktoken encoding (see below)

Encoding options

ctoken uses tiktoken-rs to estimate actual tokens in files, and supports these encoding used by OpenAI models

Name	Models
o200k_base (default)	GPT-5 series, o1/o3/o4 series, gpt-4o, gpt-4.5, gpt-4.1, codex-*
cl100k_base	gpt-4, gpt-3.5-turbo, text-embedding-ada-002, text-embedding-3-*
p50k_base	Code models, text-davinci-002, text-davinci-003
p50k_edit	Edit models like text-davinci-edit-001, code-davinci-edit-001
r50k_base	GPT-3 models like davinci

Note: for different LLM providers, token calculation might skughtly differ. Therefore this tool should be used for rough comparison (e.g. "how much this file/folder is bigger in terms of tokens than that one"), rather than precise estimation.

Profile system

On first run, ctoken creates ~/.config/ctoken/profiles.toml with built-in profiles for common project types: java, c_cpp, typescript, python, rust, go.

# Use a profile
ctoken . -p typescript

# Restore built-in profiles to defaults (prompts before changing)
ctoken . --recreate-profiles

Edit ~/.config/ctoken/profiles.toml directly to add custom profiles or tweak existing ones:

[myproject]
include = ["**/*.rs", "**/*.toml", "docs/**/*.md"]
exclude = ["target/**"]

New built-in profiles added in later versions are appended automatically without overwriting your customizations.

Performance notes

• Uses all CPU cores for tokenization (via rayon).
• Files are read fully into memory. Very large files (50+ MB) will use proportionate RAM.
• Binary files are detected by extension or by scanning the first 8 KB for NUL bytes, and skipped.
• Symlinks are never followed.

License

Apache-2.0