A tcpdump-like CLI that queries firefly records in ESnet's Elasticsearch and prints one line per firefly document.
ffdump [options] [expression...]
No install step: the ffdump launcher adds its own directory to sys.path.
Requires Python 3.9+ with requests (stdlib only otherwise; tests use unittest).
The Elasticsearch endpoint is not baked in: point ffdump at a cluster via the
environment (or --es-url):
$ export FFDUMP_ES_URL=https://your-elasticsearch-host:9200Commands that connect fail with exit code 2 (no Elasticsearch endpoint: ...)
if neither is set; --dry-run works without one.
$ ffdump -c 5 # last 5 records of the last 15 minutes
$ ffdump --last 1h exp cms and dst site '*FNAL*'
$ ffdump host 198.51.100.0/24 and port 1094 # traffic of a net on port 1094
$ ffdump src host 2001:db8:0:8420::13c -v # verbose (-vv: every field)
$ ffdump -c 5 -n # raw IPs (reverse DNS is on by default)
$ ffdump --epoch -c 5 # unix epoch timestamps (same as -tt)
$ ffdump --src 10.0.0.1 --dport 443 --state end # shortcut options (ANDed)
$ ffdump -s 2026-11-12T08:00:00Z -u '2026-11-12 08:30' end 'bytes > 1G' -H
$ ffdump appname xrootd and not site '*CERN*' -tt
$ ffdump -ttt -c 20 exp atlas # delta between lines
$ ffdump --json -c 2 | jq .meta.src_ip # JSON, one flat doc per line
$ ffdump --csv --fields ts,src_ip,dst_ip,total_bytes --last 5m end
$ ffdump --raw -c 3 # raw syslog messages
$ ffdump --stats --last 10m exp cms # per-field top values + total
$ ffdump --list dst_site --top 20 --last 1h # distinct values with counts
$ ffdump --schema # every queryable field + type
$ ffdump -f --interval 5 exp cms # follow (like tail -f); Ctrl-C stops
$ ffdump --all --last 1h exp atlas > out.txt # everything (PIT + search_after)
$ ffdump --dry-run --last 1h host 203.0.113.4 # show the ES query, don't runVolume is ~2.4M docs/hour, so the default window is the last 15 minutes and the default count is 100. Always filter server-side.
tcpdump-like; combine with and/or/not (&&/||/!) and parentheses;
juxtaposition means AND. Quote globs/parens for the shell.
| Prim | Matches |
|---|---|
[dir] host H |
IP, CIDR, or hostname (resolved via DNS) |
[dir] net CIDR |
network |
[dir] port N|N-M |
port or range |
[dir] site NAME |
CRIC name/rc_site/net_site (case-insensitive, * ? wildcards) |
dir VAL |
shorthand for dir host VAL |
exp/experiment, act/activity NAME|ID |
experiment / activity |
state start|end|ongoing, or bare start/end/ongoing |
flow state |
shost/syslog-host H |
emitting host (incl. legacy field, wildcards) |
appname/app-name N |
syslog app name |
application N |
meta.application |
proto tcp|udp, bare tcp/udp |
protocol |
ip, ip6, afi 4|6 |
address family |
collector X, flowid ID |
collector id, flow id |
bytes|sent|rcvd|duration|rtt|thr OP NUM |
OP in > >= < <= = !=; NUM may have k/M/G/T suffix (bytes = total_bytes, falling back to sent+received) |
lucene "QUERY" |
raw query_string |
FIELD:VALUE |
any token with : that isn't an IPv6 address → query_string |
dir is src, dst, src or dst, src and dst (no dir = either side).
TS SRC.SPORT (SRCSITE) > DST.DPORT (DSTSITE) STATE EXP ACT SYSLOG_HOST APPNAME [k=v extras] rcv=R sent=S
Times are always UTC; missing values print as -.
- Endpoints are reverse-resolved to hostnames by default (cached per IP,
1-second timeout, falls back to the raw address);
-n/--no-resolvekeeps raw IP addresses. rcv=N sent=Nare trailing key=value fields, styled likeapp=/proto=/dur=, and always come last.-qcore only; default addsapp proto dur rtt;-vadds throughput, totals, CRIC tier/country/asn per side, collector, flow id,storage.*;-vvflattens every remaining_sourceleaf ask=v.-tdrop the ts,-tt/--epochunix epoch seconds,-tttdelta since previous line,-ttttdefault format.-Hhuman-readable bytes; colour by state on a TTY (suppress with--no-color).--json(flat_sourceplus_id/_index),--raw(the syslogmessage),--csv,--fields a,b,c(FlowRecord attribute names).
- Time:
-s/--since,-u/--until,-l/--last DUR(default15m),--time-field @timestamp|start|end. Times: unix epoch,2026-11-12 08:00[:05],2026-11-12T08:00:00Z,2026-11-12, or relative (15m,2h,3d,now-2h,now). - Limits/modes:
-c/--count N(default 100),--all,--sort asc|desc(default asc),-f/--follow,--interval S,--stats,--list FIELD,--schema(dump field names + types),--top N. - Shortcuts (ANDed with the expression):
--src --dst --host --port --sport --dport --site --src-site --dst-site --exp --act --state --syslog-host --app-name --application. - Output extras:
-n/--no-resolve(keep raw IPs),--epoch(unix timestamps). - Conn:
--es-url(envFFDUMP_ES_URL; required to connect),--index,--timeout,--insecure,--debug(print the query to stderr),--dry-run.
Exit codes: 0 ok, 1 Elasticsearch/network error, 2 usage/expression error.
Ctrl-C exits quietly.
ffdump executable launcher
ffdumplib/
cli.py argument parsing, run modes, follow loop
client.py thin requests-based ES client (PIT + search_after paging)
timeparse.py parse_time(): UTC / epoch / relative time -> ES value
expr.py tokenizer, parser, AST -> ES bool DSL
query.py build_query / build_sort / FIELD_MAP
record.py normalize(hit) -> FlowRecord (defensive)
fmt.py line/json/raw/csv/fields formatters, stats printer
tests/ unittest suites (live tests hit the real cluster)
DESIGN.md design and interface contract
$ python3 -m unittest discover -s tests # unit tests (fully offline)Live smoke tests (which query a real cluster) are intentionally kept out of
this repository; they run as a separate module with FFDUMP_LIVE=1 and
FFDUMP_ES_URL set.