Skip to content

About

firefly-explorer command line tool

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ffdump

A tcpdump-like CLI that queries firefly records in ESnet's Elasticsearch and prints one line per firefly document.

ffdump [options] [expression...]

No install step: the ffdump launcher adds its own directory to sys.path. Requires Python 3.9+ with requests (stdlib only otherwise; tests use unittest).

The Elasticsearch endpoint is not baked in: point ffdump at a cluster via the environment (or --es-url):

$ export FFDUMP_ES_URL=https://your-elasticsearch-host:9200

Commands that connect fail with exit code 2 (no Elasticsearch endpoint: ...) if neither is set; --dry-run works without one.

Quick start

$ ffdump -c 5                                   # last 5 records of the last 15 minutes
$ ffdump --last 1h exp cms and dst site '*FNAL*'
$ ffdump host 198.51.100.0/24 and port 1094      # traffic of a net on port 1094
$ ffdump src host 2001:db8:0:8420::13c -v        # verbose (-vv: every field)
$ ffdump -c 5 -n                                 # raw IPs (reverse DNS is on by default)
$ ffdump --epoch -c 5                            # unix epoch timestamps (same as -tt)
$ ffdump --src 10.0.0.1 --dport 443 --state end  # shortcut options (ANDed)
$ ffdump -s 2026-11-12T08:00:00Z -u '2026-11-12 08:30' end 'bytes > 1G' -H
$ ffdump appname xrootd and not site '*CERN*' -tt
$ ffdump -ttt -c 20 exp atlas                    # delta between lines
$ ffdump --json -c 2 | jq .meta.src_ip           # JSON, one flat doc per line
$ ffdump --csv --fields ts,src_ip,dst_ip,total_bytes --last 5m end
$ ffdump --raw -c 3                              # raw syslog messages
$ ffdump --stats --last 10m exp cms              # per-field top values + total
$ ffdump --list dst_site --top 20 --last 1h      # distinct values with counts
$ ffdump --schema                                 # every queryable field + type
$ ffdump -f --interval 5 exp cms                 # follow (like tail -f); Ctrl-C stops
$ ffdump --all --last 1h exp atlas > out.txt     # everything (PIT + search_after)
$ ffdump --dry-run --last 1h host 203.0.113.4        # show the ES query, don't run

Volume is ~2.4M docs/hour, so the default window is the last 15 minutes and the default count is 100. Always filter server-side.

Expression grammar

tcpdump-like; combine with and/or/not (&&/||/!) and parentheses; juxtaposition means AND. Quote globs/parens for the shell.

Prim Matches
[dir] host H IP, CIDR, or hostname (resolved via DNS)
[dir] net CIDR network
[dir] port N|N-M port or range
[dir] site NAME CRIC name/rc_site/net_site (case-insensitive, * ? wildcards)
dir VAL shorthand for dir host VAL
exp/experiment, act/activity NAME|ID experiment / activity
state start|end|ongoing, or bare start/end/ongoing flow state
shost/syslog-host H emitting host (incl. legacy field, wildcards)
appname/app-name N syslog app name
application N meta.application
proto tcp|udp, bare tcp/udp protocol
ip, ip6, afi 4|6 address family
collector X, flowid ID collector id, flow id
bytes|sent|rcvd|duration|rtt|thr OP NUM OP in > >= < <= = !=; NUM may have k/M/G/T suffix (bytes = total_bytes, falling back to sent+received)
lucene "QUERY" raw query_string
FIELD:VALUE any token with : that isn't an IPv6 address → query_string

dir is src, dst, src or dst, src and dst (no dir = either side).

Output

TS SRC.SPORT (SRCSITE) > DST.DPORT (DSTSITE) STATE EXP ACT SYSLOG_HOST APPNAME [k=v extras] rcv=R sent=S

Times are always UTC; missing values print as -.

  • Endpoints are reverse-resolved to hostnames by default (cached per IP, 1-second timeout, falls back to the raw address); -n / --no-resolve keeps raw IP addresses.
  • rcv=N sent=N are trailing key=value fields, styled like app=/proto=/dur=, and always come last.
  • -q core only; default adds app proto dur rtt; -v adds throughput, totals, CRIC tier/country/asn per side, collector, flow id, storage.*; -vv flattens every remaining _source leaf as k=v.
  • -t drop the ts, -tt / --epoch unix epoch seconds, -ttt delta since previous line, -tttt default format.
  • -H human-readable bytes; colour by state on a TTY (suppress with --no-color).
  • --json (flat _source plus _id/_index), --raw (the syslog message), --csv, --fields a,b,c (FlowRecord attribute names).

Options

  • Time: -s/--since, -u/--until, -l/--last DUR (default 15m), --time-field @timestamp|start|end. Times: unix epoch, 2026-11-12 08:00[:05], 2026-11-12T08:00:00Z, 2026-11-12, or relative (15m, 2h, 3d, now-2h, now).
  • Limits/modes: -c/--count N (default 100), --all, --sort asc|desc (default asc), -f/--follow, --interval S, --stats, --list FIELD, --schema (dump field names + types), --top N.
  • Shortcuts (ANDed with the expression): --src --dst --host --port --sport --dport --site --src-site --dst-site --exp --act --state --syslog-host --app-name --application.
  • Output extras: -n/--no-resolve (keep raw IPs), --epoch (unix timestamps).
  • Conn: --es-url (env FFDUMP_ES_URL; required to connect), --index, --timeout, --insecure, --debug (print the query to stderr), --dry-run.

Exit codes: 0 ok, 1 Elasticsearch/network error, 2 usage/expression error. Ctrl-C exits quietly.

Layout

ffdump              executable launcher
ffdumplib/
  cli.py            argument parsing, run modes, follow loop
  client.py         thin requests-based ES client (PIT + search_after paging)
  timeparse.py      parse_time(): UTC / epoch / relative time -> ES value
  expr.py           tokenizer, parser, AST -> ES bool DSL
  query.py          build_query / build_sort / FIELD_MAP
  record.py         normalize(hit) -> FlowRecord (defensive)
  fmt.py            line/json/raw/csv/fields formatters, stats printer
tests/              unittest suites (live tests hit the real cluster)
DESIGN.md           design and interface contract

Tests

$ python3 -m unittest discover -s tests        # unit tests (fully offline)

Live smoke tests (which query a real cluster) are intentionally kept out of this repository; they run as a separate module with FFDUMP_LIVE=1 and FFDUMP_ES_URL set.

About

firefly-explorer command line tool

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages