A Network Traffic Feature Extraction Tool

Overview

This tool is engineered for robust and efficient feature extraction, particularly for applications such as network intrusion detection systems, among others. Leveraging Rust language and eBPF, it excels in processing high volumes of network traffic with remarkable speed and throughput. (When your traffic is already captured, don't worry! It also has a build in pcap reader which is also amazingly fast.) With various pre-defined feature sets and the ability to create custom feature sets, RustiFlow offers a versatile solution for network security applications.

Key Features

• High Throughput: Utilizes Rust and the Aya library for eBPF program compilation and execution, ensuring exceptional performance and resource efficiency.
• Versatile Feature Sets: Offers a variety of pre-defined feature sets (flows) and the flexibility to create custom feature sets tailored to specific requirements. An example of the custom flow is shown here.
• Pcap File Support: Facilitates packet analysis from classic pcap and pcapng files, compatible with both Linux and Windows generated files. A capture that ends inside a record is read up to that record with a warning; a corrupt record stops the read with an error naming its file offset, after exporting earlier packets. Timestamp fractions must be below 1,000,000 for microsecond pcap or 1,000,000,000 for nanosecond pcap; invalid fractions are rejected, not normalized. Out-of-range pcapng timestamps are also rejected. Sorted offline CSV rows are deterministic across runs and thread counts. Expiration scans use one capture clock shared by all shards, including a final scan at the last readable record's timestamp.
• Diverse Output Options: Features can be outputted to the console, a CSV file, or other formats with minimal effort.
• Richer TCP Quality Signals: The RustiFlow feature set exports duplicate ACK counts, zero-window observations, and TCP close style in addition to the existing lifecycle and retransmission fields.
• Endpoint-Aware IP Context: The RustiFlow feature set exports ip_version, endpoint IP scope, and coarse path_locality derived from normalized addresses without expanding the eBPF event payload.

Feature sets

See the wiki for the different feature sets available.

Bulk durations are exported in milliseconds, including fractional milliseconds. Bulk rates are packet bytes per second, measured using integer microsecond timestamps. A bulk needs at least four nonzero-length packets; an idle gap greater than exactly one second ends it. Packet-byte accounting and direction-switch rules are unchanged. A bulk with identical timestamps has an undefined rate exported as zero, following the shared rate convention. Positive bulk count with zero total duration identifies that fallback; it must not be interpreted as measured zero throughput.

Supported Packet/Header Coverage

RustiFlow currently extracts flows from the following protocol/header combinations:

Layer	Offline pcap	Realtime eBPF
Link	Ethernet, Linux cooked capture, 802.1Q VLAN	Ethernet
Network	IPv4, IPv6	IPv4, IPv6
IPv6 extras	Extension headers supported before transport parsing	Extension headers supported before transport parsing
Transport	TCP, UDP, ICMP, ICMPv6	TCP, UDP, ICMP, ICMPv6

Notes:

• Realtime support is Linux-only.
• Offline and realtime aim to expose the same flow semantics, but timestamp and packet-length sources can differ slightly.
• Realtime VLAN parsing is not implemented yet.

Architecture

Realtime processing

Offline PCAP processing

Installing a release:

Every release consumer (the installer, the Debian package and both container images) installs the same release bundle, rustiflow-<version>-x86_64-linux.tar.gz. It holds the rustiflow binary, the two eBPF objects that realtime capture loads, a systemd template unit, example configuration, an install.sh, a MANIFEST recording how it was built, and SHA256SUMS. Releases target x86_64 Linux with glibc 2.36 or newer (Debian bookworm, Ubuntu 24.04 and later) and need nothing beyond glibc at runtime. They are built for x86-64-v3 CPUs (AVX2, BMI2, FMA, MOVBE: Intel Haswell, AMD Zen and newer); MANIFEST records the CPU as target_cpu, and the installer refuses a host CPU that cannot run it.

Installing the bundle

tar -xzf rustiflow-<version>-x86_64-linux.tar.gz
sudo sh rustiflow-<version>-x86_64-linux/install.sh            # into /usr/local
sudo sh rustiflow-<version>-x86_64-linux/install.sh --prefix /opt/rustiflow

The installer verifies the bundle checksums, installs bin/rustiflow, the eBPF objects in lib/rustiflow/, examples in share/rustiflow/ and lib/systemd/system/rustiflow@.service under the prefix, and never writes to /etc, so reinstalling keeps your configuration. Check an installation later with:

cd /usr/local && sha256sum -c share/rustiflow/INSTALLED.sha256

The binary finds its eBPF objects in <prefix>/lib/rustiflow/ next to its own <prefix>/bin/, so keep that layout or point RUSTIFLOW_EBPF_DIR at the directory holding rustiflow-ebpf-ipv4 and rustiflow-ebpf-ipv6.

Debian package

debian/ packages a bundle without rebuilding it, installing into /usr:

RUSTIFLOW_BUNDLE=/path/to/rustiflow-<version>-x86_64-linux.tar.gz dpkg-buildpackage -b -uc -us
sudo apt install ../rustiflow_<version>-1_amd64.deb

RUSTIFLOW_BUNDLE defaults to target/dist/. The bundle version must match debian/changelog. Use a bundle built on bookworm (see below): one built on a newer host needs a newer glibc than Debian provides.

Running realtime capture as a service

rustiflow@<interface>.service runs realtime capture on one interface with only the CAP_BPF, CAP_NET_ADMIN, CAP_NET_RAW and CAP_PERFMON capabilities (CAP_NET_RAW is used only by --promiscuous). It is installed but never enabled:

sudo install -D -m 0644 /usr/share/rustiflow/config.toml.example /etc/rustiflow/eth0.toml
sudo install -D -m 0644 /usr/share/rustiflow/environment.example /etc/rustiflow/eth0.env  # optional
sudo systemctl daemon-reload
sudo systemctl enable --now rustiflow@eth0.service

On a tap or SPAN/mirror port, set RUSTIFLOW_REALTIME_ARGS=--promiscuous in /etc/rustiflow/eth0.env; see Promiscuous capture.

Use /usr/local/share/rustiflow/... for a bundle installed into the default prefix. Relative CSV export paths in the config are written below /var/lib/rustiflow/eth0/.

Building a release bundle

The canonical bundle is built in the Dockerfile's bookworm dist stage, so the binary runs on every supported distribution:

docker build --target bundle \
  --build-arg RUSTIFLOW_COMMIT=$(git rev-parse HEAD) \
  --build-arg SOURCE_DATE_EPOCH=$(git log -1 --format=%ct) \
  --output type=local,dest=target/dist .

cargo xtask dist builds the same layout on the current host into target/dist/, and cargo xtask install [--prefix DIR] installs that staged bundle without rebuilding. A host build only runs on systems with the same or newer glibc as the build host.

The release profile uses fat LTO, one codegen unit and stripped symbols, and leaves out the profiling feature. --target-cpu (default x86-64-v3) sets the CPU the binary is tuned for and requires. native tunes for the build host, and MANIFEST then names its model, for example znver3. Use a lower level such as x86-64-v2 for older CPUs. In the container build, pass it as --build-arg RUSTIFLOW_TARGET_CPU=<cpu>.

cargo xtask dist --target-cpu native   # bundle for this machine only

Using commands:

You can then run the binary with the following commands displayed on the help menu.

When using command-line arguments, both --features (-f) and --output (-o) are required unless --config-file (-c) is provided. Missing options produce a usage error. Running without arguments still opens the TUI.

Active and idle measurements

Idle statistics count gaps longer than five seconds that end with another packet in the same flow. Trailing silence at expiration or shutdown does not count as an observed idle period. Active periods, flow duration, and end time remain based on packet timestamps. This rule applies to offline and realtime exports.

Using the tui interface:

If you want a more graphical interface, you can use the tui interface by just running rustiflow without any arguments. This will open a field where you can enter a configuration file you want to edit or you can choose to start new. After that, the following interface will show up:

NOTE: When using the save button, you will be prompted for a filename. You can reuse this file with following command:

rustiflow --config-file <filename> realtime <interface> [--ingress-only] [--promiscuous]

rustiflow -c <filename> pcap <path to pcap file>

After saving the configuration file, you can safely reset without changing the configuration file.

Using the configuration file:

This is an example of a configuration file that you can use to run the tool with the --config-file option.

[config]
features = "CIDDS"
active_timeout = 522
idle_timeout = 885855
early_export = 25
expiration_check_interval = 0

[output]
output = "Csv"
export_path = "path/to/output.csv"
header = false
drop_contaminant_features = true

Example 2:

[config]
features = "Nfstream"
active_timeout = 3600
idle_timeout = 120
early_export = 10
expiration_check_interval = 60
threads = 8

[output]
output = "Print"
header = true
drop_contaminant_features = false

For realtime CSV exports, the live packet graph is now off by default so the standard CLI path keeps the best throughput-oriented behavior. Enable it only when you want the graph:

[output]
packet_graph = true

or with:

rustiflow ... -o csv --export-path out.csv --packet-graph realtime <interface>

Promiscuous capture

Realtime capture sees only the frames the network card passes up to the kernel. A physical card drops unicast frames addressed to other MAC addresses, so on a tap, a SPAN/mirror port or a backbone link most of the traffic never reaches RustiFlow. Pass --promiscuous there:

rustiflow ... realtime --promiscuous <interface>

A host that only captures its own traffic does not need it.

RustiFlow does not set the interface's PROMISC flag. It joins a promiscuous membership on a packet socket that receives no packets, and the kernel counts memberships (promiscuity in ip -d link show <interface>). The kernel drops the membership when the socket closes, also when RustiFlow crashes or is killed, so the interface returns to its previous state. A promisc on set by an operator or another tool stays in place.

The option needs CAP_NET_RAW and the AF_PACKET socket family, in addition to what realtime capture already needs. Configuration files store only the [config] and [output] sections, so pass the flag on the command line or through RUSTIFLOW_REALTIME_ARGS for the systemd unit.

Using the Container:

Make sure that you don't use docker desktop and that you don't have it installed on your machine. If you have this setup, it will not work as intended as the --network host will not link the container to the host network, but to the network of a VM that docker desktop uses.

• Build the Container:

  docker build -t rustiflow .                     # Ubuntu 24.04 runtime
  docker build --target slim -t rustiflow:slim .  # Debian bookworm-slim runtime

  Both images build the release bundle from source and install it. To install an existing bundle instead, pass the directory holding its tarball: docker build --build-context bundle=target/dist -t rustiflow .
• Run the Container:

  docker run --rm --network host -v /path/on/host:/app rustiflow [ARGS]

  Run it with the --privileged flag if you want to capture traffic in real-time.
• Example:

  docker run --rm --network host -v /home/user/pcap:/app rustiflow \
    -f basic \
    -o print \
    pcap /app/pcap.pcap

  docker run --rm --privileged --network host -v /home/user/output:/app rustiflow \
    -f cic \
    -o csv \
    --export-path /app/output.csv \
    realtime enp5s0

Notes:

• The current CLI uses flags such as -f basic and -o csv; the older positional examples are no longer correct.
• Realtime capture in a container still depends on Linux host support for eBPF and tc, so --privileged --network host remains the practical baseline for local testing.

Installation Guide for development

Prerequisites:

• Rust Installation:

  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

• Nightly Rust Toolchain:

  rustup install stable
  rustup toolchain install nightly --component rust-src

bpf Linker Installation:

Use the upstream prebuilt binary. Building current bpf-linker releases with cargo install requires a matching system LLVM installation; Rust's llvm-tools-preview component alone is not sufficient.

• For Linux x86_64:

  sudo apt install zstd # Debian/Ubuntu; use your distribution's package manager elsewhere
  curl -fL --retry 3 https://github.com/aya-rs/bpf-linker/releases/download/v0.12.0/bpf-linker-x86_64-unknown-linux-musl.tar.zst -o /tmp/bpf-linker.tar.zst
  mkdir -p "$HOME/.cargo/bin"
  tar --zstd -xf /tmp/bpf-linker.tar.zst -C "$HOME/.cargo/bin"
  rm /tmp/bpf-linker.tar.zst
  bpf-linker --version

• For macOS/Linux (Other Architectures): choose the matching archive from the upstream releases, or follow the upstream installation guide.
• Ubuntu 20.04 LTS Specific:

  sudo apt install linux-tools-5.8.0-63-generic
  export PATH=/usr/lib/linux-tools/5.8.0-63-generic:$PATH

Building the Project

cargo xtask enforces the committed lockfiles for the helper and every build it launches. If a manifest change requires a lockfile update, update and commit the affected lockfile explicitly before rebuilding. Direct Cargo commands need --locked for the same enforcement.

• eBPF Programs:

  cargo xtask ebpf-ipv4
  cargo xtask ebpf-ipv6

• User Space Programs:

  cargo build

• With flamegraph profiling (Linux only, pulls in pprof; enables RUSTIFLOW_PROFILE_FLAMEGRAPH=<file.svg>). The profiling profile is the release profile with symbols and line tables kept, so it also suits perf; the binary is target/profiling/rustiflow:

  cargo build --profile profiling --features profiling

Fragmented and encrypted packets

Offline and realtime capture skip non-initial IPv4 and IPv6 fragments because those fragments do not contain the original transport header. Initial fragments remain eligible for extraction; RustiFlow does not reassemble fragmented packets. IPv6 AH uses its four-byte length units. IPv6 ESP is skipped because its encrypted payload cannot be parsed as a transport header.

TCP flow lifecycle

A flow first observed as SYN-ACK uses that packet's destination as the forward endpoint. Other flows use the first observed packet's source. Handshake and FIN acknowledgments tolerate reordered observations using TCP sequence arithmetic. The four_way_fin close style includes a three-segment FIN, FIN|ACK, ACK close; simultaneous_fin requires both FIN packets to carry ACKs that do not acknowledge the peer's FIN. Packets observed after a completed flow was exported start a new flow, including retransmitted FINs. This preserves observed packet counts.

Retransmission counters retain at most 1,024 merged TCP sequence ranges per direction. The lowest sequence ranges are evicted first; retransmissions that overlap only evicted history may be missed. Sequence numbers are unwrapped across 32-bit rollover, assuming observations are less than half the sequence space apart. These counters do not perform full TCP stream reassembly.

Out-of-order packet timing

Flow and directional durations use the earliest and latest observed packet timestamps. IAT statistics follow processing order with a nondecreasing timestamp reference: late packets contribute a zero interval and do not move that reference backwards. Active/idle periods also advance monotonically; completed periods are not reconstructed. Bulk durations retain the timestamp bounds of their current bulk. These bounded streaming statistics do not reconstruct chronological IATs. The same rules apply to offline and realtime input.

Running the Project in dev mode

cargo xtask run -- [OPTIONS] <COMMAND>

Usage Instructions

Command Help:

rustiflow help

Usage: rustiflow [OPTIONS] <COMMAND>

Commands:
  realtime      Real-time feature extraction
  pcap          Feature extraction from a pcap file
  replay-bench  Benchmark the realtime userspace pipeline with synthetic eBPF events (no capture, no root needed). Prints throughput and CPU time per packet
  help          Print this message or the help of the given subcommand(s)

Options:
  -c, --config-file <CONFIG_FILE>
          Configuration file path

  -f, --features <FEATURES>
          The feature set to use (required if no config file is provided)

          Possible values:
          - basic:     A basic flow that stores the basic features of a flow
          - cic:       Represents the CIC Flow, giving 90 features
          - cidds:     Represents the CIDDS Flow, giving 10 features
          - nfstream:  Represents a nfstream inspired flow, giving 71 features
          - rustiflow: Represents the Rusti Flow, giving 203 features
          - custom:    Represents a flow that you can implement yourself

      --active-timeout <ACTIVE_TIMEOUT>
          The maximum time a flow is allowed to last in seconds (optional)

          [default: 3600]

      --idle-timeout <IDLE_TIMEOUT>
          The maximum time with no packets for a flow in seconds (optional)

          [default: 120]

      --early-export <EARLY_EXPORT>
          Export a cumulative snapshot of each open flow every N seconds after its first packet (optional; must be at least 1 and smaller than the active timeout). Snapshots are triggered by packets and have flow_expire_cause None

      --expiration-check-interval <EXPIRATION_CHECK_INTERVAL>
          Interval (in seconds) for checking and expiring flows in the flowtable. This represents how often the flowtable should be scanned to remove inactive flows

          [default: 60]

      --threads <THREADS>
          The numbers of threads to use for processing packets (optional) (default: realtime uses 12, capped at the number of logical CPUs; pcap uses 5; maximum number of logical CPUs)

      -o, --output <OUTPUT>
              Output method (required if no config file is provided)

              Possible values:
              - print: The output will be printed to the console
              - csv:   The output will be written to a CSV file

          --export-path <EXPORT_PATH>
              File path for output (used if method is Csv)

          --packet-graph
              Enable the realtime packet graph for CSV exports
              [default: false]

          --header
              Whether to export the feature header

          --drop-contaminant-features
              Whether to drop contaminant features

      -h, --help
              Print help (see a summary with '-h')

      -V, --version
              Print version

rustiflow realtime --help

Real-time feature extraction

Usage: rustiflow realtime [OPTIONS] <INTERFACE>

Arguments:
  <INTERFACE>  The network interface to capture packets from

Options:
      --ingress-only  Whether to capture only ingress packets
      --promiscuous   Hold the interface in promiscuous mode while capturing, for taps, SPAN/mirror ports and backbone links (needs CAP_NET_RAW)
  -h, --help          Print help

Early export

--early-export N (or early_export = N in the configuration file) emits periodic snapshots of long-lived flows before their final export:

• Snapshots fall on a fixed per-flow schedule: first packet + N, + 2N, ... The first packet at or after a boundary exports one snapshot, so a flow produces about duration / N snapshots rather than one per packet. Boundaries that pass without packets are skipped, and a flow that goes quiet gets no further snapshots until it expires.
• Each snapshot is cumulative: it covers the flow from its first packet up to and including the triggering packet, with the same columns as a final row.
• Snapshots are closed like a final export (open bulks and the current active period are counted, TCP close style reflects the FINs seen so far) but keep flow_expire_cause None. Final rows always carry a real expiration cause.
• The flow itself stays open; its final row repeats everything in the snapshots.
• N must be at least 1 and smaller than --active-timeout. Omit the flag to disable early export.

Replay benchmark

replay-bench measures the realtime userspace pipeline without capturing traffic or loading eBPF programs, so it needs no root and no traffic generator. It generates synthetic eBPF events in memory, then drains them through the same per-source batching, dispatch, shard flow tables and CSV export as realtime:

cargo build --release
./target/release/rustiflow -f rustiflow -o csv --export-path /dev/null --threads 12 \
  replay-bench --packets 50000000 --flows 100000

It prints one result line on stderr:

Replay: packets=50000000 threads=12 wall_s=3.251 mpps=15.378 cpu_s=43.531 cpu_ns_per_packet=870.6 cpus_busy=13.39

• mpps is events per second of wall time, from the first replayed event until the output writer closes. It includes the final export of every open flow, so use far more packets than flows.
• cpu_ns_per_packet is the CPU time of the whole process (all tokio workers) over the same interval, divided by the packet count. Event generation is excluded.
• Events are spread uniformly at random over --flows flows. Even flows are TCP with in-order data segments in both directions, odd flows are UDP, and no flow terminates. All events of a flow go to one of --sources sources, which are evenly balanced, unlike real eBPF queue selection.
• --virtual-pps spaces event timestamps, which sets the virtual time seen by timeouts, expiration checks and --early-export.
• The feature set, --threads, timeouts, early export, output options and the RUSTIFLOW_REALTIME_* and RUSTIFLOW_PROFILE_* environment variables apply as for realtime. Events take 40 (IPv4) or 72 (IPv6) bytes of memory each.

Logging in both development or using the binary

Development

RUST_LOG=info cargo xtask run --

Run the focused Rust test suite with:

cargo test -p rustiflow

If you also have the sibling concap repository and a reachable Kubernetes cluster, you can run a tiny ConCap-backed smoke check and then reprocess the downloaded pcap with your current local RustiFlow checkout:

./scripts/concap_smoke.sh ../concap nmap-tcp-syn-version.yaml

Binary

sudo RUST_LOG=info rustiflow

Note: For specific logging levels, adjust RUST_LOG to error for error messages, and debug for debug messages. If you don't want any additional logs, just remove RUST_LOG=info.

---