processor_fts: launch documentation, runnable examples and example tests

README.md

@@ -44,6 +44,9 @@ To track upcoming milestones, visit the [project roadmap](https://github.com/flu
 - 📊 **Monitoring**: Expose internal metrics over HTTP/Prometheus
 - 🧩 **Extensibility**:
   - Write plugins in **C**, filters in **Lua**, and outputs in **Go**
+  - Write telemetry policies in Lua for logs, metrics, and traces with
+    [Fluent Typed Scripting (FTS)](plugins/processor_fts/README.md)
+    (`master` branch, not yet in a release)
 - 🔌 **Supports Logs, Metrics, and Traces** with unified processing and delivery
 
 ---

documentation/wasm-browser.md

@@ -597,6 +597,11 @@ preserves JSON nulls. Edit the script directly in the YAML `code: |` block.
 Traditional `pipeline.filters` Lua configuration is supported too, as are the
 three-argument and five-argument (metadata-aware) callbacks.
 
+The same interpreter runs the `fts` processor (Fluent Typed Scripting) for logs,
+metrics and traces; `jit: true` is rejected. See the
+[FTS guide](../plugins/processor_fts/README.md#native-and-browser-wasm-runtimes)
+for the differences from native builds.
+
 The browser uses **portable Lua 5.4.9, not LuaJIT**. `FLB_WASM_LUA=ON` is the
 browser default; native builds still use `FLB_LUAJIT` and the existing bundled
 LuaJIT. The build downloads Lua from lua.org with a pinned SHA-256 into the

examples/fts/README.md

@@ -0,0 +1,89 @@
+# Fluent Typed Scripting (FTS) examples
+
+Runnable policies for the [`fts` processor](../../plugins/processor_fts/README.md).
+The OpenTelemetry examples share one layout; the quickstart has only a
+configuration with an inline policy and its expected output.
+
+| File                   | Content |
+| ---------------------- | ------- |
+| `fluent-bit.yaml`      | Pipeline: `opentelemetry` input, `fts` processor, `stdout` output in `otlp_json` format. |
+| `policy.lua`           | The FTS policy. |
+| `input.json`           | OTLP/HTTP JSON request sent to the input. |
+| `expected-output.json` | What Fluent Bit prints, as a JSON array with one entry per printed line. |
+
+| Example | Signal | Shows |
+| ------- | ------ | ----- |
+| [`quickstart`](quickstart/) | Logs | Inline policy on a `dummy` input. No other tools needed. |
+| [`logs-redaction`](logs-redaction/) | Logs | Key and free-form redaction, dropping records, severity and resource enrichment. |
+| [`metrics-normalization`](metrics-normalization/) | Metrics | Unit conversion and aggregation that reduces series cardinality. |
+| [`traces-filtering`](traces-filtering/) | Traces | Attribute migration, query-string removal, event redaction and selective span drop. |
+
+## Requirements
+
+A Fluent Bit binary built with FTS. FTS is in the `master` branch and is not
+part of any published release yet; see
+[Availability](../../plugins/processor_fts/README.md#availability). To build it:
+
+```shell
+cmake -S . -B build
+cmake --build build -j8
+./build/bin/fluent-bit --help | grep fts
+```
+
+The OTLP examples also need `curl` and `python3`.
+
+## Run the quickstart
+
+```shell
+cd examples/fts/quickstart
+../../../build/bin/fluent-bit -c fluent-bit.yaml
+```
+
+Stop it with Ctrl-C. Expected output:
+
+```json
+{"user":"ana","password":"[REDACTED]","path":"/login","status":200,"environment":"dev"}
+```
+
+## Run an OTLP example
+
+The helper starts Fluent Bit, posts `input.json`, prints the output and
+compares it with `expected-output.json`:
+
+```shell
+examples/fts/run.sh examples/fts/logs-redaction
+```
+
+It uses `build/bin/fluent-bit` unless you pass another binary as the second
+argument or set `FLUENT_BIT_BINARY`. The examples listen on `127.0.0.1:4318`,
+so that port must be free.
+
+To run an example by hand, start Fluent Bit from the example directory, because
+relative `script` paths are resolved from the working directory:
+
+```shell
+cd examples/fts/logs-redaction
+../../../build/bin/fluent-bit -c fluent-bit.yaml
+```
+
+In a second terminal, send the input to the endpoint of the signal (`logs`,
+`metrics` or `traces`):
+
+```shell
+curl -H 'Content-Type: application/json' --data-binary @input.json \
+     http://127.0.0.1:4318/v1/logs
+```
+
+Fluent Bit prints each processed chunk as one OTLP JSON line.
+
+## Reading the output
+
+- The `otlp_json` format of the `stdout` output, like the `opentelemetry`
+  output, uses the `log` or `message` key of a map body as the whole body and
+  omits the other keys. The logs example uses a `msg` key so that the complete
+  redacted body is visible.
+- Metric families can be printed in a different order than they were sent:
+  the internal metrics representation groups families by type. This happens
+  with or without FTS.
+- These examples are regression tested by
+  `tests/integration/scenarios/processor_fts/tests/test_fts_examples.py`.

examples/fts/logs-redaction/README.md

@@ -0,0 +1,59 @@
+# FTS logs: redaction and enrichment
+
+[`policy.lua`](policy.lua) processes the OpenTelemetry logs of a `checkout`
+service:
+
+1. Drops records whose `http.route` attribute is `/healthz`.
+2. Replaces the values of `password`, `authorization`, `token` and `card` keys
+   anywhere in the body with `[REDACTED]`.
+3. Masks e-mail addresses and `token=` values inside the free-form `msg` text.
+4. Replaces the `user.email` attribute with `user.email_domain`.
+5. Sets the OpenTelemetry severity from the body `level` field.
+6. Adds `deployment.environment.name` to the resource and `policy.name` to
+   the log attributes.
+
+## Run
+
+```shell
+examples/fts/run.sh examples/fts/logs-redaction
+```
+
+Or by hand, from this directory:
+
+```shell
+../../../build/bin/fluent-bit -c fluent-bit.yaml
+# in another terminal
+curl -H 'Content-Type: application/json' --data-binary @input.json \
+     http://127.0.0.1:4318/v1/logs
+```
+
+## Expected output
+
+The complete output is in [`expected-output.json`](expected-output.json).
+[`input.json`](input.json) has three records; the health check is dropped and
+the other two become:
+
+| Field | Input | Output |
+| ----- | ----- | ------ |
+| `user.password` | `hunter2` | `[REDACTED]` |
+| `headers.Authorization` | `Bearer abc123` | `[REDACTED]` (matching ignores case) |
+| `card` | map with `number` and `expiry` | `[REDACTED]` (the whole value is replaced) |
+| `db_password` | `not-a-match` | unchanged (keys must match exactly) |
+| `msg` | `login succeeded for ana@example.com` | `login succeeded for <email>` |
+| `msg` | `payment failed token=tok_live_42` | `payment failed token=[REDACTED]` |
+| `attempt` | `intValue` 3 | `intValue` 3 (integer type preserved) |
+| attribute `user.email` | `ana@example.com` | removed; `user.email_domain` is `example.com` |
+| severity | none | `severityText` `info`/`error`, `severityNumber` 9/17 |
+| resource | `service.name` | adds `deployment.environment.name: production` |
+
+## Notes
+
+- `fb.redact` matches whole keys, so `db_password` is kept. List every key name
+  you need, or walk the map yourself when you need prefix or pattern matches.
+- Key redaction does not read strings. Secrets inside text, such as the `msg`
+  field here, need explicit string rules like `scrub()`, and those rules only
+  catch the formats they describe.
+- `fb.redact` only visits the value it is given. This policy redacts the body;
+  attributes, resource attributes and metadata need their own calls.
+- The processor keeps the default `error_mode: drop`, so a record whose policy
+  fails is removed instead of forwarded unredacted.

examples/fts/logs-redaction/expected-output.json

@@ -0,0 +1,176 @@
+[
+  {
+    "resourceLogs": [
+      {
+        "resource": {
+          "attributes": [
+            {
+              "key": "service.name",
+              "value": {
+                "stringValue": "checkout"
+              }
+            },
+            {
+              "key": "deployment.environment.name",
+              "value": {
+                "stringValue": "production"
+              }
+            }
+          ]
+        },
+        "scopeLogs": [
+          {
+            "scope": {
+              "name": "checkout.http",
+              "version": "1.4.0"
+            },
+            "logRecords": [
+              {
+                "timeUnixNano": "1767225600000000000",
+                "observedTimeUnixNano": "1767225600000000000",
+                "severityNumber": 9,
+                "severityText": "info",
+                "attributes": [
+                  {
+                    "key": "http.route",
+                    "value": {
+                      "stringValue": "/login"
+                    }
+                  },
+                  {
+                    "key": "user.email_domain",
+                    "value": {
+                      "stringValue": "example.com"
+                    }
+                  },
+                  {
+                    "key": "policy.name",
+                    "value": {
+                      "stringValue": "logs-redaction"
+                    }
+                  }
+                ],
+                "body": {
+                  "kvlistValue": {
+                    "values": [
+                      {
+                        "key": "msg",
+                        "value": {
+                          "stringValue": "login succeeded for <email>"
+                        }
+                      },
+                      {
+                        "key": "level",
+                        "value": {
+                          "stringValue": "info"
+                        }
+                      },
+                      {
+                        "key": "user",
+                        "value": {
+                          "kvlistValue": {
+                            "values": [
+                              {
+                                "key": "name",
+                                "value": {
+                                  "stringValue": "ana"
+                                }
+                              },
+                              {
+                                "key": "password",
+                                "value": {
+                                  "stringValue": "[REDACTED]"
+                                }
+                              }
+                            ]
+                          }
+                        }
+                      },
+                      {
+                        "key": "headers",
+                        "value": {
+                          "kvlistValue": {
+                            "values": [
+                              {
+                                "key": "Authorization",
+                                "value": {
+                                  "stringValue": "[REDACTED]"
+                                }
+                              },
+                              {
+                                "key": "x-request-id",
+                                "value": {
+                                  "stringValue": "req-1"
+                                }
+                              }
+                            ]
+                          }
+                        }
+                      }
+                    ]
+                  }
+                }
+              },
+              {
+                "timeUnixNano": "1767225602000000000",
+                "observedTimeUnixNano": "1767225602000000000",
+                "severityNumber": 17,
+                "severityText": "error",
+                "attributes": [
+                  {
+                    "key": "http.route",
+                    "value": {
+                      "stringValue": "/pay"
+                    }
+                  },
+                  {
+                    "key": "policy.name",
+                    "value": {
+                      "stringValue": "logs-redaction"
+                    }
+                  }
+                ],
+                "body": {
+                  "kvlistValue": {
+                    "values": [
+                      {
+                        "key": "msg",
+                        "value": {
+                          "stringValue": "payment failed token=[REDACTED]"
+                        }
+                      },
+                      {
+                        "key": "level",
+                        "value": {
+                          "stringValue": "error"
+                        }
+                      },
+                      {
+                        "key": "attempt",
+                        "value": {
+                          "intValue": "3"
+                        }
+                      },
+                      {
+                        "key": "card",
+                        "value": {
+                          "stringValue": "[REDACTED]"
+                        }
+                      },
+                      {
+                        "key": "db_password",
+                        "value": {
+                          "stringValue": "not-a-match"
+                        }
+                      }
+                    ]
+                  }
+                }
+              }
+            ]
+          }
+        ]
+      }
+    ]
+  }
+]

examples/fts/logs-redaction/fluent-bit.yaml

@@ -0,0 +1,18 @@
+service:
+  flush: 1
+  log_level: info
+
+pipeline:
+  inputs:
+    - name: opentelemetry
+      listen: 127.0.0.1
+      port: 4318
+      processors:
+        logs:
+          - name: fts
+            script: policy.lua
+
+  outputs:
+    - name: stdout
+      match: '*'
+      format: otlp_json