Conversation
Log event size now sums only the bytes of body values, recursing into nested maps and lists, instead of the external size of the whole body. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BJGFLWZjY5iLrQ8GXLTFR5
| @doc """ | ||
| Size in bytes of an event body's values, excluding map keys. | ||
| """ | ||
| @spec body_byte_size(term()) :: non_neg_integer() | ||
| def body_byte_size(value) when is_map(value) and not is_struct(value) do | ||
| Enum.reduce(value, 0, fn {_k, v}, acc -> acc + body_byte_size(v) end) | ||
| end | ||
|
|
||
| def body_byte_size(value) when is_list(value) do | ||
| Enum.reduce(value, 0, fn v, acc -> acc + body_byte_size(v) end) | ||
| end | ||
|
|
||
| def body_byte_size(value), do: :erlang.external_size(value) |
There was a problem hiding this comment.
| @doc """ | |
| Size in bytes of an event body's values, excluding map keys. | |
| """ | |
| @spec body_byte_size(term()) :: non_neg_integer() | |
| def body_byte_size(value) when is_map(value) and not is_struct(value) do | |
| Enum.reduce(value, 0, fn {_k, v}, acc -> acc + body_byte_size(v) end) | |
| end | |
| def body_byte_size(value) when is_list(value) do | |
| Enum.reduce(value, 0, fn v, acc -> acc + body_byte_size(v) end) | |
| end | |
| def body_byte_size(value), do: :erlang.external_size(value) | |
| @doc """ | |
| Approximate wire size in bytes of an event body's values, excluding map keys | |
| and structural punctuation. | |
| """ | |
| @spec body_byte_size(term()) :: non_neg_integer() | |
| def body_byte_size(value) when is_binary(value), do: byte_size(value) | |
| def body_byte_size(value) when is_integer(value), do: byte_size(Integer.to_string(value)) | |
| def body_byte_size(value) when is_float(value), do: byte_size(Float.to_string(value)) | |
| def body_byte_size(value) when is_atom_value(value), do: byte_size(Atom.to_string(value)) | |
| def body_byte_size(value) when is_boolean(value), do: 1 | |
| def body_byte_size(nil), do: 0 | |
| def body_byte_size(value) when is_map(value) and not is_struct(value) do | |
| :maps.fold(fn _k, v, acc -> acc + body_byte_size(v) end, 0, value) | |
| end | |
| def body_byte_size(value) when is_list(value) do | |
| Enum.reduce(value, 0, fn v, acc -> acc + body_byte_size(v) end) | |
| end | |
| def body_byte_size(value), do: :erlang.external_size(value) |
🤖
per-leaf
external_sizeis wrong in both directions. Each leaf re-pays the ETF version byte + type tag: a 1-byte integer costs 3, a 5-byte binary costs 11.Enum.to_list(1..100)measures 300 vs 104 before. Metric histogram bodies are numeric arrays, so this inflates them ~3x.
|
|
||
| row = | ||
| {id, gen_tid, gen_event_id, :erlang.external_size(event.body), event.retries || 0, | ||
| {id, gen_tid, gen_event_id, LogEvent.body_byte_size(event.body), event.retries || 0, |
There was a problem hiding this comment.
🤖
sizeisn't telemetry-only. It's the byte budget forbigquery/pipeline.ex:236-249(@max_batch_length6MB, sized to stay under BQ's hard 10MB limit,@max_retries 0) andspool/producer_pipeline.ex:240-254(32MB / 12MB early flush). A realistic log body measures 149 vs 300 before, so both splitters now pack ~2x the real payload. Fix: keep the pointer'ssizetransport-accurate and compute the key-excluded number at the telemetry site (bigquery/pipeline.ex:674), since the two uses genuinely need different definitions.
| @doc """ | ||
| Size in bytes of an event body's values, excluding map keys. | ||
| """ | ||
| @spec body_byte_size(term()) :: non_neg_integer() |
There was a problem hiding this comment.
Worth getting a couple basic tests around this function added to test/logflare/log_event_test.exs as well.
Log event size now sums only the bytes of body values, recursing into
nested maps and lists, instead of the external size of the whole body.
Does a full body iteration so it isn't the most performant, to be fixed in a follow-up.