Skip to content

fix: Exclude map keys from ingested byte accounting - #3967

Open
Ziinc wants to merge 1 commit into
mainfrom
claude/log-event-bytes-values-1y327e
Open

Ziinc wants to merge 1 commit into
mainfrom
claude/log-event-bytes-values-1y327e

Conversation

@Ziinc

@Ziinc Ziinc commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Log event size now sums only the bytes of body values, recursing into
nested maps and lists, instead of the external size of the whole body.

Does a full body iteration so it isn't the most performant, to be fixed in a follow-up.

Log event size now sums only the bytes of body values, recursing into
nested maps and lists, instead of the external size of the whole body.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJGFLWZjY5iLrQ8GXLTFR5
@Ziinc Ziinc changed the title Exclude map keys from ingested byte accounting fix: Exclude map keys from ingested byte accounting Sep 11, 2026
Comment on lines +418 to +430
@doc """
Size in bytes of an event body's values, excluding map keys.
"""
@spec body_byte_size(term()) :: non_neg_integer()
def body_byte_size(value) when is_map(value) and not is_struct(value) do
Enum.reduce(value, 0, fn {_k, v}, acc -> acc + body_byte_size(v) end)
end

def body_byte_size(value) when is_list(value) do
Enum.reduce(value, 0, fn v, acc -> acc + body_byte_size(v) end)
end

def body_byte_size(value), do: :erlang.external_size(value)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@doc """
Size in bytes of an event body's values, excluding map keys.
"""
@spec body_byte_size(term()) :: non_neg_integer()
def body_byte_size(value) when is_map(value) and not is_struct(value) do
Enum.reduce(value, 0, fn {_k, v}, acc -> acc + body_byte_size(v) end)
end
def body_byte_size(value) when is_list(value) do
Enum.reduce(value, 0, fn v, acc -> acc + body_byte_size(v) end)
end
def body_byte_size(value), do: :erlang.external_size(value)
@doc """
Approximate wire size in bytes of an event body's values, excluding map keys
and structural punctuation.
"""
@spec body_byte_size(term()) :: non_neg_integer()
def body_byte_size(value) when is_binary(value), do: byte_size(value)
def body_byte_size(value) when is_integer(value), do: byte_size(Integer.to_string(value))
def body_byte_size(value) when is_float(value), do: byte_size(Float.to_string(value))
def body_byte_size(value) when is_atom_value(value), do: byte_size(Atom.to_string(value))
def body_byte_size(value) when is_boolean(value), do: 1
def body_byte_size(nil), do: 0
def body_byte_size(value) when is_map(value) and not is_struct(value) do
:maps.fold(fn _k, v, acc -> acc + body_byte_size(v) end, 0, value)
end
def body_byte_size(value) when is_list(value) do
Enum.reduce(value, 0, fn v, acc -> acc + body_byte_size(v) end)
end
def body_byte_size(value), do: :erlang.external_size(value)

🤖

per-leaf external_size is wrong in both directions. Each leaf re-pays the ETF version byte + type tag: a 1-byte integer costs 3, a 5-byte binary costs 11. Enum.to_list(1..100) measures 300 vs 104 before. Metric histogram bodies are numeric arrays, so this inflates them ~3x.


row =
{id, gen_tid, gen_event_id, :erlang.external_size(event.body), event.retries || 0,
{id, gen_tid, gen_event_id, LogEvent.body_byte_size(event.body), event.retries || 0,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖

size isn't telemetry-only. It's the byte budget for bigquery/pipeline.ex:236-249 (@max_batch_length 6MB, sized to stay under BQ's hard 10MB limit, @max_retries 0) and spool/producer_pipeline.ex:240-254 (32MB / 12MB early flush). A realistic log body measures 149 vs 300 before, so both splitters now pack ~2x the real payload. Fix: keep the pointer's size transport-accurate and compute the key-excluded number at the telemetry site (bigquery/pipeline.ex:674), since the two uses genuinely need different definitions.

@doc """
Size in bytes of an event body's values, excluding map keys.
"""
@spec body_byte_size(term()) :: non_neg_integer()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth getting a couple basic tests around this function added to test/logflare/log_event_test.exs as well.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants