Carry coarse label-source provenance into Under the Hood reports - #52
Open
Pitchfork-and-Torch wants to merge 26 commits into
Open
Carry coarse label-source provenance into Under the Hood reports#52Pitchfork-and-Torch wants to merge 26 commits into
Pitchfork-and-Torch wants to merge 26 commits into
Conversation
TweetSafetyLabelEvent already has SafetyLabelSource (BotMaker vs Tool), but the daily job dropped it and the report had no field to show whether a label was applied manually. Persist a coarse category only — no rule_id or actor_ldap — and emit it on each post-label row. Fixes xai-org#39 Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Replace other/unavailable with unknown. reportJson clamps to the four tokens. GrokAnnotationAction is not on the published spam.rtf IDL the jobs compile against, so it is not matched and folds to unknown. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Persist only automated|manual|llm so snapshot/flag/unmapped rows share one unset key and do not split into duplicate unknown report rows. Detect a GrokAnnotation* union via Product prefix so llm works when that variant exists without naming a case that may be absent from spam.rtf. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Treat stored unknown/other/unavailable as unset so earlier daily rows do not split aggregates. Also match GrokAnnotation on the union class name when the value is not a Product. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Group-by already coalesced leftover tokens; the kept row still carried the raw string. Normalize after reduce so month assembly always sees automated|manual|llm or unset, including single-row groups. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Count keys used the raw last-apply source. Leftover tokens that persist to unset could split a label, then collapse at write and drop counts. Normalize in actionAgg and the count key so one persistable token is counted once. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Trim and lowercase so Automated / ' llm ' do not miss the allowlist, split aggregates, or render as unknown. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
UthDailyPostLabel is constructed with named args everywhere else in this lane. Reconstruct the row the same way so source rewrite does not depend on a case-class copy method the IDL compiler may not emit. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Positional UthPostLabelAggregate(label, days, source) depends on field order. Use named args and persistToken so the MH row matches the daily rewrite path. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Leftover month rows stored as unset vs unknown/other collapse to the same report token. Keep one row per (label, source), taking the larger post count so the same posts are not shown twice. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
groupBy after source dedupe left row order hash-dependent. Sort so the same month aggregate always emits a stable reportJson list. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Match the last simple class segment with startsWith(GrokAnnotation) instead of contains(), so an unrelated name cannot be classified llm. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Flag inputs have no SafetyLabelSource. Set source = None at both daily and backfill writes so a future thrift default cannot invent provenance. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
max(posts) dropped disjoint leftover days after source clamp. Union days and keep the larger carried/removed per day, matching month assembly, then sum. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
groupBy returns a Map; flatMap there expects key/value pairs and can drop or fail to type the report rows. Convert toList first. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
String.split('$') does not split inner-class names. Take the segment
after the last '.' or '$' and also check getName so binary names like
SafetyLabelSource$GrokAnnotationAction still map to llm.
Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Nested tuple accessors can silently pick the wrong slot if the group shape changes. Destructure (name, source) and day rows by case. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
postLabelSource was missing its function closer, so formatPercentage was nested inside it. Post-label about/effect must use the raw label id, matching the account path; display name stays on label only. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
latestAsOf grouped by persistable source, so an earlier unset row could survive next to a later sourced rewrite of the same user/day/label. Keep every source from the latest asOf only. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
lastIndexOf('$') on GrokAnnotationAction$ (Scala module suffix) left
an empty segment and missed llm. Take the last non-empty . / $ piece.
Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
mergeActionAgg took the later apply's source even when that row was an unset snapshot duplicate, wiping BotMaker/Tool/Grok provenance. Last apply still wins the timestamp; fall back to the other persistable source when the later row is unset. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
groupBy still used t._1 / t._2 after raw was added as a third tuple field. Name the display name and source in the key, same as the leftover flatMap. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Class.getSimpleName can throw InternalError on Scala $ names and would fail the daily job for an unmapped SafetyLabelSource. getName plus the last non-empty segment already yields the simple class name. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
Same-day leftover rows still used best._4 / best._5 after source was added to the tuple. Take carried/removed by pattern match. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
mergeActionAgg fell back to an earlier persistable source for any later unset apply, so a later unmapped event inherited BotMaker/Tool provenance. Thread isSnapshot and fall back only for snapshot rows. Co-authored-by: Jon Bailey <Pitchfork-and-Torch@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #39.
README says Under the Hood should show whether labels were manually applied outside automated systems. The daily job already reads
TweetSafetyLabelEvent.labelbut kept only timestamps;SafetyLabelSourcewas dropped, and the serving types had no field to carry it.reportJson.sourceis one of four tokens:automated←BotMakerActionmanual←ToolActionllm← a Grok/LLM union member, detected byproductPrefixorgetNameplus the last non-empty./$segment (GrokAnnotation*), skipping a trailing synthetic$andgetSimpleName(can throw on Scala$names), so this compiles if that case is absent fromspam.rtfunknown← unset snapshot / tweet-flag / unmapped-source rows (serving clamp)Jobs persist only
automated|manual|llmafter trim/lowercase. Unmapped, missing, leftover tokens, and tweet-flag rows stay unset (source = Noneis set explicitly on flag writes). Daily counts clamp the last-apply source before grouping. When an event apply and an unset snapshot apply are merged, the later timestamp wins; a persistable event source is kept only if the later row is a snapshot gap-fill (isSnapshot), not a later unmapped event.latestAsOftreats the newest asOf as a full rewrite of(user, authored day, label): it keeps every persistable source from that asOf and drops older asOf rows so a later sourced rewrite cannot sit beside an earlier unset row. The month MH writer reconstructs rows with named Scrooge fields and pattern-matches leftover same-daycarried/removed(notbest._4/best._5). Serving clamps any other stored string tounknown, merges leftover rows that clamp to the same(label, source)by unioning days (larger carried/removed per day, then sum;groupBy.toListbeforeflatMapso Map.flatMap cannot drop rows), pattern-matches that leftover group key as(name, source)rather thant._1/t._2, and sorts that list by label then source soreportJsonorder is stable.postLabelSourceis a closed function (match + function closer beforeformatPercentage). Post-labelabout/effectlook up by the raw label id, same as the account path; displaynameis only written tolabel.Does not emit
rule_id,actor_ldap,agent_tool, or VF-client type names. Snapshot and tweet-flag inputs stay unset (no invented source).Based on current
xai-org/x-algorithmmain (c65aa179). Lane isunder-the-hood/only.Test plan
reportJsonpath.under-the-hood/has no published build/test target in this snapshot; no job was executed.source(ids 3 / 10) is backward-compatible with existing parquet/MH rows.reportJson.sourcecan only beautomated|manual|llm|unknown.unknownrow per label, including after daily count,latestAsOf, and reportJson day-union.(user, day, label)and does not emit both.automatedandmanualrows for one label both survivelatestAsOf.source = None.Automated/ padded tokens normalize to the allowlist instead ofunknown.(label, source).groupBykeys(name, source)by pattern match, nott._1/t._2.carried/removed, notbest._4/best._5.productPrefixorgetNameplus the last non-empty./$segment, notgetSimpleNameorString.split("$").postLabelSourcecloses beforeformatPercentage.postLabelAbout/postLabelEffectreceive the raw label id, not the display name.