Add experimental slate-level diversity reranking

[therufet721]

Summary

• replace plain top-K selection in the Phoenix candidate pipeline with an experimental slate-diversity selector
• cap both original-content authors and serving/retweeting authors near the top of the slate
• avoid adjacent posts with identical non-empty semantic-ID vectors
• bound relevance displacement with a configurable lookahead and relax semantic diversity before author caps when necessary
• keep the feature disabled by default for controlled evaluation
• extract the dependency-free algorithm into a standalone local Cargo test harness

Why

Individually relevant posts can still produce a repetitive feed when the top-ranked slate contains repeated authors or adjacent content from the same semantic cluster. This change adds an interpretable, bounded reranking layer while preserving score order whenever the constraints allow it.

Interaction with existing soft author diversity

RankingScorer already supports EnableAuthorDiversity, which computes SlateContext.k in its pre-diversity score order and applies a soft score multiplier using AuthorDiversityDecay and AuthorDiversityFloor. This PR intentionally adds an independently configurable hard guardrail rather than replacing that mechanism: a soft decay discourages repetition but cannot guarantee a maximum.

The selector does not reuse SlateContext.k because that value only counts the raw serving/retweeting author_id, reflects an earlier pool ordering, and can become stale after VMRanker changes scores. It also is not a count of the selector's dynamically chosen prefix. The selector therefore maintains fresh counts for both the serving author and the original author of retweeted content.

Before enabling broadly, evaluation should compare three cells: existing soft decay only, hard cap only, and soft decay plus hard cap. Incremental reorder rate and score/rank displacement from the hard selector should be measured to detect over-diversification.

Configuration

• EnableSlateDiversity (default: false)
• SlateDiversityAuthorWindowSize (default: 20)
• SlateDiversityMaxPostsPerAuthor (default: 2)
• EnableSlateSemanticDiversity (default: true within an enabled treatment)
• SlateDiversityMaxLookahead (default: 10)

Validation

9 passed; 0 failed

The standalone harness compiles the same dependency-free engine used by the Home Mixer adapter. Tests cover original-author and serving-author caps, exact semantic-vector matching, partial-vector overlap, fallback ordering, lookahead displacement, degenerate configurations, and 500 deterministic randomized candidate pools verifying that candidates are never lost or duplicated.

Also validated with Rustfmt, cargo clippy --all-targets -- -D warnings, and git diff --check.

Limitations and follow-ups

• This checkout does not include a Cargo manifest or internal dependencies for home-mixer, so the adapter still requires internal CI compilation.
• Diversity is applied before post-selection visibility and conversation filters. Those filters can remove an intervening post, so semantic adjacency is currently a best-effort pre-filter property rather than a final served-feed guarantee.
• Complete semantic-vector equality is intentionally conservative, but the appropriate SID hierarchy/prefix should be confirmed with the SID owner before enabling semantic diversity.
• Semantic-ID coverage and constraint-relaxation metrics are not included in this change.

This PR is intentionally draft and the feature remains off by default.

[aialphawolf]

Thank you you don't know how much I appreciate this!

[NetworkCats]

LGTM