Repository navigation

[FP8 5/5] Accelerate cached prediction with context splitting - #1107

Draft

aw471 wants to merge 5 commits into

fp8-stack/02-kumotabularfrom

fp8-stack/cached-predict

Draft

[FP8 5/5] Accelerate cached prediction with context splitting#1107
aw471 wants to merge 5 commits into
fp8-stack/02-kumotabularfrom
fp8-stack/cached-predict

Commits on Oct 26, 2026
- Accelerate cached FP8 attention with context partitions
  aw471
  committed
  d2434c7
- Use measured context thresholds for cached attention
  aw471
  committed
  a17ecb4
- Use a 64K cutoff for aligned cached attention
  aw471
  committed
  6a9e92c
- Use a shared measured cutoff for cached FP8 attention
  aw471
  committed
  9fb927c
- Retain cached attention speedups below 64K context
  aw471
  committed
  12048bc