Repository navigation
Read the Docs builds have been killed after about 24 minutes since 30 October #40
Description
What happens
Every Read the Docs build since 2026-10-30 has been stopped at about 1420 s, with sphinx-build reporting no exit code and no output. Every build up to 2026-10-24 finished in about 10 minutes.
| Build | Version | Commit | Date | Seconds | Result |
|---|---|---|---|---|---|
| 35378900 | stable | 24672b1 |
10-24 | 663 | passed |
| 35378009 | PR 8 | c33d051 |
10-24 | 573 | passed |
| 35532209 | PR 5 | 5649fdf |
10-30 | 1427 | failed |
| 35639324 | PR 9 | d0ff0fd |
11-04 | 1411 | failed |
| 35639601 | PR 38 | 440c7d3 |
11-04 | 1428 | failed |
PR 5 only bumped GitHub Actions, and main has had no commit since 10-24, so the manual's content does not explain it.
Why it matters
latest and stable have not been rebuilt since 10-24, so they still serve. If the same thing happens to them, the published manual stops updating at the next merge or release, without the merge being blocked: this check is not one the main ruleset requires.
Context
- The same build passes elsewhere on the same commits:
poe docslocally in about 4.5 minutes on an Apple Silicon Mac, and the CI gate on Linux and macOS. - A suspicion, not a finding:
.readthedocs.yamlinstallsuv latest. The API logs of the builds above show uv 0.13.5 in the passing one and 0.13.6 in the failing one, with the same Python, 3.14.6, anduv syncoutput of the same length. That is the only difference visible in them. A change on Read the Docs' side, or a hang or memory limit in the gallery, would look the same. - The API returns no output for the killed command, so where it stopped is not known. The time limit that applies is not known either.
To do
- Look at the build page while logged in, for a stated reason such as a time or memory limit.
- Pin uv in
.readthedocs.yamlto 0.13.5, and rebuild a pull request. Not tried yet. - If that does not change it, add timing and output flushing to find where the build stops.
- Needs a decision: by the maintainers: whether to require this check on
mainonce it is reliable.
Must not change: warnings stay errors, and the gallery still executes.
Done when
A pull request's build and the next build of latest both finish, in about the old time or with the reason for the new one stated.
Activity
jtravs commentedon Nov 4, 2026
This weakens the suspicion above: the local
poe docsthat passes in about 4.5 minutes also ran on uv 0.13.6 (Homebrew build, Apple Silicon). So uv 0.13.6 alone does not make the build hang. Pinning uv is still a cheap test, but look at the build page first for a stated limit.jtravs commentedon Nov 4, 2026
Confirmed by the build page, logged in: "Build terminated due to time out." Read the Docs documents a limit of 15 minutes and 7 GB. A clean local build (Apple Silicon, 245 s) spends about 21 s starting up, 72 s on the gallery, 16 s reading and 133 s writing about 2000 HTML pages, with no single slow step. The 24 October builds took about 10 minutes, so the killed ones are at least 2.7 times slower on the same content. Whether they are slow or hung is still not known, because a killed build keeps no output.
jtravs commentedon Nov 4, 2026
Result. Sending the build's output to a file fixes it, and #41 does that.
latestat046a506with the unchanged config: killed at 1448 s. Prepare the next release notes #42, unchanged too: killed at 1473 s.So the build is neither hung nor newly slow, and uv is not the cause. The suspicion above is withdrawn.
Not established: the mechanism. A stall writing to a full pipe would fit but has not been observed, and neither what changed around late October nor why kills come at about 1420 to 1470 s rather than the documented 900 s is known.
latestfailed, so the published manual still shows the 24 October build until #41 merges.jtravs commentedon Nov 4, 2026
Correction to the comment above: sending the output to a file does not fix this. The same config with only an empty commit added, 35643041, was killed at 1451 s, so I drew that conclusion from too little.
Since 30 October: output straight to Read the Docs, 0 of 11 builds passed; output to a file, 2 of 3. The redirect may help, but a pass depends on something else as well, and the API records nothing that separates the pass from the fail: not uv, not the content, and it does not name the builder.
Still true: a passing build takes about 10 minutes, with steady progress and a peak near 1.5 GB. Unknown: the cause.
lateststill fails, so the published manual stays at the 24 October build.jtravs commentedon Nov 4, 2026
Another explanation ruled out. With the log kept in a file and only its last 20 lines going to Read the Docs, 35644770 was still killed, at exactly 900 s, so output volume is not the cause.
Also: 35644212 copied its output to a file and was set to kill everything at 780 s and exit. The command never returned, and the build ended as
cancelledat 1410 s with no output. Kills come at either 900 s or about 1420 to 1470 s.Next: repeat a build that prints progress and stops itself at 780 s, to get a trace of one that does not finish.
jtravs commentedon Nov 4, 2026
Where this stands. The evidence now points away from optmat's build, and I'm stopping the experiments from inside a build, since they can't separate what's left.
A self-stopping script does not return. 35645214 ran a foreground loop that depends on nothing but
sleepand a file: at 780 s it dumps state, prints its log and exits. The command never returned and left no output, and the build ended at 1459 s. 35644212, with a similar hard stop, ended the same way at 1410 s, ascancelled. So in a failing build, either the command's output stream stops being read or the whole container stops running. Sphinx being slow can't produce this. I can't tell those two apart from inside the build, and neither has been observed directly.Builds since 30 October (corrects the 0 of 11 above, which was a miscount):
exit 1), 35642042. Killed: 35643041, 35644770, 35645214Excluded, broken by mistakes in my diagnostic scripts: 35640711, 35641608, 35640759.
Ruled out or unsupported: optmat's content (
mainwas unchanged from 24 October to the first failure, and identical content both passes and fails); uv's version alone; output volume; memory, in the two traced passing builds (about 1.5 GB of 7.7 GB).Unexplained: kills come at 900 s or at about 1410 to 1470 s, when the documented limit is 900 s. 2 of 5 against 0 of 10 could be chance, so the redirect is not shown to help.
Next: ask Read the Docs, who can see the builders. #41 stays a draft, for its evidence.
jtravs commentedon Nov 4, 2026
Result: #44 makes the build pass, and corrects what I wrote above. All 6 builds of its tree passed. The cause of the stalls is still unknown.
0b747d9d2988a55148912495738b3727b72f24b553They ran one after another on the evening of 2026-11-04, with the unchanged config (output straight to Read the Docs), which had passed 0 of 10 since 30 October, counting 35647177 from #39. Sphinx took 185 s in 35653799 and 282 s in 35653170, against 632 s in 35642042.
What changed. Sphinx copies the whole navigation tree for every page it writes. The 763 source records in a hidden toctree took 177 s of a 271 s profiled local build, without re-running the gallery. #44 takes them out of the tree.
Corrections to the comments above:
Suspicion, not a finding: something changed on Read the Docs around 29 October that stops long builds. solprop passed builds of up to 776 s during that period, so there is no clean threshold. Its commit blames memory, at 8.8 GB on its side, but the two traced optmat passes peaked near 1.5 GB, so memory is not the shared cause.
Also unexplained: 35648470 (
latest, the #39 merge) and 35648479 (PR 42) ended ascancelledat about 215 s. The maintainer did not cancel them, and no newer build of those versions explains it.Still to do: merge #44 and confirm that
latestbuilds, which is this issue's Done when. A long build could still be exposed if the manual grows. #41 stays a draft until then, for its evidence.jtravs commentedon Nov 4, 2026
Done when is met. PR 44's build (35653969, 236 s) and
latestat the merge88f50f7(35655332, 251 s) both finished, and the published manual now serveswanted.html. Across #44's tree and the merge that is 8 of 8.The cause of the stalls is still not known, and a much longer build could meet it again. The maintainers' decision on requiring this check on
mainis still open; I would not require it while the stalls are unexplained.35655341, the release pull request #42, failed at 224 s for a separate reason: a broken link from a changelog fragment, shown in its log.