docs: document the Task API - #1094
Open
adithya-s-k wants to merge 2 commits into
Open
Conversation
The Task API landed in huggingface#726 as part of the external-env importer work and has never been documented: `TaskProvider` and the four task request models were absent from the API reference, and `docs/` had no mention of the `/{env_name}/splits`, `/tasks`, `/num_tasks`, `/task`, `/task_range` routes. Adds a `guides/task-api.md` guide covering the protocol, the HTTP routes and their error semantics, split normalization, `env_name` resolution, the fresh-instance-per-request lifecycle, task selection via `reset(split=, index=)`, streaming caveats, and the client-side helper pattern. Also adds the missing autodoc entries and reformats the `TaskProvider` docstrings into the HF doc-builder style used by the reference docs.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Task API — the optional discovery layer that lets a dataset-backed environment publish its tasks and splits — has been in the codebase since #726 but has never been documented. Before this PR:
docs/had zero mentions of the Task API or any of its routesTaskProviderwas missing fromreference/core.md, as wereListTasksRequest,NumTasksRequest,GetTaskRequest, andGetTaskRangeRequestIt shipped in #726 as one bullet of the external-env importer work (
976c4121, "Adds ORS-compatible task/split APIs to the OpenEnv server") and hasn't been touched since. The only implementations in-repo are the generatedors_environment.py.tplwrapper and #1003 (LaTeX OCR), which is what surfaced the gap.Changes
New:
docs/source/guides/task-api.md— follows the structure ofguides/mcp-environment-lifecycle.md:TaskProviderstructural protocol, and the routesHTTPEnvServerderives from itenv_name→ 404, missing method /NotImplementedError→ 501,IndexError→ 400env_nameresolution — defaults to the factory's class name and is matched case-insensitively, so implementers should pass it explicitly{"name", "type"}, withtypepreserved only fortrain/validation/testResetRequest'sextra="allow", filtered against the env'sreset()signature, soreset(split=..., index=...)is what actually starts an episodelist_tasks()whilenum_tasks()keeps reporting the honest totalcurlequivalentsdocs/source/reference/core.md— adds the missing[[autodoc]]entries forTaskProviderand the four request models, plus a cross-link from the intro.docs/source/_toctree.yml— "The Task API" under Advanced Guides.src/openenv/core/env_server/interfaces.py— reformats theTaskProviderdocstrings into the HF doc-builder style used by the reference docs (Args/Returns/Raises,*optional*,Examples:), now that they're rendered by autodoc. No behavior change.Notes for reviewers
http_server.py:1062-1151,_get_valid_kwargs,_normalize_split,_call_task_method). [feature] Import external envs (ors, verifiers) #726 shipped no design doc, so the guide states the docstring's own side-effect-free / fresh-instance rules as the contract — happy to strengthen or soften that wording if there's a stronger intended contract.doc-builderisn't in the dev env here, so the rendered output is unverified;_toctree.ymlparses and all autodoc targets import.Verification
usort check/ruff format --check/ruff checkclean on the touched filepytest tests/core/test_task_api.py— 8 passedNote
Low Risk
Documentation and docstring-only updates with no changes to server or protocol behavior.
Overview
Adds first-class documentation for the Task API — the optional
TaskProviderdiscovery layer and the HTTP routesHTTPEnvServerexposes under/{env_name}/….A new Advanced Guides page (
guides/task-api.md) explains when to implement the five task methods, that discovery is metadata-only (episodes still start viareset(split=..., index=...)), route tables and error semantics (404/501/400), split normalization, client patterns, and a pre-ship checklist. The docs site nav and Core API reference now link to it and autodocTaskProviderplusListTasksRequest,NumTasksRequest,GetTaskRequest, andGetTaskRangeRequest.TaskProviderdocstrings ininterfaces.pyare expanded into Hugging Face doc-builder style (Args/Returns/Raises, examples) so they render in the reference; no runtime behavior changes.Reviewed by Cursor Bugbot for commit 3ba7e28. Bugbot is set up for automated code reviews on this repo. Configure here.