dark factory is a pipeline that lets a language model write code on its own, inside a virtual machine that exists for one task and is deleted when the task ends, judged by tests it never sees. A task with hidden acceptance tests goes through preflight, execution in a throwaway machine, verification, a push, and a second fresh machine that runs the hidden tests; the run ends as exactly one outcome in an append-only log. Three parts live here. The runner is that pipeline. The bench is its measuring instrument: the tools that validate a task set against its reference solutions and deliberately broken copies, the frozen limits a comparison runs under, and the arms (the ways a model is put to a task). The templates are the Go and Python starting trees a task is laid over. The task set itself, with its hidden tests, is the repository dark-tasks; three example tasks stay here so the tools run without another checkout.

What you need

Running the runner needs what runner/host.toml names: a Proxmox host with a VM template, a Gitea instance, and an OpenAI-compatible model endpoint. Every value in that file is an example for a local machine; replace it with your own. A container backend is available: docs/docker.md builds the runner image, starts Gitea and the runner on one internal network with compose.yaml, and spawns each sandbox as a sibling container, so the runner can be deployed without Proxmox. A third backend, lxc, makes each sandbox a full clone of a snapshot of a real container on the Proxmox host, named by sandbox_container and sandbox_snapshot in runner/host.toml. Its sandbox_pool (default "", no Proxmox pool) is the pool each clone is placed in, so the throwaways stay grouped, and its sandbox_allow_in (default "") is a comma-separated list of "<ipv4>:<tcp port>" clients the clone's default-drop firewall lets in besides the service host. On the container backend a sandbox reaches only Gitea and the gates, the reverse proxies in front of the model endpoint and the trace collector, and DARK_CONF_DIR keeps the three toml files, with your own endpoints and model ids, outside the checkout. A tier written claude:<model-id> runs Claude Code as the executor through one more gate, which holds the token so that no sandbox does; docs/docker.md has the setup.

The bench tools need Python 3, bash, coreutils and git. The task check reads the task set dark-tasks from wherever DARK_TASKS (or a path given as its argument) points; a clone is never found by its place on disk alone. With neither, the same command checks the three example tasks kept here.

Quick start

python3 -m venv .venv && . .venv/bin/activate
pip install '.[dev]'                         # dark console script, plus pytest: verify.sh needs it
dark --help                                  # the commands the runner offers
dark check-config                            # config from --conf, $DARK_CONF, else ./runner here
(cd runner && bash verify.sh)                # tests, config check, no binaries
python3 bench/tools/check_tasks.py           # the three example tasks: 0 problems

For real use, clone dark-tasks and point DARK_TASKS at the clone:

git clone https://github.com/majgull/dark-tasks $HOME/src/dark-tasks
export DARK_TASKS=$HOME/src/dark-tasks
python3 bench/tools/check_tasks.py           # every task in the set
python3 bench/tools/validate.py              # reference solutions pass, starting trees fail (needs Go)

DARK_TASKS takes several task sets at once, : separated, such as $HOME/src/dark-tasks:$HOME/src/my-tasks, so a public set and a private one are checked in one run; a task name found in two sets is refused, naming both, rather than one silently winning.

dark user --task <task.toml> --tier <id> runs one user-arm task: a browser sandbox checks a deployed URL step by step. The example is dark user --task bench/tasks/hello-user/task.toml --tier example-small: three steps a stranger can check on https://example.com, with example-small the placeholder tier in runner/models.toml.

dark desktop --task <task.toml> --tier <id> --target lab runs one desktop-arm task: the task's own image as a lab sandbox takes its numbered steps against a desktop, and hidden checks judge the result.

dark demo <records> cuts that run's records into a captioned demo.mp4 (needs ffmpeg with libass and libx264; --voice adds speech with piper).

python3 -m dark long --task <dir> --tier <id> runs one long-arm task: a session executor works in the task's several repositories and a reviewer session judges the branches it pushed.

docs/fresh-container.md records a fresh container running this Quick start.

Layout

runner/      the pipeline: the dark/ package, models.toml, budgets.toml, host.toml, ops/, tests/
bench/       tools/ that validate a task set, frozen/ limits, arms.toml; tasks/ holds three examples, mutants/ one
templates/   the Go and Python starting trees a task is laid over
docs/        concepts.md (the terms the code uses), history.md (where it came from) and fresh-container.md

Documentation

• docs/concepts.md defines every term the code uses.
• docs/mcp.md says how to start and register the MCP server.
• docs/history.md says where the code came from.
• Contributing says how to run the tests and the task checks.
• dark-tasks is the 31-task set with its hidden tests.
• dark-paper is a frozen study of the tool on its first task set.

License

The whole repository is dedicated to the public domain under CC0 1.0 Universal; LICENSE is its text.