Conversation
Route recordings by source format, add staged MCAP-to-FB conversion, and select FB worker concurrency from explicit GPU profiles with a conservative fallback. Constraint: 5060-class workers must not inherit concurrency tuned for 5090-class hardware Rejected: Fixed 2/2 FB parallelism | It can exhaust VRAM on lower-memory GPUs Confidence: high Scope-risk: moderate Directive: Keep numeric environment overrides available until cooperative runtime throttling exists Tested: 67 targeted backend tests; frontend production build; diff whitespace check; RTX 5060 Ti profile smoke selected 1/1 Not-tested: Full suite is blocked by existing tests/test_config.py expectation that omits the current raw dataset source
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
auto,mcap, orfb) and report skipped mixed-format recordings through job progress.mcap_to_fb_convertjob that rewrites legacy MCAP recordings into the Data Foundry FB layout using staging and atomic promotion.data_foundry-conversion-workerGPU image via the Docker API while preserving cancellation and destination rollback behavior.parallel=3,nvenc_parallel=3parallel=3,nvenc_parallel=21/1FB_CONVERTER_PARALLEL,FB_CONVERTER_NVENC_PARALLEL, andFB_CONVERTER_GPU_MODELas operational overrides.Why
The previous FB wrapper always passed
2/2, so lower-memory GPUs inherited concurrency chosen for higher-end workers and could fail with GPU OOM. The converter also lacked a first-class distinction between legacy MCAP recordings and the Data Foundry FB input layout.This change makes format selection explicit and gives lower-tier or unknown GPUs a conservative default while retaining tuned 4090/5090 profiles and manual overrides.
Impact
convertjob.data_foundry-conversion-workerimage and Docker socket access from the converter service.mcap_to_fb_convertthrough idempotent schema compatibility statements.Validation
uv run pytest -q tests/test_converter_input_format.py tests/test_fb_converter.py tests/test_mcap_to_fb.py tests/test_mcap_to_fb_job.py tests/test_converter_queue_adapter.py tests/test_jobs_router.py tests/test_converter_dockerfile.py tests/test_ui_service_docker_ops.py tests/test_converter_mem_limit.py— 67 passed.npm run build— TypeScript and Vite production build passed.git diff --check— passed.parallel=1,nvenc_parallel=1.Known validation gaps
tests/test_config.py: the test expectsdataset_sources == ["lerobot", "lerobot_test"], while current settings also contain"raw". The first-failure run reported 37 passed and 19 skipped before that assertion.