[Typing] Fix functional bugs and add defensive runtime improvements (Part 2: Runtime Fixes) - #5425
Open
letitz wants to merge 18 commits into
Open
[Typing] Fix functional bugs and add defensive runtime improvements (Part 2: Runtime Fixes)#5425letitz wants to merge 18 commits into
letitz wants to merge 18 commits into
Conversation
letitz
force-pushed
the
typing-runtime-fixes
branch
11 times, most recently
from
August 14, 2026 03:46
cc057f3 to
35bebf9
Compare
This commit completes the migration of ClusterFuzz's Python dependency management system from Pipenv to uv. pipx is used to install uv locally. Architecture & Configs: - Created root pyproject.toml with 5 dependency groups, centralizing dependency definitions in a single file (from two Pipfiles) and removing the need for the hack where appengine packages were modelled as dev dependencies in src/Pipfile. - Added setuptools==80.9.0 to project dependencies for pkg_resources API support. - Pinned isort~=5.13.2 and pylint~=2.17.7 in dev dependencies to support line length options. - Shared pinned Google client dependencies in vendor-appengine to avoid version mismatches with vendor-core. - Deleted legacy Pipfile* and requirements.txt files. Butler & Runtime Improvements: - Updated _install_third_party in src/local/butler/common.py to use uv and avoid materializing requirements.txt into the repo, which are duplicative of `uv.lock` contents anyway. - Updated src/local/butler/py_unittest.py to preserve src/third_party in sys.path while adding src/appengine/third_party, ensuring both App Engine and core libraries are importable during App Engine unit tests. This fixes an issue where the previous setup assumed third-party packages were not only vendored into third_party directories, but also installed into the virtual environment. - Updated virtual environment guards in src/local/butler/guard.py and CASP CLI wrapper in cli/casp/src/casp/utils/container.py. Scripts, CI, Docker & Docs: - Updated setup scripts in local/ to install pipx and uv and activate .venv. - Updated CI workflows to do the same. - Updated Docker build scripts and Dockerfiles similarly. - Updated bot setup scripts (configs/test/bot/setup/*.bash, windows-init.ps1). - Updated developer and agent documentation. - Updated local/tests/install_deps_test_linux.bash to verify fresh checkout environment setup with uv (.venv, third_party vendoring, bower components, and uv run butler.py lint / py_unittest).
TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
- pyproject.toml: Removed duplicate root dependencies already in vendor-core (cryptography, future, google-cloud-ndb, protobuf, psutil, crcmod); added conditional pywinauto dependency for win32 under dev group. - windows-init.ps1: Reverted redundant PATH modification (pipx ensurepath handles PATH); updated uv sync to use --frozen. - Dockerfiles: Removed redundant --upgrade and --no-deps flags from uv pip install -e /data. - local/install_deps_linux.bash: Made optional gcloud apt packages conditional on rodete and added b/414408644 and b/484368884 references. - local/install_deps_macos.bash: Removed redundant pipx check (handled by Brewfile); added --frozen to uv sync. - local/install_python_deps_linux.bash: Removed extra source .venv/bin/activate call after nodeenv. - src/local/butler/py_unittest.py: Removed unnecessary site.addsitedir call. - CI/Bot setup scripts: Added --frozen flag to all automated uv sync invocations. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
- pyproject.toml: Moved setuptools==80.9.0 to [dependency-groups] dev. - local/tests/ci_tests.bash & setup scripts: Reverted --frozen flag from uv sync. - local/install_deps_linux.bash: Removed || true from apt-get install on rodete. - Dockerfile: Added explanatory comment for uv and python3.11 symlinks. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
- pyproject.toml: Moved vendor-core, vendor-appengine, and vendor-platform-specific to [project.optional-dependencies] extras: core, appengine, platform-specific. Added comment explaining special handling for platform-specific binary wheel downloads via pip download in common.py _install_platform_pip. - src/local/butler/common.py: Updated _install_third_party and _install_platform_pip to use uv export --extra. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
- local/README.md, AGENTS.md, kubernetes_e2e_test.bash, devcontainer.json: Removed redundant explicit pipx/uv installation steps as local/install_deps.bash handles uv setup. - pyproject.toml: Removed redundant packages from appengine optional dependencies group that are already included in core. Updated comment on platform-specific extras to clarify vendoring behavior. - uv.lock: Updated lockfile. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
…ests.bash TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
- Configured [[tool.uv.index]] with PyPI (https://pypi.org/simple) as the default index in pyproject.toml. - Regenerated uv.lock so all package sources and wheel URLs point to PyPI/files.pythonhosted.org rather than Google internal pkg.dev. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
The pre-built ci docker image does not have uv installed. Install pipx and use it to install uv before executing the CI test pipeline. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
Avoid Cloud Build substitution parsing errors on shell environment variables by extracting the CI setup and test invocation into a dedicated script. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
Parameterize local/tests/ci_tests_inner.bash to accept the test runner command (defaulting to local/tests/run_tests) and reuse it in cloudbuild.yaml. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
letitz
force-pushed
the
typing-runtime-fixes
branch
from
August 14, 2026 13:50
35bebf9 to
db94461
Compare
letitz
force-pushed
the
typing-runtime-fixes
branch
from
August 14, 2026 18:41
db94461 to
f87535f
Compare
This commit introduces Python type annotations across the codebase with zero runtime behavioral impact. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
- Restore bug fixes and defensive handling across 35 modules in src/appengine and src/clusterfuzz - Remove temporary type ignore comments from typing-mechanical - Add proper null-safety guards and assertions for runtime integrity
letitz
force-pushed
the
typing-runtime-fixes
branch
from
August 14, 2026 19:26
f87535f to
15d58d7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This is Part 2 of the static typing rollout for ClusterFuzz, built on top of Part 1 (#5424 /
letitz:typing-mechanical).While Part 1 focused strictly on mechanical type annotations with zero runtime impact, this PR addresses real bugs, edge cases, and missing defensive handling uncovered by static type checking across the codebase.
Summary of Changes
None/ missing attribute access on optional Datastore and API models.AttributeErrorandTypeErrorexceptions in task dispatchers and platforms.# type: ignoresuppressions from Part 1 with proper defensive checks and validations.tasks/__init__.py,bisection.py, andfuzzer_selection.py.Verification
uv run ty check: 0 errorsuv run butler.py lint --type-check: 0 errors, 0 warningspython butler.py py_unittest -t core -m: 2,221 tests passed (0 errors, 0 failures)python butler.py py_unittest -t appengine -m: 742 tests passed (0 errors, 0 failures)