Skip to content

feat(benchmark): Add vllm_performance benchmark for prithvi - #155

Open
mgazz wants to merge 6 commits into
IBM:mainfrom
mgazz:prithvi
Open

feat(benchmark): Add vllm_performance benchmark for prithvi#155
mgazz wants to merge 6 commits into
IBM:mainfrom
mgazz:prithvi

Conversation

@mgazz

@mgazz mgazz commented Jun 29, 2026

Copy link
Copy Markdown
Member

No description provided.

@DRL-NextGen

DRL-NextGen commented Jun 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Checks Summary

Last run: 2026-08-31T15:52:40.066Z

Mend Unified Agent vulnerability scan found 5 vulnerabilities:

Severity Identifier Package Details Fix
🔺 High CVE-2026-58659 lightning-2.6.5-py3-none-any.whl
PyTorch Lightning through 2.6.5, fixed in commit d710d68, contains a remote code execution vulnerabi...PyTorch Lightning through 2.6.5, fixed in commit d710d68, contains a remote code execution vulnerability in the _load_state function that imports and executes attacker-controlled module names from checkpoint _instantiator hyperparameters. Attackers can craft malicious checkpoint files that bypass weights_only=True protections to execute arbitrary code when LightningModule.load_from_checkpoint is called.
 Not Available
🔺 High CVE-2026-65918 torchvision-0.28.0-cp312-cp312-manylinux_2_28_x86_64.whl
PyTorch torchvision through 0.28.0, fixed in commit 4e05dc2, contains an out-of-bounds heap read vul...PyTorch torchvision through 0.28.0, fixed in commit 4e05dc2, contains an out-of-bounds heap read vulnerability in the GIF decoder's read_from_tensor callback that passes unclamped length to memcpy. Attackers can supply malicious or truncated GIF files to cause denial of service via segmentation fault or disclose adjacent heap memory contents.
 Not Available
🔷 Medium CVE-2026-59890 setuptools-80.10.2-py3-none-any.whl
setuptools is a package that allows users to download, build, install, upgrade, and uninstall Python...setuptools is a package that allows users to download, build, install, upgrade, and uninstall Python packages. Prior to 83.0.0, FileList applied MANIFEST.in exclude, global-exclude, recursive-exclude, and prune directives by matching compiled glob patterns against on-disk file names without Unicode normalization, so on macOS APFS or HFS+ an NFD file name could bypass an NFC exclusion rule and be packed into a source distribution. This issue is fixed in version 83.0.0.
 Upgrade to version setuptools - 83.0.0,setuptools - 83.0.0,https://github.com/pypa/setuptools.git - 83.0.0
🔷 Medium CVE-2026-65920 diffusers-0.39.0-py3-none-any.whl
Diffusers through 0.39.0, fixed in commit cee298c, contains a path traversal vulnerability in the _g...Diffusers through 0.39.0, fixed in commit cee298c, contains a path traversal vulnerability in the _get_checkpoint_shard_files function that allows attackers to read arbitrary files by supplying malicious weight_map values in model index JSON. Attackers can use ../ sequences or absolute paths in weight_map entries to escape the model directory and read safetensors files outside the intended location during model loading.
 Not Available
🔷 Medium CVE-2026-75104 transformers-5.14.1-py3-none-any.whl
Hugging Face Transformers fails to validate shard filenames in checkpoint index files, allowing atta...Hugging Face Transformers fails to validate shard filenames in checkpoint index files, allowing attackers to read arbitrary files outside the model directory. Attackers can supply malicious index files with parent-directory references or absolute paths that are joined without validation, enabling file disclosure and filesystem reconnaissance.
 Not Available

@christian-pinto christian-pinto added the ci Enable CI integration label Jun 30, 2026
@christian-pinto

christian-pinto commented Jun 30, 2026

Copy link
Copy Markdown
Member

@mgazz We have an issue currently blocking our CI. I will have tests and benchmarks test runs done as soon as we get the CI going again.

@christian-pinto

christian-pinto commented Jul 3, 2026

Copy link
Copy Markdown
Member

@mgazz please merge with the latest master and then run

git restore --source origin/main -- uv.lock
uv lock

Then add the new uv.lock and commit. Make sure the pre-commit hooks are installed becuase they will re-generate the requirement files.

@christian-pinto

christian-pinto commented Jul 3, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

✅ Benchmark Run (2026-07-03 13:01 UTC)

Status Duration Ray Job ID Instance
✅ 45s raysubmit_eBX6vxZcWXP7rCCq packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@DRL-NextGen

DRL-NextGen commented Jul 3, 2026

Copy link
Copy Markdown
Member

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

@christian-pinto

christian-pinto commented Jul 8, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

✅ Benchmark Run (2026-07-08 08:33 UTC)

Status Duration Ray Job ID Instance
✅ - raysubmit_kCSfr6uEANHdn6qW packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@DRL-NextGen

DRL-NextGen commented Jul 8, 2026

Copy link
Copy Markdown
Member

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

@christian-pinto

christian-pinto commented Jul 8, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Jul 8, 2026

Copy link
Copy Markdown
Member

🔁 Benchmark Run (2026-07-08 08:47 UTC)

Status Ray Job ID Instance
🔄 raysubmit_7U4393QH8DGxgrWh packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@christian-pinto

christian-pinto commented Jul 8, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Jul 8, 2026

Copy link
Copy Markdown
Member

🔁 Benchmark Run (2026-07-08 09:07 UTC)

Status Ray Job ID Instance
❌ - packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@christian-pinto

christian-pinto commented Jul 8, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

✅ Benchmark Run (2026-07-08 09:48 UTC)

Status Duration Ray Job ID Instance
✅ 175s raysubmit_h8hjk6QypreGPMdE packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@DRL-NextGen

DRL-NextGen commented Jul 8, 2026

Copy link
Copy Markdown
Member

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

@christian-pinto

christian-pinto commented Jul 15, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Jul 15, 2026

Copy link
Copy Markdown
Member

🔁 Benchmark Run (2026-07-15 09:53 UTC)

Status Ray Job ID Instance
❌ - packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@christian-pinto

christian-pinto commented Jul 15, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

✅ Benchmark Run (2026-07-15 10:13 UTC)

Status Duration Ray Job ID Instance
✅ - raysubmit_xiPY8v5tG6iww75x packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@DRL-NextGen

DRL-NextGen commented Jul 15, 2026

Copy link
Copy Markdown
Member

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

@DRL-NextGen

DRL-NextGen commented Jul 15, 2026

Copy link
Copy Markdown
Member

🔁 Benchmark Run (2026-07-15 10:14 UTC)

Status Ray Job ID Instance
🔄 raysubmit_Z9NN62AP4aewAPgR packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@christian-pinto

christian-pinto commented Jul 15, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Jul 15, 2026

Copy link
Copy Markdown
Member

🔁 Benchmark Run (2026-07-15 10:40 UTC)

Status Ray Job ID Instance
🔄 raysubmit_P2A2J3DyeAnUAii3 packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@christian-pinto

christian-pinto commented Jul 15, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

✅ Benchmark Run (2026-07-15 10:49 UTC)

Status Duration Ray Job ID Instance
✅ - raysubmit_PWj6xWHrqk2HCMTJ packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@DRL-NextGen

DRL-NextGen commented Jul 15, 2026

Copy link
Copy Markdown
Member

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

@mgazz

mgazz commented Jul 20, 2026

Copy link
Copy Markdown
Member Author

Currently blocked on this vLLM PR to restore ability to run vllm benchmarks without tokeniser

mgazz added 3 commits August 5, 2026 14:06
@mgazz
feat(benchmark): Add vllm_performance benchmark for prithvi
fe626e6 
Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
@mgazz
build(benchmark): bump terratorch benchmark package version to >=1.9.6
90759c3 
Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
@mgazz
build(benchmark): update Prithvi benchmark package
d7c66e7 
- bump ado-vllm-performance version to 1.13.2
- update experiment name to geospatial-vllm-bench-deployment-custom-dataset

Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
mgazz added 2 commits August 6, 2026 13:38
@mgazz
fix(benchmark): update Prithvi experiment indentifier
0b1dbb5 
Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
@mgazz
fix(benchmark): add experimentVersion to the Prithvi experiment
c64418e 
Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
@mgazz

mgazz commented Aug 6, 2026

Copy link
Copy Markdown
Member Author

/run_benchmarks

1 similar comment
@christian-pinto

christian-pinto commented Aug 17, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Aug 17, 2026

Copy link
Copy Markdown
Member

🔁 Benchmark Run (2026-08-17 13:19 UTC)

Status Ray Job ID Instance
❌ - packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@christian-pinto

christian-pinto commented Aug 17, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Aug 17, 2026

Copy link
Copy Markdown
Member

🔁 Benchmark Run (2026-08-17 13:41 UTC)

Status Ray Job ID Instance
❌ - packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@christian-pinto

christian-pinto commented Aug 18, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Aug 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

✅ Benchmark Run (2026-08-18 08:22 UTC)

Status Duration Ray Job ID Instance
✅ 250s raysubmit_nzDYLn6PB2USPV8N packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@DRL-NextGen

DRL-NextGen commented Aug 18, 2026

Copy link
Copy Markdown
Member

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

@mgazz
build(benchmark): bump ado-vllm-performance version
15a7326 
Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
@christian-pinto

christian-pinto commented Aug 31, 2026

Copy link
Copy Markdown
Member

/run_benchmarks

@DRL-NextGen

DRL-NextGen commented Aug 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

✅ Benchmark Run (2026-08-31 15:52 UTC)

Status Duration Ray Job ID Instance
✅ - raysubmit_47fvgswCrqCZUUw8 packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

@DRL-NextGen

DRL-NextGen commented Aug 31, 2026

Copy link
Copy Markdown
Member

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@christian-pinto christian-pinto Awaiting requested review from christian-pinto

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ci Enable CI integration

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mgazz @DRL-NextGen @christian-pinto