Skip to content

feat(storage): Add full object checksum validation for appendable uploads finalize#16245

Merged
v-pratap merged 4 commits into
googleapis:mainfrom
v-pratap:implement-final-proposal-clean
Jul 14, 2026
Merged

feat(storage): Add full object checksum validation for appendable uploads finalize#16245
v-pratap merged 4 commits into
googleapis:mainfrom
v-pratap:implement-final-proposal-clean

Conversation

@v-pratap

@v-pratap v-pratap commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

#GeminiAlert

  1. is_append == true
    For appendable uploads, the internal HashFunction is broken. It only sees the chunks uploaded in the current stream, not the full object, so its locally computed hashes are wrong. Because it's broken, we MUST bypass it (by setting action = kFinalize) and manually inject the checksum ourselves. Since we are doing it manually, we want to respect the user's expected checksum regardless of whether they set it at the start of the upload (options_) OR dynamically at the end (current_options). Therefore, we pull the value from merged to cover both bases.

  2. is_append == false (Normal Uploads)
    For normal uploads, the internal HashFunction works perfectly! It computes the hashes over the whole object, validates them locally, and injects them into the gRPC request. HOWEVER, the HashFunction is initialized at the start of the upload. It only knows about options_. It has absolutely zero knowledge of current_options (options passed dynamically at the exact moment of Finalize()).

If the user provided the checksum at the start (options_), we WANT the HashFunction to handle it. If we checked merged here, we would mistakenly bypass the HashFunction. BUT, if the user provided the checksum dynamically at the end (current_options), the HashFunction doesn't know about it. It will compute its own hash and aggressively overwrite whatever the user wanted! Therefore, for normal uploads, we only check current_options. If the user provided a last-second checksum, we bypass the HashFunction and manually inject it. Otherwise, we leave the HashFunction alone to do its job.

Summary:

We use merged for appends because we must manually inject the checksum no matter when it was provided.
We use current_options for normal uploads because we only want to manually inject it if the user provided a last-second checksum that the HashFunction doesn't know about.

@v-pratap v-pratap requested review from a team as code owners July 10, 2026 08:53
@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Jul 10, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors checksum handling and hash function management during appendable uploads. The review feedback identifies critical issues where removing the Crc32cHashFunction constructor and its recreation logic breaks checksum validation for resumed non-appendable uploads. Additionally, the reviewer recommends simplifying the UpdateChecksums signature, returning an explicit error if requested CRC32C validation cannot be performed due to missing metadata, and adding unit tests to cover the new validation paths.

Comment thread google/cloud/storage/internal/async/writer_connection_resumed.cc
Comment thread google/cloud/storage/internal/hash_function_impl.h
Comment thread google/cloud/storage/internal/async/writer_connection_impl.h Outdated
Comment thread google/cloud/storage/internal/async/writer_connection_impl.cc Outdated
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.01980% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.27%. Comparing base (ce70c7d) to head (76976b5).

Files with missing lines Patch % Lines
...rage/internal/async/writer_connection_impl_test.cc 97.79% 4 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #16245    +/-   ##
========================================
  Coverage   92.26%   92.27%            
========================================
  Files        2214     2214            
  Lines      206307   206505   +198     
========================================
+ Hits       190355   190546   +191     
- Misses      15952    15959     +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@v-pratap v-pratap force-pushed the implement-final-proposal-clean branch 6 times, most recently from a354dfa to 04ce29b Compare July 13, 2026 09:24
Comment thread google/cloud/storage/internal/async/writer_connection_impl.cc Outdated
Comment thread google/cloud/storage/internal/async/writer_connection_impl.cc
@v-pratap v-pratap force-pushed the implement-final-proposal-clean branch from 04ce29b to a77ed47 Compare July 13, 2026 11:19
@v-pratap v-pratap requested a review from kalragauri July 13, 2026 11:24
@v-pratap v-pratap force-pushed the implement-final-proposal-clean branch from a77ed47 to 69c280b Compare July 14, 2026 06:18
@v-pratap v-pratap force-pushed the implement-final-proposal-clean branch from 69c280b to 76976b5 Compare July 14, 2026 08:22
@v-pratap v-pratap merged commit f1be536 into googleapis:main Jul 14, 2026
62 of 63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants