BigQueryReadClient::ReadRows fails with `UNKNOWN: Error occurred when fetching oauth2 token` after CreateReadSession succeeds

[dharanad]

Client google-cloud-cpp v2.22.0

What Happened

Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: BigQuery ReadRows failed for stream projects/rising-woods-374602/locations/us/sessions/CAISDG80QTJWdmc5UnpMSRoCamMaAm55/streams/CM8RGgJqYxoCbnkoAg: UNKNOWN: Error occurred when fetching oauth2 token. (rows delivered by this data source before the failure: 0)

I'm not filing this as a bug yet. I want to understand the failure mode.

1. Under what conditions does this occur?
2. Is this a client side or service side issue ? how do I confirm which one it is?

Is UNKNOWN error type retried by the default BigQueryReadConnection retry policy? If a transient token fetch failure lands as UNKNOWN, it looks like it would bypass retry and kill the read outright. Is that expected, and is there a supported way to make token-fetch failures retryable ?

[Ruchirtripathi]

The credentials are valid, as confirmed by the successful CreateReadSession.
The failure during ReadRows is most likely happening later when the long-running gRPC stream needs to refresh the OAuth2 token or reconnect.
A temporary network/DNS issue, timeout, or dropped gRPC connection can cause that token refresh to fail.
If running on GCP, temporary overload or rate-limiting of the metadata server could also be responsible.
So the issue appears to be transient connectivity/token refresh rather than incorrect credentials.

This is a client-side (or client-environment) issue. The BigQuery service never actually saw the ReadRows request because the gRPC library on the client
machine refused to dispatch it without a valid authentication token.

You can confirm this and see the exact underlying cause (e.g., DNS failure, HTTP 503 from the metadata server, connection refused) by enabling verbose
gRPC tracing on your client. Set the following environment variables before running your application:

export GRPC_TRACE=oauth2_security_connector,secure_endpoint,http
export GRPC_VERBOSITY=DEBUG

This will print out the exact HTTP requests being made by gRPC to fetch the token and the specific errors they encounter.

It's more of an edge case than a strict bug, However the intersection of these two behaviors results in transient network errors to the metadata server manifesting as fatal stream errors. It is a known pain point, and a valid feature request on that issue would be to ask if their interceptors can internally map this specific "fetching oauth2 token" UNKNOWN error into an UNAVAILABLE status so that it is seamlessly handled by the default retry policies.