CURLOPT_NOPROXY is set unconditionally, silently overriding the user's no_proxy environment variable

[ankitsambyal]

What component of google-cloud-cpp is this related to?

google/cloud/internal/curl_impl.cc, so it affects every REST-based client. I hit it through GCS (google/cloud/storage).

Describe the bug

Since v2.38.0, CurlImpl::MakeRequest unconditionally sets CURLOPT_NOPROXY to the literal "metadata.google.internal" on every request:

#ifndef GOOGLE_CLOUD_CPP_WINDOWS_BAZEL_CI_WORKAROUND
#if CURL_AT_LEAST_VERSION(7, 19, 4)
status = handle_.SetOption(CURLOPT_NOPROXY, "metadata.google.internal");
if (!status.ok()) return OnTransferError(context, std::move(status));
#endif
#endif
Per the libcurl documentation, CURLOPT_NOPROXY overrides the no_proxy environment variable rather than adding to it: "If there is an environment variable called no_proxy (or NO_PROXY), it is used if the CURLOPT_NOPROXY option is not set."

The effect is that the user's entire no_proxy list is discarded, while http_proxy/https_proxy continue to be honoured. On a machine behind a corporate proxy where no_proxy includes .googleapis.com, all GCS traffic that previously went direct is now tunnelled through the proxy. For bulk object reads this is a large throughput and latency regression.

This appears to be an unintended side effect of #15145 ("fix: Bypass proxy for GCE metadata server requests"). The intent was clearly to keep metadata-server probes off the proxy, but the option is applied in MakeRequest, which serves every request from every client, not just metadata requests.

To Reproduce

With these environment variables set:

http_proxy=http://proxy.example.com:3128
https_proxy=http://proxy.example.com:3128
no_proxy=.googleapis.com,localhost
Build a storage::Client with default options (no ProxyOption set).
Read any object.
Observe that the request goes through the proxy, despite .googleapis.com being listed in no_proxy.
The behaviour is reproducible directly with the curl CLI, since --noproxy maps to CURLOPT_NOPROXY. With the flag, mimicking the library:

$ curl -v --noproxy metadata.google.internal https://storage.googleapis.com/

• Uses proxy env variable https_proxy == 'http://proxy.example.com:3128'

CONNECT storage.googleapis.com:443 HTTP/1.1

• CONNECT tunnel established, response 200
  Without it, no_proxy is honoured and the connection is direct:

$ curl -v https://storage.googleapis.com/

• Uses proxy env variable no_proxy == '.googleapis.com,localhost'
  Note that the no_proxy line disappears entirely in the first case — libcurl never consults it.

Expected behavior

Setting up the metadata-server bypass should not discard a user-supplied no_proxy. Two options that would both work:

1. Scope the CURLOPT_NOPROXY call to the metadata credential requests it was meant for, rather than applying it in MakeRequest for all traffic. This seems closest to the original intent of fix: Bypass proxy for GCE metadata server requests #15145.
2. Merge rather than replace: read no_proxy/NO_PROXY from the environment and append metadata.google.internal to it before calling SetOption.

Operating system: Rocky Linux 9.7

What compiler and version are you using? gcc 13.4.0

What version of google-cloud-cpp are you using? Reproduced on 2.39.0 and 2.43.0. I bisected the released tags: google/cloud/internal/curl_impl.cc has no CURLOPT_NOPROXY call at v2.37.0, and does from v2.38.0 onward. The call is still present on main.

Additional context

Users behind a proxy currently have no way to work around this from the API, since the option is set inside the library after user Options are applied. The only mitigations are to unset the proxy environment variables for the affected process, or pin to a version below 2.38.