Skip to content

Use the s3fs profile argument as the boto3 profile name - #985

Merged
laughingman7743 merged 2 commits into
masterfrom
fix/983-s3fs-profile-arg
Oct 3, 2026
Merged

laughingman7743 merged 2 commits into
masterfrom
fix/983-s3fs-profile-arg

Conversation

@laughingman7743

Copy link
Copy Markdown
Member

WHAT

S3FileSystem created without a connection now uses s3fs's profile argument as the boto3 profile_name.
When both are given, profile_name wins.
AioS3FileSystem and fsspec.filesystem("s3", profile=...) (with PyAthena registered for s3://) go through the same client construction, so they also pick up the profile.

  • pyathena/filesystem/s3.py: _get_client_compatible_with_s3fs() maps profile to profile_name, and its docstring lists profile.
  • docs/filesystem.md: adds a named-profile example next to the other s3fs-compatible arguments.
  • tests/pyathena/filesystem/test_s3.py: test_get_client_compatible_with_s3fs_profile builds clients from temporary config and credentials files and checks which profile's access key signs requests.

Behavior change for users who passed profile= and relied on it being ignored: requests are now signed with the named profile, and an unknown profile raises botocore's ProfileNotFound at construction, as profile_name= already did.

WHY

Closes #983.
s3fs names the argument profile, while boto3 names it profile_name.
Only the keys in Connection._SESSION_PASSING_ARGS reach the boto3 Session, so profile= was silently dropped and the default credential chain was used.

TEST

Tested commit: 9cc6d01

  • just format and just lint: passed.
  • just docs lint: passed.
  • uv run pytest --noconftest -q tests/pyathena/filesystem/test_s3.py -k test_get_client_compatible_with_s3fs with dummy Athena environment variables: 5 passed.
    With the source change reverted, the {"profile": "other"} case fails (signs with DEFAULTKEY).
  • Manual check with temporary profile files: AioS3FileSystem(profile="other") and fsspec.filesystem("s3", profile="other") sign with the other profile.
  • No live AWS tests were run locally; the change only affects client construction, which the offline test covers.

🤖 Generated with Claude Code

S3FileSystem accepted s3fs's constructor arguments without a connection,
but dropped s3fs's profile argument because boto3 names it profile_name,
so requests were signed with the default credential chain. Map profile to
profile_name when profile_name is not given.

Closes #983

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread pyathena/filesystem/s3.py
}
kwargs.update(creds)
client_kwargs.update(creds)
if profile := kwargs.pop("profile", None):

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (implementation behavior, public contracts, simplicity, regression coverage)

Base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 9cc6d013516cb94011ac159d99eddee851f8ea24. Result: CLEAN.

Covered inventory: pyathena/filesystem/s3.py (_get_client_compatible_with_s3fs() and its callers), docs/filesystem.md, tests/pyathena/filesystem/test_s3.py.

  • Behavior: profile is popped from the local kwargs copy only. S3FileSystem.__init__() still passes the original kwargs to AbstractFileSystem, so profile stays in storage_options and the instance-cache token still separates filesystems by profile, as before. profile_name wins when both are given (setdefault).
  • Callers: the only caller is S3FileSystem.__init__() without a connection (s3.py:180). AioS3FileSystem.__init__() forwards **kwargs to the internal S3FileSystem (s3_async.py:113-123), and fsspec.filesystem("s3", ...) resolves to S3FileSystem once registered; both checked manually to sign with the other profile. With a connection, all s3fs arguments are ignored, as documented in the constructor docstring; unchanged.
  • Failure path: an unknown profile now raises botocore ProfileNotFound at construction (checked), the same as profile_name=. anon=True with profile still builds the session with the profile and signs nothing; s3fs also builds its session from the profile.
  • Test quality: the parametrized test builds clients from temporary config/credentials files with env credentials removed, and fails for {"profile": "other"} with the source change reverted (signs with DEFAULTKEY). Limitation: it reads the resolved access key through botocore's private _request_signer._credentials; botocore has no public accessor on a client.

Comment thread docs/filesystem.md
fs = S3FileSystem(key="YOUR_ACCESS_KEY", secret="YOUR_SECRET_KEY")

# Or with a named profile.
fs = S3FileSystem(profile="YOUR_PROFILE")

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (factual claims, caller compatibility, AWS operational effects, documentation)

Base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 9cc6d013516cb94011ac159d99eddee851f8ea24. Result: CLEAN.

Claims checked:

  • "s3fs names the argument profile": s3fs S3FileSystem keeps unknown constructor kwargs in self.kwargs and builds aiobotocore.session.AioSession(**self.kwargs) (fsspec/s3fs s3fs/core.py, set_session()), and its own docstring example is S3FileSystem(profile="<profile name>"). boto3 Session names it profile_name (Connection._SESSION_PASSING_ARGS, connection.py:85-92). Holds.
  • "AioS3FileSystem and fsspec.filesystem("s3", profile=...) pick up the profile": checked with temporary profile files; both sign with OTHERKEY.
  • "profile_name wins when both are given": covered by the parametrized test case.
  • "An unknown profile raises ProfileNotFound at construction, as profile_name= did": checked for profile=; both go through the same Session(profile_name=...) call.
  • Docstring "used as profile_name when profile_name is not given": an explicit profile_name=None counts as given and keeps the default chain; acceptable and matches the wording.

Existing callers: users who passed profile= and relied on it being ignored now sign with that profile (or get ProfileNotFound). This is the bug fix itself; listed in the PR body as a behavior change for the release notes. profile_name= callers and connection= callers are unchanged.

AWS operator: no new requests. Credential resolution for the profile (including assume-role or SSO profiles) is the same as for profile_name=; botocore defers role assumption until the first request.

Documentation: the new example sits with the other s3fs-compatible constructor arguments in docs/filesystem.md; docs/usage.md:819 covers connection-level profile_name and stays correct. No other docs list the s3fs arguments.

Evidence: offline test locally (5 passed; the profile case fails with the source change reverted). Live AWS tests not run locally; CI runs once Ready.

monkeypatch.setenv("AWS_SHARED_CREDENTIALS_FILE", str(credentials))

fs = S3FileSystem(region_name="us-east-1", skip_instance_cache=True, **kwargs)
assert fs._client._request_signer._credentials.access_key == expected

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed reviewer result)

Reviewer: OpenAI Codex CLI 0.160.0, model gpt-6-astra (different model from the Claude author), codex exec -s read-only --ephemeral, session 01a100c5-77f5-7ca2-9921-624835c35050.
Base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 9cc6d013516cb94011ac159d99eddee851f8ea24; reviewed from a detached snapshot of the head with the literal diff, without the PR number, description, commit message or self-review records. Snapshot and PR worktree verified unchanged afterwards.
Static review only; the reviewer did not run tests.

Result: CLEAN.

Reviewer output:

Covered:

  • Profile alias normalization and explicit profile_name precedence.
  • fsspec storage-option forwarding, instance caching, connection-based callers, and AioS3FileSystem.
  • Regression assertions, temporary profile files, AWS environment cleanup, and CI/session setup.
  • Docstring and documentation accuracy.

CLEAN — No actionable defects found. Distinct credential keys make the test detect an ignored alias; environment cleanup and skip_instance_cache=True prevent ordinary CI credentials or cached instances from masking it.

Static source review only. No edits, builds, tests, or GitHub access.

@laughingman7743
laughingman7743 marked this pull request as ready for review October 3, 2026 08:01
@laughingman7743
laughingman7743 merged commit 65a305b into master Oct 3, 2026
9 checks passed
@laughingman7743
laughingman7743 deleted the fix/983-s3fs-profile-arg branch October 3, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3FileSystem ignores the s3fs profile argument

1 participant