Use the s3fs profile argument as the boto3 profile name - #985
Conversation
S3FileSystem accepted s3fs's constructor arguments without a connection, but dropped s3fs's profile argument because boto3 names it profile_name, so requests were signed with the default credential chain. Map profile to profile_name when profile_name is not given. Closes #983 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| } | ||
| kwargs.update(creds) | ||
| client_kwargs.update(creds) | ||
| if profile := kwargs.pop("profile", None): |
There was a problem hiding this comment.
Self-review round one (implementation behavior, public contracts, simplicity, regression coverage)
Base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 9cc6d013516cb94011ac159d99eddee851f8ea24. Result: CLEAN.
Covered inventory: pyathena/filesystem/s3.py (_get_client_compatible_with_s3fs() and its callers), docs/filesystem.md, tests/pyathena/filesystem/test_s3.py.
- Behavior:
profileis popped from the localkwargscopy only.S3FileSystem.__init__()still passes the originalkwargstoAbstractFileSystem, soprofilestays instorage_optionsand the instance-cache token still separates filesystems by profile, as before.profile_namewins when both are given (setdefault). - Callers: the only caller is
S3FileSystem.__init__()without aconnection(s3.py:180).AioS3FileSystem.__init__()forwards**kwargsto the internalS3FileSystem(s3_async.py:113-123), andfsspec.filesystem("s3", ...)resolves toS3FileSystemonce registered; both checked manually to sign with theotherprofile. With aconnection, all s3fs arguments are ignored, as documented in the constructor docstring; unchanged. - Failure path: an unknown profile now raises botocore
ProfileNotFoundat construction (checked), the same asprofile_name=.anon=Truewithprofilestill builds the session with the profile and signs nothing; s3fs also builds its session from the profile. - Test quality: the parametrized test builds clients from temporary config/credentials files with env credentials removed, and fails for
{"profile": "other"}with the source change reverted (signs withDEFAULTKEY). Limitation: it reads the resolved access key through botocore's private_request_signer._credentials; botocore has no public accessor on a client.
| fs = S3FileSystem(key="YOUR_ACCESS_KEY", secret="YOUR_SECRET_KEY") | ||
|
|
||
| # Or with a named profile. | ||
| fs = S3FileSystem(profile="YOUR_PROFILE") |
There was a problem hiding this comment.
Self-review round two (factual claims, caller compatibility, AWS operational effects, documentation)
Base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 9cc6d013516cb94011ac159d99eddee851f8ea24. Result: CLEAN.
Claims checked:
- "s3fs names the argument
profile": s3fsS3FileSystemkeeps unknown constructor kwargs inself.kwargsand buildsaiobotocore.session.AioSession(**self.kwargs)(fsspec/s3fss3fs/core.py,set_session()), and its own docstring example isS3FileSystem(profile="<profile name>"). boto3Sessionnames itprofile_name(Connection._SESSION_PASSING_ARGS, connection.py:85-92). Holds. - "
AioS3FileSystemandfsspec.filesystem("s3", profile=...)pick up the profile": checked with temporary profile files; both sign withOTHERKEY. - "
profile_namewins when both are given": covered by the parametrized test case. - "An unknown profile raises
ProfileNotFoundat construction, asprofile_name=did": checked forprofile=; both go through the sameSession(profile_name=...)call. - Docstring "used as
profile_namewhenprofile_nameis not given": an explicitprofile_name=Nonecounts as given and keeps the default chain; acceptable and matches the wording.
Existing callers: users who passed profile= and relied on it being ignored now sign with that profile (or get ProfileNotFound). This is the bug fix itself; listed in the PR body as a behavior change for the release notes. profile_name= callers and connection= callers are unchanged.
AWS operator: no new requests. Credential resolution for the profile (including assume-role or SSO profiles) is the same as for profile_name=; botocore defers role assumption until the first request.
Documentation: the new example sits with the other s3fs-compatible constructor arguments in docs/filesystem.md; docs/usage.md:819 covers connection-level profile_name and stays correct. No other docs list the s3fs arguments.
Evidence: offline test locally (5 passed; the profile case fails with the source change reverted). Live AWS tests not run locally; CI runs once Ready.
| monkeypatch.setenv("AWS_SHARED_CREDENTIALS_FILE", str(credentials)) | ||
|
|
||
| fs = S3FileSystem(region_name="us-east-1", skip_instance_cache=True, **kwargs) | ||
| assert fs._client._request_signer._credentials.access_key == expected |
There was a problem hiding this comment.
Independent review (relayed reviewer result)
Reviewer: OpenAI Codex CLI 0.160.0, model gpt-6-astra (different model from the Claude author), codex exec -s read-only --ephemeral, session 01a100c5-77f5-7ca2-9921-624835c35050.
Base aa0fc9146f4e683d6cdf96b894f963f9dd8f7abe, head 9cc6d013516cb94011ac159d99eddee851f8ea24; reviewed from a detached snapshot of the head with the literal diff, without the PR number, description, commit message or self-review records. Snapshot and PR worktree verified unchanged afterwards.
Static review only; the reviewer did not run tests.
Result: CLEAN.
Reviewer output:
Covered:
- Profile alias normalization and explicit
profile_nameprecedence.- fsspec storage-option forwarding, instance caching, connection-based callers, and
AioS3FileSystem.- Regression assertions, temporary profile files, AWS environment cleanup, and CI/session setup.
- Docstring and documentation accuracy.
CLEAN — No actionable defects found. Distinct credential keys make the test detect an ignored alias; environment cleanup and
skip_instance_cache=Trueprevent ordinary CI credentials or cached instances from masking it.Static source review only. No edits, builds, tests, or GitHub access.
WHAT
S3FileSystemcreated without aconnectionnow uses s3fs'sprofileargument as the boto3profile_name.When both are given,
profile_namewins.AioS3FileSystemandfsspec.filesystem("s3", profile=...)(with PyAthena registered fors3://) go through the same client construction, so they also pick up the profile.pyathena/filesystem/s3.py:_get_client_compatible_with_s3fs()mapsprofiletoprofile_name, and its docstring listsprofile.docs/filesystem.md: adds a named-profile example next to the other s3fs-compatible arguments.tests/pyathena/filesystem/test_s3.py:test_get_client_compatible_with_s3fs_profilebuilds clients from temporary config and credentials files and checks which profile's access key signs requests.Behavior change for users who passed
profile=and relied on it being ignored: requests are now signed with the named profile, and an unknown profile raises botocore'sProfileNotFoundat construction, asprofile_name=already did.WHY
Closes #983.
s3fs names the argument
profile, while boto3 names itprofile_name.Only the keys in
Connection._SESSION_PASSING_ARGSreach the boto3Session, soprofile=was silently dropped and the default credential chain was used.TEST
Tested commit: 9cc6d01
just formatandjust lint: passed.just docs lint: passed.uv run pytest --noconftest -q tests/pyathena/filesystem/test_s3.py -k test_get_client_compatible_with_s3fswith dummy Athena environment variables: 5 passed.With the source change reverted, the
{"profile": "other"}case fails (signs withDEFAULTKEY).AioS3FileSystem(profile="other")andfsspec.filesystem("s3", profile="other")sign with theotherprofile.🤖 Generated with Claude Code