Skip to content

fix(client): apply mask to dataset and score payloads #1905

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
asaraog wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fix(client): apply mask to dataset and score payloads #1905

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f332185
test: mask applies to dataset and score payloads
asaraog Sep 26, 2026
7a9e93a
fix(client): apply mask to dataset and score payloads
asaraog Sep 26, 2026
041ce37
test: enable tracing so create_score reaches the score queue
asaraog Sep 26, 2026
aae4ea4
fix(client): serialize non-string masked score comments
asaraog Sep 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 22 additions & 4 deletions langfuse/_client/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"""

import asyncio
import json
import logging
import os
import re
Expand Down Expand Up @@ -1949,6 +1950,20 @@ def create_score(
environment: Optional[str] = None,
) -> None: ...

def _apply_mask(self, data: Any) -> Any:
"""Apply the configured mask to data sent outside a span, matching span masking."""
if data is None or not self._mask:
return data
try:
return self._mask(data=data)
except Exception as e:
langfuse_logger.error(
"Masking error: Custom mask function threw exception when processing "
"data. Using fallback masking. Error: %s",
e,
)
return "<fully masked due to failed mask function>"

def create_score(
self,
*,
Expand Down Expand Up @@ -2018,6 +2033,9 @@ def create_score(
return

score_id = score_id or self._create_observation_id()
comment = self._apply_mask(comment)
if comment is not None and not isinstance(comment, str):
comment = json.dumps(comment)

try:
new_body = ScoreBody(
Expand Down Expand Up @@ -3550,7 +3568,7 @@ def create_dataset(
result = self.api.datasets.create(
name=name,
description=description,
metadata=metadata,
metadata=self._apply_mask(metadata),
input_schema=input_schema,
expected_output_schema=expected_output_schema,
)
Expand Down Expand Up @@ -3649,9 +3667,9 @@ def create_dataset_item(

result = self.api.dataset_items.create(
dataset_name=dataset_name,
input=input,
expected_output=expected_output,
metadata=metadata,
input=self._apply_mask(input),
expected_output=self._apply_mask(expected_output),
Comment on lines +3670 to +3671

@greptile-apps greptile-apps Bot Sep 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Masking can invalidate dataset items
If a dataset requires an object-shaped input or expected output, a mask that replaces the value with a string sends an invalid item to the API. The fallback used when masking fails is also a string. An item that met the dataset schema before masking is then rejected.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: langfuse/_client/client.py
Line: 3666-3667

Comment:
**Masking can invalidate dataset items**
If a dataset requires an object-shaped input or expected output, a mask that replaces the value with a string sends an invalid item to the API. The fallback used when masking fails is also a string. An item that met the dataset schema before masking is then rejected.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

@asaraog asaraog Sep 26, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same contract as span input/output masking; a mask that changes the shape affects spans the same way.

@greptile-apps greptile-apps Bot Sep 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That’s fair. The mask callback has the same intentionally unconstrained contract for span input/output, so changing the shape—and using the string fallback when the callback raises—is expected behavior here as well. A dataset schema can therefore reject masked data, but that is an existing consequence of the masking contract rather than a defect introduced by this change. I’m withdrawing this finding.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

metadata=self._apply_mask(metadata),
Comment on lines +3670 to +3672

@greptile-apps greptile-apps Bot Sep 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Masking removes uploaded media references
When a dataset item contains LangfuseMedia, this method uploads the media before applying the mask. If the mask replaces the field, the created item no longer contains the media reference, leaving the uploaded attachment unlinked to the item's payload.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: langfuse/_client/client.py
Line: 3666-3668

Comment:
**Masking removes uploaded media references**
When a dataset item contains `LangfuseMedia`, this method uploads the media before applying the mask. If the mask replaces the field, the created item no longer contains the media reference, leaving the uploaded attachment unlinked to the item's payload.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

@asaraog asaraog Sep 26, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same order as spans: _process_media_and_apply_mask processes media first, then masks (span.py L552).

@greptile-apps greptile-apps Bot Sep 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right that create_dataset_item follows the established span behavior: media is processed first and the resulting payload is then passed through the mask. This ordering is consistent with the existing masking contract, so I'll withdraw the media-linking concern.

source_trace_id=source_trace_id,
source_observation_id=source_observation_id,
status=status,
Expand Down
53 changes: 53 additions & 0 deletions tests/unit/test_mask_api_payloads.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
from unittest.mock import Mock

from langfuse import Langfuse


def _client():
client = Langfuse(
public_key="pk",
secret_key="sk",
host="https://mock-host.com",
tracing_enabled=False,
mask=lambda data: "masked",
)
client.api = Mock()
client._resources.add_score_task = Mock()
return client


def test_create_dataset_item_masks_payload():
client = _client()
client.create_dataset_item(
dataset_name="ds", input="secret", expected_output="secret", metadata="secret"
)
kwargs = client.api.dataset_items.create.call_args.kwargs
assert (kwargs["input"], kwargs["expected_output"], kwargs["metadata"]) == (
"masked",
"masked",
"masked",
)


def test_create_dataset_masks_metadata():
client = _client()
client.create_dataset(name="ds", metadata="secret")
assert client.api.datasets.create.call_args.kwargs["metadata"] == "masked"


def test_create_score_masks_comment():
client = _client()
client._tracing_enabled = True # create_score is a no-op otherwise
client.create_score(name="s", value=1, trace_id="t" * 32, comment="secret")
assert (
client._resources.add_score_task.call_args.args[0]["body"].comment == "masked"
)


def test_create_score_serializes_non_string_masked_comment():
client = _client()
client._tracing_enabled = True
client._mask = lambda data: {"redacted": True}
client.create_score(name="s", value=1, trace_id="t" * 32, comment="secret")
body = client._resources.add_score_task.call_args.args[0]["body"]
assert body.comment == '{"redacted": true}'