Skip to content

MANIFEST_SPEC says a row filter takes exactly one operator, but the loader accepts several and keeps NaN rows for the negated ones #999

Closed

Description

@yakew7

Where: faircode/manifest.py RowFilter (__post_init__ and apply()), faircode/MANIFEST_SPEC.md (section row_filters), tests/test_manifest.py.

Current gap: the spec says every filter needs "exactly one of the operators below", but __post_init__ only rejects a filter with none, and apply() ANDs whatever is set, so isin plus equals in one block silently narrows twice. The spec also does not say what happens to a missing value: not_isin and not_equals keep NaN rows (NaN != "A" is true), while isin and equals drop them, which #977's tests now pin down but the document never states.

Repro:

>>> import pandas as pd; from faircode.manifest import RowFilter
>>> df = pd.DataFrame({"race": ["A", "B", "C", None, "A"]})
>>> RowFilter(column="race", not_isin=["C"], equals="A").apply(df).index.tolist()   # accepted, two operators
[0, 4]
>>> RowFilter(column="race", not_equals="A").apply(df).index.tolist()               # the NaN row (3) is kept
[1, 2, 3]

Suggested approach: decide between "reject more than one operator" and "document AND semantics", state the NaN rule for each operator in MANIFEST_SPEC, and add a validation test for whichever is chosen.

Activity

  1. added
    bugSomething isn't working
    documentationImprovements or additions to documentation
    help wantedExtra attention is needed
    on Oct 15, 2026
  2. yakew7 commented on Oct 16, 2026

    @yakew7
    OwnerAuthor

    Fixed in 15a52c2 (on main). Full suite green (941 passed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdocumentationImprovements or additions to documentationhelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions