Skip to content

Reject negative ignore_rows and ignore_columns in CSVDocumentCleaner - #13088

Open
Premvkmishra wants to merge 1 commit into
deepset-ai:mainfrom
Premvkmishra:fix-csv-cleaner-negative-ignore-counts
Open

Premvkmishra wants to merge 1 commit into
deepset-ai:mainfrom
Premvkmishra:fix-csv-cleaner-negative-ignore-counts

Conversation

@Premvkmishra

Copy link
Copy Markdown

Related Issues

Proposed Changes:

In CSVDocumentCleaner.__init__, negative values for ignore_rows or ignore_columns were stored unvalidated. In _clean_df, pandas iloc[ignore_rows:, ignore_columns:] treated negative integers as offsets from the end of the DataFrame, leading to silent data loss without restoring ignored sections.

  • Added validation in CSVDocumentCleaner.__init__ to raise a ValueError if ignore_rows < 0 or ignore_columns < 0, matching the style of CSVDocumentSplitter and DocumentSplitter.
  • Updated :raises ValueError: entry in the component __init__ docstring and updated parameter descriptions in docs-website/docs/pipeline-components/preprocessors/csvdocumentcleaner.mdx.
  • Created a reno release note with fixes and upgrade sections.

How did you test it?

  • Added unit tests (test_negative_ignore_rows_raises, test_negative_ignore_columns_raises, and test_negative_ignore_values_raise_from_dict) in test/components/preprocessors/test_csv_document_cleaner.py.
  • Ran unit tests via hatch run test:unit test/components/preprocessors/test_csv_document_cleaner.py (24 passed).
  • Executed a mutation check to confirm the new tests fail when validation is omitted and pass when restored.
  • Verified hatch run test:types (mypy) and hatch run fmt (ruff).

Notes for the reviewer

  • Validation is performed strictly in __init__ before attribute assignment.
  • Zero (0) and positive values behave unchanged.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

@Premvkmishra
Premvkmishra requested a review from a team as a code owner October 2, 2026 14:36
@Premvkmishra
Premvkmishra requested review from julian-risch and removed request for a team October 2, 2026 14:36
@vercel

vercel Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

@Premvkmishra is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Oct 2, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CSVDocumentCleaner silently discards data for negative ignore counts

2 participants