Research: is bats-assert/bats-support adoption worth it (possibly as a submodule, like bats-core)?

[jnasbyupgrade]

Context

While adding stub-based test coverage for check-stale-expected (pgxntool #61 / pgxntool-test #31), we researched whether an existing BATS-ecosystem assertion library should replace or supplement test/lib/assertions.bash (our homegrown assert_success/assert_contains/etc.). This issue captures that research for future reference. It intentionally does not recommend a decision — there's more investigation needed first (see Open Questions below).

What bats-assert/bats-support actually provide

• bats-support: shared infrastructure other helper libraries build on — a fail function for standardized error reporting, output formatting (two-column key/value, multi-line with indentation/line counts), and caller-context checks (batslib_is_caller, useful for restricting a helper to setup/teardown-only use).
• bats-assert (depends on bats-support): assert/refute (expression truth), assert_equal/assert_not_equal (expected-vs-actual diff display), assert_success/assert_failure (optional expected-status arg; auto-shows $status/$output on failure), assert_output/refute_output (literal/partial/regex matching, multi-line-aware), assert_line/refute_line (by index or search), assert_regex/refute_regex, assert_stderr/assert_stderr_line (needs run --separate-stderr).

Comparison to our homegrown test/lib/assertions.bash

Rough feature parity for the common cases: assert_success/assert_failure/assert_failure_with_status already show $status/$output on failure, similar to bats-assert's equivalents.

Concrete gap found and already fixed (in the PR that prompted this issue): assert_contains/assert_not_contains previously produced no failure message at all — just echo "$haystack" | grep -qF "$needle" returning bare exit status, so a failure showed BATS's generic "command failed" trace with no indication of what was actually being compared. bats-assert's equivalent (assert_output --partial) always shows expected-vs-actual. This is exactly the kind of gap that using (or at least modeling closely after) a community-maintained, more battle-tested library might have avoided from the start. We patched this specific gap directly in assertions.bash rather than adopting bats-assert wholesale, since a full migration is a large, mechanical, high-risk rewrite across ~19 test files and hundreds of call sites with no other functional bug being fixed.

The submodule angle (not yet explored)

test/bats (bats-core itself) is already included as a git submodule (see .gitmodules), not vendored/copied source. This is a materially different adoption path than copy-pasting bats-assert/bats-support's source into the repo or install-scripting them in CI:

• A submodule tracks upstream directly and can be version-pinned/updated via normal git submodule workflows, same as bats-core already is.
• It avoids "vendoring drift" (copied source silently diverging from upstream, no clear update path).
• It's a much lower-friction change to structure than migrating actual test code, since it doesn't touch any existing .bats file.

This distinction (submodule vs. vendor/copy) wasn't factored into the "don't migrate" conclusion above — that conclusion was about the cost of rewriting test files, not about whether adding the library itself is cheap or expensive. Those are separable questions.

Open questions for further research

1. Loading mechanics: how would a submodule-based bats-assert/bats-support actually get loaded into test files without colliding with the existing homegrown assertions.bash (function name collisions? load order? does helpers.bash need to change how/what it loads)?
2. Partial/gradual adoption: is it workable for new test files to use bats-assert while existing files keep the homegrown helpers, or does that create confusing inconsistency (two assertion styles to know, two sets of failure-message formats in the same suite)? Is there a sensible migration path if a full switch is ever desired, or would partial adoption become permanent?
3. Maintenance implications: version-pinning strategy for a second external submodule (does it track bats-core's release cadence, or independently?), and what breaks if bats-assert/bats-support's own API changes between versions.
4. Scope: would adoption make sense only for new test files (like check-stale-expected-script.bats), or is there a case for touching existing files too, and if so how much of that risk is worth taking for the diagnostic-quality improvement alone (versus just continuing to patch homegrown gaps as they're found, as we did for assert_contains in this PR)?

References

• bats-core/bats-assert
• bats-core/bats-support
• test/lib/assertions.bash (homegrown assertions this would potentially supplement/replace)
• .gitmodules (existing bats-core submodule precedent)