Skip to content

Preprocess USFM files before conversion to handle unsupported markers - #147

Open
KallynGowdy wants to merge 1 commit into
mainfrom
claude/kind-cray-zuhz5x
Open

KallynGowdy wants to merge 1 commit into
mainfrom
claude/kind-cray-zuhz5x

Conversation

@KallynGowdy

Copy link
Copy Markdown
Collaborator

Summary

This PR adds preprocessing of USFM files before conversion to USX3 format to handle USFM markers that BibleMultiConverter doesn't understand, specifically the \ref marker used for cross-references.

Key Changes

  • Added USFM preprocessing functions:

    • preprocessUsfm(): Removes \ref markers while preserving their display text, handling both simple and nested forms
    • preprocessUsfmDirectory(): Applies preprocessing to all USFM files in a directory while copying other files unchanged
    • USFM_PREPROCESSING_VERSION: Constant to track preprocessing version for cache invalidation
  • Updated conversion workflow:

    • convertUsfmToUsx3() now preprocesses USFM files in a temporary directory before passing them to BibleMultiConverter
    • Temporary directory is cleaned up in a finally block to ensure proper resource cleanup
  • Updated cache tracking:

    • fetchBsbTranslation() now includes preprocessing version in the source marker to invalidate cached conversions when preprocessing logic changes
  • Added comprehensive test coverage:

    • conversion.spec.ts with 8 test cases covering various \ref marker formats and edge cases
    • Tests verify preprocessing handles pipes, nested markers, attribute targets, and multiple references

Implementation Details

  • Uses Node.js fs/promises for async file operations and os.tmpdir() for temporary directory creation
  • Regex pattern USFM_REF_REGEX matches both simple (\ref TEXT|TARGET\ref*) and nested (\+ref ...\+ref*) reference markers
  • Only processes files with .usfm or .sfm extensions
  • Returns count of changed files for logging purposes
  • Preserves non-USFM files unchanged during preprocessing

BibleMultiConverter 0.0.9 doesn't know \ref, so it kept both sides of
the pipe ("Psalm 91:11–12|PSA 91:11-12") and added a space where the
marker was ("( 1 Enoch 1:9|1:9)"). convertUsfmToUsx3() now copies the
USFM files to a temporary directory and replaces \ref TEXT|TARGET\ref*
(and \+ref) with TEXT before running the converter, so every fetcher
that converts USFM to USX3 gets the fix.

The BSB marker file now records the preprocessing version so existing
BSB downloads are converted again.

Fixes #144

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019LsURyVgHxWYTXpbFoYSPV

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants