Repository navigation
Preprocess USFM files before conversion to handle unsupported markers - #147
Open
KallynGowdy wants to merge 1 commit into
Open
KallynGowdy wants to merge 1 commit into
KallynGowdy wants to merge 1 commit into
Conversation
BibleMultiConverter 0.0.9 doesn't know \ref, so it kept both sides of
the pipe ("Psalm 91:11–12|PSA 91:11-12") and added a space where the
marker was ("( 1 Enoch 1:9|1:9)"). convertUsfmToUsx3() now copies the
USFM files to a temporary directory and replaces \ref TEXT|TARGET\ref*
(and \+ref) with TEXT before running the converter, so every fetcher
that converts USFM to USX3 gets the fix.
The BSB marker file now records the preprocessing version so existing
BSB downloads are converted again.
Fixes #144
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019LsURyVgHxWYTXpbFoYSPV
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds preprocessing of USFM files before conversion to USX3 format to handle USFM markers that BibleMultiConverter doesn't understand, specifically the
\refmarker used for cross-references.Key Changes
Added USFM preprocessing functions:
preprocessUsfm(): Removes\refmarkers while preserving their display text, handling both simple and nested formspreprocessUsfmDirectory(): Applies preprocessing to all USFM files in a directory while copying other files unchangedUSFM_PREPROCESSING_VERSION: Constant to track preprocessing version for cache invalidationUpdated conversion workflow:
convertUsfmToUsx3()now preprocesses USFM files in a temporary directory before passing them to BibleMultiConverterUpdated cache tracking:
fetchBsbTranslation()now includes preprocessing version in the source marker to invalidate cached conversions when preprocessing logic changesAdded comprehensive test coverage:
conversion.spec.tswith 8 test cases covering various\refmarker formats and edge casesImplementation Details
fs/promisesfor async file operations andos.tmpdir()for temporary directory creationUSFM_REF_REGEXmatches both simple (\ref TEXT|TARGET\ref*) and nested (\+ref ...\+ref*) reference markers.usfmor.sfmextensions