Feature/mdf data sources

[jonathanb-db]

Summary

Adds 3 DataSources for MF4 binary file loading, required data readers as well as a quickstart guide ("QUICKSTART.md" in the data_sources subdirectory). This is the first of many planned data sources to be added to impulse. The introduced readers are optimized to work with spark and use a stripe-memory reading approach to optimize IO-calls and general throughput compared to other solutions available. Please review the "README.md" in the code directory for details.

Currently this feature is experimental, since some parts of the mf4 standard are not yet implemented, missing functionality shall be added in the future once required. For a full list of current limitations review "KNOWN_LIMITATIONS.md" in the sub directory.

A Databricks Industry Solution that uses these data sources for a end-to-end ingestion pipeline is coming soon.

Changes

• Added 3 Data sources to load mf4 files into a spark data frame
• Added required readers to interpret mf4

Test Plan

• Unit tests added/updated
• Manual testing completed
• Documentation updated (if applicable)

Checklist

• Code follows project style guidelines
• Self-review completed
• No new linter warnings introduced

[tombonfert]

The databricks pypi-proxy is not reachable from the GH runner and this might be the reason why the lint check fails. Please check the uv.lock file of Impulse in the main branch: https://github.com/databrickslabs/impulse/blob/main/uv.lock

[jonathanb-db]

Thanks @tombonfert , just pushed with correct pypi

[codecov]

Codecov Report

❌ Patch coverage is 88.99225% with 213 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.76%. Comparing base (8457893) to head (5577a5f).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/impulse_data_sources/mdf/converter.py	20.21%	146 Missing ⚠️
src/impulse_data_sources/mdf/mdf4_reader.py	95.25%	9 Missing and 8 partials ⚠️
src/impulse_data_sources/mdf/mdf_blocks.py	94.23%	6 Missing and 11 partials ⚠️
src/impulse_data_sources/mdf/mdf_decode.py	95.63%	4 Missing and 9 partials ⚠️
src/impulse_data_sources/mdf/datasources.py	95.85%	7 Missing and 1 partial ⚠️
src/impulse_data_sources/mdf/arrow_emit.py	98.37%	3 Missing and 4 partials ⚠️
src/impulse_data_sources/mdf/bin_packer.py	97.05%	2 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #69      +/-   ##
==========================================
+ Coverage   88.67%   88.76%   +0.08%     
==========================================
  Files          60       69       +9     
  Lines        5008     6943    +1935     
  Branches      596      903     +307     
==========================================
+ Hits         4441     6163    +1722     
- Misses        461      638     +177     
- Partials      106      142      +36     

Flag	Coverage Δ
data_sources	88.99% <88.99%> (?)
query_engine	84.57% <ø> (ø)
reporting	94.19% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/impulse_data_sources/mdf/schemas.py	100.00% <100.00%> (ø)
src/impulse_data_sources/mdf/udf_helpers.py	100.00% <100.00%> (ø)
src/impulse_data_sources/mdf/bin_packer.py	97.05% <97.05%> (ø)
src/impulse_data_sources/mdf/arrow_emit.py	98.37% <98.37%> (ø)
src/impulse_data_sources/mdf/datasources.py	95.85% <95.85%> (ø)
src/impulse_data_sources/mdf/mdf_decode.py	95.63% <95.63%> (ø)
src/impulse_data_sources/mdf/mdf4_reader.py	95.25% <95.25%> (ø)
src/impulse_data_sources/mdf/mdf_blocks.py	94.23% <94.23%> (ø)
src/impulse_data_sources/mdf/converter.py	20.21% <20.21%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tombonfert]

please remove this file.

[tombonfert]

can we integrate this part into the existing docs of Impulse?

[tombonfert]

please integrate into the existing docs of Impulse

[tombonfert]

please create a new section in the existing docs of Impulse under https://databrickslabs.github.io/impulse/docs/references/data_sources ?