[AreWeDreaming]

Read imas_composer (DIII-D) data from Julia: awkward reader, PythonCall extension, env recipe

First step toward filling a dd in memory from DIII-D data through imas_composer (toksearch backend), with Python called in-process via PythonCall. This PR adds the data handover and a working environment. The dd loader itself comes next.

Changes

• awkward_from_buffers(form, len, buffers) (src/io.jl): pure Julia reader for awkward's buffer format. It returns nested vectors of zero-copy views into the flat buffers.
  • Supports NumpyArray, ListOffsetArray (incl. strings), RegularArray and ByteMaskedArray; masked floats become NaN.
  • Raises on anything else.
  • Not exported.
• ext/IMASddPythonCallExt.jl: adds awkward_from_buffers(::Py) for the dicts returned by imas_composer.julia.julia_value (GA-FDP/imas_composer ssd/julia_interface). PythonCall is a weakdep, so nothing changes for packages that don't load it.
• test/runtests_awkward.jl: hand-written buffers covering every node class, nesting up to RegularArray > ListOffsetArray > ListOffsetArray, strings, masks (valid_when true/false) and the error paths. No Python needed.
• deploy/container/: recipe for a dev env (conda env + Julia project + smoke test). Findings it encodes:
  • AWS C libraries: HDF5_jll loads aws_c_*_jll, which share the soname .so.1.0.0 with conda's copies (pulled in via pyspark → pyarrow). Newer conda versions break import awkward in-process. environment_julia_pins.yaml caps them at the JLL versions; the only cost is pyarrow 25 → 24.
  • TLS: Julia's libcurl has no default CA path, so mdsip-fdp's HTTPS tunnel fails certificate verification. The env sets FDP_TUNNEL_CAINFO.
  • Not pinned: curl and OpenSSL. Pinning them would downgrade mdsip-fdp and xrdcl-pelican-fdp, and the newer conda builds work against Julia's.
  • CondaPkg: the Null backend is set via [preferences.CondaPkg], so PythonCall uses the activated env.
• imas_composer submodule added.

Testing

• Pkg.test(): full suite passes, including 14 new awkward_from_buffers tests.
• deploy/container/smoke_test.jl in the pinned env, with no LD_PRELOAD, shot 205055:
  • all imports OK, and the fdp tunnel gets through TLS;
  • all 84 awkward fields across 15 IDS rebuilt in Julia match ak.to_list.

Follow-ups

• dd loader in the extension: map flat imas_composer paths onto the dd struct arrays and convert float32 → Float64.
• Ray workers are untested in-process.
• Bump the imas_composer submodule once ssd/julia_interface is merged.

🤖 Generated with Claude Code

[AreWeDreaming]

@jmcclena Please try these instructions

[codecov]

Codecov Report

❌ Patch coverage is 86.20690% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.97%. Comparing base (3574519) to head (52c0292).

Files with missing lines	Patch %	Lines
ext/IMASddPythonCallExt.jl	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #110      +/-   ##
==========================================
+ Coverage   43.93%   43.97%   +0.03%     
==========================================
  Files          16       17       +1     
  Lines       31551    31580      +29     
==========================================
+ Hits        13862    13887      +25     
- Misses      17689    17693       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.