[aw471]

Summary

Add exact UTF-8 integer keys and known-length metadata for native CUDA preprocessing.

Description of the change

There is no fixed string-length limit. Arrow ingestion records the maximum byte length; raw/cuDF storage remains unknown. Copies, views, concatenation, compiler reconstruction and new pickles preserve that distinction. Unknown bounds retain the existing consumer fallback instead of treating the entire buffer as one string.

This PR adds shared infrastructure for #1275 and #1272; it does not add a standalone sorting or lookup dispatch.

Validation

Downstream GPU measurements on one RTX PRO 6000. Warm timings include packing, exclude compilation startup and input construction. Torch peaks are per-call; cuDF/RMM peaks are shown separately and are not added to Torch peaks.

Consumer	Main eager, ms	Native eager, ms	Native compiled, ms	Torch peak MiB: main → native eager → compiled
Sort, 35 strings × 128 bytes	2.169	2.169 (same cuDF path)	0.133	0.063 → 0.063 → 0.046
Lookup, 35/6/4 categories × 128 bytes	6.157	1.348	0.138	0.032 → 0.146 → 0.062

Sort has no separate native eager dispatch; its eager path remains cuDF. For lookup, original cuDF separately used 0.026 MiB RMM and native used zero. These are downstream examples, not a standalone speedup claim for the key representation or a universal memory claim. Exact outputs, zero/unknown bounds, copying, concatenation, compiler reconstruction and old/new pickle compatibility passed; the combined stack passed 75 focused CUDA tests.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[codecov]

Codecov Report

❌ Patch coverage is 95.38462% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.09%. Comparing base (6379704) to head (e053cc3).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
sdm/tensor/var_len.py	91.42%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1274      +/-   ##
==========================================
+ Coverage   83.00%   83.09%   +0.08%     
==========================================
  Files         176      177       +1     
  Lines       11394    11447      +53     
==========================================
+ Hits         9458     9512      +54     
+ Misses       1936     1935       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.