This cumulative release preserves the complete v3.2 runtime and adds an optional Parquet/Arrow scan layer. The scan pushes column projection into the Parquet reader, streams bounded RecordBatch windows, supports direct zero-copy borrowing of non-null Float64 arrays, and materializes only the requested columns when QuantForge-owned buffers are required.
Enable the feature with --features parquet-arrow (Arrow/Parquet Rust crates 60.x). Example:
cargo test --release --features parquet-arrow
cargo run --release --features parquet-arrow --example parquet_projected_scan -- data.parquet close,RSI_14 timestampmmap alone is not a substitute for Parquet decoding: compressed Parquet pages must be decoded before their values can be used as numeric arrays. Use Arrow array views when possible and call materialize_f64 only when the selected numeric type or null policy requires owned contiguous f64 values.
The reader is behind an optional feature; the default QuantForge build does not gain Arrow/Parquet compile dependencies. See docs/V3_3_PARQUET_ARROW.md.
QuantForge v1.9 is an incremental, cumulative release built on v1.8. It adds first-class generic numeric columns and caller-owned direct-output batch execution, while retaining the existing compiler, SIMD, JIT, cache, backtest, planner and benchmark stack.
QuantForge is an experimental high-performance numerical execution engine built around columnar data, typed SSA-like IR, global CSE, register-aware planning, SIMD execution and parallel chunking.
This cumulative release preserves the full v1.5 compiler, register planner, SIMD paths, batch executor, tests, benchmarks and documentation, then adds a native-kernel runtime and optional executable-code experiment.
When enabled and the CPU advertises FMA, a single-use pattern such as:
(A * B) + C
can be selected as:
Fma(A, B, C)
Likewise:
(A * B) - C
becomes an FMS-style instruction candidate.
FMA is explicitly policy-controlled:
Off: never request FMA.Auto: use FMA only when the detected CPU supports it.On: request FMA, but still fall back if unsupported.
The scalar reference semantics remain authoritative. v1.5 exposes the specialized plan but deliberately does not replace the verified SIMD executor with generated machine code yet.
The long-term hot path is:
Expr → optimized DAG → typed SSA → instruction selection → kernel → SIMD → multicore
The native layer is the bridge between the optimizer and future general machine-code generation. It provides CPU-aware signatures, specialized-plan caching, stable dispatch and an isolated optional JIT experiment.
Run:
cargo test --release
cargo bench --bench v1_5_kernelThis environment does not provide rustc/cargo, so no compilation or benchmark numbers are claimed here.
v1.6 is intentionally cumulative: it preserves all v1.5 modules, tests, benches and documentation and adds a native-kernel capability layer on top.
New components:
src/native.rs: native kernel signatures, CPU capability detection, plan cache and stable AOT dispatch surface.src/native_jit.rs: optionaljitfeature with a real x86-64 executable-code experiment for an affine f64 micro-kernel.tests/v1_6.rs: cache/capability regression tests.benches/v1_6_native.rs: specialization/cache and baseline benchmark harness.
Build the cumulative default project:
cargo test --release
cargo bench --bench v1_6_nativeThe optional JIT experiment is enabled with:
cargo test --release --features jitThe JIT is intentionally narrow in v1.6. It does not claim to compile arbitrary QuantForge expressions to machine code yet. The general expression engine continues to use the verified scalar/SIMD paths. This separation lets the project benchmark native code generation without weakening numerical correctness.
The v1.6 archive is built directly from the v1.5 tree. No v1.5 source, test, benchmark or documentation file was removed. The release adds src/native.rs, optional src/native_jit.rs, tests/v1_6.rs and benches/v1_6_native.rs.
Optional --features jit adds an x86-64 JIT kernel path with XMM register allocation, constant hoisting, explicit divide-by-zero handling, FMA lowering for safe single-use patterns, executable kernel caching, and fallback to the verified SIMD/scalar path. See docs/V1_7_NATIVE_BACKEND.md.
v1.8 adds an optional AVX2 native backend (--features simd-jit). It lowers a
numeric SSA-like program to executable machine code that processes four f64
lanes per iteration, with scalar-tail fallback. The backend performs register
reuse, invariant constant hoisting and optional single-use mul + add → FMA
lowering. Unsupported semantics fall back instead of being silently changed.
Benchmark:
cargo bench --release --features simd-jit --bench v1_8_simd_jit
cargo test --release --features simd-jitThis cumulative release adds generic ColumnId inputs and direct-output batch execution so large numeric pipelines can avoid intermediate result copies. The optimized path keeps the scalar reference implementation available for equivalence testing and falls back from native JIT when its fixed-input ABI cannot safely represent dynamic columns.
Benchmark: cargo bench --bench v1_9_direct_output
Multi-output native AVX2 fusion, direct stores, generic ColumnId input resolution, and scalar-tail verification.
QuantForge v2.1 adds an optional x86-64 AVX2 multi-output JIT backend. It generates one executable kernel for multiple f64 roots, shares SSA subexpressions, and stores each root directly into caller-owned output buffers. Unsupported operations fall back to the verified v2.0 execution paths.
QuantForge now calibrates the host at runtime and uses arithmetic intensity, measured streaming bandwidth, logical CPU count and ISA detection to select execution granularity. This is a planning signal, not a benchmark guarantee.
The runtime can calibrate streaming memory bandwidth once per process and build a workload-aware dispatch decision from rows, input/output columns and estimated arithmetic intensity. This is a planning layer; real performance still requires benchmark validation on the target CPU.
The v2.5 layer profiles the actual compiled batch, measures the actual native fused kernel, and applies the winning threads/chunk/block plan rather than tuning a synthetic proxy. See docs/V2_5_REAL_KERNEL_AUTOTUNING.md.
QuantForge can profile a real hot-path closure with best-effort hardware counters and wall-clock timing. Linux uses perf_event_open for cycles/instructions/cache-misses; Windows exposes thread-cycle telemetry; restricted systems fall back cleanly to wall-clock timing.
Metrics include IPC and cache-miss rate when the underlying counters are available.
let report = executor.profile_batch_once(&expressions, &data)?;
println!("elapsed={:?} IPC={:?} cache-miss-rate={:?}", report.elapsed, report.ipc, report.cache_miss_rate);Counter availability is platform/permission dependent; QuantForge always retains the wall-clock path as a portable baseline.
The runtime now models logical CPUs, physical cores, sockets, Linux NUMA nodes and
shared cache groups where available. Parallel telemetry can pin workers to a
NUMA-balanced CPU map using profile_parallel_chunks_affine(), while reporting
whether the OS accepted each affinity request.
v3.0 adds aligned buffers, reusable scratch arenas/pools, zero-copy streaming windows, optional file-backed mmap, parallel first-touch and optional Linux huge-page hints. These are memory primitives and planning hooks; actual speedups must be measured on the target workload and CPU. See docs/V3_0_MEMORY_ARCHITECTURE.md.
Adds NUMA-aware worker placement, best-effort CPU affinity, node-partitioned scratch pools, and affinity-driven first-touch. See docs/V3_1_NUMA_RUNTIME.md.
v3.2 adds MemoryLayoutFingerprint and ArtifactStore. The persistent cache stores
portable optimization metadata and measured tuning plans, not executable machine
code. Artifact files are atomically written and checksum validated. Native kernels
must still be regenerated and validated for the current process/ABI/ISA.
Benchmark:
cargo bench --release --bench v3_2_artifactsTests:
cargo test --release --test v3_2ProjectedBatch::execute_binary_f64_into consumes existing non-null Float64 Arrow buffers by borrowed slices and writes results directly into a caller-owned output slice. It supports Add, Subtract, Multiply, Divide, Min and Max. Large batches use Rayon chunks; small batches use a scalar loop that the release compiler may auto-vectorize. This is direct-buffer execution, not a claim that Parquet compressed bytes are zero-copy or that explicit SIMD has been proven.
The direct path rejects null-containing or non-Float64 columns rather than silently materializing/casting them. Use the explicit materialization APIs for other types/null policies.
The optional parquet-arrow feature provides projected, streaming Parquet scans. QuantForge requests only named columns, receives bounded Arrow RecordBatch windows, allows a no-copy borrowed view for non-null Float64 arrays, and materializes values to Table/DataBlock only when required by the selected backend. This does not claim zero-copy from compressed Parquet page bytes: page decoding is still required. See docs/V3_3_PARQUET_ARROW.md and docs/V3_3_RELEASE_NOTES.md.