Tensors in .NET

[ericstj]

Goal: Provide a type for use as both exchange and interop that represents multi-dimensional data if a single primitive Type. Implement arithmetic and linear algebra operations so that the type can serve as a sufficient basis for data preparation and as an input and output to neural networks.

• Explore memory layouts and how they play into interop
• Explore high-level Tensor design
  • Abstract? Type heirarchy? Construction patterns.
  • Dense vs sparse
  • Slices
  • Dimension order
  • Expected outcome: design document draft in dotnet/designs
• Explore Tensor arithmetic design
• Explore Tensor construction design
• Explore Tensor loading and saving
• TODO: Further break down Tensor implementation into smaller chunks
• Tensor API proposal
  • Expected outcome: API review
• Tensor core types implementation
  • Expected outcome: PR
• TODO: Further break down Tensor implementation into smaller chunks
• Performance tests
• Interop with OnnxRuntime
• Interop with TorchSharp
• Expose in ML.NET

[hez2010]

I hope we can take #89730 into consider together as it's critical for achieving high performance linear algebra operations on arbitrary ND tensors. Such feature has been a fundamental of libraries like the native implementation part of pytorch.

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Goal: Provide a type for use as both exchange and interop that represents multi-dimensional data if a single primitive Type. Implement arithmetic and linear algebra operations so that the type can serve as a sufficient basis for data preparation and as an input and output to neural networks.

• Explore memory layouts and how they play into interop
• Explore high-level Tensor design
  • Abstract? Type heirarchy? Construction patterns.
  • Dense vs sparse
  • Slices
  • Dimension order
  • Expected outcome: design document draft in dotnet/designs
• Explore Tensor arithmetic design
• Explore Tensor construction design
• Explore Tensor loading and saving
• TODO: Further break down Tensor implementation into smaller chunks
• Tensor API proposal
  • Expected outcome: API review
• Tensor core types implementation
  • Expected outcome: PR
• TODO: Further break down Tensor implementation into smaller chunks
• Performance tests
• Interop with OnnxRuntime
• Interop with TorchSharp
• Expose in ML.NET

Author:	ericstj
Assignees:	-
Labels:	Epic, area-Meta, untriaged, needs-area-label
Milestone:	-

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Goal: Provide a type for use as both exchange and interop that represents multi-dimensional data if a single primitive Type. Implement arithmetic and linear algebra operations so that the type can serve as a sufficient basis for data preparation and as an input and output to neural networks.

• Explore memory layouts and how they play into interop
• Explore high-level Tensor design
  • Abstract? Type heirarchy? Construction patterns.
  • Dense vs sparse
  • Slices
  • Dimension order
  • Expected outcome: design document draft in dotnet/designs
• Explore Tensor arithmetic design
• Explore Tensor construction design
• Explore Tensor loading and saving
• TODO: Further break down Tensor implementation into smaller chunks
• Tensor API proposal
  • Expected outcome: API review
• Tensor core types implementation
  • Expected outcome: PR
• TODO: Further break down Tensor implementation into smaller chunks
• Performance tests
• Interop with OnnxRuntime
• Interop with TorchSharp
• Expose in ML.NET

Author:	ericstj
Assignees:	-
Labels:	Epic, area-Meta, area-System.Numerics, untriaged, needs-area-label
Milestone:	-

Tagging subscribers to this area: @dotnet/area-system-numerics-tensors
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Goal: Provide a type for use as both exchange and interop that represents multi-dimensional data if a single primitive Type. Implement arithmetic and linear algebra operations so that the type can serve as a sufficient basis for data preparation and as an input and output to neural networks.

• Explore memory layouts and how they play into interop
• Explore high-level Tensor design
  • Abstract? Type heirarchy? Construction patterns.
  • Dense vs sparse
  • Slices
  • Dimension order
  • Expected outcome: design document draft in dotnet/designs
• Explore Tensor arithmetic design
• Explore Tensor construction design
• Explore Tensor loading and saving
• TODO: Further break down Tensor implementation into smaller chunks
• Tensor API proposal
  • Expected outcome: API review
• Tensor core types implementation
  • Expected outcome: PR
• TODO: Further break down Tensor implementation into smaller chunks
• Performance tests
• Interop with OnnxRuntime
• Interop with TorchSharp
• Expose in ML.NET

Author:	ericstj
Assignees:	-
Labels:	Epic, area-Meta, area-System.Numerics.Tensors, untriaged, needs-area-label
Milestone:	-

[Shuenhoy]

I found my original comments somehow away from my actual intentions. The key point is not ET, layz evaluation, or any specific techniques. It is the ability of eliminating intermediate arrays (known as deforestation, or fusion https://en.wikipedia.org/wiki/Deforestation_(computer_science)). Users can always use lowlevel primitives to manually archive this. But if the highlevel api can not handle this, its usages will be largely limited.

---

original comments

For the arithmetic and linear algebra operations, I suggest considering the "expression template"(ET) techniques commonly used in C++ world (eg. eigen3 and blaze), which is a kind of lazy evaluations that can reduce the allocation of temp object. This is helpful especially for a GC language.

For a quick impression, v1 + v2 in ET will produce a VecAdd<Vec, Vec> instead of a Vec. Similarly, m * (v1 + v2) will produce a MatMul<Mat, VecAdd<Vec, Vec>>. The computation is encoded in the type, and will get evaluated when assigned to a concrete Vec type.

Last month, as part of attemp at rebasing my research from C++ to .NET, I made a experiment with ET in .NET via generics: https://github.com/Shuenhoy/DotnetETExp . The (not sufficient but illustrative) results look promising and suggest ET may also be helpful in .NET world.

However, there are serveral barries that prevent me further investigating:

• Operator overloading does not support generic parameter (Generic operators support in C#. csharplang#813). The arithmetic in ET can happen between unlimited types, instead of finite types eg. Vector Matrix.
• I then turned to F#, which supports some kinds of generic operators. However, I still cannot have generic parameter on op_Implicit, which means I cannot have the computation evaluated at assignment, but have to manually evaluate it like let out = v1 + v2 |> eval
• Generic constraint is not part of overload signature (Proposal: allow generic constraints to be part of the signature for overloading resolution csharplang#2013). This preven definition of left multiple and right multiple in ET:

// ...
member (*.) (left: 'scalar & #INumberBase<'scalar>, right: 'MatExp & #IMatExp<'matExp, 'scalar> ) = // ...
member (*.) (left: 'matExp & #IMatExp<'matExp, 'scalar>, right:  'scalar & #INumberBase<'scalar> ) = // ...

There is an workaround to use a wrapper struct like type MatExp<'inner, 'scalar when IMatExp<'inner> and INumberBase<'scalar>>. But then we have to manually do the wrap.

• Lacks of const generics, as @hez2010 has mentioned (Feature Proposal: Const Generics #89730). Const generics have two benefits at least: inline allocate small matrix and prevent some potential bugs. For example, in graphics, it is common to use a Matrix<double, Dynamic, 3> V to store all vertices. Then V.Row(x) must be a expression of dimension 1*3.
• Ref struct cannot implement interface ([Proposal]: Let ref structs implement interfaces and substitute into type parameters (VS 17.11, .NET 9) csharplang#7608) . As small matrix is allocated inline, their references need to be store in the expresstion type struct like VecAdd to avoid copy.
• Lacks of (const) generic specialization. This can unify fix sized and dynamic sized matrix in the form Matrix<Scalar, DimRow, DimCol> and simplify the API.

Some other features like existential types (dotnet/csharplang#5556) may also be useful. But I cannot provide more information for now. In case someone is interested, I have uploaded my attempts with F# here https://github.com/Shuenhoy/Furzn/ .

These feature requrests have been existed for a while and some may require changes in runtime and even metadata. I do not expect they can be all implemented any sooner. But I hope .NET foundation can take a closer look on them for a potentialy better presentation of Tensors and Linear algebra in .NET.

[tannergooding]

Graph optimizations like that are a separate/independent concern and are not something that should be part of the default type experience.

None of the major tensor libraries force such handling. They all allow trivial direct usage and provide a separate way to do lazy evaluation over an expression tree in order to allow dynamic code generation for additional performance.