Figure out the portable commonality of the AVX2 and aarch64 implementations in https://github.com/simdutf/simdutf/ and port them to core::simd and possible some vendor operations.
This should probably go into a separate crate to allow use without the non-UTF encodings supported by encoding_rs.
Instead of first validating the whole input, perhaps piecewise UTF-8 validation from the simdutf8 crate either needs to be exposed to the UTF-8 to UTF-16 conversion on copypasted.
Since the UTF-16 conversions use very large lookup tables, application-context benchmarking is probably needed to understand the cache effects and to decide what input is long enough for the extra cache evictions to be worth it.
Figure out the portable commonality of the AVX2 and aarch64 implementations in https://github.com/simdutf/simdutf/ and port them to
core::simdand possible some vendor operations.This should probably go into a separate crate to allow use without the non-UTF encodings supported by encoding_rs.
Instead of first validating the whole input, perhaps piecewise UTF-8 validation from the simdutf8 crate either needs to be exposed to the UTF-8 to UTF-16 conversion on copypasted.
Since the UTF-16 conversions use very large lookup tables, application-context benchmarking is probably needed to understand the cache effects and to decide what input is long enough for the extra cache evictions to be worth it.