Issues

• Missing Cargo.lock

  mlc-ai/tokenizers-cpp#96 · Filter by author xksc opened on Apr 10, 2026

• The Encode and EncodeBatch methods methods unnecessarily require std::string instances

  mlc-ai/tokenizers-cpp#95 · Filter by author gh-andre opened on Mar 27, 2026

• Building from the release tag v0.1.1 will fail with errors

  mlc-ai/tokenizers-cpp#94 · Filter by author gh-andre opened on Mar 10, 2026

• linker error when using aarch64 tokenizer_c.a

  mlc-ai/tokenizers-cpp#91 · Filter by author NaveenMittal0 opened on Dec 12, 2025

• Any plans for publishing ubuntu debians, or portable binaries?

  mlc-ai/tokenizers-cpp#89 · Filter by author blozano-tt opened on Oct 22, 2025

• When compiling with rustc 1.90.0 (rustup version 1.28.2), an error was encountered

  mlc-ai/tokenizers-cpp#88 · Filter by author yangxudong opened on Sep 24, 2025

• Finish implementation of MLC_ENABLE_SENTENCEPIECE_TOKENIZER

  mlc-ai/tokenizers-cpp#84 · Filter by author WilliamTambellini opened on Jul 28, 2025
  1

• Upgrade to official sentencepiece 0.2

  mlc-ai/tokenizers-cpp#83 · Filter by author WilliamTambellini opened on Jul 22, 2025

• Call for a new release (0.1.2?)

  mlc-ai/tokenizers-cpp#82 · Filter by author WilliamTambellini opened on Jul 17, 2025

• Add support for generating dynamic library

  mlc-ai/tokenizers-cpp#74 · Filter by author PARTHSQL opened on Jun 11, 2025

• rustc E0308 error when compiling version v.0.1.0

  mlc-ai/tokenizers-cpp#72 · Filter by author gaokao123 opened on May 28, 2025

• Does it support Wordpiece tokenizer?

  mlc-ai/tokenizers-cpp#71 · Filter by author yangxudong opened on May 19, 2025

• Android Cross Compile Error! (error: linker cc not found)

  mlc-ai/tokenizers-cpp#63 · Filter by author huangzhengxiang opened on Feb 28, 2025

• Problem using #include "tokenizers_c.h"

  mlc-ai/tokenizers-cpp#61 · Filter by author doogunwo opened on Feb 13, 2025
  1

• Sentence piece llama2 its not printing space

  mlc-ai/tokenizers-cpp#58 · Filter by author naveenmittal04 opened on Feb 3, 2025
  1

• How to build in android platform?

  mlc-ai/tokenizers-cpp#56 · Filter by author jixiedaima opened on Jan 16, 2025

• link error in tokenizers_c.lib

  mlc-ai/tokenizers-cpp#54 · Filter by author Caio-lima-santos opened on Jan 7, 2025

• Building for size

  mlc-ai/tokenizers-cpp#51 · Filter by author stellaraccident opened on Nov 26, 2024
  1

• Abort is not a great error handling strategy

  mlc-ai/tokenizers-cpp#50 · Filter by author stellaraccident opened on Nov 26, 2024
  1

• AddressSanitizer: heap-use-after-free on addres xxx

  mlc-ai/tokenizers-cpp#48 · Filter by author flyinskyin2013 opened on Nov 15, 2024

• Why MLC_ENABLE_SENTENCEPIECE_TOKENIZER OFF by default?

  mlc-ai/tokenizers-cpp#45 · Filter by author korciuch opened on Oct 13, 2024
  6

• does it support multi - thread decode ?

  mlc-ai/tokenizers-cpp#40 · Filter by author Vincent-syr opened on Jul 26, 2024
  3

• Not building on IOS. Always get ld: library 'System' not found.

  mlc-ai/tokenizers-cpp#39 · Filter by author g1henx opened on Jul 3, 2024

• tokenizer for triton inference server

  mlc-ai/tokenizers-cpp#35 · Filter by author geraldstanje opened on Jun 1, 2024

• Fail to import in Node.js

  mlc-ai/tokenizers-cpp#32 · Filter by author zcbenz opened on May 5, 2024
  1