gpt_tokenizer 0.1.1
SDKdartflutter
Platformandroidioswindowslinuxmacosweb
The implementation of BPE algorithm for sentence encode/decode
This package is used for flutter_gpt_tokenizer, and should not be used directly.
It implements the BPE algorithm of OpenAI tiktoken using Rust