toggle the display of search filters (inactive)toggle the display of search filters (active)

Platforms

Android
iOS
Linux
macOS
Web
Windows

SDKs

Dart
Flutter
Results 18 packages
Sort by PUB-AZS

gpt_tokenizer

1
likes
120
points
228
downloads
The implementation of BPE algorithm for sentence encode/decode

v 0.1.1 (1 years ago)

flutter_gpt_tokenizer

11
likes
150
points
205
downloads
Encode/decode/count the tokens for your ChatGPT prompt(sentence/words).

v 0.1.1 (1 years ago)

sqlite3_arabic_tokenizer_flutter

3
likes
150
points
151
downloads
SQLite3 Arabic Tokenizer

v 0.0.3 (1 years ago)

tiktoken_tokenizer_gpt4o_o1

2
likes
140
points
108
downloads
OpenAI's Tiktoken tokenizer for models: GPT-4, GPT-4o, GPT-4o-mini, o1, o1-mini, and o1-preview.#tokenizer#openai#gpt#gpt-4o#tiktoken

v 1.2.1 (1 years ago)

bpe

0
likes
130
points
99
downloads
Fast tokenizer for CL100kBase and O2O100kBase. Streaming count estimators

v 1.0.2 (1 years ago)

tiny_segmenter_dart

2
likes
125
points
89
downloads
A compact Japanese text tokenizer for Dart. TinySegmenter is a Japanese word segmentation library based on the original JavaScript implementation by Taku Kudo.#japanese#text-processing#tokenizer#nlp#segmentation

v 1.0.1 (1 years ago)

japanese_word_tokenizer

9
likes
150
points
60
downloads
A lightweight Japanese word tokenizer library for Dart and Flutter based on wakachigaki

v 0.0.3 (1 years ago)

libsimple_flutter

1
likes
130
points
51
downloads
sqlite fts5 libsimple tokenizer plugin.libsimple version 20230628

v 0.0.10 (1 years ago)

signwriting

2
likes
145
points
39
downloads
Dart utilities for SignWriting formats, tokenizer and visualizer.#sign#signwriting

v 1.2.1 (1 years ago)

tokenizer

6
likes
140
points
38
downloads
Converts a stream of strings into a stream of tokens

v 1.0.0 (1 years ago)