Add fast tokenizer
#34
by
TimeRobber
- opened
No description provided.
Closing as I want to remove " - Trained by huggingface/tokenizers". Change has been integrated in our repo: https://github.com/huggingface/tokenizers/pull/1166 I'll open another PR to get fast tokenizer working.