seanghay

Update README.md

95d936b verified 4 months ago

preview code

raw

history blame contribute delete

310 Bytes

metadata

library_name: transformers
tags: []

XLM Roberta Tokenizer trained with 162M tokens of Khmer text.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("seanghay/xlm-roberta-khmer-32k-tokenizer")
tokenizer.tokenize("សួស្ដីកម្ពុជា!")