sarashina2-7b / tokenizer_config.json

Commit History

use LlamaTokenizer
a647f24

ryo0634 commited on

Use tool tokens
f240094

kajyuuen commited on

Use FastTokenizer
065e10a

kajyuuen commited on

add tokenizer_config
14d5c52

sho-takase commited on