Update tokenizer_config.json
#7
by
ybelkada
- opened
No description provided.
Muennighoff
changed pull request status to
merged
It's annoying that we're missing indent here ... Whoever is writing this file should probably have indent
option set.
Just to be sure, I think tokenizer_config.json
comes from transformers
, and it's supposed to be fixed for quite a while: https://github.com/huggingface/transformers/blame/862888a35834527fed61beaf42373423ffdbd216/src/transformers/tokenization_utils_base.py#L2155
@ybelkada
did you perhaps change this manually instead of going through transformers?