tokenization
#23
by
caterpillarman
- opened
How does this thing tokenize? I'm trying to train a model to speak Sanskrit. If I add word tokens, will the model find them even though they have a higher index than the individual character tokens? I would fine tune, of course, once I figure out how to do that with mistral based models.