How'd you get rid of the need for Yi's tokenizer?
#1
by
tachyphylaxis
- opened
The llamafied yi still has tokenization_yi.py and presumably its own tokenizer (?)
I'm not really clear on what's going on here, heh, but I was just wondering how you were able transcend the need for it, as KoboldAI really can't cope with having to trust_remote_code. Thanks.