openlm-research

/

open_llama_7b

like 126

Follow

OpenLM Research 173

# sample code to repro the bug
>>> tokenizer = LlamaTokenizer.from_pretrained(‘huggingface/open_llama_7b’)
>>> tokenizer.encode('London')
[1, 2516]
>>> tokenizer.decode(2516)
'London'
>>> tokenizer.decode(20719)
'London'
>>>  tokenizer.decode(2516)==tokenizer.decode(20719)
True