ValueError: Tokenizer class LlamaTokenizer does not exist or is not currently imported. (while using Transformers)

#23
by ShifraSec - opened

Background:

Using OpenPlayGround to connection to the model and getting eternal server error 500, so started to investigate my connection to the model.

Steps to reproduce

run a python REPL (or while in code)...

>> from transformers import AutoTokenizer, AutoModelForCausalLM

fine then ...

>>> tokenizer = AutoTokenizer.from_pretrained("anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g")

gives me this error:

ValueError: Tokenizer class LlamaTokenizer does not exist or is not currently imported. 

Notes:

  1. tried to:

    pip install --upgrade transformers
    

still didn't work.

  1. My system info:

    OS: MacOS M1 13.3 
    
    Pip version: pip 23.0.1
    
    python version: Python 3.11.0
    

Sign up or log in to comment