Getting "trust_remote_code" Error when Running SageMaker Deploy Code Sample
#27
by
garystafford
- opened
When deploying the model to Amazon SageMaker using the supplied code in the Deploy tab I am getting the following error, on the predictor.predict call:
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
"code": 400,
"type": "InternalServerException",
"message": "Loading /.sagemaker/mms/models/tiiuae__falcon-40b-instruct requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code\u003dTrue` to remove this error."
}
You can use this in your model_fn function in inference.py
tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-40b-instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b-instruct", trust_remote_code=True,torch_dtype=torch.bfloat16, device_map="auto")
I'm struggling with this as well. In Sagemaker Studio, I have created a file code/inference.py, and put in it:
from transformers import pipeline
def model_fn(model_dir):
tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-40b-instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b-instruct", trust_remote_code=True,torch_dtype=torch.bfloat16, device_map="auto")
return pipeline(
"question-answering",
model=model,
tokenizer=tokenizer,
torch_dtype="auto",
trust_remote_code=True,
device_map="auto",
)
But I continue to get the error regarding "trust_remote_code." Can someone please explain a bit more on exactly what is required to get falcon to work as a Sagemaker endpoint? I'd really appreciate it.
Here's the guide by Phil Schmid to deploy this model on sage maker... Works like a charm
FalconLLM
changed discussion status to
closed