tiiuae

/

falcon-7b

like 1.08k

Follow

Technology Innovation Institute 924

Same here :(

    277 
    278         if isinstance(model, str):
--> 279             raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
    280 
    281     framework = "tf" if "keras.engine.training.Model" in str(inspect.getmro(model.__class__)) else "pt"

ValueError: Could not load model tiiuae/falcon-7b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.auto.modeling_tf_auto.TFAutoModelForCausalLM'>).

Adding this line before pipeline should fix it
model = AutoModelForCausalLM.from_pretrained(model, trust_remote_code=True)

While loading it in 8bit the model gives this error:

TypeError Traceback (most recent call last)
Cell In[5], line 3
1 from transformers import AutoModelForCausalLM
----> 3 model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-7b",trust_remote_code=True,load_in_8bit = True,device_map='auto')

File /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:462, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
458 class_ref = config.auto_map[cls.name]
459 model_class = get_class_from_dynamic_module(
460 class_ref, pretrained_model_name_or_path, **hub_kwargs, **kwargs
461 )
--> 462 return model_class.from_pretrained(
463 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
464 )
465 elif type(config) in cls._model_mapping.keys():
466 model_class = _get_model_class(config, cls._model_mapping)

File /opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py:2828, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
2826 # Dispatch model with hooks on all devices if necessary
2827 if device_map is not None:
-> 2828 dispatch_model(model, device_map=device_map, offload_dir=offload_folder, offload_index=offload_index)
2830 if output_loading_info:
2831 if loading_info is None:

TypeError: dispatch_model() got an unexpected keyword argument 'offload_index'

Base on similar issues in accelerate, you might need to upgrade your version of the library by running pip install --upgrade accelerate: https://github.com/huggingface/peft/issues/186

I am getting the following error where I already changed to float32:

File "falcon_7b.py", line 32, in
pipeline_ = pipeline("text-generation", model=model_folder, device=0, trust_remote_code=True, model_kwargs={"torch_dtype": torch.float32})
File "python3.10/site-packages/transformers/pipelines/init.py", line 727, in pipeline
framework, model = infer_framework_load_model(
File "python3.10/site-packages/transformers/pipelines/base.py", line 266, in infer_framework_load_model
raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
ValueError: Could not load model ../models/falcon_7b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>,).

I have similar issue even after adding:

model = AutoModelForCausalLM.from_pretrained(model, trust_remote_code=True)

Error:

HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is
96: 'RWForCausalLM(
  (transformer): RWModel(
    (word_embeddings): Embedding(65024, 4544)
    (h): ModuleList(
      (0-31): 32 x DecoderLayer(
        (input_layernorm): LayerNorm((4544,), eps=1e-05, elementwise_affine=True)
        (self_attention): Attention(
          (maybe_rotary): RotaryEmbedding()
          (query_key_value): Linear(in_features=4544, out_features=4672, bias=False)
          (dense): Linear(in_features=4544, out_features=4544, bias=False)
          (attention_dropout): Dropout(p=0.0, inplace=False)
        )
        (mlp): MLP(
          (dense_h_to_4h): Linear(in_features=4544, out_features=18176, bias=False)
          (act): GELU(approximate='none')
          (dense_4h_to_h): Linear(in_features=18176, out_features=4544, bias=False)
        )
      )
    )
    (ln_f): LayerNorm((4544,), eps=1e-05, elementwise_affine=True)
  )
  (lm_head): Linear(in_features=4544, out_features=65024, bias=False)
)'.