Some weights were not initialized
#1
by
dreamerdeo
- opened
First of all, thanks for your great work!
When I attempt to predict, I encounter the following issues.
Some weights of LlamaForCausalLM were not initialized from the model checkpoint at SeaLLMs/SeaLLM-7B-Hybrid and are newly initialized: ['model.layers.19.self_attn.k_proj.weight', 'model.layers.13.mlp.up_proj.weight', 'model.layers.17.mlp.gate_proj.weight', 'mode
l.layers.0.self_attn.v_proj.weight', 'model.layers.5.self_attn.k_proj.weight', 'model.layers.1.self_attn.v_proj.weight', 'model.layers.9.self_attn.k_proj.weight', 'model.layers.9.mlp.gate_proj.weight', 'model.layers.10.self_attn.q_proj.weight', 'model.layers.24
The code is
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('SeaLLMs/SeaLLM-7B-Hybrid')
Could you give me a help? Thanks!
Thanks for your interest. It's a bug, let me upload a patch.
Fix. kindly download the latest revision
nxphi47
changed discussion status to
closed
Thanks for your fix. But when I re-run the code on transformers==4.35.2, I still have the problem
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('SeaLLMs/SeaLLM-7B-Hybrid')
The problem still exists
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
return model_class.from_pretrained(
File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3480, in from_pretrained
) = cls._load_pretrained_model(
File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3856, in _load_pretrained_model
state_dict = load_state_dict(shard_file)
File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 469, in load_state_dict
if metadata.get("format") not in ["pt", "tf", "flax"]:
AttributeError: 'NoneType' object has no attribute 'get'
Could you conduct a further check on this?
Thanks!
Hi, I am also getting the same problem as dreamerdeo, using transformers==4.31.0 here