open-llm-leaderboard/open_llm_leaderboard · Model submitted a while ago (over a week) but not showing on the leaderboard

Feb 21

hi,
I submitted my model : https://huggingface.co/luciodery/Bonsai-PrunedPhi-1.8B over a week ago and I can't seem to find it in either the pending queue or the running evaluations queue.
This is even though I submitted the model without any issue.
Any ideas what could be going wrong ? Or is it just that the wait times are currently really high ?

clefourrier

Open LLM Leaderboard org Feb 21

Hi!
Can you follow the steps in the FAQ of the leaderboard (About tab) and point us to the request file of your model so we can investigate?

luciodery

Feb 22

Thanks @clefourrier -- here is the request file :
https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/luciodery/Bonsai-PrunedPhi-1.8B_eval_request_False_float16_Original.json

Unsure why the status is failed. Any ideas what could be the reason ?

clefourrier

Open LLM Leaderboard org Feb 22

Hi!
Thanks for the link to the request files!

Just checked the logs, it would appear that your model has a size problem - did you make sure you could load it with the AutoClasses?

(
  File "lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
  File "lib/python3.10/site-packages/transformers/modeling_utils.py", line 3852, in from_pretrained
    ) = cls._load_pretrained_model(
  File "lib/python3.10/site-packages/transformers/modeling_utils.py", line 4286, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "lib/python3.10/site-packages/transformers/modeling_utils.py", line 807, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "lib/python3.10/site-packages/accelerate/utils/modeling.py", line 281, in set_module_tensor_to_device
    raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([6620]) in "bias" (which has shape torch.Size([10240])), this look incorrect.

clefourrier changed discussion status to closed Feb 26