Spaces:
Running
Guanaco-13B
Hi Tim,
I'm just getting started with this stuff and could use a little help.
I'm trying to use guanaco-13b and if I've understood correctly, I need to merge the adapter_model.bin file with the original LLaMa files in the HF format. Is this the same process as applying the delta for the Vicuna model?
To provide some context, when I apply the delta for the Vicuna model, I use this script from FastChat:
python3 -m fastchat.model.apply_delta
--base-model-path /path/to/llama-7b
--target-model-path /path/to/output/vicuna-7b
--delta-path lmsys/vicuna-7b-delta-v1.1
I'm wondering if this script could also be used for Guanaco LLM? I attempted to run it but ran into an error saying that it needs a config.json file. I tried to use your adapter_config.json file but it didn't work. Would you be able to provide some guidance on how to resolve this issue?
Hey! Part of the advantages of QLoRA is that we can now distribute adapters directly. You can use your LLaMA weights and add the adapters to the model without having to go through the step of applying the delta.
The HF blog post has colab notebook examples on how to fine-tune.
I am currently trying to create a 13B equivalent of the guanaco-33b-merged model available at https://huggingface.co/timdettmers/guanaco-33b-merged. I wish to use this model with the "oogabooga" text generation web UI.
I was successful in using the guanaco-33b model, largely because Tim shared the merged version. However, for guanaco-13b, I've only been able to find the adapter_config.json and adapter_model.bin files.
I've gone through the blogpost and the Colab notebook, but I couldn't find any specific guidance on how to generate a merged version of the guanaco-13b model based on the adapter_model.bin file.
Could you please provide me with some instructions or tips on how to achieve this? Any help would be greatly appreciated.
EDIT:
Merged versions can be found here:
https://huggingface.co/TheBloke/guanaco-7B-HF
https://huggingface.co/TheBloke/guanaco-13B-HF
https://huggingface.co/TheBloke/guanaco-65B-HF
@dmeight Thank you for sharing your experience.
I'm trying to achieve the step you concluded. I'm referring to when you wrote "I was successful in using the guanaco-33b model, largely because Tim shared the merged version." Can you please share the steps you took to have that model running?
I would really appreciate it if you could share it.
Thank you.