rinna/vicuna-13b-delta-finetuned-langchain-MRKL

NOTE: This "delta model" cannot be used directly.
Users have to apply it on top of the original LLaMA weights to get actual vicuna-13b-finetuned-langchain-MRKL weights.
See https://github.com/rinnakk/vicuna-13b-delta-finetuned-langchain-MRKL#model-weights for instructions.

vicuna-13b-finetuned-langchain-MRKL

Model details

Model type: vicuna-13b-finetuned-langchain-MRKL is an open-source chatbot trained by fine-tuning vicuna-13b on 15 examples with langchain-MRKL format.

Model Usage:

To obtain the correct model, plese run apply_delta.py first.(https://github.com/rinnakk/vicuna-13b-delta-finetuned-langchain-MRKL/blob/main/model/apply_delta.py) See instructions https://github.com/rinnakk/vicuna-13b-delta-finetuned-langchain-MRKL#model-weights

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("path/to/vicuna-13b-finetuned-langchain-MRKL")
model = AutoModelForCausalLM.from_pretrained("path/to/vicuna-13b-finetuned-langchain-MRKL")
model.cuda()

prompt = """Answer the following questions as best you can. You have access to the following tools:

Search: useful for when you need to answer questions about current events
Calculator: useful for when you need to answer questions about math

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [Search, Calculator]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin!

Question: The current age of the President of the United States multiplied by 0.5.
Thought:"""

input_ids = tokenizer(prompt, return_tensors='pt').input_ids.to("cuda")
tokens =  model.generate(input_ids,min_length = 5, max_new_tokens=128,do_sample = True, temperature = 0.7, top_p = 0.9)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

output(The tokens after "Thought:"):

I need to find the current age of the President and then multiply it by 0.5
Action: Search
Action Input: Who is the President of the United States?

if you launched a httpserver with the model and installed langchain(https://github.com/hwchase17/langchain), you can modify demo.py with your httpserver's ip&port and your SERPAPI_API_KEY, then run it.(https://github.com/rinnakk/vicuna-13b-delta-finetuned-langchain-MRKL/blob/main/demo.py)
you can also try this by Jupyter Notebook. https://github.com/rinnakk/vicuna-13b-delta-finetuned-langchain-MRKL/blob/main/demo.ipynb

Where to send questions or comments about the model:

https://github.com/rinnakk/vicuna-13b-delta-finetuned-langchain-MRKL/issues

Training dataset

train only one epoch on mix data (sharegpt + 32*my.json + moss-003-sft-data)

Evaluation

• demo for langchain-MRKL: https://github.com/rinnakk/vicuna-13b-delta-finetuned-langchain-MRKL/blob/main/demo.ipynb
• No evaluation set. Because we don't think we improved the ability of model. we just make model fit langchain-MRKL strictly.
• We just want to show vicuna-13b's powerful ability about thinking and action.

Major Improvement

• support langchain-MRKL(agent= "zero-shot-react-description")
• very fast because of stritcly format(it doesn't generate redundant tokens)

Author

Qu Peng (https://huggingface.co/PengQu)