This repository comes with LoRA checkpoint to make LLaMA into a chatbot like language model. The checkpoint is the output of instruction following fine-tuning process with the following settings on 8xA100(40G) DGX system. In this fine-tuning process, the length of tokens is set to 1024 since GPT tends to generates long texts
- Training script: borrowed from the official Alpaca-LoRA implementation
- Training script:
python finetune.py \
--base_model='decapoda-research/llama-13b-hf' \
--data_path='alpaca_data_gpt4.json' \
--num_epochs=10 \
--cutoff_len=1024 \
--group_by_length \
--output_dir='./gpt4-alpaca-lora-13b' \
--lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
--lora_r=16 \
--batch_size=... \
--micro_batch_size=...
You can find how the training went from W&B report here.
Example output
- Example #1
Instruction: List all Canadian provinces in alphabetical order.
Response: List of Canadian provinces in alphabetical order:
Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Northwest Territories, Nova Scotia, Nunavut, Ontario, Prince Edward Island, Quebec, Saskatchewan.
Instruction: Which ones are on the east side?
Response: Eastern provinces of Canada are: Newfoundland and Labrador, Nova Scotia, Prince Edward Island, New Brunswick.
Instruction: What foods are famous in each province on the east side?
Response: Newfoundland and Labrador: Jiggs Dinner, Screech, Partridgeberry Wine, Flipper Pie, Nanny’s Cookies. Nova Scotia: Donair, Pizza Burger, Apple Pie, Halifax Donair. New Brunswick:
Fried Dough (Malpeque Doughnuts), Poutine, Acadian Foods (Cote-de-beouf, potato chips, cabbage rolls, pickled beets). Prince Edward Island: Coleslaw, Lobster, Potato Bread, Blueberry Pies, Butter Tarts.
Instruction: What about sightseeing? or landmarks? list one per province
Response: Newfoundland and Labrador: Cabot Tower (St. John’s), Terra Nova National Park, Gros Morne National Park, Signal Hill. Nova Scotia: Cabot Trail, Peggy’s Cove, Lighthouses, Halifax Waterfront. New Brunswick: Fundy National Park, Bay of Fundy Tides,
Loyalist House, Hopewell Rocks. Prince Edward Island: Green Gables Shore, Cavendish National Park, Anne of Green Gables House.
- Example #2
Instruction: Tell me about alpacas.
Response: Alpacas are a type of llama that live in the Andes mountains in South America. They are native to Peru, Chile, Bolivia, and Ecuador, where they are valued for their fiber. Alpacas are related to camels and they are cousins of the llama. Alpacas have woolly fur that comes in many colors and they are very gentle creatures. They are considered the smaller cousins of the llama and they are about the size of a medium dog. Alpacas have long necks and long, lush fur around their necks, which is known as their “ruff.” They also have long lashes that help keep dust and debris from their eyes. Alpacas have hooves that help them move easily through the rocky terrain of the Andes mountains. They make a soft “humbing” sound when they are content. Alpacas live in herds, and females are called “huacas,” and males are called “chancos.” Alpacas can live for up to 20 years. Alpacas can come in a variety of colors, including white, black, brown,
Instruction: What other animals are living in the same area?
Response: Alpacas live in the Andes mountains, and in that region there are many other animals. There are condors, vicunas, pumas, rheas, foxes, chinchillas, and more.
Instruction: Are they the same species?
Response: Alpacas and llamas are both members of the camelid family, but there are differences. Alpacas have longer necks, smaller ears, and shorter, thicker wool. Llamas are larger and their wool is coarser.
- Example #3
Instruction: Tell me about the king of France in 2019.
Response: In 2019, King of France is not applicable, as France does not have a monarch. It has been a republic since the 1790s and its constitution prohibits the monarchy from being reinstated. The President of France is currently Emmanuel Macron.
Instruction: What about before him?
Response: Before Emmanuel Macron, there were 5 other presidents: Nicolas Sarkozy, Jacques Chirac, François Mitterrand, Valéry Giscard d’Estaing, and Georges Pompidou.
- Example #4
Instruction: Write a Python program that prints the first 10 Fibonacci numbers.
Response:
def fibonacci_numbers():
a, b = 0, 1
for i in range(10):
a, b = b, a + b
print(a, b)
fibonacci_numbers()
Output: 0 1 1 2 3 5 8 13 21 34
Instruction: Could you explain how the code works?
Response: First, the function fibonacci_numbers() is defined. The range function creates a range of numbers, in this case between 0 and 10.
Inside the loop, we initialize two variables a and b to 0 and 1 respectively. a, b represent the previous and current Fibonacci numbers respectively. For each number in the range, we add the previous Fibonacci number b to the current one a.
At the end of the loop, the a and b variables will hold the Fibonacci numbers from 0 to 10. This is then printed on the screen.
Instruction: What is recursion?
Response: Recursion is a way of defining a method in which the method calls itself until a certain condition is met. In Python, recursion is used when there is a task that can be broken into smaller tasks that can be done recursively. A recursive function can be defined as any function that calls itself in the body of the function.