Update README.md

README.md

@@ -1,34 +1,95 @@
 ---
 license: llama3
-library_name: peft
+library_name: transformers
 tags:
-- generated_from_trainer
+- text-generation-inference
+- Dora
+- Qdora
+- peft
 base_model: meta-llama/Meta-Llama-3-8B
 model-index:
 - name: llama-3-8B-fine-tuned-dora
   results: []
+datasets:
+- timdettmers/openassistant-guanaco
 ---
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
 # llama-3-8B-fine-tuned-dora
+<img src="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e32b89a-9146-4004-81b4-18c20a913df0_1920x1080.jpeg" alt="im" width="700" />
+
+This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on [openassistant-guanaco](https://huggingface.co/datasets/timdettmers/openassistant-guanaco) dataset.
+For LoraConfig we set the `use_dora=True` for the Dora decomposition and comparison with Lora. 
+
+## Inference
+```python
+import os
+from os.path import exists, join, isdir
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, GenerationConfig
+from peft import PeftModel
+from peft.tuners.lora import LoraLayer
+import accelerate
+
+# Update variables!
+max_new_tokens = 100
+top_p = 0.9
+temperature=0.7
+user_question = "What is  central limit theorem?"
+
+# Base model
+model_name_or_path = 'meta-llama/Meta-Llama-3-8B' # Change it to 'YOUR_BASE_MODEL'
+adapter_path = 'ShirinYamani/llama-3-8B-fine-tuned-dora' # Change it to 'YOUR_ADAPTER_PATH'
+
+tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
+# if you wanna use LLaMA HF then fix the early conversion issues.
+tokenizer.bos_token_id = 1
+
+# Load the model (use bf16 for faster inference)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name_or_path,
+    torch_dtype=torch.bfloat16,
+    device_map={"": 0},
+    quantization_config=BitsAndBytesConfig(
+        load_in_4bit=True,
+        bnb_4bit_compute_dtype=torch.bfloat16,
+        bnb_4bit_use_double_quant=True,
+        bnb_4bit_quant_type='nf4',
+    )
+)
+
+model = PeftModel.from_pretrained(model, adapter_path)
+model.eval()
+
+prompt = (
+    "A chat between a curious human and an artificial intelligence assistant. "
+    "The assistant gives helpful, detailed, and polite answers to the user's questions. "
+    "### Human: {user_question}"
+    "### Assistant: "
+)
+
+def generate(model, user_question, max_new_tokens=max_new_tokens, top_p=top_p, temperature=temperature):
+    inputs = tokenizer(prompt.format(user_question=user_question), return_tensors="pt").to('cuda')
+
+    outputs = model.generate(
+        **inputs,
+        generation_config=GenerationConfig(
+            do_sample=True,
+            max_new_tokens=max_new_tokens,
+            top_p=top_p,
+            temperature=temperature,
+        )
+    )
+
+    text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    #print(text)
+    return text
+
+generate(model, user_question)
+```
 
-This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on an unknown dataset.
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
 
 ### Training hyperparameters
 
@@ -45,9 +106,6 @@ The following hyperparameters were used during training:
 - training_steps: 10
 - mixed_precision_training: Native AMP
 
-### Training results
-
-
 
 ### Framework versions