|
--- |
|
license: llama3 |
|
library_name: transformers |
|
tags: [] |
|
--- |
|
|
|
# Dracarys-Llama-3.1-70B-Instruct |
|
|
|
### Built with Meta Llama 3 |
|
|
|
# Introduction |
|
|
|
We introduce the latest in the Smaug series, the Dracarys family of finetunes targeting coding performance improvements |
|
across a variety of base models. |
|
|
|
This variant is a finetune of [meta-llama/Meta-Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) |
|
|
|
Compared to meta-llama/Meta-Llama-3.1-70B-Instruct, Dracarys has better LiveCodeBench scores (see evaluation results below). |
|
|
|
### Model Description |
|
|
|
- **Developed by:** [Abacus.AI](https://abacus.ai) |
|
- **License:** https://llama.meta.com/llama3/license/ |
|
- **Finetuned from model:** [meta-llama/Meta-Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct). |
|
|
|
## How to use |
|
|
|
The prompt format is unchanged from Llama 3 70B Instruct (see evaluations for prompt details for LCB) |
|
|
|
### Use with transformers |
|
|
|
See the snippet below for usage with Transformers: |
|
|
|
```python |
|
import transformers |
|
import torch |
|
|
|
model_id = "abacusai/Dracarys-72B-Instruct" |
|
|
|
pipeline = transformers.pipeline( |
|
"text-generation", |
|
model=model_id, |
|
model_kwargs={"torch_dtype": torch.bfloat16}, |
|
device_map="auto", |
|
) |
|
|
|
messages = [ |
|
{"role": "system", "content": "You are data science coding assistant that generates Python code using Pandas and Numpy."}, |
|
{"role": "user", "content": "Write code to select rows from the dataframe `df` having the maximum `temp` for each `city`"}, |
|
] |
|
|
|
prompt = pipeline.tokenizer.apply_chat_template( |
|
messages, |
|
tokenize=False, |
|
add_generation_prompt=True |
|
) |
|
|
|
terminators = [ |
|
pipeline.tokenizer.eos_token_id, |
|
pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>"), |
|
pipeline.tokenizer.convert_tokens_to_ids("<|end_of_text|>"), |
|
] |
|
|
|
outputs = pipeline( |
|
prompt, |
|
max_new_tokens=256, |
|
eos_token_id=terminators, |
|
do_sample=True, |
|
temperature=0.6, |
|
top_p=0.9, |
|
) |
|
print(outputs[0]["generated_text"][len(prompt):]) |
|
``` |
|
|
|
# Evaluation Results |
|
|
|
|
|
## LiveCodeBench |
|
|
|
| Model | Code Generation | Code Execution |Test Output Prediction | |
|
|-------------------------------------|-----------------|----------------|-----------------------| |
|
| **Dracarys-Llama-3.1-70B-Instruct** | **33.34** | **48.329** | **49.90** | |
|
| Meta-Llama-3.1-70B-Instruct | 32.23 | 48.768 | 41.40 | |
|
|
|
## Breakdown of LiveCodeBench CodeGeneration |
|
|
|
| Model | Easy | Medium | Hard | |
|
|-------------------------------------|-----------------|----------------|-----------------------| |
|
| **Dracarys-Llama-3.1-70B-Instruct** | **71.89** | 17.30 | **4.23** | |
|
| Meta-Llama-3.1-70B-Instruct | 68.4 | 17.99 | 3.57 | |
|
|
|
## Breakdown of LiveCodeBench TestOutputPrediction |
|
|
|
| Model | Easy | Medium | Hard | |
|
|-------------------------------------|-----------------|----------------|-----------------------| |
|
| **Dracarys-Llama-3.1-70B-Instruct** | **60.88** | **44.53** | **39.30** | |
|
| Meta-Llama-3.1-70B-Instruct | 51.22 | 35.91 | 34.30 | |
|
|
|
## LiveBench(July update) |
|
|
|
| Model | Global Average | Coding Average | Reasoning Average| Mathematics Average | Data Analysis Average | Language Average | IF Average | |
|
|-------------------------------------|----------------|----------------|------------------|---------------------|-----------------------|------------------|-------------| |
|
| **Dracarys-Llama-3.1-70B-Instruct** | **48.67** | **35.23** | **44.0** | **45.68** | 48 | 41.77 | 77.37 | |
|
| Meta-Llama-3.1-70B-Instruct | 48.44 | 32.67 | 40.67 | 45.58 | 50.29 | 42.36 | 79.08 | |
