README.md

---
license: apache-2.0
language:
- ko
- en
pipeline_tag: text-generation
tags:
- arxiv:2405.15640
- gecko
- pretrained
library_name: transformers
---

<div align="center">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/5f2fb0137e583543386213d6/MoO9R898b32tV2IyIamYN.png" style="width: 65%">
  <h1>GECKO: Generative Language Model for English, Code and Korean</h1>
</div>

<p align="center" style="margin-top: 0px;">
  <a href="https://arxiv.org/pdf/2405.15640">
    <img src="https://github.com/alpayariyak/openchat/blob/master/assets/arxiv-logomark-small-square-border.png?raw=true" alt="ArXiv Logo" style="width:20px; vertical-align: middle; display: inline-block; margin-right: 5px; margin-left: 5px; margin-top: 0px; margin-bottom: 0px;"/>
    <span class="link-text" style="margin-right: 5px;">Paper</span>
  </a> |
  <a href="">
    <img src="https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png" alt="GitHub Logo" style="width:20px; vertical-align: middle; display: inline-block; margin-right: 5px; margin-left: 5px; margin-top: 0px; margin-bottom: 0px;"/>
    <span class="link-text" style=" margin-right: 5px;">GitHub</span>
  </a> |
</p>

## GECKO-7B
GECKO is a 7B parameter deconder-only transformer pretrained on Korean, English and code. It is trained on 200 billion tokens and use terabytes of Korean corpus. GECKO is an open-source model released under Apache 2.0 License. For more details about our model, please read [our technical report](https://arxiv.org/pdf/2405.15640).


## Model Details

GECKO is a generative language model using Llama architecture. Therefore, our model is easlily integrated with other frameworks which support Llama.

|  | Training Data | Params | Content Length | GQA | Tokens | LR |
| --- | --- | --- | --- | --- | --- | --- |
| GECKO | A mix of publicly available online data | 7B | 8k | X | 200B | 3.0 x 10-4 |

## Usage

~14GB RAM is the required minimum memory size with half-precision like float16 or bfloat16.

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = 'kifai/GECKO-7B'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

text = """이 HTML 코드가 어떤 기능을 하는지 설명하고, 그 설명을 영어로 제공해주세요.
\```html
<button onclick="alert('Welcome!')">Click Me</button>
\```
"""
inputs = tokenizer(text, return_tensors='pt')['input_ids'].to('cuda')
output = model.generate(inputs, max_new_tokens=512, repetition_penalty=1.2)
print(tokenizer.decode(output[0], skip_special_tokens=True))
# 이 HTML 코드가 어떤 기능을 하는지 설명하고, 그 설명을 영어로 제공해주세요.
# \```html
# <button onclick="alert('Welcome!')">Click Me</button>
# \```
# 
# ## Description
# 
# This is a button that will display the message "Welcome!" when clicked.
# 
# ## Expected Output
# 
# The expected output should be:
# 
# \```text
# Welcome!
# \```
```

## Limitation

GECKO is a generative language model that comes with some risks. Its testing has mainly been conducted in Korean and has not covered all possible scenarios. As with all large language models, the outputs from GECKO cannot be predicted in advance and might somtimes be inaccurate, biased, or otherwise problematic. Therefore, developers should conduct safety testing and fine-tune model for the intended uses before deploying it.

## License

GECKO is released under Apache 2.0 license.

## Citation

```
@misc{oh2024gecko,
      title={GECKO: Generative Language Model for English, Code and Korean}, 
      author={Sungwoo Oh and Donggyu Kim},
      year={2024},
      eprint={2405.15640},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

## Acknowledgement

The training is supported by [**TPU Research Cloud**](https://sites.research.google/trc/) program.

## Contact

We look forward to hearing you and collaborating with us

- Sungwoo Oh [[LinkedIn](https://www.linkedin.com/in/sackoh/)]
- Donggyu Kim