---
license: mit
datasets:
- DDSC/reddit-da
- uonlp/CulturaX
language:
- da
---

# Model Card for the Danoliterate Baseline 7B Model

A base model with the same architecture as LlaMa 2 7B but trained from scratch on a combination of Danish datasets for 20K updates (655M tokens.)
## Model Details

### Model Description

As test model part of the thesis [Are GLLMs Danoliterate? Benchmarking Generative NLP in Danish](https://sorenmulli.github.io/thesis/thesis.pdf) with relevant details in Sections 4.1, 5.1 and 6.1.

- **Developed by:** Søren Vejlgaard Holm under supervision from Lars Kai Hansen and Martin Carsten Nielsen.
- **Model type:** Base, autoregressive LLM with LLaMa 2 7B architecture.
- **Language(s) (NLP):** Danish
- **License:** MIT

## Uses

This model is strictly a research artifact for investigating the effect of pre-training a model from scratch and is not intended to be applied directly.


## Bias, Risks, and Limitations

The model has been trained on a large corpus on uncurated internet content and can thus possible generate problematic content.

## Training Details

### Training Data

The pretraining mix contained The Danish Gigaword + Danish Reddit corpora as compiled by the Danish Data Science Community as well as the Danish subset of CulturaX.
For more details, see Section 4.1 in [the thesis](https://sorenmulli.github.io/thesis/thesis.pdf).

### Training Procedure

See Sections 5.1 and 6.1 in [the thesis](https://sorenmulli.github.io/thesis/thesis.pdf)


## Evaluation

On the [Danoliterate LLM Benchmark](https://danoliterate.compute.dtu.dk/), this model gets an index score of 13 as of June 2024.


## Model Card Contact

Contact Søren Vejlgaard Holm at swiho@dtu.dk or swh@alvenir.ai.