File size: 1,806 Bytes
874c0f8 44bac6a 7866dc9 44bac6a 7866dc9 44bac6a 874c0f8 7866dc9 874c0f8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
---
language:
- en
tags:
- NLP
license: mit
datasets:
- TristanBehrens/bach_garland_2024-100K
base_model: None
---
# Bach Garland xLSTM - An xLSTM model trained on Johann Sebastian Bach Style music
Say Hello on [LinkedIn](https://www.linkedin.com/dr-tristan-behrens-734967a2/) and [X](https://x.com/DrTBehrens).
This is a xLSTM model trained on music by Johann Sebastian Bach. It includes all pieces of Bach's music that can be played on church organ. The samples come in the prototypical Garland notation.
The dataset contains 100K samples and comes with a total token count of 144M.
## How to use
1. Clone this repository and follow the installation instructions: https://github.com/AI-Guru/helibrunna/
2. Open and run the notebook `examples/music.ipynb`. Do not forget to add the id of this model.
3. Enjoy!
## Training
Trained with [Helibrunna](https://github.com/AI-Guru/helibrunna) by [Dr. Tristan Behrens](https://de.linkedin.com/dr-tristan-behrens-734967a2).
## Configuration
```
training:
model_name: bach_garland_xlstm
batch_size: 4
lr: 0.001
lr_warmup_steps: 5000
lr_decay_until_steps: 50000
lr_decay_factor: 0.001
weight_decay: 0.1
amp_precision: bfloat16
weight_precision: float32
enable_mixed_precision: true
num_epochs: 4
output_dir: output/bach_garland_xlstm
save_every_step: 500
log_every_step: 10
wandb_project: bach_garland_xlstm
torch_compile: false
model:
num_blocks: 4
embedding_dim: 64
mlstm_block:
mlstm:
num_heads: 4
slstm_block:
slstm:
num_heads: 4
slstm_at:
- 2
context_length: 4096
vocab_size: 178
dataset:
hugging_face_id: TristanBehrens/bach_garland_2024-100K
tokenizer:
type: whitespace
fill_token: '[EOS]'
```
|