---
tags:
- text2text-generation
- definition-modeling
metrics:
- rouge
model-index:
- name: mt0-definition-en-xl
  results: []
language:
- en
widget:
- text: "He ate a sweet apple. What is the definition of apple?"
  example_title: "Definition generation"
- text: "The paper contains a number of original ideas about color perception. What is the definition of original?"
  example_title: "Definition generation"
license: cc-by-sa-4.0
datasets:
- marksverdhei/wordnet-definitions-en-2021
---

# mt0-definition-en-xl

This model is a version of [mt0-xl](https://huggingface.co/bigscience/mt0-xl) fine-tuned on English WordNet, CodWoE and Oxford.

It achieves the following results on the evaluation set:
- Loss: 1.7210
- Rouge1: 41.5067
- Rouge2: 23.7149
- Rougel: 39.138
- Rougelsum: 39.1647
- Gen Len: 15.1578

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 4
- total_train_batch_size: 128
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20.0

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Rouge1  | Rouge2  | Rougel  | Rougelsum | Gen Len |
|:-------------:|:-----:|:-----:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
| 2.1171        | 1.0   | 1370  | 1.8175          | 27.0261 | 8.6429  | 25.2826 | 25.2952   | 11.8798 |
| 1.8186        | 2.0   | 2740  | 1.7112          | 29.1583 | 9.9747  | 27.3432 | 27.3647   | 11.7919 |
| 1.643         | 3.0   | 4110  | 1.6442          | 30.9045 | 11.2256 | 28.7826 | 28.788    | 12.4125 |
| 1.499         | 4.0   | 5480  | 1.5978          | 32.1126 | 12.6674 | 29.97   | 29.9843   | 12.3129 |
| 1.3772        | 5.0   | 6850  | 1.5720          | 33.6113 | 13.8451 | 31.3468 | 31.3599   | 12.6887 |
| 1.2742        | 6.0   | 8220  | 1.5564          | 34.4899 | 15.1005 | 32.3177 | 32.3291   | 12.2003 |
| 1.1785        | 7.0   | 9590  | 1.5466          | 35.4729 | 16.2035 | 33.2166 | 33.2295   | 12.4487 |
| 1.0941        | 8.0   | 10960 | 1.5571          | 36.4885 | 17.5396 | 34.2494 | 34.2759   | 12.7543 |
| 1.0202        | 9.0   | 12330 | 1.5541          | 37.4019 | 18.5568 | 35.1341 | 35.1473   | 12.8603 |
| 0.9552        | 10.0  | 13700 | 1.5642          | 38.127  | 19.4057 | 35.9008 | 35.9163   | 12.6987 |
| 0.8963        | 11.0  | 15070 | 1.5772          | 38.5073 | 20.0584 | 36.3304 | 36.3399   | 12.7052 |
| 0.8443        | 12.0  | 16440 | 1.5955          | 39.2323 | 20.9237 | 36.9863 | 37.0049   | 13.0395 |
| 0.7982        | 13.0  | 17810 | 1.6089          | 39.7947 | 21.6422 | 37.5619 | 37.5815   | 13.1400 |
| 0.7586        | 14.0  | 19180 | 1.6293          | 40.2922 | 22.2301 | 38.0755 | 38.0757   | 12.8589 |
| 0.7234        | 15.0  | 20550 | 1.6493          | 40.6358 | 22.5355 | 38.3523 | 38.3659   | 13.1102 |
| 0.6946        | 16.0  | 21920 | 1.6701          | 40.7708 | 22.906  | 38.5037 | 38.5174   | 13.1035 |
| 0.6688        | 17.0  | 23290 | 1.6902          | 41.0847 | 23.1663 | 38.8126 | 38.8149   | 13.2951 |
| 0.6484        | 18.0  | 24660 | 1.7005          | 41.2075 | 23.3967 | 38.9529 | 38.9545   | 13.2707 |
| 0.6342        | 19.0  | 26030 | 1.7116          | 41.2454 | 23.5187 | 39.0203 | 39.0396   | 13.2173 |
| 0.6234        | 20.0  | 27400 | 1.7210          | 41.3073 | 23.5691 | 39.0662 | 39.074    | 13.2558 |


### Framework versions

- Transformers 4.30.2
- Pytorch 1.13.1+rocm5.2
- Datasets 2.12.0
- Tokenizers 0.12.1