---
pipeline_tag: translation
language: multilingual
library_name: transformers
license: cc-by-nc-sa-4.0
---
🛡️ Guardians of the Machine Translation Meta-Evaluation:
Sentinel Metrics Fall In!
This repository contains the **SENTINELSRC** metric model trained on Direct Assessments (DA) annotations. For details on how to use our sentinel metric models, check our [GitHub repository](https://github.com/SapienzaNLP/guardians-mt-eval).
## Usage
After having installed our repository package, you can use this model within Python in the following way:
```python
from sentinel_metric import download_model, load_from_checkpoint
model_path = download_model("sapienzanlp/sentinel-src-da")
model = load_from_checkpoint(model_path)
data = [
{"src": "E.T.打电话回家"},
{"src": "托托,我感觉我们已经不在堪萨斯了"}
]
output = model.predict(data, batch_size=8, gpus=1)
```
Output:
```python
# Segment scores
>>> output.scores
[0.524381160736084, 0.4951918423175812]
# System score
>>> output.system_score
0.5097865015268326
```
## Cite this work
This work has been published at [ACL 2024 (Main Conference)](https://aclanthology.org/2024.acl-long.856/). If you use any part, please consider citing our paper as follows:
```bibtex
@inproceedings{perrella-etal-2024-guardians,
title = "Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!",
author = "Perrella, Stefano and Proietti, Lorenzo and Scir{\`e}, Alessandro and Barba, Edoardo and Navigli, Roberto",
editor = "Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = aug,
year = "2024",
address = "Bangkok, Thailand", publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-long.856",
pages = "16216--16244",
}
```
## License
This work is licensed under [Creative Commons Attribution-ShareAlike-NonCommercial 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).