|
--- |
|
language: |
|
- ru |
|
tags: |
|
- sentence-similarity |
|
- text-classification |
|
datasets: |
|
- merionum/ru_paraphraser |
|
- RuPAWS |
|
--- |
|
|
|
|
|
This is a cross-encoder model trained to predict semantic equivalence of two Russian sentences. |
|
|
|
It classifies text pairs as paraphrases (class 1) or non-paraphrases (class 0). Its scores can be used as a metric of content preservation for paraphrasing or text style transfer. |
|
|
|
It is a [sberbank-ai/ruRoberta-large](https://huggingface.co/sberbank-ai/ruRoberta-large) model fine-tuned on a union of 3 datasets: |
|
1. `RuPAWS`: https://github.com/ivkrotova/rupaws_dataset based on Quora and QQP; |
|
2. `ru_paraphraser`: https://huggingface.co/merionum/ru_paraphraser; |
|
3. Results of the manual check of content preservation for the [RUSSE-2022](https://www.dialog-21.ru/media/5755/dementievadplusetal105.pdf) text detoxification dataset collection (`content_5.tsv`). |
|
|
|
The task was formulated as binary classification: whether the two sentences have the same meaning (1) or different (0). |
|
|
|
The table shows the training dataset size after duplication (joining `text1 + text2` and `text2 + text1` pairs): |
|
|
|
source \ label | 0 | 1 |
|
-- | -- | -- |
|
detox | 1412| 3843 |
|
paraphraser |5539 | 1688 |
|
rupaws_qqp |1112 | 792 |
|
rupaws_wiki |3526 | 2166 |
|
|
|
The model was trained with Adam optimizer and the following hyperparameters: |
|
|
|
``` |
|
learning_rate = 1e-5 |
|
batch_size = 8 |
|
gradient_accumulation_steps = 4 |
|
n_epochs = 3 |
|
max_grad_norm = 1.0 |
|
``` |
|
|
|
After training, the model had the following ROC AUC scores on the test sets: |
|
set | ROC AUC |
|
- | - |
|
detox | 0.857112 |
|
paraphraser | 0.858465 |
|
rupaws_qqp | 0.859195 |
|
rupaws_wiki | 0.906121 |
|
|
|
Example usage: |
|
|
|
```Python |
|
import torch |
|
from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
|
|
model = AutoModelForSequenceClassification.from_pretrained('SkolkovoInstitute/ruRoberta-large-paraphrase-v1') |
|
tokenizer = AutoTokenizer.from_pretrained('SkolkovoInstitute/ruRoberta-large-paraphrase-v1') |
|
|
|
def get_similarity(text1, text2): |
|
""" Predict the probability that two Russian sentences are paraphrases of each other. """ |
|
with torch.inference_mode(): |
|
batch = tokenizer( |
|
text1, text2, |
|
truncation=True, max_length=model.config.max_position_embeddings, return_tensors='pt', |
|
).to(model.device) |
|
proba = torch.softmax(model(**batch).logits, -1) |
|
return proba[0][1].item() |
|
|
|
print(get_similarity('Я тебя люблю', 'Ты мне нравишься')) # 0.9798 |
|
print(get_similarity('Я тебя люблю', 'Я тебя ненавижу')) # 0.0008 |
|
``` |
