Calculating Accuracy of the model
Since I'm new into NLP, idk how to calculate the accuracy of the model you've developed.
I got your code from github, trained it on the same dataset and I think the validation loss is greater.
Please guide me if you can, TIA!
I generated this code below:
from sklearn.metrics import accuracy_score
predictions = trainer.predict(val_dataset).predictions
references = val_dataset ["query"]
accuracy = accuracy_ score(references, predictions)
print(f"Accuracy: {accuracy: .4f}")
Do you think it should work fine? Please let me know if there are any suggestions or modifications required. Thanks!
since this is a text generative task, I think bleu score is more appropriate than accuracy
you can check this for more information
https://huggingface.co/spaces/evaluate-metric/bleu
Thanks a lot. You are right, bleu score is more suitable for generative tasks.
I calculated the bleu score and rouge score, here are the results:
{'bleu': 69.11695994149133, 'rouge1': 93.32338243439364, 'rouge2': 87.46108869509003, 'rougeL': 91.20303118257938}
I think these are pretty good.