Update Space (evaluate main: 940d6dee)

README.md

@@ -12,9 +12,9 @@ tags:
 - metric
 description: >-
   Perplexity (PPL) is one of the most common metrics for evaluating language models.
-  It is defined as the exponentiated average negative log-likelihood of a sequence.
+  It is defined as the exponentiated average negative log-likelihood of a sequence, calculated with exponent base `e`.
   
-  For more information, see https://huggingface.co/docs/transformers/perplexity
+  For more information on perplexity, see [this tutorial](https://huggingface.co/docs/transformers/perplexity).
 ---
 
 # Metric Card for Perplexity
@@ -22,10 +22,11 @@ description: >-
 ## Metric Description
 Given a model and an input text sequence, perplexity measures how likely the model is to generate the input text sequence.
 
-As a metric, it can be used to evaluate how well the model has learned the distribution of the text it was trained on
+As a metric, it can be used to evaluate how well the model has learned the distribution of the text it was trained on.
 
+In this case, `model_id` should be the trained model to be evaluated, and the input texts should be the text that the model was trained on.
 
-In this case, the model input should be the trained model to be evaluated, and the input texts should be the text that the model was trained on.
+This implementation of perplexity is calculated with log base `e`, as in `perplexity = e**(sum(losses) / num_tokenized_tokens)`, following recent convention in deep learning frameworks.
 
 ## Intended Uses
 Any language generation task.
@@ -43,10 +44,10 @@ results = perplexity.compute(predictions=predictions, model_id='gpt2')
 ### Inputs
 - **model_id** (str): model used for calculating Perplexity. NOTE: Perplexity can only be calculated for causal language models.
     - This includes models such as gpt2, causal variations of bert, causal versions of t5, and more (the full list can be found in the AutoModelForCausalLM documentation here: https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModelForCausalLM )
-- **predictions** (list of str): input text, each separate text snippet is one list entry.
+- **predictions** (list of str): input text, where each separate text snippet is one list entry.
 - **batch_size** (int): the batch size to run texts through the model. Defaults to 16.
 - **add_start_token** (bool): whether to add the start token to the texts, so the perplexity can include the probability of the first word. Defaults to True.
-- **device** (str): device to run on, defaults to 'cuda' when available
+- **device** (str): device to run on, defaults to `cuda` when available
 
 ### Output Values
 This metric outputs a dictionary with the perplexity scores for the text input in the list, and the average perplexity.
@@ -56,7 +57,7 @@ If one of the input texts is longer than the max input length of the model, then
 {'perplexities': [8.182524681091309, 33.42122268676758, 27.012239456176758], 'mean_perplexity': 22.871995608011883}
 ```
 
-This metric's range is 0 and up. A lower score is better.
+The range of this metric is [0, inf). A lower score is better.
 
 #### Values from Popular Papers
 
@@ -72,9 +73,9 @@ results = perplexity.compute(model_id='gpt2',
 print(list(results.keys()))
 >>>['perplexities', 'mean_perplexity']
 print(round(results["mean_perplexity"], 2))
->>>78.22
+>>>646.74
 print(round(results["perplexities"][0], 2))
->>>11.11
+>>>32.25
 ```
 Calculating perplexity on predictions loaded in from a dataset:
 ```python
@@ -88,9 +89,9 @@ results = perplexity.compute(model_id='gpt2',
 print(list(results.keys()))
 >>>['perplexities', 'mean_perplexity']
 print(round(results["mean_perplexity"], 2))
->>>60.35
+>>>576.76
 print(round(results["perplexities"][0], 2))
->>>81.12
+>>>889.28
 ```
 
 ## Limitations and Bias

perplexity.py

@@ -29,7 +29,7 @@ _CITATION = """\
 
 _DESCRIPTION = """
 Perplexity (PPL) is one of the most common metrics for evaluating language models.
-It is defined as the exponentiated average negative log-likelihood of a sequence.
+It is defined as the exponentiated average negative log-likelihood of a sequence, calculated with exponent base `e`.
 
 For more information, see https://huggingface.co/docs/transformers/perplexity
 """
@@ -78,9 +78,9 @@ Examples:
         >>> print(list(results.keys()))
         ['perplexities', 'mean_perplexity']
         >>> print(round(results["mean_perplexity"], 2)) # doctest: +SKIP
-        60.35
+        576.76
         >>> print(round(results["perplexities"][0], 2)) # doctest: +SKIP
-        81.12
+        889.28
 """
 
 
@@ -180,7 +180,7 @@ class Perplexity(evaluate.Metric):
             shift_labels = labels[..., 1:].contiguous()
             shift_attention_mask_batch = attn_mask[..., 1:].contiguous()
 
-            perplexity_batch = torch.exp2(
+            perplexity_batch = torch.exp(
                 (loss_fct(shift_logits.transpose(1, 2), shift_labels) * shift_attention_mask_batch).sum(1)
                 / shift_attention_mask_batch.sum(1)
             )

requirements.txt

@@ -1,4 +1,4 @@
-git+https://github.com/huggingface/evaluate@288e417936483b1b48ac2dc64a9d9c80ae0ed7e6
+git+https://github.com/huggingface/evaluate@940d6dee3b4a23eabb0c81e4117c9533cd7c458a
 torch
 torch
 transformers