WIP: Add real model cards as test cases

.gitignore

@@ -0,0 +1,3 @@
+flagged/
+gradio_cached_examples/
+.idea/

tests/cards/Helsinki-NLP___opus-mt-en-es.md

@@ -0,0 +1,93 @@
+### eng-spa
+
+* source group: English 
+* target group: Spanish 
+*  OPUS readme: [eng-spa](https://github.com/Helsinki-NLP/Tatoeba-Challenge/tree/master/models/eng-spa/README.md)
+
+*  model: transformer
+* source language(s): eng
+* target language(s): spa
+* model: transformer
+* pre-processing: normalization + SentencePiece (spm32k,spm32k)
+* download original weights: [opus-2020-08-18.zip](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.zip)
+* test set translations: [opus-2020-08-18.test.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.test.txt)
+* test set scores: [opus-2020-08-18.eval.txt](https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.eval.txt)
+
+## Benchmarks
+
+| testset               | BLEU  | chr-F |
+|-----------------------|-------|-------|
+| newssyscomb2009-engspa.eng.spa 	| 31.0 	| 0.583 |
+| news-test2008-engspa.eng.spa 	| 29.7 	| 0.564 |
+| newstest2009-engspa.eng.spa 	| 30.2 	| 0.578 |
+| newstest2010-engspa.eng.spa 	| 36.9 	| 0.620 |
+| newstest2011-engspa.eng.spa 	| 38.2 	| 0.619 |
+| newstest2012-engspa.eng.spa 	| 39.0 	| 0.625 |
+| newstest2013-engspa.eng.spa 	| 35.0 	| 0.598 |
+| Tatoeba-test.eng.spa 	| 54.9 	| 0.721 |
+
+
+### System Info: 
+- hf_name: eng-spa
+
+- source_languages: eng
+
+- target_languages: spa
+
+- opus_readme_url: https://github.com/Helsinki-NLP/Tatoeba-Challenge/tree/master/models/eng-spa/README.md
+
+- original_repo: Tatoeba-Challenge
+
+- tags: ['translation']
+
+- languages: ['en', 'es']
+
+- src_constituents: {'eng'}
+
+- tgt_constituents: {'spa'}
+
+- src_multilingual: False
+
+- tgt_multilingual: False
+
+- prepro:  normalization + SentencePiece (spm32k,spm32k)
+
+- url_model: https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.zip
+
+- url_test_set: https://object.pouta.csc.fi/Tatoeba-MT-models/eng-spa/opus-2020-08-18.test.txt
+
+- src_alpha3: eng
+
+- tgt_alpha3: spa
+
+- short_pair: en-es
+
+- chrF2_score: 0.721
+
+- bleu: 54.9
+
+- brevity_penalty: 0.978
+
+- ref_len: 77311.0
+
+- src_name: English
+
+- tgt_name: Spanish
+
+- train_date: 2020-08-18 00:00:00
+
+- src_alpha2: en
+
+- tgt_alpha2: es
+
+- prefer_old: False
+
+- long_pair: eng-spa
+
+- helsinki_git_sha: d2f0910c89026c34a44e331e785dec1e0faa7b82
+
+- transformers_git_sha: f7af09b4524b784d67ae8526f0e2fcc6f5ed0de9
+
+- port_machine: brutasse
+
+- port_time: 2020-08-24-18:20

tests/cards/StanfordAIMI___stanford-deidentifier-base.md

@@ -0,0 +1,23 @@
+Stanford de-identifier was trained on a variety of radiology and biomedical documents with the goal of automatising the de-identification process while reaching satisfactory accuracy for use in production. Manuscript in-proceedings. 
+
+These model weights are the recommended ones among all available deidentifier weights.
+
+Associated github repo: https://github.com/MIDRC/Stanford_Penn_Deidentifier
+
+## Citation
+
+```bibtex
+@article{10.1093/jamia/ocac219,
+    author = {Chambon, Pierre J and Wu, Christopher and Steinkamp, Jackson M and Adleberg, Jason and Cook, Tessa S and Langlotz, Curtis P},
+    title = "{Automated deidentification of radiology reports combining transformer and “hide in plain sight” rule-based methods}",
+    journal = {Journal of the American Medical Informatics Association},
+    year = {2022},
+    month = {11},
+    abstract = "{To develop an automated deidentification pipeline for radiology reports that detect protected health information (PHI) entities and replaces them with realistic surrogates “hiding in plain sight.”In this retrospective study, 999 chest X-ray and CT reports collected between November 2019 and November 2020 were annotated for PHI at the token level and combined with 3001 X-rays and 2193 medical notes previously labeled, forming a large multi-institutional and cross-domain dataset of 6193 documents. Two radiology test sets, from a known and a new institution, as well as i2b2 2006 and 2014 test sets, served as an evaluation set to estimate model performance and to compare it with previously released deidentification tools. Several PHI detection models were developed based on different training datasets, fine-tuning approaches and data augmentation techniques, and a synthetic PHI generation algorithm. These models were compared using metrics such as precision, recall and F1 score, as well as paired samples Wilcoxon tests.Our best PHI detection model achieves 97.9 F1 score on radiology reports from a known institution, 99.6 from a new institution, 99.5 on i2b2 2006, and 98.9 on i2b2 2014. On reports from a known institution, it achieves 99.1 recall of detecting the core of each PHI span.Our model outperforms all deidentifiers it was compared to on all test sets as well as human labelers on i2b2 2014 data. It enables accurate and automatic deidentification of radiology reports.A transformer-based deidentification pipeline can achieve state-of-the-art performance for deidentifying radiology reports and other medical documents.}",
+    issn = {1527-974X},
+    doi = {10.1093/jamia/ocac219},
+    url = {https://doi.org/10.1093/jamia/ocac219},
+    note = {ocac219},
+    eprint = {https://academic.oup.com/jamia/advance-article-pdf/doi/10.1093/jamia/ocac219/47220191/ocac219.pdf},
+}
+```

tests/cards/albert-base-v2.md

@@ -0,0 +1,263 @@
+# ALBERT Base v2
+
+Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
+[this paper](https://arxiv.org/abs/1909.11942) and first released in
+[this repository](https://github.com/google-research/albert). This model, as all ALBERT models, is uncased: it does not make a difference
+between english and English.
+
+Disclaimer: The team releasing ALBERT did not write a model card for this model so this model card has been written by
+the Hugging Face team.
+
+## Model description
+
+ALBERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
+was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
+was pretrained with two objectives:
+
+- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
+  the entire masked sentence through the model and has to predict the masked words. This is different from traditional
+  recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
+  GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
+  sentence.
+- Sentence Ordering Prediction (SOP): ALBERT uses a pretraining loss based on predicting the ordering of two consecutive segments of text.
+
+This way, the model learns an inner representation of the English language that can then be used to extract features
+useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
+classifier using the features produced by the ALBERT model as inputs.
+
+ALBERT is particular in that it shares its layers across its Transformer. Therefore, all layers have the same weights. Using repeating layers results in a small memory footprint, however, the computational cost remains similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same number of (repeating) layers.
+
+This is the second version of the base model. Version 2 is different from version 1 due to different dropout rates, additional training data, and longer training. It has better results in nearly all downstream tasks.
+
+This model has the following configuration:
+
+- 12 repeating layers
+- 128 embedding dimension
+- 768 hidden dimension
+- 12 attention heads
+- 11M parameters
+
+## Intended uses & limitations
+
+You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
+be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=albert) to look for
+fine-tuned versions on a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='albert-base-v2')
+>>> unmasker("Hello I'm a [MASK] model.")
+[
+   {
+      "sequence":"[CLS] hello i'm a modeling model.[SEP]",
+      "score":0.05816134437918663,
+      "token":12807,
+      "token_str":"▁modeling"
+   },
+   {
+      "sequence":"[CLS] hello i'm a modelling model.[SEP]",
+      "score":0.03748830780386925,
+      "token":23089,
+      "token_str":"▁modelling"
+   },
+   {
+      "sequence":"[CLS] hello i'm a model model.[SEP]",
+      "score":0.033725276589393616,
+      "token":1061,
+      "token_str":"▁model"
+   },
+   {
+      "sequence":"[CLS] hello i'm a runway model.[SEP]",
+      "score":0.017313428223133087,
+      "token":8014,
+      "token_str":"▁runway"
+   },
+   {
+      "sequence":"[CLS] hello i'm a lingerie model.[SEP]",
+      "score":0.014405295252799988,
+      "token":29104,
+      "token_str":"▁lingerie"
+   }
+]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import AlbertTokenizer, AlbertModel
+tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
+model = AlbertModel.from_pretrained("albert-base-v2")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import AlbertTokenizer, TFAlbertModel
+tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2'')
+model = TFAlbertModel.from_pretrained("albert-base-v2)
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+### Limitations and bias
+
+Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
+predictions:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='albert-base-v2')
+>>> unmasker("The man worked as a [MASK].")
+
+[
+   {
+      "sequence":"[CLS] the man worked as a chauffeur.[SEP]",
+      "score":0.029577180743217468,
+      "token":28744,
+      "token_str":"▁chauffeur"
+   },
+   {
+      "sequence":"[CLS] the man worked as a janitor.[SEP]",
+      "score":0.028865724802017212,
+      "token":29477,
+      "token_str":"▁janitor"
+   },
+   {
+      "sequence":"[CLS] the man worked as a shoemaker.[SEP]",
+      "score":0.02581118606030941,
+      "token":29024,
+      "token_str":"▁shoemaker"
+   },
+   {
+      "sequence":"[CLS] the man worked as a blacksmith.[SEP]",
+      "score":0.01849772222340107,
+      "token":21238,
+      "token_str":"▁blacksmith"
+   },
+   {
+      "sequence":"[CLS] the man worked as a lawyer.[SEP]",
+      "score":0.01820771023631096,
+      "token":3672,
+      "token_str":"▁lawyer"
+   }
+]
+
+>>> unmasker("The woman worked as a [MASK].")
+
+[
+   {
+      "sequence":"[CLS] the woman worked as a receptionist.[SEP]",
+      "score":0.04604868218302727,
+      "token":25331,
+      "token_str":"▁receptionist"
+   },
+   {
+      "sequence":"[CLS] the woman worked as a janitor.[SEP]",
+      "score":0.028220869600772858,
+      "token":29477,
+      "token_str":"▁janitor"
+   },
+   {
+      "sequence":"[CLS] the woman worked as a paramedic.[SEP]",
+      "score":0.0261906236410141,
+      "token":23386,
+      "token_str":"▁paramedic"
+   },
+   {
+      "sequence":"[CLS] the woman worked as a chauffeur.[SEP]",
+      "score":0.024797942489385605,
+      "token":28744,
+      "token_str":"▁chauffeur"
+   },
+   {
+      "sequence":"[CLS] the woman worked as a waitress.[SEP]",
+      "score":0.024124596267938614,
+      "token":13678,
+      "token_str":"▁waitress"
+   }
+]
+```
+
+This bias will also affect all fine-tuned versions of this model.
+
+## Training data
+
+The ALBERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
+unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
+headers).
+
+## Training procedure
+
+### Preprocessing
+
+The texts are lowercased and tokenized using SentencePiece and a vocabulary size of 30,000. The inputs of the model are
+then of the form:
+
+```
+[CLS] Sentence A [SEP] Sentence B [SEP]
+```
+
+### Training
+
+The ALBERT procedure follows the BERT setup.
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+## Evaluation results
+
+When fine-tuned on downstream tasks, the ALBERT models achieve the following results:
+
+|                | Average  | SQuAD1.1 | SQuAD2.0 | MNLI     | SST-2    | RACE     |
+|----------------|----------|----------|----------|----------|----------|----------|
+|V2              |
+|ALBERT-base     |82.3      |90.2/83.2 |82.1/79.3 |84.6      |92.9      |66.8      |
+|ALBERT-large    |85.7      |91.8/85.2 |84.9/81.8 |86.5      |94.9      |75.2      |
+|ALBERT-xlarge   |87.9      |92.9/86.4 |87.9/84.1 |87.9      |95.4      |80.7      |
+|ALBERT-xxlarge  |90.9      |94.6/89.1 |89.8/86.9 |90.6      |96.8      |86.8      |
+|V1              |
+|ALBERT-base     |80.1      |89.3/82.3 | 80.0/77.1|81.6      |90.3      | 64.0     |
+|ALBERT-large    |82.4      |90.6/83.9 | 82.3/79.4|83.5      |91.7      | 68.5     |
+|ALBERT-xlarge   |85.5      |92.5/86.1 | 86.1/83.1|86.4      |92.4      | 74.8     |
+|ALBERT-xxlarge  |91.0      |94.8/89.3 | 90.2/87.4|90.8      |96.9      | 86.5     |
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{DBLP:journals/corr/abs-1909-11942,
+  author    = {Zhenzhong Lan and
+               Mingda Chen and
+               Sebastian Goodman and
+               Kevin Gimpel and
+               Piyush Sharma and
+               Radu Soricut},
+  title     = {{ALBERT:} {A} Lite {BERT} for Self-supervised Learning of Language
+               Representations},
+  journal   = {CoRR},
+  volume    = {abs/1909.11942},
+  year      = {2019},
+  url       = {http://arxiv.org/abs/1909.11942},
+  archivePrefix = {arXiv},
+  eprint    = {1909.11942},
+  timestamp = {Fri, 27 Sep 2019 13:04:21 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1909-11942.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```

tests/cards/bert-base-cased.md

@@ -0,0 +1,220 @@
+# BERT base model (cased)
+
+Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
+[this paper](https://arxiv.org/abs/1810.04805) and first released in
+[this repository](https://github.com/google-research/bert). This model is case-sensitive: it makes a difference between
+english and English.
+
+Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by
+the Hugging Face team.
+
+## Model description
+
+BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
+was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
+was pretrained with two objectives:
+
+- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
+  the entire masked sentence through the model and has to predict the masked words. This is different from traditional
+  recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
+  GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
+  sentence.
+- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes
+  they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to
+  predict if the two sentences were following each other or not.
+
+This way, the model learns an inner representation of the English language that can then be used to extract features
+useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
+classifier using the features produced by the BERT model as inputs.
+
+## Intended uses & limitations
+
+You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
+be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=bert) to look for
+fine-tuned versions on a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='bert-base-cased')
+>>> unmasker("Hello I'm a [MASK] model.")
+
+[{'sequence': "[CLS] Hello I'm a fashion model. [SEP]",
+  'score': 0.09019174426794052,
+  'token': 4633,
+  'token_str': 'fashion'},
+ {'sequence': "[CLS] Hello I'm a new model. [SEP]",
+  'score': 0.06349995732307434,
+  'token': 1207,
+  'token_str': 'new'},
+ {'sequence': "[CLS] Hello I'm a male model. [SEP]",
+  'score': 0.06228214129805565,
+  'token': 2581,
+  'token_str': 'male'},
+ {'sequence': "[CLS] Hello I'm a professional model. [SEP]",
+  'score': 0.0441727414727211,
+  'token': 1848,
+  'token_str': 'professional'},
+ {'sequence': "[CLS] Hello I'm a super model. [SEP]",
+  'score': 0.03326151892542839,
+  'token': 7688,
+  'token_str': 'super'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import BertTokenizer, BertModel
+tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
+model = BertModel.from_pretrained("bert-base-cased")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import BertTokenizer, TFBertModel
+tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
+model = TFBertModel.from_pretrained("bert-base-cased")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+### Limitations and bias
+
+Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
+predictions:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='bert-base-cased')
+>>> unmasker("The man worked as a [MASK].")
+
+[{'sequence': '[CLS] The man worked as a lawyer. [SEP]',
+  'score': 0.04804691672325134,
+  'token': 4545,
+  'token_str': 'lawyer'},
+ {'sequence': '[CLS] The man worked as a waiter. [SEP]',
+  'score': 0.037494491785764694,
+  'token': 17989,
+  'token_str': 'waiter'},
+ {'sequence': '[CLS] The man worked as a cop. [SEP]',
+  'score': 0.035512614995241165,
+  'token': 9947,
+  'token_str': 'cop'},
+ {'sequence': '[CLS] The man worked as a detective. [SEP]',
+  'score': 0.031271643936634064,
+  'token': 9140,
+  'token_str': 'detective'},
+ {'sequence': '[CLS] The man worked as a doctor. [SEP]',
+  'score': 0.027423162013292313,
+  'token': 3995,
+  'token_str': 'doctor'}]
+
+>>> unmasker("The woman worked as a [MASK].")
+
+[{'sequence': '[CLS] The woman worked as a nurse. [SEP]',
+  'score': 0.16927455365657806,
+  'token': 7439,
+  'token_str': 'nurse'},
+ {'sequence': '[CLS] The woman worked as a waitress. [SEP]',
+  'score': 0.1501094549894333,
+  'token': 15098,
+  'token_str': 'waitress'},
+ {'sequence': '[CLS] The woman worked as a maid. [SEP]',
+  'score': 0.05600163713097572,
+  'token': 13487,
+  'token_str': 'maid'},
+ {'sequence': '[CLS] The woman worked as a housekeeper. [SEP]',
+  'score': 0.04838843643665314,
+  'token': 26458,
+  'token_str': 'housekeeper'},
+ {'sequence': '[CLS] The woman worked as a cook. [SEP]',
+  'score': 0.029980547726154327,
+  'token': 9834,
+  'token_str': 'cook'}]
+```
+
+This bias will also affect all fine-tuned versions of this model.
+
+## Training data
+
+The BERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
+unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
+headers).
+
+## Training procedure
+
+### Preprocessing
+
+The texts are tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are then of the form:
+
+```
+[CLS] Sentence A [SEP] Sentence B [SEP]
+```
+
+With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus and in
+the other cases, it's another random sentence in the corpus. Note that what is considered a sentence here is a
+consecutive span of text usually longer than a single sentence. The only constrain is that the result with the two
+"sentences" has a combined length of less than 512 tokens.
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+### Pretraining
+
+The model was trained on 4 cloud TPUs in Pod configuration (16 TPU chips total) for one million steps with a batch size
+of 256. The sequence length was limited to 128 tokens for 90% of the steps and 512 for the remaining 10%. The optimizer
+used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01,
+learning rate warmup for 10,000 steps and linear decay of the learning rate after.
+
+## Evaluation results
+
+When fine-tuned on downstream tasks, this model achieves the following results:
+
+Glue test results:
+
+| Task | MNLI-(m/mm) | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  | Average |
+|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
+|      | 84.6/83.4   | 71.2 | 90.5 | 93.5  | 52.1 | 85.8  | 88.9 | 66.4 | 79.6    |
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{DBLP:journals/corr/abs-1810-04805,
+  author    = {Jacob Devlin and
+               Ming{-}Wei Chang and
+               Kenton Lee and
+               Kristina Toutanova},
+  title     = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language
+               Understanding},
+  journal   = {CoRR},
+  volume    = {abs/1810.04805},
+  year      = {2018},
+  url       = {http://arxiv.org/abs/1810.04805},
+  archivePrefix = {arXiv},
+  eprint    = {1810.04805},
+  timestamp = {Tue, 30 Oct 2018 20:39:56 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<a href="https://huggingface.co/exbert/?model=bert-base-cased">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>

tests/cards/bert-base-multilingual-cased.md

@@ -0,0 +1,145 @@
+# BERT multilingual base model (cased)
+
+Pretrained model on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective.
+It was introduced in [this paper](https://arxiv.org/abs/1810.04805) and first released in
+[this repository](https://github.com/google-research/bert). This model is case sensitive: it makes a difference
+between english and English.
+
+Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by
+the Hugging Face team.
+
+## Model description
+
+BERT is a transformers model pretrained on a large corpus of multilingual data in a self-supervised fashion. This means
+it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
+was pretrained with two objectives:
+
+- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
+  the entire masked sentence through the model and has to predict the masked words. This is different from traditional
+  recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
+  GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
+  sentence.
+- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes
+  they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to
+  predict if the two sentences were following each other or not.
+
+This way, the model learns an inner representation of the languages in the training set that can then be used to
+extract features useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a
+standard classifier using the features produced by the BERT model as inputs.
+
+## Intended uses & limitations
+
+You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
+be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=bert) to look for
+fine-tuned versions on a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='bert-base-multilingual-cased')
+>>> unmasker("Hello I'm a [MASK] model.")
+
+[{'sequence': "[CLS] Hello I'm a model model. [SEP]",
+  'score': 0.10182085633277893,
+  'token': 13192,
+  'token_str': 'model'},
+ {'sequence': "[CLS] Hello I'm a world model. [SEP]",
+  'score': 0.052126359194517136,
+  'token': 11356,
+  'token_str': 'world'},
+ {'sequence': "[CLS] Hello I'm a data model. [SEP]",
+  'score': 0.048930276185274124,
+  'token': 11165,
+  'token_str': 'data'},
+ {'sequence': "[CLS] Hello I'm a flight model. [SEP]",
+  'score': 0.02036019042134285,
+  'token': 23578,
+  'token_str': 'flight'},
+ {'sequence': "[CLS] Hello I'm a business model. [SEP]",
+  'score': 0.020079681649804115,
+  'token': 14155,
+  'token_str': 'business'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import BertTokenizer, BertModel
+tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
+model = BertModel.from_pretrained("bert-base-multilingual-cased")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import BertTokenizer, TFBertModel
+tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
+model = TFBertModel.from_pretrained("bert-base-multilingual-cased")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+## Training data
+
+The BERT model was pretrained on the 104 languages with the largest Wikipedias. You can find the complete list
+[here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages).
+
+## Training procedure
+
+### Preprocessing
+
+The texts are lowercased and tokenized using WordPiece and a shared vocabulary size of 110,000. The languages with a
+larger Wikipedia are under-sampled and the ones with lower resources are oversampled. For languages like Chinese,
+Japanese Kanji and Korean Hanja that don't have space, a CJK Unicode block is added around every character. 
+
+The inputs of the model are then of the form:
+
+```
+[CLS] Sentence A [SEP] Sentence B [SEP]
+```
+
+With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus and in
+the other cases, it's another random sentence in the corpus. Note that what is considered a sentence here is a
+consecutive span of text usually longer than a single sentence. The only constrain is that the result with the two
+"sentences" has a combined length of less than 512 tokens.
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{DBLP:journals/corr/abs-1810-04805,
+  author    = {Jacob Devlin and
+               Ming{-}Wei Chang and
+               Kenton Lee and
+               Kristina Toutanova},
+  title     = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language
+               Understanding},
+  journal   = {CoRR},
+  volume    = {abs/1810.04805},
+  year      = {2018},
+  url       = {http://arxiv.org/abs/1810.04805},
+  archivePrefix = {arXiv},
+  eprint    = {1810.04805},
+  timestamp = {Tue, 30 Oct 2018 20:39:56 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```

tests/cards/bert-base-uncased.md

@@ -0,0 +1,241 @@
+# BERT base model (uncased)
+
+Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
+[this paper](https://arxiv.org/abs/1810.04805) and first released in
+[this repository](https://github.com/google-research/bert). This model is uncased: it does not make a difference
+between english and English.
+
+Disclaimer: The team releasing BERT did not write a model card for this model so this model card has been written by
+the Hugging Face team.
+
+## Model description
+
+BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it
+was pretrained on the raw texts only, with no humans labeling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
+was pretrained with two objectives:
+
+- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
+  the entire masked sentence through the model and has to predict the masked words. This is different from traditional
+  recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
+  GPT which internally masks the future tokens. It allows the model to learn a bidirectional representation of the
+  sentence.
+- Next sentence prediction (NSP): the models concatenates two masked sentences as inputs during pretraining. Sometimes
+  they correspond to sentences that were next to each other in the original text, sometimes not. The model then has to
+  predict if the two sentences were following each other or not.
+
+This way, the model learns an inner representation of the English language that can then be used to extract features
+useful for downstream tasks: if you have a dataset of labeled sentences, for instance, you can train a standard
+classifier using the features produced by the BERT model as inputs.
+
+## Model variations
+
+BERT has originally been released in base and large variations, for cased and uncased input text. The uncased models also strips out an accent markers.  
+Chinese and multilingual uncased and cased versions followed shortly after.  
+Modified preprocessing with whole word masking has replaced subpiece masking in a following work, with the release of two models.  
+Other 24 smaller models are released afterward.  
+
+The detailed release history can be found on the [google-research/bert readme](https://github.com/google-research/bert/blob/master/README.md) on github.
+
+| Model | #params | Language |
+|------------------------|--------------------------------|-------|
+| [`bert-base-uncased`](https://huggingface.co/bert-base-uncased) | 110M   | English |
+| [`bert-large-uncased`](https://huggingface.co/bert-large-uncased)              | 340M    | English | sub 
+| [`bert-base-cased`](https://huggingface.co/bert-base-cased)        | 110M    | English |
+| [`bert-large-cased`](https://huggingface.co/bert-large-cased) | 340M    |  English |
+| [`bert-base-chinese`](https://huggingface.co/bert-base-chinese) | 110M    | Chinese |
+| [`bert-base-multilingual-cased`](https://huggingface.co/bert-base-multilingual-cased) | 110M | Multiple |
+| [`bert-large-uncased-whole-word-masking`](https://huggingface.co/bert-large-uncased-whole-word-masking) | 340M | English |
+| [`bert-large-cased-whole-word-masking`](https://huggingface.co/bert-large-cased-whole-word-masking) | 340M | English |
+
+## Intended uses & limitations
+
+You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
+be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=bert) to look for
+fine-tuned versions of a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='bert-base-uncased')
+>>> unmasker("Hello I'm a [MASK] model.")
+
+[{'sequence': "[CLS] hello i'm a fashion model. [SEP]",
+  'score': 0.1073106899857521,
+  'token': 4827,
+  'token_str': 'fashion'},
+ {'sequence': "[CLS] hello i'm a role model. [SEP]",
+  'score': 0.08774490654468536,
+  'token': 2535,
+  'token_str': 'role'},
+ {'sequence': "[CLS] hello i'm a new model. [SEP]",
+  'score': 0.05338378623127937,
+  'token': 2047,
+  'token_str': 'new'},
+ {'sequence': "[CLS] hello i'm a super model. [SEP]",
+  'score': 0.04667217284440994,
+  'token': 3565,
+  'token_str': 'super'},
+ {'sequence': "[CLS] hello i'm a fine model. [SEP]",
+  'score': 0.027095865458250046,
+  'token': 2986,
+  'token_str': 'fine'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import BertTokenizer, BertModel
+tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
+model = BertModel.from_pretrained("bert-base-uncased")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import BertTokenizer, TFBertModel
+tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
+model = TFBertModel.from_pretrained("bert-base-uncased")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+### Limitations and bias
+
+Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
+predictions:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='bert-base-uncased')
+>>> unmasker("The man worked as a [MASK].")
+
+[{'sequence': '[CLS] the man worked as a carpenter. [SEP]',
+  'score': 0.09747550636529922,
+  'token': 10533,
+  'token_str': 'carpenter'},
+ {'sequence': '[CLS] the man worked as a waiter. [SEP]',
+  'score': 0.0523831807076931,
+  'token': 15610,
+  'token_str': 'waiter'},
+ {'sequence': '[CLS] the man worked as a barber. [SEP]',
+  'score': 0.04962705448269844,
+  'token': 13362,
+  'token_str': 'barber'},
+ {'sequence': '[CLS] the man worked as a mechanic. [SEP]',
+  'score': 0.03788609802722931,
+  'token': 15893,
+  'token_str': 'mechanic'},
+ {'sequence': '[CLS] the man worked as a salesman. [SEP]',
+  'score': 0.037680890411138535,
+  'token': 18968,
+  'token_str': 'salesman'}]
+
+>>> unmasker("The woman worked as a [MASK].")
+
+[{'sequence': '[CLS] the woman worked as a nurse. [SEP]',
+  'score': 0.21981462836265564,
+  'token': 6821,
+  'token_str': 'nurse'},
+ {'sequence': '[CLS] the woman worked as a waitress. [SEP]',
+  'score': 0.1597415804862976,
+  'token': 13877,
+  'token_str': 'waitress'},
+ {'sequence': '[CLS] the woman worked as a maid. [SEP]',
+  'score': 0.1154729500412941,
+  'token': 10850,
+  'token_str': 'maid'},
+ {'sequence': '[CLS] the woman worked as a prostitute. [SEP]',
+  'score': 0.037968918681144714,
+  'token': 19215,
+  'token_str': 'prostitute'},
+ {'sequence': '[CLS] the woman worked as a cook. [SEP]',
+  'score': 0.03042375110089779,
+  'token': 5660,
+  'token_str': 'cook'}]
+```
+
+This bias will also affect all fine-tuned versions of this model.
+
+## Training data
+
+The BERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
+unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
+headers).
+
+## Training procedure
+
+### Preprocessing
+
+The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
+then of the form:
+
+```
+[CLS] Sentence A [SEP] Sentence B [SEP]
+```
+
+With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus, and in
+the other cases, it's another random sentence in the corpus. Note that what is considered a sentence here is a
+consecutive span of text usually longer than a single sentence. The only constrain is that the result with the two
+"sentences" has a combined length of less than 512 tokens.
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+### Pretraining
+
+The model was trained on 4 cloud TPUs in Pod configuration (16 TPU chips total) for one million steps with a batch size
+of 256. The sequence length was limited to 128 tokens for 90% of the steps and 512 for the remaining 10%. The optimizer
+used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01,
+learning rate warmup for 10,000 steps and linear decay of the learning rate after.
+
+## Evaluation results
+
+When fine-tuned on downstream tasks, this model achieves the following results:
+
+Glue test results:
+
+| Task | MNLI-(m/mm) | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  | Average |
+|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
+|      | 84.6/83.4   | 71.2 | 90.5 | 93.5  | 52.1 | 85.8  | 88.9 | 66.4 | 79.6    |
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{DBLP:journals/corr/abs-1810-04805,
+  author    = {Jacob Devlin and
+               Ming{-}Wei Chang and
+               Kenton Lee and
+               Kristina Toutanova},
+  title     = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language
+               Understanding},
+  journal   = {CoRR},
+  volume    = {abs/1810.04805},
+  year      = {2018},
+  url       = {http://arxiv.org/abs/1810.04805},
+  archivePrefix = {arXiv},
+  eprint    = {1810.04805},
+  timestamp = {Tue, 30 Oct 2018 20:39:56 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<a href="https://huggingface.co/exbert/?model=bert-base-uncased">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>

tests/cards/cl-tohoku___bert-base-japanese-whole-word-masking.md

@@ -0,0 +1,37 @@
+# BERT base Japanese (IPA dictionary, whole word masking enabled)
+
+This is a [BERT](https://github.com/google-research/bert) model pretrained on texts in the Japanese language.
+
+This version of the model processes input texts with word-level tokenization based on the IPA dictionary, followed by the WordPiece subword tokenization.
+Additionally, the model is trained with the whole word masking enabled for the masked language modeling (MLM) objective.
+
+The codes for the pretraining are available at [cl-tohoku/bert-japanese](https://github.com/cl-tohoku/bert-japanese/tree/v1.0).
+
+## Model architecture
+
+The model architecture is the same as the original BERT base model; 12 layers, 768 dimensions of hidden states, and 12 attention heads.
+
+## Training Data
+
+The model is trained on Japanese Wikipedia as of September 1, 2019.
+To generate the training corpus, [WikiExtractor](https://github.com/attardi/wikiextractor) is used to extract plain texts from a dump file of Wikipedia articles.
+The text files used for the training are 2.6GB in size, consisting of approximately 17M sentences.
+
+## Tokenization
+
+The texts are first tokenized by [MeCab](https://taku910.github.io/mecab/) morphological parser with the IPA dictionary and then split into subwords by the WordPiece algorithm.
+The vocabulary size is 32000.
+
+## Training
+
+The model is trained with the same configuration as the original BERT; 512 tokens per instance, 256 instances per batch, and 1M training steps.
+
+For the training of the MLM (masked language modeling) objective, we introduced the **Whole Word Masking** in which all of the subword tokens corresponding to a single word (tokenized by MeCab) are masked at once.
+
+## Licenses
+
+The pretrained models are distributed under the terms of the [Creative Commons Attribution-ShareAlike 3.0](https://creativecommons.org/licenses/by-sa/3.0/).
+
+## Acknowledgments
+
+For training models, we used Cloud TPUs provided by [TensorFlow Research Cloud](https://www.tensorflow.org/tfrc/) program.

tests/cards/distilbert-base-cased-distilled-squad.md

@@ -0,0 +1,179 @@
+# DistilBERT base cased distilled SQuAD
+
+## Table of Contents
+- [Model Details](#model-details)
+- [How To Get Started With the Model](#how-to-get-started-with-the-model)
+- [Uses](#uses)
+- [Risks, Limitations and Biases](#risks-limitations-and-biases)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Environmental Impact](#environmental-impact)
+- [Technical Specifications](#technical-specifications)
+- [Citation Information](#citation-information)
+- [Model Card Authors](#model-card-authors)
+
+## Model Details
+
+**Model Description:** The DistilBERT model was proposed in the blog post [Smaller, faster, cheaper, lighter: Introducing DistilBERT, adistilled version of BERT](https://medium.com/huggingface/distilbert-8cf3380435b5), and the paper [DistilBERT, adistilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108). DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than *bert-base-uncased*, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark.
+
+This model is a fine-tune checkpoint of [DistilBERT-base-cased](https://huggingface.co/distilbert-base-cased), fine-tuned using (a second step of) knowledge distillation on [SQuAD v1.1](https://huggingface.co/datasets/squad). 
+
+- **Developed by:** Hugging Face
+- **Model Type:** Transformer-based language model
+- **Language(s):** English 
+- **License:** Apache 2.0
+- **Related Models:** [DistilBERT-base-cased](https://huggingface.co/distilbert-base-cased)
+- **Resources for more information:**
+  - See [this repository](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation) for more about Distil\* (a class of compressed models including this model)
+  - See [Sanh et al. (2019)](https://arxiv.org/abs/1910.01108) for more information about knowledge distillation and the training procedure
+
+## How to Get Started with the Model 
+
+Use the code below to get started with the model. 
+
+```python
+>>> from transformers import pipeline
+>>> question_answerer = pipeline("question-answering", model='distilbert-base-cased-distilled-squad')
+
+>>> context = r"""
+... Extractive Question Answering is the task of extracting an answer from a text given a question. An example     of a
+... question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune
+... a model on a SQuAD task, you may leverage the examples/pytorch/question-answering/run_squad.py script.
+... """
+
+>>> result = question_answerer(question="What is a good example of a question answering dataset?",     context=context)
+>>> print(
+... f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}"
+...)
+
+Answer: 'SQuAD dataset', score: 0.5152, start: 147, end: 160
+```
+
+Here is how to use this model in PyTorch:
+
+```python
+from transformers import DistilBertTokenizer, DistilBertModel
+import torch
+tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased-distilled-squad')
+model = DistilBertModel.from_pretrained('distilbert-base-cased-distilled-squad')
+
+question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
+
+inputs = tokenizer(question, text, return_tensors="pt")
+with torch.no_grad():
+    outputs = model(**inputs)
+
+print(outputs)
+```
+
+And in TensorFlow: 
+
+```python
+from transformers import DistilBertTokenizer, TFDistilBertForQuestionAnswering
+import tensorflow as tf
+
+tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")
+model = TFDistilBertForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")
+
+question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
+
+inputs = tokenizer(question, text, return_tensors="tf")
+outputs = model(**inputs)
+
+answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
+answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])
+
+predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
+tokenizer.decode(predict_answer_tokens)
+```
+
+## Uses
+
+This model can be used for question answering.
+
+#### Misuse and Out-of-scope Use
+
+The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
+
+## Risks, Limitations and Biases
+
+**CONTENT WARNING: Readers should be aware that language generated by this model can be disturbing or offensive to some and can propagate historical and current stereotypes.**
+
+Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:
+
+
+```python
+>>> from transformers import pipeline
+>>> question_answerer = pipeline("question-answering", model='distilbert-base-cased-distilled-squad')
+
+>>> context = r"""
+... Alice is sitting on the bench. Bob is sitting next to her.
+... """
+
+>>> result = question_answerer(question="Who is the CEO?", context=context)
+>>> print(
+... f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}"
+...)
+
+Answer: 'Bob', score: 0.7527, start: 32, end: 35
+```
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
+
+## Training
+
+#### Training Data
+
+The [distilbert-base-cased model](https://huggingface.co/distilbert-base-cased) was trained using the same data as the [distilbert-base-uncased model](https://huggingface.co/distilbert-base-uncased). The [distilbert-base-uncased model](https://huggingface.co/distilbert-base-uncased) model describes it's training data as: 
+
+> DistilBERT pretrained on the same data as BERT, which is [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers).
+
+To learn more about the SQuAD v1.1 dataset, see the [SQuAD v1.1 data card](https://huggingface.co/datasets/squad).
+
+#### Training Procedure
+
+##### Preprocessing
+
+See the [distilbert-base-cased model card](https://huggingface.co/distilbert-base-cased) for further details.
+
+##### Pretraining
+
+See the [distilbert-base-cased model card](https://huggingface.co/distilbert-base-cased) for further details. 
+
+## Evaluation
+
+As discussed in the [model repository](https://github.com/huggingface/transformers/blob/main/examples/research_projects/distillation/README.md)
+
+> This model reaches a F1 score of 87.1 on the [SQuAD v1.1] dev set (for comparison, BERT bert-base-cased version reaches a F1 score of 88.7).	
+
+## Environmental Impact
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). We present the hardware type and hours used based on the [associated paper](https://arxiv.org/pdf/1910.01108.pdf). Note that these details are just for training DistilBERT, not including the fine-tuning with SQuAD.
+
+- **Hardware Type:** 8 16GB V100 GPUs
+- **Hours used:** 90 hours
+- **Cloud Provider:** Unknown
+- **Compute Region:** Unknown
+- **Carbon Emitted:** Unknown
+
+## Technical Specifications
+
+See the [associated paper](https://arxiv.org/abs/1910.01108) for details on the modeling architecture, objective, compute infrastructure, and training details.
+
+## Citation Information
+
+```bibtex
+@inproceedings{sanh2019distilbert,
+  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
+  author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},
+  booktitle={NeurIPS EMC^2 Workshop},
+  year={2019}
+}
+```
+
+APA: 
+- Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
+
+## Model Card Authors
+
+This model card was written by the Hugging Face team.

tests/cards/distilbert-base-uncased-finetuned-sst-2-english.md

@@ -0,0 +1,80 @@
+# DistilBERT base uncased finetuned SST-2
+
+## Table of Contents
+- [Model Details](#model-details)
+- [How to Get Started With the Model](#how-to-get-started-with-the-model)
+- [Uses](#uses)
+- [Risks, Limitations and Biases](#risks-limitations-and-biases)
+- [Training](#training)
+
+## Model Details
+**Model Description:** This model is a fine-tune checkpoint of [DistilBERT-base-uncased](https://huggingface.co/distilbert-base-uncased), fine-tuned on SST-2.
+This model reaches an accuracy of 91.3 on the dev set (for comparison, Bert bert-base-uncased version reaches an accuracy of 92.7).
+- **Developed by:** Hugging Face
+- **Model Type:** Text Classification
+- **Language(s):** English
+- **License:** Apache-2.0
+- **Parent Model:** For more details about DistilBERT, we encourage users to check out [this model card](https://huggingface.co/distilbert-base-uncased).
+- **Resources for more information:**
+    - [Model Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/distilbert#transformers.DistilBertForSequenceClassification)
+
+## How to Get Started With the Model
+
+Example of single-label classification:
+​​
+```python
+import torch
+from transformers import DistilBertTokenizer, DistilBertForSequenceClassification
+
+tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-uncased")
+model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased")
+
+inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
+with torch.no_grad():
+    logits = model(**inputs).logits
+
+predicted_class_id = logits.argmax().item()
+model.config.id2label[predicted_class_id]
+
+```
+
+## Uses
+
+#### Direct Use
+
+This model can be used for  topic classification. You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to be fine-tuned on a downstream task. See the model hub to look for fine-tuned versions on a task that interests you.
+
+#### Misuse and Out-of-scope Use
+The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
+
+
+## Risks, Limitations and Biases
+
+Based on a few experimentations, we observed that this model could produce biased predictions that target underrepresented populations.
+
+For instance, for sentences like `This film was filmed in COUNTRY`, this binary classification model will give radically different probabilities for the positive label depending on the country (0.89 if the country is France, but 0.08 if the country is Afghanistan) when nothing in the input indicates such a strong semantic shift. In this [colab](https://colab.research.google.com/gist/ageron/fb2f64fb145b4bc7c49efc97e5f114d3/biasmap.ipynb), [Aurélien Géron](https://twitter.com/aureliengeron) made an interesting map plotting these probabilities for each country.
+
+<img src="https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/resolve/main/map.jpeg" alt="Map of positive probabilities per country." width="500"/>
+
+We strongly advise users to thoroughly probe these aspects on their use-cases in order to evaluate the risks of this model. We recommend looking at the following bias evaluation datasets as a place to start: [WinoBias](https://huggingface.co/datasets/wino_bias), [WinoGender](https://huggingface.co/datasets/super_glue), [Stereoset](https://huggingface.co/datasets/stereoset).
+
+
+
+# Training
+
+
+#### Training Data
+
+
+The authors use the following Stanford Sentiment Treebank([sst2](https://huggingface.co/datasets/sst2)) corpora for the model.
+
+#### Training Procedure
+
+###### Fine-tuning hyper-parameters
+
+
+- learning_rate = 1e-5
+- batch_size = 32
+- warmup = 600
+- max_seq_length = 128
+- num_train_epochs = 3.0

tests/cards/distilbert-base-uncased.md

@@ -0,0 +1,208 @@
+# DistilBERT base model (uncased)
+
+This model is a distilled version of the [BERT base model](https://huggingface.co/bert-base-uncased). It was
+introduced in [this paper](https://arxiv.org/abs/1910.01108). The code for the distillation process can be found
+[here](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation). This model is uncased: it does
+not make a difference between english and English.
+
+## Model description
+
+DistilBERT is a transformers model, smaller and faster than BERT, which was pretrained on the same corpus in a
+self-supervised fashion, using the BERT base model as a teacher. This means it was pretrained on the raw texts only,
+with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic
+process to generate inputs and labels from those texts using the BERT base model. More precisely, it was pretrained
+with three objectives:
+
+- Distillation loss: the model was trained to return the same probabilities as the BERT base model.
+- Masked language modeling (MLM): this is part of the original training loss of the BERT base model. When taking a
+  sentence, the model randomly masks 15% of the words in the input then run the entire masked sentence through the
+  model and has to predict the masked words. This is different from traditional recurrent neural networks (RNNs) that
+  usually see the words one after the other, or from autoregressive models like GPT which internally mask the future
+  tokens. It allows the model to learn a bidirectional representation of the sentence.
+- Cosine embedding loss: the model was also trained to generate hidden states as close as possible as the BERT base
+  model.
+
+This way, the model learns the same inner representation of the English language than its teacher model, while being
+faster for inference or downstream tasks.
+
+## Intended uses & limitations
+
+You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
+be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=distilbert) to look for
+fine-tuned versions on a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='distilbert-base-uncased')
+>>> unmasker("Hello I'm a [MASK] model.")
+
+[{'sequence': "[CLS] hello i'm a role model. [SEP]",
+  'score': 0.05292855575680733,
+  'token': 2535,
+  'token_str': 'role'},
+ {'sequence': "[CLS] hello i'm a fashion model. [SEP]",
+  'score': 0.03968575969338417,
+  'token': 4827,
+  'token_str': 'fashion'},
+ {'sequence': "[CLS] hello i'm a business model. [SEP]",
+  'score': 0.034743521362543106,
+  'token': 2449,
+  'token_str': 'business'},
+ {'sequence': "[CLS] hello i'm a model model. [SEP]",
+  'score': 0.03462274372577667,
+  'token': 2944,
+  'token_str': 'model'},
+ {'sequence': "[CLS] hello i'm a modeling model. [SEP]",
+  'score': 0.018145186826586723,
+  'token': 11643,
+  'token_str': 'modeling'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import DistilBertTokenizer, DistilBertModel
+tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+model = DistilBertModel.from_pretrained("distilbert-base-uncased")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import DistilBertTokenizer, TFDistilBertModel
+tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
+model = TFDistilBertModel.from_pretrained("distilbert-base-uncased")
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+### Limitations and bias
+
+Even if the training data used for this model could be characterized as fairly neutral, this model can have biased
+predictions. It also inherits some of
+[the bias of its teacher model](https://huggingface.co/bert-base-uncased#limitations-and-bias).
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='distilbert-base-uncased')
+>>> unmasker("The White man worked as a [MASK].")
+
+[{'sequence': '[CLS] the white man worked as a blacksmith. [SEP]',
+  'score': 0.1235365942120552,
+  'token': 20987,
+  'token_str': 'blacksmith'},
+ {'sequence': '[CLS] the white man worked as a carpenter. [SEP]',
+  'score': 0.10142576694488525,
+  'token': 10533,
+  'token_str': 'carpenter'},
+ {'sequence': '[CLS] the white man worked as a farmer. [SEP]',
+  'score': 0.04985016956925392,
+  'token': 7500,
+  'token_str': 'farmer'},
+ {'sequence': '[CLS] the white man worked as a miner. [SEP]',
+  'score': 0.03932540491223335,
+  'token': 18594,
+  'token_str': 'miner'},
+ {'sequence': '[CLS] the white man worked as a butcher. [SEP]',
+  'score': 0.03351764753460884,
+  'token': 14998,
+  'token_str': 'butcher'}]
+
+>>> unmasker("The Black woman worked as a [MASK].")
+
+[{'sequence': '[CLS] the black woman worked as a waitress. [SEP]',
+  'score': 0.13283951580524445,
+  'token': 13877,
+  'token_str': 'waitress'},
+ {'sequence': '[CLS] the black woman worked as a nurse. [SEP]',
+  'score': 0.12586183845996857,
+  'token': 6821,
+  'token_str': 'nurse'},
+ {'sequence': '[CLS] the black woman worked as a maid. [SEP]',
+  'score': 0.11708822101354599,
+  'token': 10850,
+  'token_str': 'maid'},
+ {'sequence': '[CLS] the black woman worked as a prostitute. [SEP]',
+  'score': 0.11499975621700287,
+  'token': 19215,
+  'token_str': 'prostitute'},
+ {'sequence': '[CLS] the black woman worked as a housekeeper. [SEP]',
+  'score': 0.04722772538661957,
+  'token': 22583,
+  'token_str': 'housekeeper'}]
+```
+
+This bias will also affect all fine-tuned versions of this model.
+
+## Training data
+
+DistilBERT pretrained on the same data as BERT, which is [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset
+consisting of 11,038 unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia)
+(excluding lists, tables and headers).
+
+## Training procedure
+
+### Preprocessing
+
+The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
+then of the form:
+
+```
+[CLS] Sentence A [SEP] Sentence B [SEP]
+```
+
+With probability 0.5, sentence A and sentence B correspond to two consecutive sentences in the original corpus and in
+the other cases, it's another random sentence in the corpus. Note that what is considered a sentence here is a
+consecutive span of text usually longer than a single sentence. The only constrain is that the result with the two
+"sentences" has a combined length of less than 512 tokens.
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+### Pretraining
+
+The model was trained on 8 16 GB V100 for 90 hours. See the
+[training code](https://github.com/huggingface/transformers/tree/master/examples/distillation) for all hyperparameters
+details.
+
+## Evaluation results
+
+When fine-tuned on downstream tasks, this model achieves the following results:
+
+Glue test results:
+
+| Task | MNLI | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  |
+|:----:|:----:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|
+|      | 82.2 | 88.5 | 89.2 | 91.3  | 51.3 | 85.8  | 87.5 | 59.9 |
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{Sanh2019DistilBERTAD,
+  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
+  author={Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf},
+  journal={ArXiv},
+  year={2019},
+  volume={abs/1910.01108}
+}
+```
+
+<a href="https://huggingface.co/exbert/?model=distilbert-base-uncased">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>

tests/cards/distilroberta-base.md

@@ -0,0 +1,175 @@
+# Model Card for DistilRoBERTa base
+
+# Table of Contents
+
+1. [Model Details](#model-details)
+2. [Uses](#uses)
+3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
+4. [Training Details](#training-details)
+5. [Evaluation](#evaluation)
+6. [Environmental Impact](#environmental-impact)
+7. [Citation](#citation)
+8. [How To Get Started With the Model](#how-to-get-started-with-the-model)
+
+# Model Details
+
+## Model Description
+
+This model is a distilled version of the [RoBERTa-base model](https://huggingface.co/roberta-base). It follows the same training procedure as [DistilBERT](https://huggingface.co/distilbert-base-uncased).
+The code for the distillation process can be found [here](https://github.com/huggingface/transformers/tree/master/examples/distillation).
+This model is case-sensitive: it makes a difference between english and English.
+
+The model has 6 layers, 768 dimension and 12 heads, totalizing 82M parameters (compared to 125M parameters for RoBERTa-base).
+On average DistilRoBERTa is twice as fast as Roberta-base.
+
+We encourage users of this model card to check out the [RoBERTa-base model card](https://huggingface.co/roberta-base) to learn more about usage, limitations and potential biases.
+
+- **Developed by:** Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf (Hugging Face)
+- **Model type:** Transformer-based language model
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+- **Related Models:** [RoBERTa-base model card](https://huggingface.co/roberta-base)
+- **Resources for more information:** 
+  - [GitHub Repository](https://github.com/huggingface/transformers/blob/main/examples/research_projects/distillation/README.md)
+  - [Associated Paper](https://arxiv.org/abs/1910.01108)
+  
+# Uses
+
+## Direct Use and Downstream Use
+
+You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=roberta) to look for fine-tuned versions on a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at model like GPT2.
+
+## Out of Scope Use
+
+The model should not be used to intentionally create hostile or alienating environments for people. The model was not trained to be factual or true representations of people or events, and therefore using the models to generate such content is out-of-scope for the abilities of this model.
+
+# Bias, Risks, and Limitations
+
+Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='distilroberta-base')
+>>> unmasker("The man worked as a <mask>.")
+[{'score': 0.1237526461482048,
+  'sequence': 'The man worked as a waiter.',
+  'token': 38233,
+  'token_str': ' waiter'},
+ {'score': 0.08968018740415573,
+  'sequence': 'The man worked as a waitress.',
+  'token': 35698,
+  'token_str': ' waitress'},
+ {'score': 0.08387645334005356,
+  'sequence': 'The man worked as a bartender.',
+  'token': 33080,
+  'token_str': ' bartender'},
+ {'score': 0.061059024184942245,
+  'sequence': 'The man worked as a mechanic.',
+  'token': 25682,
+  'token_str': ' mechanic'},
+ {'score': 0.03804653510451317,
+  'sequence': 'The man worked as a courier.',
+  'token': 37171,
+  'token_str': ' courier'}]
+  
+>>> unmasker("The woman worked as a <mask>.")
+[{'score': 0.23149248957633972,
+  'sequence': 'The woman worked as a waitress.',
+  'token': 35698,
+  'token_str': ' waitress'},
+ {'score': 0.07563332468271255,
+  'sequence': 'The woman worked as a waiter.',
+  'token': 38233,
+  'token_str': ' waiter'},
+ {'score': 0.06983394920825958,
+  'sequence': 'The woman worked as a bartender.',
+  'token': 33080,
+  'token_str': ' bartender'},
+ {'score': 0.05411609262228012,
+  'sequence': 'The woman worked as a nurse.',
+  'token': 9008,
+  'token_str': ' nurse'},
+ {'score': 0.04995106905698776,
+  'sequence': 'The woman worked as a maid.',
+  'token': 29754,
+  'token_str': ' maid'}]
+```
+
+## Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
+
+# Training Details
+
+DistilRoBERTa was pre-trained on [OpenWebTextCorpus](https://skylion007.github.io/OpenWebTextCorpus/), a reproduction of OpenAI's WebText dataset (it is ~4 times less training data than the teacher RoBERTa). See the [roberta-base model card](https://huggingface.co/roberta-base/blob/main/README.md) for further details on training.
+
+# Evaluation
+
+When fine-tuned on downstream tasks, this model achieves the following results (see [GitHub Repo](https://github.com/huggingface/transformers/blob/main/examples/research_projects/distillation/README.md)):
+
+Glue test results:
+
+| Task | MNLI | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  |
+|:----:|:----:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|
+|      | 84.0 | 89.4 | 90.8 | 92.5  | 59.3 | 88.3  | 86.6 | 67.9 |
+
+# Environmental Impact
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** More information needed
+- **Hours used:** More information needed
+- **Cloud Provider:** More information needed
+- **Compute Region:** More information needed
+- **Carbon Emitted:** More information needed
+
+# Citation
+
+```bibtex
+@article{Sanh2019DistilBERTAD,
+  title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
+  author={Victor Sanh and Lysandre Debut and Julien Chaumond and Thomas Wolf},
+  journal={ArXiv},
+  year={2019},
+  volume={abs/1910.01108}
+}
+```
+
+APA
+- Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
+
+# How to Get Started With the Model
+
+You can use the model directly with a pipeline for masked language modeling: 
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='distilroberta-base')
+>>> unmasker("Hello I'm a <mask> model.")
+[{'score': 0.04673689603805542,
+  'sequence': "Hello I'm a business model.",
+  'token': 265,
+  'token_str': ' business'},
+ {'score': 0.03846118599176407,
+  'sequence': "Hello I'm a freelance model.",
+  'token': 18150,
+  'token_str': ' freelance'},
+ {'score': 0.03308931365609169,
+  'sequence': "Hello I'm a fashion model.",
+  'token': 2734,
+  'token_str': ' fashion'},
+ {'score': 0.03018997237086296,
+  'sequence': "Hello I'm a role model.",
+  'token': 774,
+  'token_str': ' role'},
+ {'score': 0.02111748233437538,
+  'sequence': "Hello I'm a Playboy model.",
+  'token': 24526,
+  'token_str': ' Playboy'}]
+```
+
+<a href="https://huggingface.co/exbert/?model=distilroberta-base">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>

tests/cards/emilyalsentzer___Bio_ClinicalBERT.md

@@ -0,0 +1,37 @@
+# ClinicalBERT - Bio + Clinical BERT Model
+
+The [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) paper contains four unique clinicalBERT models: initialized with BERT-Base (`cased_L-12_H-768_A-12`) or BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`) & trained on either all MIMIC notes or only discharge summaries. 
+
+This model card describes the Bio+Clinical BERT model, which was initialized from [BioBERT](https://arxiv.org/abs/1901.08746) & trained on all MIMIC notes. 
+
+## Pretraining Data
+The `Bio_ClinicalBERT` model was trained on all notes from [MIMIC III](https://www.nature.com/articles/sdata201635), a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see [here](https://mimic.physionet.org/). All notes from the `NOTEEVENTS` table were included (~880M words).
+
+## Model Pretraining 
+
+### Note Preprocessing
+Each note in MIMIC was first split into sections using a rules-based section splitter (e.g. discharge summary notes were split into "History of Present Illness", "Family History", "Brief Hospital Course", etc. sections). Then each section was split into sentences using SciSpacy (`en core sci md` tokenizer). 
+
+### Pretraining Procedures
+The model was trained using code from [Google's BERT repository](https://github.com/google-research/bert) on a GeForce GTX TITAN X 12 GB GPU. Model parameters were initialized with BioBERT (`BioBERT-Base v1.0 + PubMed 200K + PMC 270K`).
+
+### Pretraining Hyperparameters
+We used a batch size of 32, a maximum sequence length of 128, and a learning rate of 5 · 10−5 for pre-training our models. The models trained on all MIMIC notes  were trained for 150,000 steps. The dup factor for duplicating input data with different masks was set to 5. All other default parameters were used (specifically, masked language model probability = 0.15
+and max predictions per sequence = 20).
+
+## How to use the model
+
+Load the model via the transformers library:
+```
+from transformers import AutoTokenizer, AutoModel
+tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
+model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
+```
+
+## More Information
+
+Refer to the original paper, [Publicly Available Clinical BERT Embeddings](https://arxiv.org/abs/1904.03323) (NAACL Clinical NLP Workshop 2019) for additional details and performance on NLI and NER tasks.
+
+## Questions?
+
+Post a Github issue on the [clinicalBERT repo](https://github.com/EmilyAlsentzer/clinicalBERT) or email [email protected] with any questions.

tests/cards/facebook___bart-large-mnli.md

@@ -0,0 +1,73 @@
+# bart-large-mnli
+
+This is the checkpoint for [bart-large](https://huggingface.co/facebook/bart-large) after being trained on the [MultiNLI (MNLI)](https://huggingface.co/datasets/multi_nli) dataset.
+
+Additional information about this model:
+- The [bart-large](https://huggingface.co/facebook/bart-large) model page
+- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
+](https://arxiv.org/abs/1910.13461)
+- [BART fairseq implementation](https://github.com/pytorch/fairseq/tree/master/fairseq/models/bart)
+
+## NLI-based Zero Shot Text Classification
+
+[Yin et al.](https://arxiv.org/abs/1909.00161) proposed a method for using pre-trained NLI models as a ready-made zero-shot sequence classifiers. The method works by posing the sequence to be classified as the NLI premise and to construct a hypothesis from each candidate label. For example, if we want to evaluate whether a sequence belongs to the class "politics", we could construct a hypothesis of `This text is about politics.`. The probabilities for entailment and contradiction are then converted to label probabilities.
+
+This method is surprisingly effective in many cases, particularly when used with larger pre-trained models like BART and Roberta. See [this blog post](https://joeddav.github.io/blog/2020/05/29/ZSL.html) for a more expansive introduction to this and other zero shot methods, and see the code snippets below for examples of using this model for zero-shot classification both with Hugging Face's built-in pipeline and with native Transformers/PyTorch code.
+
+#### With the zero-shot classification pipeline
+
+The model can be loaded with the `zero-shot-classification` pipeline like so:
+
+```python
+from transformers import pipeline
+classifier = pipeline("zero-shot-classification",
+                      model="facebook/bart-large-mnli")
+```
+
+You can then use this pipeline to classify sequences into any of the class names you specify.
+
+```python
+sequence_to_classify = "one day I will see the world"
+candidate_labels = ['travel', 'cooking', 'dancing']
+classifier(sequence_to_classify, candidate_labels)
+#{'labels': ['travel', 'dancing', 'cooking'],
+# 'scores': [0.9938651323318481, 0.0032737774308770895, 0.002861034357920289],
+# 'sequence': 'one day I will see the world'}
+```
+
+If more than one candidate label can be correct, pass `multi_class=True` to calculate each class independently:
+
+```python
+candidate_labels = ['travel', 'cooking', 'dancing', 'exploration']
+classifier(sequence_to_classify, candidate_labels, multi_class=True)
+#{'labels': ['travel', 'exploration', 'dancing', 'cooking'],
+# 'scores': [0.9945111274719238,
+#  0.9383890628814697,
+#  0.0057061901316046715,
+#  0.0018193122232332826],
+# 'sequence': 'one day I will see the world'}
+```
+
+
+#### With manual PyTorch
+
+```python
+# pose sequence as a NLI premise and label as a hypothesis
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+nli_model = AutoModelForSequenceClassification.from_pretrained('facebook/bart-large-mnli')
+tokenizer = AutoTokenizer.from_pretrained('facebook/bart-large-mnli')
+
+premise = sequence
+hypothesis = f'This example is {label}.'
+
+# run through model pre-trained on MNLI
+x = tokenizer.encode(premise, hypothesis, return_tensors='pt',
+                     truncation_strategy='only_first')
+logits = nli_model(x.to(device))[0]
+
+# we throw away "neutral" (dim 1) and take the probability of
+# "entailment" (2) as the probability of the label being true 
+entail_contradiction_logits = logits[:,[0,2]]
+probs = entail_contradiction_logits.softmax(dim=1)
+prob_label_is_true = probs[:,1]
+```

tests/cards/google___electra-base-discriminator.md

@@ -0,0 +1,29 @@
+## ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
+
+**ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
+
+For a detailed description and experimental results, please refer to our paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
+
+This repository contains code to pre-train ELECTRA, including small ELECTRA models on a single GPU. It also supports fine-tuning ELECTRA on downstream tasks including classification tasks (e.g,. [GLUE](https://gluebenchmark.com/)), QA tasks (e.g., [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)), and sequence tagging tasks (e.g., [text chunking](https://www.clips.uantwerpen.be/conll2000/chunking/)).
+
+## How to use the discriminator in `transformers`
+
+```python
+from transformers import ElectraForPreTraining, ElectraTokenizerFast
+import torch
+
+discriminator = ElectraForPreTraining.from_pretrained("google/electra-base-discriminator")
+tokenizer = ElectraTokenizerFast.from_pretrained("google/electra-base-discriminator")
+
+sentence = "The quick brown fox jumps over the lazy dog"
+fake_sentence = "The quick brown fox fake over the lazy dog"
+
+fake_tokens = tokenizer.tokenize(fake_sentence)
+fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt")
+discriminator_outputs = discriminator(fake_inputs)
+predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2)
+
+[print("%7s" % token, end="") for token in fake_tokens]
+
+[print("%7s" % int(prediction), end="") for prediction in predictions.tolist()]
+```

tests/cards/gpt2.md

@@ -0,0 +1,158 @@
+# GPT-2
+
+Test the whole generation capabilities here: https://transformer.huggingface.co/doc/gpt2-large
+
+Pretrained model on English language using a causal language modeling (CLM) objective. It was introduced in
+[this paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
+and first released at [this page](https://openai.com/blog/better-language-models/).
+
+Disclaimer: The team releasing GPT-2 also wrote a
+[model card](https://github.com/openai/gpt-2/blob/master/model_card.md) for their model. Content from this model card
+has been written by the Hugging Face team to complete the information they provided and give specific examples of bias.
+
+## Model description
+
+GPT-2 is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. This
+means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots
+of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely,
+it was trained to guess the next word in sentences.
+
+More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence,
+shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the
+predictions for the token `i` only uses the inputs from `1` to `i` but not the future tokens.
+
+This way, the model learns an inner representation of the English language that can then be used to extract features
+useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a
+prompt.
+
+This is the **smallest** version of GPT-2, with 124M parameters. 
+
+**Related Models:** [GPT-Large](https://huggingface.co/gpt2-large), [GPT-Medium](https://huggingface.co/gpt2-medium) and [GPT-XL](https://huggingface.co/gpt2-xl)
+
+## Intended uses & limitations
+
+You can use the raw model for text generation or fine-tune it to a downstream task. See the
+[model hub](https://huggingface.co/models?filter=gpt2) to look for fine-tuned versions on a task that interests you.
+
+### How to use
+
+You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we
+set a seed for reproducibility:
+
+```python
+>>> from transformers import pipeline, set_seed
+>>> generator = pipeline('text-generation', model='gpt2')
+>>> set_seed(42)
+>>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)
+
+[{'generated_text': "Hello, I'm a language model, a language for thinking, a language for expressing thoughts."},
+ {'generated_text': "Hello, I'm a language model, a compiler, a compiler library, I just want to know how I build this kind of stuff. I don"},
+ {'generated_text': "Hello, I'm a language model, and also have more than a few of your own, but I understand that they're going to need some help"},
+ {'generated_text': "Hello, I'm a language model, a system model. I want to know my language so that it might be more interesting, more user-friendly"},
+ {'generated_text': 'Hello, I\'m a language model, not a language model"\n\nThe concept of "no-tricks" comes in handy later with new'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import GPT2Tokenizer, GPT2Model
+tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
+model = GPT2Model.from_pretrained('gpt2')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import GPT2Tokenizer, TFGPT2Model
+tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
+model = TFGPT2Model.from_pretrained('gpt2')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+### Limitations and bias
+
+The training data used for this model has not been released as a dataset one can browse. We know it contains a lot of
+unfiltered content from the internet, which is far from neutral. As the openAI team themselves point out in their
+[model card](https://github.com/openai/gpt-2/blob/master/model_card.md#out-of-scope-use-cases):
+
+> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases
+> that require the generated text to be true.
+>
+> Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do
+> not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a
+> study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race,
+> and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar
+> levels of caution around use cases that are sensitive to biases around human attributes.
+
+Here's an example of how the model can have biased predictions:
+
+```python
+>>> from transformers import pipeline, set_seed
+>>> generator = pipeline('text-generation', model='gpt2')
+>>> set_seed(42)
+>>> generator("The White man worked as a", max_length=10, num_return_sequences=5)
+
+[{'generated_text': 'The White man worked as a mannequin for'},
+ {'generated_text': 'The White man worked as a maniser of the'},
+ {'generated_text': 'The White man worked as a bus conductor by day'},
+ {'generated_text': 'The White man worked as a plumber at the'},
+ {'generated_text': 'The White man worked as a journalist. He had'}]
+
+>>> set_seed(42)
+>>> generator("The Black man worked as a", max_length=10, num_return_sequences=5)
+
+[{'generated_text': 'The Black man worked as a man at a restaurant'},
+ {'generated_text': 'The Black man worked as a car salesman in a'},
+ {'generated_text': 'The Black man worked as a police sergeant at the'},
+ {'generated_text': 'The Black man worked as a man-eating monster'},
+ {'generated_text': 'The Black man worked as a slave, and was'}]
+```
+
+This bias will also affect all fine-tuned versions of this model.
+
+## Training data
+
+The OpenAI team wanted to train this model on a corpus as large as possible. To build it, they scraped all the web
+pages from outbound links on Reddit which received at least 3 karma. Note that all Wikipedia pages were removed from
+this dataset, so the model was not trained on any part of Wikipedia. The resulting dataset (called WebText) weights
+40GB of texts but has not been publicly released. You can find a list of the top 1,000 domains present in WebText
+[here](https://github.com/openai/gpt-2/blob/master/domains.txt).
+
+## Training procedure
+
+### Preprocessing
+
+The texts are tokenized using a byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a
+vocabulary size of 50,257. The inputs are sequences of 1024 consecutive tokens.
+
+The larger model was trained on 256 cloud TPU v3 cores. The training duration was not disclosed, nor were the exact
+details of training.
+
+## Evaluation results
+
+The model achieves the following results without any fine-tuning (zero-shot):
+
+| Dataset  | LAMBADA | LAMBADA | CBT-CN | CBT-NE | WikiText2 | PTB    | enwiki8 | text8  | WikiText103 | 1BW   |
+|:--------:|:-------:|:-------:|:------:|:------:|:---------:|:------:|:-------:|:------:|:-----------:|:-----:|
+| (metric) | (PPL)   | (ACC)   | (ACC)  | (ACC)  | (PPL)     | (PPL)  | (BPB)   | (BPC)  | (PPL)       | (PPL) |
+|          | 35.13   | 45.99   | 87.65  | 83.4   | 29.41     | 65.85  | 1.16    | 1,17   | 37.50       | 75.20 |
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{radford2019language,
+  title={Language Models are Unsupervised Multitask Learners},
+  author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
+  year={2019}
+}
+```
+
+<a href="https://huggingface.co/exbert/?model=gpt2">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>

tests/cards/jonatasgrosman___wav2vec2-large-xlsr-53-english.md

@@ -0,0 +1,102 @@
+# Fine-tuned XLSR-53 large model for speech recognition in English
+
+Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on English using the train and validation splits of [Common Voice 6.1](https://huggingface.co/datasets/common_voice).
+When using this model, make sure that your speech input is sampled at 16kHz.
+
+This model has been fine-tuned thanks to the GPU credits generously given by the [OVHcloud](https://www.ovhcloud.com/en/public-cloud/ai-training/) :)
+
+The script used for training can be found here: https://github.com/jonatasgrosman/wav2vec2-sprint
+
+## Usage
+
+The model can be used directly (without a language model) as follows...
+
+Using the [HuggingSound](https://github.com/jonatasgrosman/huggingsound) library:
+
+```python
+from huggingsound import SpeechRecognitionModel
+
+model = SpeechRecognitionModel("jonatasgrosman/wav2vec2-large-xlsr-53-english")
+audio_paths = ["/path/to/file.mp3", "/path/to/another_file.wav"]
+
+transcriptions = model.transcribe(audio_paths)
+```
+
+Writing your own inference script:
+
+```python
+import torch
+import librosa
+from datasets import load_dataset
+from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
+
+LANG_ID = "en"
+MODEL_ID = "jonatasgrosman/wav2vec2-large-xlsr-53-english"
+SAMPLES = 10
+
+test_dataset = load_dataset("common_voice", LANG_ID, split=f"test[:{SAMPLES}]")
+
+processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
+model = Wav2Vec2ForCTC.from_pretrained(MODEL_ID)
+
+# Preprocessing the datasets.
+# We need to read the audio files as arrays
+def speech_file_to_array_fn(batch):
+    speech_array, sampling_rate = librosa.load(batch["path"], sr=16_000)
+    batch["speech"] = speech_array
+    batch["sentence"] = batch["sentence"].upper()
+    return batch
+
+test_dataset = test_dataset.map(speech_file_to_array_fn)
+inputs = processor(test_dataset["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
+
+with torch.no_grad():
+    logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
+
+predicted_ids = torch.argmax(logits, dim=-1)
+predicted_sentences = processor.batch_decode(predicted_ids)
+
+for i, predicted_sentence in enumerate(predicted_sentences):
+    print("-" * 100)
+    print("Reference:", test_dataset[i]["sentence"])
+    print("Prediction:", predicted_sentence)
+```
+
+| Reference  | Prediction |
+| ------------- | ------------- |
+| "SHE'LL BE ALL RIGHT." | SHE'LL BE ALL RIGHT |
+| SIX | SIX |
+| "ALL'S WELL THAT ENDS WELL." | ALL AS WELL THAT ENDS WELL |
+| DO YOU MEAN IT? | DO YOU MEAN IT |
+| THE NEW PATCH IS LESS INVASIVE THAN THE OLD ONE, BUT STILL CAUSES REGRESSIONS. | THE NEW PATCH IS LESS INVASIVE THAN THE OLD ONE BUT STILL CAUSES REGRESSION |
+| HOW IS MOZILLA GOING TO HANDLE AMBIGUITIES LIKE QUEUE AND CUE? | HOW IS MOSLILLAR GOING TO HANDLE ANDBEWOOTH HIS LIKE Q AND Q |
+| "I GUESS YOU MUST THINK I'M KINDA BATTY." | RUSTIAN WASTIN PAN ONTE BATTLY |
+| NO ONE NEAR THE REMOTE MACHINE YOU COULD RING? | NO ONE NEAR THE REMOTE MACHINE YOU COULD RING |
+| SAUCE FOR THE GOOSE IS SAUCE FOR THE GANDER. | SAUCE FOR THE GUICE IS SAUCE FOR THE GONDER |
+| GROVES STARTED WRITING SONGS WHEN SHE WAS FOUR YEARS OLD. | GRAFS STARTED WRITING SONGS WHEN SHE WAS FOUR YEARS OLD |
+
+## Evaluation
+
+1. To evaluate on `mozilla-foundation/common_voice_6_0` with split `test`
+
+```bash
+python eval.py --model_id jonatasgrosman/wav2vec2-large-xlsr-53-english --dataset mozilla-foundation/common_voice_6_0 --config en --split test
+```
+
+2. To evaluate on `speech-recognition-community-v2/dev_data`
+
+```bash
+python eval.py --model_id jonatasgrosman/wav2vec2-large-xlsr-53-english --dataset speech-recognition-community-v2/dev_data --config en --split validation --chunk_length_s 5.0 --stride_length_s 1.0
+```
+
+## Citation
+If you want to cite this model you can use this:
+
+```bibtex
+@misc{grosman2021xlsr53-large-english,
+  title={Fine-tuned {XLSR}-53 large model for speech recognition in {E}nglish},
+  author={Grosman, Jonatas},
+  howpublished={\url{https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-english}},
+  year={2021}
+}
+```

tests/cards/microsoft___layoutlmv3-base.md

@@ -0,0 +1,29 @@
+# LayoutLMv3
+
+[Microsoft Document AI](https://www.microsoft.com/en-us/research/project/document-ai/) | [GitHub](https://aka.ms/layoutlmv3)
+
+## Model description
+
+LayoutLMv3 is a pre-trained multimodal Transformer for Document AI with unified text and image masking. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model. For example, LayoutLMv3 can be fine-tuned for both text-centric tasks, including form understanding, receipt understanding, and document visual question answering, and image-centric tasks such as document image classification and document layout analysis.
+
+[LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://arxiv.org/abs/2204.08387)
+Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei, ACM Multimedia 2022.
+
+## Citation
+
+If you find LayoutLM useful in your research, please cite the following paper:
+
+```
+@inproceedings{huang2022layoutlmv3,
+  author={Yupan Huang and Tengchao Lv and Lei Cui and Yutong Lu and Furu Wei},
+  title={LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking},
+  booktitle={Proceedings of the 30th ACM International Conference on Multimedia},
+  year={2022}
+}
+```
+
+## License
+
+The content of this project itself is licensed under the [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/).
+Portions of the source code are based on the [transformers](https://github.com/huggingface/transformers) project.
+[Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct)

tests/cards/openai___clip-vit-base-patch32.md

@@ -0,0 +1,136 @@
+# Model Card: CLIP
+
+Disclaimer: The model card is taken and modified from the official CLIP repository, it can be found [here](https://github.com/openai/CLIP/blob/main/model-card.md).
+
+## Model Details
+
+The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within.
+
+### Model Date
+
+January 2021
+
+### Model Type
+
+The model uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss. 
+
+The original implementation had two variants: one using a ResNet image encoder and the other using a Vision Transformer. This repository has the variant with the Vision Transformer.
+
+
+### Documents
+
+- [Blog Post](https://openai.com/blog/clip/)
+- [CLIP Paper](https://arxiv.org/abs/2103.00020)
+
+
+### Use with Transformers
+
+```python3
+from PIL import Image
+import requests
+
+from transformers import CLIPProcessor, CLIPModel
+
+model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
+processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
+
+url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+image = Image.open(requests.get(url, stream=True).raw)
+
+inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)
+
+outputs = model(**inputs)
+logits_per_image = outputs.logits_per_image # this is the image-text similarity score
+probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
+```
+
+
+## Model Use
+
+### Intended Use
+
+The model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such models - the CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis.
+
+#### Primary intended uses
+
+The primary intended users of these models are AI researchers.
+
+We primarily imagine the model will be used by researchers to better understand robustness, generalization, and other capabilities, biases, and constraints of computer vision models.
+
+### Out-of-Scope Use Cases
+
+**Any** deployed use case of the model - whether commercial or not - is currently out of scope. Non-deployed use cases such as image search in a constrained environment, are also not recommended unless there is thorough in-domain testing of the model with a specific, fixed class taxonomy. This is because our safety assessment demonstrated a high need for task specific testing especially given the variability of CLIP’s performance with different class taxonomies. This makes untested and unconstrained deployment of the model in any use case currently potentially harmful. 
+
+Certain use cases which would fall under the domain of surveillance and facial recognition are always out-of-scope regardless of performance of the model. This is because the use of artificial intelligence for tasks such as these can be premature currently given the lack of testing norms and checks to ensure its fair use.
+
+Since the model has not been purposefully trained in or evaluated on any languages other than English, its use should be limited to English language use cases.
+
+
+
+## Data
+
+The model was trained on publicly available image-caption data. This was done through a combination of crawling a handful of websites and using commonly-used pre-existing image datasets such as [YFCC100M](http://projects.dfki.uni-kl.de/yfcc100m/). A large portion of the data comes from our crawling of the internet. This means that the data is more representative of people and societies most connected to the internet which tend to skew towards more developed nations, and younger, male users.
+
+### Data Mission Statement
+
+Our goal with building this dataset was to test out robustness and generalizability in computer vision tasks. As a result, the focus was on gathering large quantities of data from different publicly-available internet data sources. The data was gathered in a mostly non-interventionist manner. However, we only crawled websites that had policies against excessively violent and adult images and allowed us to filter out such content. We do not intend for this dataset to be used as the basis for any commercial or deployed model and will not be releasing the dataset.
+
+
+
+## Performance and Limitations
+
+### Performance
+
+We have evaluated the performance of CLIP on a wide range of benchmarks across a variety of computer vision datasets such as OCR to texture recognition to fine-grained classification. The paper describes model performance on the following datasets:
+
+- Food101
+- CIFAR10   
+- CIFAR100   
+- Birdsnap
+- SUN397
+- Stanford Cars
+- FGVC Aircraft
+- VOC2007
+- DTD
+- Oxford-IIIT Pet dataset
+- Caltech101
+- Flowers102
+- MNIST   
+- SVHN 
+- IIIT5K   
+- Hateful Memes   
+- SST-2
+- UCF101
+- Kinetics700
+- Country211
+- CLEVR Counting
+- KITTI Distance
+- STL-10
+- RareAct
+- Flickr30
+- MSCOCO
+- ImageNet
+- ImageNet-A
+- ImageNet-R
+- ImageNet Sketch
+- ObjectNet (ImageNet Overlap)
+- Youtube-BB
+- ImageNet-Vid
+
+## Limitations
+
+CLIP and our analysis of it have a number of limitations. CLIP currently struggles with respect to certain tasks such as fine grained classification and counting objects. CLIP also poses issues with regards to fairness and bias which we discuss in the paper and briefly in the next section. Additionally, our approach to testing CLIP also has an important limitation- in many cases we have used linear probes to evaluate the performance of CLIP and there is evidence suggesting that linear probes can underestimate model performance.
+
+### Bias and Fairness
+
+We find that the performance of CLIP - and the specific biases it exhibits - can depend significantly on class design and the choices one makes for categories to include and exclude. We tested the risk of certain kinds of denigration with CLIP by classifying images of people from [Fairface](https://arxiv.org/abs/1908.04913) into crime-related and non-human animal categories. We found significant disparities with respect to race and gender. Additionally, we found that these disparities could shift based on how the classes were constructed. (Details captured in the Broader Impacts Section in the paper).
+
+We also tested the performance of CLIP on gender, race and age classification using the Fairface dataset (We default to using race categories as they are constructed in the Fairface dataset.) in order to assess quality of performance across different demographics. We found accuracy >96% across all races for gender classification with ‘Middle Eastern’ having the highest accuracy (98.4%) and ‘White’ having the lowest (96.5%). Additionally, CLIP averaged ~93% for racial classification and ~63% for age classification. Our use of evaluations to test for gender, race and age classification as well as denigration harms is simply to evaluate performance of the model across people and surface potential risks and not to demonstrate an endorsement/enthusiasm for such tasks.
+
+
+
+## Feedback
+
+### Where to send questions or comments about the model
+
+Please use [this Google Form](https://forms.gle/Uv7afRH5dvY34ZEs9)

tests/cards/openai___clip-vit-large-patch14.md

@@ -0,0 +1,136 @@
+# Model Card: CLIP
+
+Disclaimer: The model card is taken and modified from the official CLIP repository, it can be found [here](https://github.com/openai/CLIP/blob/main/model-card.md).
+
+## Model Details
+
+The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within.
+
+### Model Date
+
+January 2021
+
+### Model Type
+
+The base model uses a ViT-L/14 Transformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.
+
+The original implementation had two variants: one using a ResNet image encoder and the other using a Vision Transformer. This repository has the variant with the Vision Transformer.
+
+
+### Documents
+
+- [Blog Post](https://openai.com/blog/clip/)
+- [CLIP Paper](https://arxiv.org/abs/2103.00020)
+
+
+### Use with Transformers
+
+```python
+from PIL import Image
+import requests
+
+from transformers import CLIPProcessor, CLIPModel
+
+model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
+processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
+
+url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+image = Image.open(requests.get(url, stream=True).raw)
+
+inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)
+
+outputs = model(**inputs)
+logits_per_image = outputs.logits_per_image # this is the image-text similarity score
+probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
+```
+
+
+## Model Use
+
+### Intended Use
+
+The model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such models - the CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis.
+
+#### Primary intended uses
+
+The primary intended users of these models are AI researchers.
+
+We primarily imagine the model will be used by researchers to better understand robustness, generalization, and other capabilities, biases, and constraints of computer vision models.
+
+### Out-of-Scope Use Cases
+
+**Any** deployed use case of the model - whether commercial or not - is currently out of scope. Non-deployed use cases such as image search in a constrained environment, are also not recommended unless there is thorough in-domain testing of the model with a specific, fixed class taxonomy. This is because our safety assessment demonstrated a high need for task specific testing especially given the variability of CLIP’s performance with different class taxonomies. This makes untested and unconstrained deployment of the model in any use case currently potentially harmful. 
+
+Certain use cases which would fall under the domain of surveillance and facial recognition are always out-of-scope regardless of performance of the model. This is because the use of artificial intelligence for tasks such as these can be premature currently given the lack of testing norms and checks to ensure its fair use.
+
+Since the model has not been purposefully trained in or evaluated on any languages other than English, its use should be limited to English language use cases.
+
+
+
+## Data
+
+The model was trained on publicly available image-caption data. This was done through a combination of crawling a handful of websites and using commonly-used pre-existing image datasets such as [YFCC100M](http://projects.dfki.uni-kl.de/yfcc100m/). A large portion of the data comes from our crawling of the internet. This means that the data is more representative of people and societies most connected to the internet which tend to skew towards more developed nations, and younger, male users.
+
+### Data Mission Statement
+
+Our goal with building this dataset was to test out robustness and generalizability in computer vision tasks. As a result, the focus was on gathering large quantities of data from different publicly-available internet data sources. The data was gathered in a mostly non-interventionist manner. However, we only crawled websites that had policies against excessively violent and adult images and allowed us to filter out such content. We do not intend for this dataset to be used as the basis for any commercial or deployed model and will not be releasing the dataset.
+
+
+
+## Performance and Limitations
+
+### Performance
+
+We have evaluated the performance of CLIP on a wide range of benchmarks across a variety of computer vision datasets such as OCR to texture recognition to fine-grained classification. The paper describes model performance on the following datasets:
+
+- Food101
+- CIFAR10   
+- CIFAR100   
+- Birdsnap
+- SUN397
+- Stanford Cars
+- FGVC Aircraft
+- VOC2007
+- DTD
+- Oxford-IIIT Pet dataset
+- Caltech101
+- Flowers102
+- MNIST   
+- SVHN 
+- IIIT5K   
+- Hateful Memes   
+- SST-2
+- UCF101
+- Kinetics700
+- Country211
+- CLEVR Counting
+- KITTI Distance
+- STL-10
+- RareAct
+- Flickr30
+- MSCOCO
+- ImageNet
+- ImageNet-A
+- ImageNet-R
+- ImageNet Sketch
+- ObjectNet (ImageNet Overlap)
+- Youtube-BB
+- ImageNet-Vid
+
+## Limitations
+
+CLIP and our analysis of it have a number of limitations. CLIP currently struggles with respect to certain tasks such as fine grained classification and counting objects. CLIP also poses issues with regards to fairness and bias which we discuss in the paper and briefly in the next section. Additionally, our approach to testing CLIP also has an important limitation- in many cases we have used linear probes to evaluate the performance of CLIP and there is evidence suggesting that linear probes can underestimate model performance.
+
+### Bias and Fairness
+
+We find that the performance of CLIP - and the specific biases it exhibits - can depend significantly on class design and the choices one makes for categories to include and exclude. We tested the risk of certain kinds of denigration with CLIP by classifying images of people from [Fairface](https://arxiv.org/abs/1908.04913) into crime-related and non-human animal categories. We found significant disparities with respect to race and gender. Additionally, we found that these disparities could shift based on how the classes were constructed. (Details captured in the Broader Impacts Section in the paper).
+
+We also tested the performance of CLIP on gender, race and age classification using the Fairface dataset (We default to using race categories as they are constructed in the Fairface dataset.) in order to assess quality of performance across different demographics. We found accuracy >96% across all races for gender classification with ‘Middle Eastern’ having the highest accuracy (98.4%) and ‘White’ having the lowest (96.5%). Additionally, CLIP averaged ~93% for racial classification and ~63% for age classification. Our use of evaluations to test for gender, race and age classification as well as denigration harms is simply to evaluate performance of the model across people and surface potential risks and not to demonstrate an endorsement/enthusiasm for such tasks.
+
+
+
+## Feedback
+
+### Where to send questions or comments about the model
+
+Please use [this Google Form](https://forms.gle/Uv7afRH5dvY34ZEs9)

tests/cards/philschmid___bart-large-cnn-samsum.md

@@ -0,0 +1,62 @@
+## `bart-large-cnn-samsum`
+
+> If you want to use the model you should try a newer fine-tuned FLAN-T5 version [philschmid/flan-t5-base-samsum](https://huggingface.co/philschmid/flan-t5-base-samsum) out socring the BART version with `+6` on `ROGUE1` achieving `47.24`.
+
+# TRY [philschmid/flan-t5-base-samsum](https://huggingface.co/philschmid/flan-t5-base-samsum)
+
+
+This model was trained using Amazon SageMaker and the new Hugging Face Deep Learning container.
+
+For more information look at:
+- [🤗 Transformers Documentation: Amazon SageMaker](https://huggingface.co/transformers/sagemaker.html)
+- [Example Notebooks](https://github.com/huggingface/notebooks/tree/master/sagemaker)
+- [Amazon SageMaker documentation for Hugging Face](https://docs.aws.amazon.com/sagemaker/latest/dg/hugging-face.html)
+- [Python SDK SageMaker documentation for Hugging Face](https://sagemaker.readthedocs.io/en/stable/frameworks/huggingface/index.html)
+- [Deep Learning Container](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#huggingface-training-containers)
+
+## Hyperparameters
+```json
+{
+    "dataset_name": "samsum",
+    "do_eval": true,
+    "do_predict": true,
+    "do_train": true,
+    "fp16": true,
+    "learning_rate": 5e-05,
+    "model_name_or_path": "facebook/bart-large-cnn",
+    "num_train_epochs": 3,
+    "output_dir": "/opt/ml/model",
+    "per_device_eval_batch_size": 4,
+    "per_device_train_batch_size": 4,
+    "predict_with_generate": true,
+    "seed": 7
+}
+```
+
+## Usage
+```python
+from transformers import pipeline
+summarizer = pipeline("summarization", model="philschmid/bart-large-cnn-samsum")
+
+conversation = '''Jeff: Can I train a 🤗 Transformers model on Amazon SageMaker? 
+Philipp: Sure you can use the new Hugging Face Deep Learning Container. 
+Jeff: ok.
+Jeff: and how can I get started? 
+Jeff: where can I find documentation? 
+Philipp: ok, ok you can find everything here. https://huggingface.co/blog/the-partnership-amazon-sagemaker-and-hugging-face                                           
+'''
+summarizer(conversation)
+```
+
+## Results
+
+| key | value |
+| --- | ----- |
+| eval_rouge1 | 42.621 |
+| eval_rouge2 | 21.9825 |
+| eval_rougeL | 33.034 |
+| eval_rougeLsum | 39.6783 |
+| test_rouge1 | 41.3174 |
+| test_rouge2 | 20.8716 |
+| test_rougeL | 32.1337 |
+| test_rougeLsum | 38.4149 |

tests/cards/prajjwal1___bert-tiny.md

@@ -0,0 +1,46 @@
+The following model is a Pytorch pre-trained model obtained from converting Tensorflow checkpoint found in the [official Google BERT repository](https://github.com/google-research/bert). 
+
+This is one of the smaller pre-trained BERT variants, together with [bert-mini](https://huggingface.co/prajjwal1/bert-mini) [bert-small](https://huggingface.co/prajjwal1/bert-small) and [bert-medium](https://huggingface.co/prajjwal1/bert-medium). They were introduced in the study `Well-Read Students Learn Better: On the Importance of Pre-training Compact Models` ([arxiv](https://arxiv.org/abs/1908.08962)), and ported to HF for the study `Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics` ([arXiv](https://arxiv.org/abs/2110.01518)). These models are supposed to be trained on a downstream task.
+
+If you use the model, please consider citing both the papers:
+```
+@misc{bhargava2021generalization,
+      title={Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics}, 
+      author={Prajjwal Bhargava and Aleksandr Drozd and Anna Rogers},
+      year={2021},
+      eprint={2110.01518},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+
+@article{DBLP:journals/corr/abs-1908-08962,
+  author    = {Iulia Turc and
+               Ming{-}Wei Chang and
+               Kenton Lee and
+               Kristina Toutanova},
+  title     = {Well-Read Students Learn Better: The Impact of Student Initialization
+               on Knowledge Distillation},
+  journal   = {CoRR},
+  volume    = {abs/1908.08962},
+  year      = {2019},
+  url       = {http://arxiv.org/abs/1908.08962},
+  eprinttype = {arXiv},
+  eprint    = {1908.08962},
+  timestamp = {Thu, 29 Aug 2019 16:32:34 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1908-08962.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+
+```
+Config of this model:
+- `prajjwal1/bert-tiny` (L=2, H=128) [Model Link](https://huggingface.co/prajjwal1/bert-tiny)
+
+
+Other models to check out:
+- `prajjwal1/bert-mini` (L=4, H=256) [Model Link](https://huggingface.co/prajjwal1/bert-mini)
+- `prajjwal1/bert-small` (L=4, H=512) [Model Link](https://huggingface.co/prajjwal1/bert-small)
+- `prajjwal1/bert-medium` (L=8, H=512) [Model Link](https://huggingface.co/prajjwal1/bert-medium)
+
+Original Implementation and more info can be found in [this Github repository](https://github.com/prajjwal1/generalize_lm_nli).
+
+Twitter: [@prajjwal_1](https://twitter.com/prajjwal_1)

tests/cards/roberta-base.md

@@ -0,0 +1,224 @@
+# RoBERTa base model
+
+Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
+[this paper](https://arxiv.org/abs/1907.11692) and first released in
+[this repository](https://github.com/pytorch/fairseq/tree/master/examples/roberta). This model is case-sensitive: it
+makes a difference between english and English.
+
+Disclaimer: The team releasing RoBERTa did not write a model card for this model so this model card has been written by
+the Hugging Face team.
+
+## Model description
+
+RoBERTa is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means
+it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts. 
+
+More precisely, it was pretrained with the Masked language modeling (MLM) objective. Taking a sentence, the model
+randomly masks 15% of the words in the input then run the entire masked sentence through the model and has to predict
+the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one
+after the other, or from autoregressive models like GPT which internally mask the future tokens. It allows the model to
+learn a bidirectional representation of the sentence.
+
+This way, the model learns an inner representation of the English language that can then be used to extract features
+useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
+classifier using the features produced by the BERT model as inputs.
+
+## Intended uses & limitations
+
+You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
+See the [model hub](https://huggingface.co/models?filter=roberta) to look for fine-tuned versions on a task that
+interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at a model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='roberta-base')
+>>> unmasker("Hello I'm a <mask> model.")
+
+[{'sequence': "<s>Hello I'm a male model.</s>",
+  'score': 0.3306540250778198,
+  'token': 2943,
+  'token_str': 'Ġmale'},
+ {'sequence': "<s>Hello I'm a female model.</s>",
+  'score': 0.04655390977859497,
+  'token': 2182,
+  'token_str': 'Ġfemale'},
+ {'sequence': "<s>Hello I'm a professional model.</s>",
+  'score': 0.04232972860336304,
+  'token': 2038,
+  'token_str': 'Ġprofessional'},
+ {'sequence': "<s>Hello I'm a fashion model.</s>",
+  'score': 0.037216778844594955,
+  'token': 2734,
+  'token_str': 'Ġfashion'},
+ {'sequence': "<s>Hello I'm a Russian model.</s>",
+  'score': 0.03253649175167084,
+  'token': 1083,
+  'token_str': 'ĠRussian'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import RobertaTokenizer, RobertaModel
+tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
+model = RobertaModel.from_pretrained('roberta-base')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import RobertaTokenizer, TFRobertaModel
+tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
+model = TFRobertaModel.from_pretrained('roberta-base')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+### Limitations and bias
+
+The training data used for this model contains a lot of unfiltered content from the internet, which is far from
+neutral. Therefore, the model can have biased predictions:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='roberta-base')
+>>> unmasker("The man worked as a <mask>.")
+
+[{'sequence': '<s>The man worked as a mechanic.</s>',
+  'score': 0.08702439814805984,
+  'token': 25682,
+  'token_str': 'Ġmechanic'},
+ {'sequence': '<s>The man worked as a waiter.</s>',
+  'score': 0.0819653645157814,
+  'token': 38233,
+  'token_str': 'Ġwaiter'},
+ {'sequence': '<s>The man worked as a butcher.</s>',
+  'score': 0.073323555290699,
+  'token': 32364,
+  'token_str': 'Ġbutcher'},
+ {'sequence': '<s>The man worked as a miner.</s>',
+  'score': 0.046322137117385864,
+  'token': 18678,
+  'token_str': 'Ġminer'},
+ {'sequence': '<s>The man worked as a guard.</s>',
+  'score': 0.040150221437215805,
+  'token': 2510,
+  'token_str': 'Ġguard'}]
+
+>>> unmasker("The Black woman worked as a <mask>.")
+
+[{'sequence': '<s>The Black woman worked as a waitress.</s>',
+  'score': 0.22177888453006744,
+  'token': 35698,
+  'token_str': 'Ġwaitress'},
+ {'sequence': '<s>The Black woman worked as a prostitute.</s>',
+  'score': 0.19288744032382965,
+  'token': 36289,
+  'token_str': 'Ġprostitute'},
+ {'sequence': '<s>The Black woman worked as a maid.</s>',
+  'score': 0.06498628109693527,
+  'token': 29754,
+  'token_str': 'Ġmaid'},
+ {'sequence': '<s>The Black woman worked as a secretary.</s>',
+  'score': 0.05375480651855469,
+  'token': 2971,
+  'token_str': 'Ġsecretary'},
+ {'sequence': '<s>The Black woman worked as a nurse.</s>',
+  'score': 0.05245552211999893,
+  'token': 9008,
+  'token_str': 'Ġnurse'}]
+```
+
+This bias will also affect all fine-tuned versions of this model.
+
+## Training data
+
+The RoBERTa model was pretrained on the reunion of five datasets:
+- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books;
+- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers) ;
+- [CC-News](https://commoncrawl.org/2016/10/news-dataset-available/), a dataset containing 63 millions English news
+  articles crawled between September 2016 and February 2019.
+- [OpenWebText](https://github.com/jcpeterson/openwebtext), an opensource recreation of the WebText dataset used to
+  train GPT-2,
+- [Stories](https://arxiv.org/abs/1806.02847) a dataset containing a subset of CommonCrawl data filtered to match the
+  story-like style of Winograd schemas.
+
+Together these datasets weigh 160GB of text.
+
+## Training procedure
+
+### Preprocessing
+
+The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50,000. The inputs of
+the model take pieces of 512 contiguous tokens that may span over documents. The beginning of a new document is marked
+with `<s>` and the end of one by `</s>`
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `<mask>`.
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).
+
+### Pretraining
+
+The model was trained on 1024 V100 GPUs for 500K steps with a batch size of 8K and a sequence length of 512. The
+optimizer used is Adam with a learning rate of 6e-4, \\(\beta_{1} = 0.9\\), \\(\beta_{2} = 0.98\\) and
+\\(\epsilon = 1e-6\\), a weight decay of 0.01, learning rate warmup for 24,000 steps and linear decay of the learning
+rate after.
+
+## Evaluation results
+
+When fine-tuned on downstream tasks, this model achieves the following results:
+
+Glue test results:
+
+| Task | MNLI | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  |
+|:----:|:----:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|
+|      | 87.6 | 91.9 | 92.8 | 94.8  | 63.6 | 91.2  | 90.2 | 78.7 |
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{DBLP:journals/corr/abs-1907-11692,
+  author    = {Yinhan Liu and
+               Myle Ott and
+               Naman Goyal and
+               Jingfei Du and
+               Mandar Joshi and
+               Danqi Chen and
+               Omer Levy and
+               Mike Lewis and
+               Luke Zettlemoyer and
+               Veselin Stoyanov},
+  title     = {RoBERTa: {A} Robustly Optimized {BERT} Pretraining Approach},
+  journal   = {CoRR},
+  volume    = {abs/1907.11692},
+  year      = {2019},
+  url       = {http://arxiv.org/abs/1907.11692},
+  archivePrefix = {arXiv},
+  eprint    = {1907.11692},
+  timestamp = {Thu, 01 Aug 2019 08:59:33 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1907-11692.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<a href="https://huggingface.co/exbert/?model=roberta-base">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>

tests/cards/roberta-large.md

@@ -0,0 +1,225 @@
+# RoBERTa large model
+
+Pretrained model on English language using a masked language modeling (MLM) objective. It was introduced in
+[this paper](https://arxiv.org/abs/1907.11692) and first released in
+[this repository](https://github.com/pytorch/fairseq/tree/master/examples/roberta). This model is case-sensitive: it
+makes a difference between english and English.
+
+Disclaimer: The team releasing RoBERTa did not write a model card for this model so this model card has been written by
+the Hugging Face team.
+
+## Model description
+
+RoBERTa is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means
+it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts. 
+
+More precisely, it was pretrained with the Masked language modeling (MLM) objective. Taking a sentence, the model
+randomly masks 15% of the words in the input then run the entire masked sentence through the model and has to predict
+the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one
+after the other, or from autoregressive models like GPT which internally mask the future tokens. It allows the model to
+learn a bidirectional representation of the sentence.
+
+This way, the model learns an inner representation of the English language that can then be used to extract features
+useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
+classifier using the features produced by the BERT model as inputs.
+
+## Intended uses & limitations
+
+You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
+See the [model hub](https://huggingface.co/models?filter=roberta) to look for fine-tuned versions on a task that
+interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='roberta-large')
+>>> unmasker("Hello I'm a <mask> model.")
+
+[{'sequence': "<s>Hello I'm a male model.</s>",
+  'score': 0.3317350447177887,
+  'token': 2943,
+  'token_str': 'Ġmale'},
+ {'sequence': "<s>Hello I'm a fashion model.</s>",
+  'score': 0.14171843230724335,
+  'token': 2734,
+  'token_str': 'Ġfashion'},
+ {'sequence': "<s>Hello I'm a professional model.</s>",
+  'score': 0.04291723668575287,
+  'token': 2038,
+  'token_str': 'Ġprofessional'},
+ {'sequence': "<s>Hello I'm a freelance model.</s>",
+  'score': 0.02134818211197853,
+  'token': 18150,
+  'token_str': 'Ġfreelance'},
+ {'sequence': "<s>Hello I'm a young model.</s>",
+  'score': 0.021098261699080467,
+  'token': 664,
+  'token_str': 'Ġyoung'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import RobertaTokenizer, RobertaModel
+tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
+model = RobertaModel.from_pretrained('roberta-large')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import RobertaTokenizer, TFRobertaModel
+tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
+model = TFRobertaModel.from_pretrained('roberta-large')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+### Limitations and bias
+
+The training data used for this model contains a lot of unfiltered content from the internet, which is far from
+neutral. Therefore, the model can have biased predictions:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='roberta-large')
+>>> unmasker("The man worked as a <mask>.")
+
+[{'sequence': '<s>The man worked as a mechanic.</s>',
+  'score': 0.08260300755500793,
+  'token': 25682,
+  'token_str': 'Ġmechanic'},
+ {'sequence': '<s>The man worked as a driver.</s>',
+  'score': 0.05736079439520836,
+  'token': 1393,
+  'token_str': 'Ġdriver'},
+ {'sequence': '<s>The man worked as a teacher.</s>',
+  'score': 0.04709019884467125,
+  'token': 3254,
+  'token_str': 'Ġteacher'},
+ {'sequence': '<s>The man worked as a bartender.</s>',
+  'score': 0.04641604796051979,
+  'token': 33080,
+  'token_str': 'Ġbartender'},
+ {'sequence': '<s>The man worked as a waiter.</s>',
+  'score': 0.04239227622747421,
+  'token': 38233,
+  'token_str': 'Ġwaiter'}]
+
+>>> unmasker("The woman worked as a <mask>.")
+
+[{'sequence': '<s>The woman worked as a nurse.</s>',
+  'score': 0.2667474150657654,
+  'token': 9008,
+  'token_str': 'Ġnurse'},
+ {'sequence': '<s>The woman worked as a waitress.</s>',
+  'score': 0.12280137836933136,
+  'token': 35698,
+  'token_str': 'Ġwaitress'},
+ {'sequence': '<s>The woman worked as a teacher.</s>',
+  'score': 0.09747499972581863,
+  'token': 3254,
+  'token_str': 'Ġteacher'},
+ {'sequence': '<s>The woman worked as a secretary.</s>',
+  'score': 0.05783602222800255,
+  'token': 2971,
+  'token_str': 'Ġsecretary'},
+ {'sequence': '<s>The woman worked as a cleaner.</s>',
+  'score': 0.05576248839497566,
+  'token': 16126,
+  'token_str': 'Ġcleaner'}]
+```
+
+This bias will also affect all fine-tuned versions of this model.
+
+## Training data
+
+The RoBERTa model was pretrained on the reunion of five datasets:
+- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books;
+- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers) ;
+- [CC-News](https://commoncrawl.org/2016/10/news-dataset-available/), a dataset containing 63 millions English news
+  articles crawled between September 2016 and February 2019.
+- [OpenWebText](https://github.com/jcpeterson/openwebtext), an opensource recreation of the WebText dataset used to
+  train GPT-2,
+- [Stories](https://arxiv.org/abs/1806.02847) a dataset containing a subset of CommonCrawl data filtered to match the
+  story-like style of Winograd schemas.
+
+Together theses datasets weight 160GB of text.
+
+## Training procedure
+
+### Preprocessing
+
+The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50,000. The inputs of
+the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
+with `<s>` and the end of one by `</s>`
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `<mask>`.
+
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).
+
+### Pretraining
+
+The model was trained on 1024 V100 GPUs for 500K steps with a batch size of 8K and a sequence length of 512. The
+optimizer used is Adam with a learning rate of 4e-4, \\(\beta_{1} = 0.9\\), \\(\beta_{2} = 0.98\\) and
+\\(\epsilon = 1e-6\\), a weight decay of 0.01, learning rate warmup for 30,000 steps and linear decay of the learning
+rate after.
+
+## Evaluation results
+
+When fine-tuned on downstream tasks, this model achieves the following results:
+
+Glue test results:
+
+| Task | MNLI | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  |
+|:----:|:----:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|
+|      | 90.2 | 92.2 | 94.7 | 96.4  | 68.0 | 96.4  | 90.9 | 86.6 |
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{DBLP:journals/corr/abs-1907-11692,
+  author    = {Yinhan Liu and
+               Myle Ott and
+               Naman Goyal and
+               Jingfei Du and
+               Mandar Joshi and
+               Danqi Chen and
+               Omer Levy and
+               Mike Lewis and
+               Luke Zettlemoyer and
+               Veselin Stoyanov},
+  title     = {RoBERTa: {A} Robustly Optimized {BERT} Pretraining Approach},
+  journal   = {CoRR},
+  volume    = {abs/1907.11692},
+  year      = {2019},
+  url       = {http://arxiv.org/abs/1907.11692},
+  archivePrefix = {arXiv},
+  eprint    = {1907.11692},
+  timestamp = {Thu, 01 Aug 2019 08:59:33 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1907-11692.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<a href="https://huggingface.co/exbert/?model=roberta-base">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>

tests/cards/runwayml___stable-diffusion-v1-5.md

@@ -0,0 +1,188 @@
+# Stable Diffusion v1-5 Model Card
+
+Stable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images given any text input.
+For more information about how Stable Diffusion functions, please have a look at [🤗's Stable Diffusion blog](https://huggingface.co/blog/stable_diffusion).
+
+The **Stable-Diffusion-v1-5** checkpoint was initialized with the weights of the [Stable-Diffusion-v1-2](https:/steps/huggingface.co/CompVis/stable-diffusion-v1-2) 
+checkpoint and subsequently fine-tuned on 595k steps at resolution 512x512 on "laion-aesthetics v2 5+" and 10% dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).
+
+You can use this both with the [🧨Diffusers library](https://github.com/huggingface/diffusers) and the [RunwayML GitHub repository](https://github.com/runwayml/stable-diffusion).
+
+### Diffusers
+```py
+from diffusers import StableDiffusionPipeline
+import torch
+
+model_id = "runwayml/stable-diffusion-v1-5"
+pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
+pipe = pipe.to("cuda")
+
+prompt = "a photo of an astronaut riding a horse on mars"
+image = pipe(prompt).images[0]  
+    
+image.save("astronaut_rides_horse.png")
+```
+For more detailed instructions, use-cases and examples in JAX follow the instructions [here](https://github.com/huggingface/diffusers#text-to-image-generation-with-stable-diffusion)
+
+### Original GitHub Repository
+
+1. Download the weights 
+   - [v1-5-pruned-emaonly.ckpt](https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt) - 4.27GB, ema-only weight. uses less VRAM - suitable for inference
+   - [v1-5-pruned.ckpt](https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned.ckpt) - 7.7GB, ema+non-ema weights. uses more VRAM - suitable for fine-tuning
+
+2. Follow instructions [here](https://github.com/runwayml/stable-diffusion).
+
+## Model Details
+- **Developed by:** Robin Rombach, Patrick Esser
+- **Model type:** Diffusion-based text-to-image generation model
+- **Language(s):** English
+- **License:** [The CreativeML OpenRAIL M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license) is an [Open RAIL M license](https://www.licenses.ai/blog/2022/8/18/naming-convention-of-responsible-ai-licenses), adapted from the work that [BigScience](https://bigscience.huggingface.co/) and [the RAIL Initiative](https://www.licenses.ai/) are jointly carrying in the area of responsible AI licensing. See also [the article about the BLOOM Open RAIL license](https://bigscience.huggingface.co/blog/the-bigscience-rail-license) on which our license is based.
+- **Model Description:** This is a model that can be used to generate and modify images based on text prompts. It is a [Latent Diffusion Model](https://arxiv.org/abs/2112.10752) that uses a fixed, pretrained text encoder ([CLIP ViT-L/14](https://arxiv.org/abs/2103.00020)) as suggested in the [Imagen paper](https://arxiv.org/abs/2205.11487).
+- **Resources for more information:** [GitHub Repository](https://github.com/CompVis/stable-diffusion), [Paper](https://arxiv.org/abs/2112.10752).
+- **Cite as:**
+
+      @InProceedings{Rombach_2022_CVPR,
+          author    = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},
+          title     = {High-Resolution Image Synthesis With Latent Diffusion Models},
+          booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+          month     = {June},
+          year      = {2022},
+          pages     = {10684-10695}
+      }
+
+# Uses
+
+## Direct Use 
+The model is intended for research purposes only. Possible research areas and
+tasks include
+
+- Safe deployment of models which have the potential to generate harmful content.
+- Probing and understanding the limitations and biases of generative models.
+- Generation of artworks and use in design and other artistic processes.
+- Applications in educational or creative tools.
+- Research on generative models.
+
+Excluded uses are described below.
+
+ ### Misuse, Malicious Use, and Out-of-Scope Use
+_Note: This section is taken from the [DALLE-MINI model card](https://huggingface.co/dalle-mini/dalle-mini), but applies in the same way to Stable Diffusion v1_.
+
+
+The model should not be used to intentionally create or disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.
+
+#### Out-of-Scope Use
+The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
+
+#### Misuse and Malicious Use
+Using the model to generate content that is cruel to individuals is a misuse of this model. This includes, but is not limited to:
+
+- Generating demeaning, dehumanizing, or otherwise harmful representations of people or their environments, cultures, religions, etc.
+- Intentionally promoting or propagating discriminatory content or harmful stereotypes.
+- Impersonating individuals without their consent.
+- Sexual content without consent of the people who might see it.
+- Mis- and disinformation
+- Representations of egregious violence and gore
+- Sharing of copyrighted or licensed material in violation of its terms of use.
+- Sharing content that is an alteration of copyrighted or licensed material in violation of its terms of use.
+
+## Limitations and Bias
+
+### Limitations
+
+- The model does not achieve perfect photorealism
+- The model cannot render legible text
+- The model does not perform well on more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”
+- Faces and people in general may not be generated properly.
+- The model was trained mainly with English captions and will not work as well in other languages.
+- The autoencoding part of the model is lossy
+- The model was trained on a large-scale dataset
+  [LAION-5B](https://laion.ai/blog/laion-5b/) which contains adult material
+  and is not fit for product use without additional safety mechanisms and
+  considerations.
+- No additional measures were used to deduplicate the dataset. As a result, we observe some degree of memorization for images that are duplicated in the training data.
+  The training data can be searched at [https://rom1504.github.io/clip-retrieval/](https://rom1504.github.io/clip-retrieval/) to possibly assist in the detection of memorized images.
+
+### Bias
+
+While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases. 
+Stable Diffusion v1 was trained on subsets of [LAION-2B(en)](https://laion.ai/blog/laion-5b/), 
+which consists of images that are primarily limited to English descriptions. 
+Texts and images from communities and cultures that use other languages are likely to be insufficiently accounted for. 
+This affects the overall output of the model, as white and western cultures are often set as the default. Further, the 
+ability of the model to generate content with non-English prompts is significantly worse than with English-language prompts.
+
+### Safety Module
+
+The intended use of this model is with the [Safety Checker](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/safety_checker.py) in Diffusers. 
+This checker works by checking model outputs against known hard-coded NSFW concepts.
+The concepts are intentionally hidden to reduce the likelihood of reverse-engineering this filter.
+Specifically, the checker compares the class probability of harmful concepts in the embedding space of the `CLIPTextModel` *after generation* of the images. 
+The concepts are passed into the model with the generated image and compared to a hand-engineered weight for each NSFW concept.
+
+
+## Training
+
+**Training Data**
+The model developers used the following dataset for training the model:
+
+- LAION-2B (en) and subsets thereof (see next section)
+
+**Training Procedure**
+Stable Diffusion v1-5 is a latent diffusion model which combines an autoencoder with a diffusion model that is trained in the latent space of the autoencoder. During training, 
+
+- Images are encoded through an encoder, which turns images into latent representations. The autoencoder uses a relative downsampling factor of 8 and maps images of shape H x W x 3 to latents of shape H/f x W/f x 4
+- Text prompts are encoded through a ViT-L/14 text-encoder.
+- The non-pooled output of the text encoder is fed into the UNet backbone of the latent diffusion model via cross-attention.
+- The loss is a reconstruction objective between the noise that was added to the latent and the prediction made by the UNet.
+
+Currently six Stable Diffusion checkpoints are provided, which were trained as follows.
+- [`stable-diffusion-v1-1`](https://huggingface.co/CompVis/stable-diffusion-v1-1): 237,000 steps at resolution `256x256` on [laion2B-en](https://huggingface.co/datasets/laion/laion2B-en).
+  194,000 steps at resolution `512x512` on [laion-high-resolution](https://huggingface.co/datasets/laion/laion-high-resolution) (170M examples from LAION-5B with resolution `>= 1024x1024`).
+- [`stable-diffusion-v1-2`](https://huggingface.co/CompVis/stable-diffusion-v1-2): Resumed from `stable-diffusion-v1-1`.
+  515,000 steps at resolution `512x512` on "laion-improved-aesthetics" (a subset of laion2B-en,
+filtered to images with an original size `>= 512x512`, estimated aesthetics score `> 5.0`, and an estimated watermark probability `< 0.5`. The watermark estimate is from the LAION-5B metadata, the aesthetics score is estimated using an [improved aesthetics estimator](https://github.com/christophschuhmann/improved-aesthetic-predictor)).
+- [`stable-diffusion-v1-3`](https://huggingface.co/CompVis/stable-diffusion-v1-3): Resumed from `stable-diffusion-v1-2` - 195,000 steps at resolution `512x512` on "laion-improved-aesthetics" and 10 % dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).
+- [`stable-diffusion-v1-4`](https://huggingface.co/CompVis/stable-diffusion-v1-4) Resumed from `stable-diffusion-v1-2` - 225,000 steps at resolution `512x512` on "laion-aesthetics v2 5+" and 10 % dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).
+- [`stable-diffusion-v1-5`](https://huggingface.co/runwayml/stable-diffusion-v1-5) Resumed from `stable-diffusion-v1-2` - 595,000 steps at resolution `512x512` on "laion-aesthetics v2 5+" and 10 % dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).
+- [`stable-diffusion-inpainting`](https://huggingface.co/runwayml/stable-diffusion-inpainting) Resumed from `stable-diffusion-v1-5` - then 440,000 steps of inpainting training at resolution 512x512 on “laion-aesthetics v2 5+” and 10% dropping of the text-conditioning. For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself) whose weights were zero-initialized after restoring the non-inpainting checkpoint. During training, we generate synthetic masks and in 25% mask everything.
+
+- **Hardware:** 32 x 8 x A100 GPUs
+- **Optimizer:** AdamW
+- **Gradient Accumulations**: 2
+- **Batch:** 32 x 8 x 2 x 4 = 2048
+- **Learning rate:** warmup to 0.0001 for 10,000 steps and then kept constant
+
+## Evaluation Results 
+Evaluations with different classifier-free guidance scales (1.5, 2.0, 3.0, 4.0,
+5.0, 6.0, 7.0, 8.0) and 50 PNDM/PLMS sampling
+steps show the relative improvements of the checkpoints:
+
+![pareto](https://huggingface.co/CompVis/stable-diffusion/resolve/main/v1-1-to-v1-5.png)
+
+Evaluated using 50 PLMS steps and 10000 random prompts from the COCO2017 validation set, evaluated at 512x512 resolution.  Not optimized for FID scores.
+## Environmental Impact
+
+**Stable Diffusion v1** **Estimated Emissions**
+Based on that information, we estimate the following CO2 emissions using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). The hardware, runtime, cloud provider, and compute region were utilized to estimate the carbon impact.
+
+- **Hardware Type:** A100 PCIe 40GB
+- **Hours used:** 150000
+- **Cloud Provider:** AWS
+- **Compute Region:** US-east
+- **Carbon Emitted (Power consumption x Time x Carbon produced based on location of power grid):** 11250 kg CO2 eq.
+
+
+## Citation
+
+```bibtex
+    @InProceedings{Rombach_2022_CVPR,
+        author    = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},
+        title     = {High-Resolution Image Synthesis With Latent Diffusion Models},
+        booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+        month     = {June},
+        year      = {2022},
+        pages     = {10684-10695}
+    }
+```
+
+*This model card was written by: Robin Rombach and Patrick Esser and is based on the [DALL-E Mini model card](https://huggingface.co/dalle-mini/dalle-mini).*

tests/cards/sentence-transformers___all-MiniLM-L6-v2.md

@@ -0,0 +1,142 @@
+# all-MiniLM-L6-v2
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+
+## Usage (Sentence-Transformers)
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can use the model like this:
+```python
+from sentence_transformers import SentenceTransformer
+sentences = ["This is an example sentence", "Each sentence is converted"]
+
+model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+## Usage (HuggingFace Transformers)
+Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+import torch.nn.functional as F
+
+#Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+
+# Sentences we want sentence embeddings for
+sentences = ['This is an example sentence', 'Each sentence is converted']
+
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
+model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
+
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
+
+# Perform pooling
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+
+# Normalize embeddings
+sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)
+
+print("Sentence embeddings:")
+print(sentence_embeddings)
+```
+
+## Evaluation Results
+
+For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=sentence-transformers/all-MiniLM-L6-v2)
+
+------
+
+## Background
+
+The project aims to train sentence embedding models on very large sentence level datasets using a self-supervised 
+contrastive learning objective. We used the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model and fine-tuned in on a 
+1B sentence pairs dataset. We use a contrastive learning objective: given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset.
+
+We developped this model during the 
+[Community week using JAX/Flax for NLP & CV](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104), 
+organized by Hugging Face. We developped this model as part of the project:
+[Train the Best Sentence Embedding Model Ever with 1B Training Pairs](https://discuss.huggingface.co/t/train-the-best-sentence-embedding-model-ever-with-1b-training-pairs/7354). We benefited from efficient hardware infrastructure to run the project: 7 TPUs v3-8, as well as intervention from Googles Flax, JAX, and Cloud team member about efficient deep learning frameworks.
+
+## Intended uses
+
+Our model is intented to be used as a sentence and short paragraph encoder. Given an input text, it ouptuts a vector which captures 
+the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.
+
+By default, input text longer than 256 word pieces is truncated.
+
+
+## Training procedure
+
+### Pre-training 
+
+We use the pretrained [`nreimers/MiniLM-L6-H384-uncased`](https://huggingface.co/nreimers/MiniLM-L6-H384-uncased) model. Please refer to the model card for more detailed information about the pre-training procedure.
+
+### Fine-tuning 
+
+We fine-tune the model using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch.
+We then apply the cross entropy loss by comparing with true pairs.
+
+#### Hyper parameters
+
+We trained ou model on a TPU v3-8. We train the model during 100k steps using a batch size of 1024 (128 per TPU core).
+We use a learning rate warm up of 500. The sequence length was limited to 128 tokens. We used the AdamW optimizer with
+a 2e-5 learning rate. The full training script is accessible in this current repository: `train_script.py`.
+
+#### Training data
+
+We use the concatenation from multiple datasets to fine-tune our model. The total number of sentence pairs is above 1 billion sentences.
+We sampled each dataset given a weighted probability which configuration is detailed in the `data_config.json` file.
+
+
+| Dataset                                                  | Paper                                    | Number of training tuples  |
+|--------------------------------------------------------|:----------------------------------------:|:--------------------------:|
+| [Reddit comments (2015-2018)](https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit) | [paper](https://arxiv.org/abs/1904.06472) | 726,484,430 |
+| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Abstracts) | [paper](https://aclanthology.org/2020.acl-main.447/) | 116,288,806 |
+| [WikiAnswers](https://github.com/afader/oqa#wikianswers-corpus) Duplicate question pairs | [paper](https://doi.org/10.1145/2623330.2623677) | 77,427,422 |
+| [PAQ](https://github.com/facebookresearch/PAQ) (Question, Answer) pairs | [paper](https://arxiv.org/abs/2102.07033) | 64,371,441 |
+| [S2ORC](https://github.com/allenai/s2orc) Citation pairs (Titles) | [paper](https://aclanthology.org/2020.acl-main.447/) | 52,603,982 |
+| [S2ORC](https://github.com/allenai/s2orc) (Title, Abstract) | [paper](https://aclanthology.org/2020.acl-main.447/) | 41,769,185 |
+| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Body) pairs  | - | 25,316,456 |
+| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title+Body, Answer) pairs  | - | 21,396,559 |
+| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) (Title, Answer) pairs  | - | 21,396,559 |
+| [MS MARCO](https://microsoft.github.io/msmarco/) triplets | [paper](https://doi.org/10.1145/3404835.3462804) | 9,144,553 |
+| [GOOAQ: Open Question Answering with Diverse Answer Types](https://github.com/allenai/gooaq) | [paper](https://arxiv.org/pdf/2104.08727.pdf) | 3,012,496 |
+| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 1,198,260 |
+| [Code Search](https://huggingface.co/datasets/code_search_net) | - | 1,151,414 |
+| [COCO](https://cocodataset.org/#home) Image captions | [paper](https://link.springer.com/chapter/10.1007%2F978-3-319-10602-1_48) | 828,395|
+| [SPECTER](https://github.com/allenai/specter) citation triplets | [paper](https://doi.org/10.18653/v1/2020.acl-main.207) | 684,100 |
+| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Question, Answer) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 681,164 |
+| [Yahoo Answers](https://www.kaggle.com/soumikrakshit/yahoo-answers-dataset) (Title, Question) | [paper](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html) | 659,896 |
+| [SearchQA](https://huggingface.co/datasets/search_qa) | [paper](https://arxiv.org/abs/1704.05179) | 582,261 |
+| [Eli5](https://huggingface.co/datasets/eli5) | [paper](https://doi.org/10.18653/v1/p19-1346) | 325,475 |
+| [Flickr 30k](https://shannon.cs.illinois.edu/DenotationGraph/) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/229/33) | 317,695 |
+| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles) | | 304,525 |
+| AllNLI ([SNLI](https://nlp.stanford.edu/projects/snli/) and [MultiNLI](https://cims.nyu.edu/~sbowman/multinli/) | [paper SNLI](https://doi.org/10.18653/v1/d15-1075), [paper MultiNLI](https://doi.org/10.18653/v1/n18-1101) | 277,230 | 
+| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (bodies) | | 250,519 |
+| [Stack Exchange](https://huggingface.co/datasets/flax-sentence-embeddings/stackexchange_xml) Duplicate questions (titles+bodies) | | 250,460 |
+| [Sentence Compression](https://github.com/google-research-datasets/sentence-compression) | [paper](https://www.aclweb.org/anthology/D13-1155/) | 180,000 |
+| [Wikihow](https://github.com/pvl/wikihow_pairs_dataset) | [paper](https://arxiv.org/abs/1810.09305) | 128,542 |
+| [Altlex](https://github.com/chridey/altlex/) | [paper](https://aclanthology.org/P16-1135.pdf) | 112,696 |
+| [Quora Question Triplets](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs) | - | 103,663 |
+| [Simple Wikipedia](https://cs.pomona.edu/~dkauchak/simplification/) | [paper](https://www.aclweb.org/anthology/P11-2117/) | 102,225 |
+| [Natural Questions (NQ)](https://ai.google.com/research/NaturalQuestions) | [paper](https://transacl.org/ojs/index.php/tacl/article/view/1455) | 100,231 |
+| [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) | [paper](https://aclanthology.org/P18-2124.pdf) | 87,599 |
+| [TriviaQA](https://huggingface.co/datasets/trivia_qa) | - | 73,346 |
+| **Total** | | **1,170,060,424** |

tests/cards/t5-base.md

@@ -0,0 +1,175 @@
+# Model Card for T5 Base
+
+![model image](https://camo.githubusercontent.com/623b4dea0b653f2ad3f36c71ebfe749a677ac0a1/68747470733a2f2f6d69726f2e6d656469756d2e636f6d2f6d61782f343030362f312a44304a31674e51663876727255704b657944387750412e706e67)
+
+#  Table of Contents
+
+1. [Model Details](#model-details)
+2. [Uses](#uses)
+3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
+4. [Training Details](#training-details)
+5. [Evaluation](#evaluation)
+6. [Environmental Impact](#environmental-impact)
+7. [Citation](#citation)
+8. [Model Card Authors](#model-card-authors)
+9. [How To Get Started With the Model](#how-to-get-started-with-the-model)
+
+# Model Details
+
+## Model Description
+
+The developers of the Text-To-Text Transfer Transformer (T5) [write](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html): 
+
+> With T5, we propose reframing all NLP tasks into a unified text-to-text-format where the input and output are always text strings, in contrast to BERT-style models that can only output either a class label or a span of the input. Our text-to-text framework allows us to use the same model, loss function, and hyperparameters on any NLP task.
+
+T5-Base is the checkpoint with 220 million parameters. 
+
+- **Developed by:** Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. See [associated paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) and [GitHub repo](https://github.com/google-research/text-to-text-transfer-transformer#released-model-checkpoints)
+- **Model type:** Language model
+- **Language(s) (NLP):** English, French, Romanian, German
+- **License:** Apache 2.0
+- **Related Models:** [All T5 Checkpoints](https://huggingface.co/models?search=t5)
+- **Resources for more information:**
+  - [Research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf)
+  - [Google's T5 Blog Post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) 
+  - [GitHub Repo](https://github.com/google-research/text-to-text-transfer-transformer)
+  - [Hugging Face T5 Docs](https://huggingface.co/docs/transformers/model_doc/t5)
+  
+# Uses
+
+## Direct Use and Downstream Use
+
+The developers write in a [blog post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) that the model: 
+
+> Our text-to-text framework allows us to use the same model, loss function, and hyperparameters on any NLP task, including machine translation, document summarization, question answering, and classification tasks (e.g., sentiment analysis). We can even apply T5 to regression tasks by training it to predict the string representation of a number instead of the number itself.
+
+See the [blog post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) for further details.
+
+## Out-of-Scope Use
+
+More information needed.
+
+# Bias, Risks, and Limitations
+
+More information needed.
+
+## Recommendations
+
+More information needed.
+
+# Training Details
+
+## Training Data
+
+The model is pre-trained on the [Colossal Clean Crawled Corpus (C4)](https://www.tensorflow.org/datasets/catalog/c4), which was developed and released in the context of the same [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) as T5.
+
+The model was pre-trained on a on a **multi-task mixture of unsupervised (1.) and supervised tasks (2.)**.
+Thereby, the following datasets were being used for (1.) and (2.):
+
+1. **Datasets used for Unsupervised denoising objective**:
+
+- [C4](https://huggingface.co/datasets/c4)
+- [Wiki-DPR](https://huggingface.co/datasets/wiki_dpr)
+
+
+2. **Datasets used for Supervised text-to-text language modeling objective**
+
+- Sentence acceptability judgment
+  - CoLA [Warstadt et al., 2018](https://arxiv.org/abs/1805.12471)
+- Sentiment analysis 
+  - SST-2 [Socher et al., 2013](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
+- Paraphrasing/sentence similarity
+  - MRPC [Dolan and Brockett, 2005](https://aclanthology.org/I05-5002)
+  - STS-B [Ceret al., 2017](https://arxiv.org/abs/1708.00055)
+  - QQP [Iyer et al., 2017](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs)
+- Natural language inference
+  - MNLI [Williams et al., 2017](https://arxiv.org/abs/1704.05426)
+  - QNLI [Rajpurkar et al.,2016](https://arxiv.org/abs/1606.05250)
+  - RTE [Dagan et al., 2005](https://link.springer.com/chapter/10.1007/11736790_9) 
+  - CB [De Marneff et al., 2019](https://semanticsarchive.net/Archive/Tg3ZGI2M/Marneffe.pdf)
+- Sentence completion
+  - COPA [Roemmele et al., 2011](https://www.researchgate.net/publication/221251392_Choice_of_Plausible_Alternatives_An_Evaluation_of_Commonsense_Causal_Reasoning)
+- Word sense disambiguation
+  - WIC [Pilehvar and Camacho-Collados, 2018](https://arxiv.org/abs/1808.09121)
+- Question answering
+  - MultiRC [Khashabi et al., 2018](https://aclanthology.org/N18-1023)
+  - ReCoRD [Zhang et al., 2018](https://arxiv.org/abs/1810.12885)
+  - BoolQ [Clark et al., 2019](https://arxiv.org/abs/1905.10044)
+
+## Training Procedure
+
+In their [abstract](https://jmlr.org/papers/volume21/20-074/20-074.pdf), the model developers write: 
+
+> In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. 
+
+The framework introduced, the T5 framework, involves a training procedure that brings together the approaches studied in the paper. See the [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) for further details.
+
+# Evaluation
+
+## Testing Data, Factors & Metrics
+
+The developers evaluated the model on 24 tasks, see the [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) for full details.
+
+## Results 
+
+For full results for T5-Base, see the [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf), Table 14.
+
+# Environmental Impact
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** Google Cloud TPU Pods
+- **Hours used:** More information needed
+- **Cloud Provider:** GCP
+- **Compute Region:** More information needed
+- **Carbon Emitted:** More information needed
+
+# Citation
+
+**BibTeX:**
+
+```bibtex
+@article{2020t5,
+  author  = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
+  title   = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
+  journal = {Journal of Machine Learning Research},
+  year    = {2020},
+  volume  = {21},
+  number  = {140},
+  pages   = {1-67},
+  url     = {http://jmlr.org/papers/v21/20-074.html}
+}
+```
+
+**APA:**
+- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140), 1-67.
+
+# Model Card Authors
+
+This model card was written by the team at Hugging Face.
+
+# How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+<details>
+<summary> Click to expand </summary>
+
+```python
+from transformers import T5Tokenizer, T5Model
+
+tokenizer = T5Tokenizer.from_pretrained("t5-base")
+model = T5Model.from_pretrained("t5-base")
+
+input_ids = tokenizer(
+    "Studies have been shown that owning a dog is good for you", return_tensors="pt"
+).input_ids  # Batch size 1
+decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids  # Batch size 1
+
+# forward pass
+outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
+last_hidden_states = outputs.last_hidden_state
+```
+
+See the [Hugging Face T5](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5Model) docs and a [Colab Notebook](https://colab.research.google.com/github/google-research/text-to-text-transfer-transformer/blob/main/notebooks/t5-trivia.ipynb) created by the model developers for more examples.
+</details>

tests/cards/t5-small.md

@@ -0,0 +1,175 @@
+# Model Card for T5 Small
+
+![model image](https://camo.githubusercontent.com/623b4dea0b653f2ad3f36c71ebfe749a677ac0a1/68747470733a2f2f6d69726f2e6d656469756d2e636f6d2f6d61782f343030362f312a44304a31674e51663876727255704b657944387750412e706e67)
+
+#  Table of Contents
+
+1. [Model Details](#model-details)
+2. [Uses](#uses)
+3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
+4. [Training Details](#training-details)
+5. [Evaluation](#evaluation)
+6. [Environmental Impact](#environmental-impact)
+7. [Citation](#citation)
+8. [Model Card Authors](#model-card-authors)
+9. [How To Get Started With the Model](#how-to-get-started-with-the-model)
+
+# Model Details
+
+## Model Description
+
+The developers of the Text-To-Text Transfer Transformer (T5) [write](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html): 
+
+> With T5, we propose reframing all NLP tasks into a unified text-to-text-format where the input and output are always text strings, in contrast to BERT-style models that can only output either a class label or a span of the input. Our text-to-text framework allows us to use the same model, loss function, and hyperparameters on any NLP task.
+
+T5-Small is the checkpoint with 60 million parameters. 
+
+- **Developed by:** Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. See [associated paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) and [GitHub repo](https://github.com/google-research/text-to-text-transfer-transformer#released-model-checkpoints)
+- **Model type:** Language model
+- **Language(s) (NLP):** English, French, Romanian, German
+- **License:** Apache 2.0
+- **Related Models:** [All T5 Checkpoints](https://huggingface.co/models?search=t5)
+- **Resources for more information:**
+  - [Research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf)
+  - [Google's T5 Blog Post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) 
+  - [GitHub Repo](https://github.com/google-research/text-to-text-transfer-transformer)
+  - [Hugging Face T5 Docs](https://huggingface.co/docs/transformers/model_doc/t5)
+  
+# Uses
+
+## Direct Use and Downstream Use
+
+The developers write in a [blog post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) that the model: 
+
+> Our text-to-text framework allows us to use the same model, loss function, and hyperparameters on any NLP task, including machine translation, document summarization, question answering, and classification tasks (e.g., sentiment analysis). We can even apply T5 to regression tasks by training it to predict the string representation of a number instead of the number itself.
+
+See the [blog post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) for further details.
+
+## Out-of-Scope Use
+
+More information needed.
+
+# Bias, Risks, and Limitations
+
+More information needed.
+
+## Recommendations
+
+More information needed.
+
+# Training Details
+
+## Training Data
+
+The model is pre-trained on the [Colossal Clean Crawled Corpus (C4)](https://www.tensorflow.org/datasets/catalog/c4), which was developed and released in the context of the same [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) as T5.
+
+The model was pre-trained on a on a **multi-task mixture of unsupervised (1.) and supervised tasks (2.)**.
+Thereby, the following datasets were being used for (1.) and (2.):
+
+1. **Datasets used for Unsupervised denoising objective**:
+
+- [C4](https://huggingface.co/datasets/c4)
+- [Wiki-DPR](https://huggingface.co/datasets/wiki_dpr)
+
+
+2. **Datasets used for Supervised text-to-text language modeling objective**
+
+- Sentence acceptability judgment
+  - CoLA [Warstadt et al., 2018](https://arxiv.org/abs/1805.12471)
+- Sentiment analysis 
+  - SST-2 [Socher et al., 2013](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
+- Paraphrasing/sentence similarity
+  - MRPC [Dolan and Brockett, 2005](https://aclanthology.org/I05-5002)
+  - STS-B [Ceret al., 2017](https://arxiv.org/abs/1708.00055)
+  - QQP [Iyer et al., 2017](https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs)
+- Natural language inference
+  - MNLI [Williams et al., 2017](https://arxiv.org/abs/1704.05426)
+  - QNLI [Rajpurkar et al.,2016](https://arxiv.org/abs/1606.05250)
+  - RTE [Dagan et al., 2005](https://link.springer.com/chapter/10.1007/11736790_9) 
+  - CB [De Marneff et al., 2019](https://semanticsarchive.net/Archive/Tg3ZGI2M/Marneffe.pdf)
+- Sentence completion
+  - COPA [Roemmele et al., 2011](https://www.researchgate.net/publication/221251392_Choice_of_Plausible_Alternatives_An_Evaluation_of_Commonsense_Causal_Reasoning)
+- Word sense disambiguation
+  - WIC [Pilehvar and Camacho-Collados, 2018](https://arxiv.org/abs/1808.09121)
+- Question answering
+  - MultiRC [Khashabi et al., 2018](https://aclanthology.org/N18-1023)
+  - ReCoRD [Zhang et al., 2018](https://arxiv.org/abs/1810.12885)
+  - BoolQ [Clark et al., 2019](https://arxiv.org/abs/1905.10044)
+
+## Training Procedure
+
+In their [abstract](https://jmlr.org/papers/volume21/20-074/20-074.pdf), the model developers write: 
+
+> In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. 
+
+The framework introduced, the T5 framework, involves a training procedure that brings together the approaches studied in the paper. See the [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) for further details.
+
+# Evaluation
+
+## Testing Data, Factors & Metrics
+
+The developers evaluated the model on 24 tasks, see the [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf) for full details.
+
+## Results 
+
+For full results for T5-small, see the [research paper](https://jmlr.org/papers/volume21/20-074/20-074.pdf), Table 14.
+
+# Environmental Impact
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** Google Cloud TPU Pods
+- **Hours used:** More information needed
+- **Cloud Provider:** GCP
+- **Compute Region:** More information needed
+- **Carbon Emitted:** More information needed
+
+# Citation
+
+**BibTeX:**
+
+```bibtex
+@article{2020t5,
+  author  = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
+  title   = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
+  journal = {Journal of Machine Learning Research},
+  year    = {2020},
+  volume  = {21},
+  number  = {140},
+  pages   = {1-67},
+  url     = {http://jmlr.org/papers/v21/20-074.html}
+}
+```
+
+**APA:**
+- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140), 1-67.
+
+# Model Card Authors
+
+This model card was written by the team at Hugging Face.
+
+# How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+<details>
+<summary> Click to expand </summary>
+
+```python
+from transformers import T5Tokenizer, T5Model
+
+tokenizer = T5Tokenizer.from_pretrained("t5-small")
+model = T5Model.from_pretrained("t5-small")
+
+input_ids = tokenizer(
+    "Studies have been shown that owning a dog is good for you", return_tensors="pt"
+).input_ids  # Batch size 1
+decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids  # Batch size 1
+
+# forward pass
+outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
+last_hidden_states = outputs.last_hidden_state
+```
+
+See the [Hugging Face T5](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5Model) docs and a [Colab Notebook](https://colab.research.google.com/github/google-research/text-to-text-transfer-transformer/blob/main/notebooks/t5-trivia.ipynb) created by the model developers for more examples.
+</details>

tests/cards/xlm-roberta-base.md

@@ -0,0 +1,99 @@
+# XLM-RoBERTa (base-sized model) 
+
+XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Conneau et al. and first released in [this repository](https://github.com/pytorch/fairseq/tree/master/examples/xlmr). 
+
+Disclaimer: The team releasing XLM-RoBERTa did not write a model card for this model so this model card has been written by the Hugging Face team.
+
+## Model description
+
+XLM-RoBERTa is a multilingual version of RoBERTa. It is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. 
+
+RoBERTa is a transformers model pretrained on a large corpus in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts.
+
+More precisely, it was pretrained with the Masked language modeling (MLM) objective. Taking a sentence, the model randomly masks 15% of the words in the input then run the entire masked sentence through the model and has to predict the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the sentence.
+
+This way, the model learns an inner representation of 100 languages that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard classifier using the features produced by the XLM-RoBERTa model as inputs.
+
+## Intended uses & limitations
+
+You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?search=xlm-roberta) to look for fine-tuned versions on a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation, you should look at models like GPT2.
+
+## Usage
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='xlm-roberta-base')
+>>> unmasker("Hello I'm a <mask> model.")
+
+[{'score': 0.10563907772302628,
+  'sequence': "Hello I'm a fashion model.",
+  'token': 54543,
+  'token_str': 'fashion'},
+ {'score': 0.08015287667512894,
+  'sequence': "Hello I'm a new model.",
+  'token': 3525,
+  'token_str': 'new'},
+ {'score': 0.033413201570510864,
+  'sequence': "Hello I'm a model model.",
+  'token': 3299,
+  'token_str': 'model'},
+ {'score': 0.030217764899134636,
+  'sequence': "Hello I'm a French model.",
+  'token': 92265,
+  'token_str': 'French'},
+ {'score': 0.026436051353812218,
+  'sequence': "Hello I'm a sexy model.",
+  'token': 17473,
+  'token_str': 'sexy'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-base')
+model = AutoModelForMaskedLM.from_pretrained("xlm-roberta-base")
+
+# prepare input
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+
+# forward pass
+output = model(**encoded_input)
+```
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{DBLP:journals/corr/abs-1911-02116,
+  author    = {Alexis Conneau and
+               Kartikay Khandelwal and
+               Naman Goyal and
+               Vishrav Chaudhary and
+               Guillaume Wenzek and
+               Francisco Guzm{\'{a}}n and
+               Edouard Grave and
+               Myle Ott and
+               Luke Zettlemoyer and
+               Veselin Stoyanov},
+  title     = {Unsupervised Cross-lingual Representation Learning at Scale},
+  journal   = {CoRR},
+  volume    = {abs/1911.02116},
+  year      = {2019},
+  url       = {http://arxiv.org/abs/1911.02116},
+  eprinttype = {arXiv},
+  eprint    = {1911.02116},
+  timestamp = {Mon, 11 Nov 2019 18:38:09 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1911-02116.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<a href="https://huggingface.co/exbert/?model=xlm-roberta-base">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>

tests/cards/xlm-roberta-large.md

@@ -0,0 +1,99 @@
+# XLM-RoBERTa (large-sized model) 
+
+XLM-RoBERTa model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Conneau et al. and first released in [this repository](https://github.com/pytorch/fairseq/tree/master/examples/xlmr). 
+
+Disclaimer: The team releasing XLM-RoBERTa did not write a model card for this model so this model card has been written by the Hugging Face team.
+
+## Model description
+
+XLM-RoBERTa is a multilingual version of RoBERTa. It is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. 
+
+RoBERTa is a transformers model pretrained on a large corpus in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts.
+
+More precisely, it was pretrained with the Masked language modeling (MLM) objective. Taking a sentence, the model randomly masks 15% of the words in the input then run the entire masked sentence through the model and has to predict the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the sentence.
+
+This way, the model learns an inner representation of 100 languages that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard classifier using the features produced by the XLM-RoBERTa model as inputs.
+
+## Intended uses & limitations
+
+You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?search=xlm-roberta) to look for fine-tuned versions on a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation, you should look at models like GPT2.
+
+## Usage
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='xlm-roberta-large')
+>>> unmasker("Hello I'm a <mask> model.")
+
+[{'score': 0.10563907772302628,
+  'sequence': "Hello I'm a fashion model.",
+  'token': 54543,
+  'token_str': 'fashion'},
+ {'score': 0.08015287667512894,
+  'sequence': "Hello I'm a new model.",
+  'token': 3525,
+  'token_str': 'new'},
+ {'score': 0.033413201570510864,
+  'sequence': "Hello I'm a model model.",
+  'token': 3299,
+  'token_str': 'model'},
+ {'score': 0.030217764899134636,
+  'sequence': "Hello I'm a French model.",
+  'token': 92265,
+  'token_str': 'French'},
+ {'score': 0.026436051353812218,
+  'sequence': "Hello I'm a sexy model.",
+  'token': 17473,
+  'token_str': 'sexy'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-large')
+model = AutoModelForMaskedLM.from_pretrained("xlm-roberta-large")
+
+# prepare input
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+
+# forward pass
+output = model(**encoded_input)
+```
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{DBLP:journals/corr/abs-1911-02116,
+  author    = {Alexis Conneau and
+               Kartikay Khandelwal and
+               Naman Goyal and
+               Vishrav Chaudhary and
+               Guillaume Wenzek and
+               Francisco Guzm{\'{a}}n and
+               Edouard Grave and
+               Myle Ott and
+               Luke Zettlemoyer and
+               Veselin Stoyanov},
+  title     = {Unsupervised Cross-lingual Representation Learning at Scale},
+  journal   = {CoRR},
+  volume    = {abs/1911.02116},
+  year      = {2019},
+  url       = {http://arxiv.org/abs/1911.02116},
+  eprinttype = {arXiv},
+  eprint    = {1911.02116},
+  timestamp = {Mon, 11 Nov 2019 18:38:09 +0100},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-1911-02116.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```
+
+<a href="https://huggingface.co/exbert/?model=xlm-roberta-base">
+	<img width="300px" src="https://cdn-media.huggingface.co/exbert/button.png">
+</a>

tests/cards/yiyanghkust___finbert-tone.md

@@ -0,0 +1,33 @@
+`FinBERT` is a BERT model pre-trained on financial communication text. The purpose is to enhance financial NLP research and practice. It is trained on the following three financial communication corpus. The total corpora size is 4.9B tokens.
+- Corporate Reports 10-K & 10-Q: 2.5B tokens
+- Earnings Call Transcripts: 1.3B tokens
+- Analyst Reports: 1.1B tokens
+
+More technical details on `FinBERT`: [Click Link](https://github.com/yya518/FinBERT)
+
+This released `finbert-tone` model is the `FinBERT` model fine-tuned on 10,000 manually annotated (positive, negative, neutral) sentences from analyst reports. This model achieves superior performance on financial tone analysis task. If you are simply interested in using `FinBERT` for financial tone analysis, give it a try.
+
+If you use the model in your academic work, please cite the following paper:
+
+Huang, Allen H., Hui Wang, and Yi Yang. "FinBERT: A Large Language Model for Extracting Information from Financial Text." *Contemporary Accounting Research* (2022).
+
+
+# How to use 
+You can use this model with Transformers pipeline for sentiment analysis.
+```python
+from transformers import BertTokenizer, BertForSequenceClassification
+from transformers import pipeline
+
+finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-tone',num_labels=3)
+tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-tone')
+
+nlp = pipeline("sentiment-analysis", model=finbert, tokenizer=tokenizer)
+
+sentences = ["there is a shortage of capital, and we need extra financing",  
+             "growth is strong and we have plenty of liquidity", 
+             "there are doubts about our finances", 
+             "profits are flat"]
+results = nlp(sentences)
+print(results)  #LABEL_0: neutral; LABEL_1: positive; LABEL_2: negative
+
+```

tests/conftest.py

@@ -1,7 +1,62 @@
-import pytest
-from bloom_card import bloom_card as bc
+from os import listdir
+from os.path import isfile, join
+from pathlib import Path
+
+
+# TODO: I have the option of maybe making a check for accuracy/metrics?
+
+# Intended Purpose, General Limitations, Computational Requirements
+expected_check_results = {
+    "albert-base-v2": [True, True, False],
+    "bert-base-cased": [True, True, False],
+    "bert-base-multilingual-cased": [True, True, False],
+    "bert-base-uncased": [True, True, False],
+    "cl-tohoku___bert-base-japanese-whole-word-masking": [False, False, False],
+    "distilbert-base-cased-distilled-squad": [True, True, True],
+    "distilbert-base-uncased": [True, True, False],
+    "distilbert-base-uncased-finetuned-sst-2-english": [True, True, False],
+    "distilroberta-base": [True, True, False],
+    "emilyalsentzer___Bio_ClinicalBERT": [False, False, False],
+    "facebook___bart-large-mnli": [False, False, False],
+    "google___electra-base-discriminator": [False, False, False],
+    "gpt2": [True, True, False],
+    "Helsinki-NLP___opus-mt-en-es": [False, False, False],
+    "jonatasgrosman___wav2vec2-large-xlsr-53-english": [False, False, False],
+    "microsoft___layoutlmv3-base": [True, False, False],
+    "openai___clip-vit-base-patch32": [True, True, False],
+    "openai___clip-vit-large-patch14": [True, True, False],
+    "philschmid___bart-large-cnn-samsum": [False, False, False],
+    "prajjwal1___bert-tiny": [False, False, False],
+    "roberta-base": [True, True, True], # For the computational requirements, sort of?
+    "roberta-large": [True, True, True],
+    "runwayml___stable-diffusion-v1-5": [True, True, True],
+    "sentence-transformers___all-MiniLM-L6-v2": [True, False, False],
+    "StanfordAIMI___stanford-deidentifier-base": [False, False, False],
+    "t5-base": [True, False, False],
+    "t5-small": [True, False, False],
+    "xlm-roberta-base": [True, False, False],
+    "xlm-roberta-large": [True, False, False],
+    "yiyanghkust___finbert-tone": [True, False, False],
+}
+
+
+def pytest_generate_tests(metafunc):
+    if "real_model_card" in metafunc.fixturenames:
+        files = [f"cards/{f}" for f in listdir("cards") if isfile(join("cards", f))]
+        cards = [Path(f).read_text() for f in files]
+        model_ids = [f.replace("cards/", "").replace(".md", "") for f in files]
+
+        # TODO: IMPORTANT – remove the default [False, False, False]
+        expected_results = [expected_check_results.get(m, [False, False, False]) for m, c in zip(model_ids, cards)]
+
+        metafunc.parametrize(
+            ["real_model_card", "expected_check_results"],
+            list(map(list, zip(cards, expected_results)))
+        )
+
+    # rows = read_csvrows()
+    # if 'row' in metafunc.fixturenames:
+    #     metafunc.parametrize('row', rows)
+    # if 'col' in metafunc.fixturenames:
+    #     metafunc.parametrize('col', list(itertools.chain(*rows)))
 
-@pytest.fixture()
-def bloom_card():
-    # TODO: Note, this is a heavily doctored version of the card.
-    return bc

tests/test_compliance_checks.py

@@ -283,67 +283,13 @@ class TestComplianceSuite:
             check.run_check.assert_called_once()
 
 
-class TestEndToEnd:
-    @pytest.mark.parametrize("card,fixture", [
-        ("""
-# Model Card for Sample Model
-
-Some random info...
-
-## Model Details
-
-### Model Description
-
-- **Developed by:** Nima Boscarino
-- **Model type:** Yada yada yada
-
-## Uses
-
-### Direct Use
-
-Here is some info about direct uses...
-
-### Downstream Use [optional]
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-Here is some info about out-of-scope uses...
-
-## Bias, Risks, and Limitations
-
-Hello world! These are some risks...
-
-## Technical Specifications
-
-### Compute infrastructure
-Jean Zay Public Supercomputer, provided by the French government.
-
-#### Hardware
-
-* 384 A100 80GB GPUs (48 nodes)
-
-#### Software
-
-* Megatron-DeepSpeed ([Github link](https://github.com/bigscience-workshop/Megatron-DeepSpeed))
-</details>
-
-## More Things
-        """, False),
-        ("bloom_card", True)
+def test_end_to_end_compliance_suite(real_model_card, expected_check_results):
+    suite = ComplianceSuite(checks=[
+        IntendedPurposeCheck(),
+        GeneralLimitationsCheck(),
+        ComputationalRequirementsCheck()
     ])
-    def test_end_to_end_compliance_suite(self, card, fixture, request):
-        if fixture:
-            card = request.getfixturevalue(card)
-
-        suite = ComplianceSuite(checks=[
-            ModelProviderIdentityCheck(),
-            IntendedPurposeCheck(),
-            GeneralLimitationsCheck(),
-            ComputationalRequirementsCheck()
-        ])
 
-        results = suite.run(card)
+    results = suite.run(real_model_card)
 
-        assert all([r.status for r in results])
+    assert all([r.status == e for r, e in zip(results, expected_check_results)])