Update README.md

README.md

@@ -9,7 +9,8 @@ widget:
 - text: Fui a la librería a comprar un <mask>.
 ---
 
-- [Version v1](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1) (default): July 26th, 2021
+- [Version v2](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v2) (default): April 28th, 2022
+- [Version v1](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1): July 26th, 2021
 - [Version v1-512](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/v1-512): July 26th, 2021
 - [Version beta](https://huggingface.co/bertin-project/bertin-roberta-base-spanish/tree/beta): July 15th, 2021
 
@@ -26,6 +27,62 @@ This is part of the
 
 The aim of this project was to pre-train a RoBERTa-base model from scratch during the Flax/JAX Community Event, in which Google Cloud provided free TPUv3-8 to do the training using Huggingface's Flax implementations of their library.
 
+
+## Team members
+
+- Javier de la Rosa ([versae](https://huggingface.co/versae))
+- Eduardo González ([edugp](https://huggingface.co/edugp))
+- Paulo Villegas ([paulo](https://huggingface.co/paulo))
+- Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
+- Manu Romero ([mrm8488](https://huggingface.co/))
+- María Grandury ([mariagrandury](https://huggingface.co/))
+
+
+## Citation and Related Information
+
+To cite this model:
+```bibtex
+@article{BERTIN,
+	author = {Javier De la Rosa y Eduardo G. Ponferrada y Manu Romero y Paulo Villegas y Pablo González de Prado Salas y María Grandury},
+	title = {BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling},
+	journal = {Procesamiento del Lenguaje Natural},
+	volume = {68},
+	number = {0},
+	year = {2022},
+	keywords = {},
+	abstract = {The pre-training of large language models usually requires massive amounts of resources, both in terms of computation and data. Frequently used web sources such as Common Crawl might contain enough noise to make this pretraining sub-optimal. In this work, we experiment with different sampling methods from the Spanish version of mC4, and present a novel data-centric technique which we name perplexity sampling that enables the pre-training of language models in roughly half the amount of steps and using one fifth of the data. The resulting models are comparable to the current state-of-the-art, and even achieve better results for certain tasks. Our work is proof of the versatility of Transformers, and paves the way for small teams to train their models on a limited budget.},
+	issn = {1989-7553},
+	url = {http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6403},
+	pages = {13--23}
+}
+```
+
+If you use this model, we would love to hear about it! Reach out on twitter, GitHub, Discord, or shoot us an email.
+
+## Team
+
+- Javier de la Rosa ([versae](https://huggingface.co/versae))
+- Eduardo González ([edugp](https://huggingface.co/edugp))
+- Paulo Villegas ([paulo](https://huggingface.co/paulo))
+- Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
+- Manu Romero ([mrm8488](https://huggingface.co/))
+- María Grandury ([mariagrandury](https://huggingface.co/))
+
+## Acknowledgements
+
+This project would not have been possible without compute generously provided by the National Library of Norway and Google through the
+[TPU Research Cloud](https://sites.research.google/trc/), as well as the Cloud TPU team for providing early access to the [Cloud TPU VM](https://cloud.google.com/blog/products/compute/introducing-cloud-tpu-vms) Alpha. And specially, to [Stella Biderman](https://www.stellabiderman.com) for her general openness, and [Ben Wang](https://github.com/kingoflolz/mesh-transformer-jax) for the main codebase.
+
+## Disclaimer
+
+The models published in this repository are intended for a generalist purpose and are available to third parties. These models may have bias and/or any other undesirable distortions. When third parties, deploy or provide systems and/or services to other parties using any of these models (or using systems based on these models) or become users of the models, they should note that it is their responsibility to mitigate the risks arising from their use and, in any event, to comply with applicable regulations, including regulations regarding the use of artificial intelligence. In no event shall the owner of the models be liable for any results arising from the use made by third parties of these models.
+
+<hr>
+
+
+<details>
+<summary>Full report</summary>
+
 # Motivation
 
 According to [Wikipedia](https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers), Spanish is the second most-spoken language in the world by native speakers (>470 million speakers), only after Chinese, and the fourth including those who speak it as a second language. However, most NLP research is still mainly available in English. Relevant contributions like BERT, XLNet or GPT2 sometimes take years to be available in Spanish and, when they do, it is often via multilingual versions which are not as performant as the English alternative.
@@ -450,15 +507,6 @@ At a personal level, the experience has been incredible for all of us. We believ
 
 Given our good results, on par with those of large corporations, we hope our work will inspire and set the basis for more small teams to play and experiment with language models on smaller subsets of huge datasets.
 
-## Team members
-
-- Javier de la Rosa ([versae](https://huggingface.co/versae))
-- Eduardo González ([edugp](https://huggingface.co/edugp))
-- Paulo Villegas ([paulo](https://huggingface.co/paulo))
-- Pablo González de Prado ([Pablogps](https://huggingface.co/Pablogps))
-- Manu Romero ([mrm8488](https://huggingface.co/))
-- María Grandury ([mariagrandury](https://huggingface.co/))
-
 ## Useful links
 
 - [Community Week timeline](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104#summary-timeline-calendar-6)
@@ -468,14 +516,4 @@ Given our good results, on par with those of large corporations, we hope our wor
 - [Masked Language Modelling example scripts](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)
 - [Model Repository](https://huggingface.co/flax-community/bertin-roberta-large-spanish/)
 
-## References
-
-- Heafield, K. (2011). KenLM: faster and smaller language model queries. Proceedings of the EMNLP2011 Sixth Workshop on Statistical Machine Translation.
-
-- Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2021). Deduplicating Training Data Makes Language Models Better. arXiv preprint arXiv:2107.06499.
-
-- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
-
-- Ney, H., Essen, U., & Kneser, R. (1994). On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language, 8(1), 1-38.
-
-- Wenzek, G., Lachaux, M. A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., & Grave, E. (2019). Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359.
+</details>