Update README.md

README.md

@@ -1,3 +1,57 @@
 ---
+language: 
+  - en
 license: apache-2.0
+
+tags:
+- bart
+- biobart
+- biomedical
+
+inference: true
+
+widget:
+- text: "Influenza is a <mask> disease."
+- type: "text-generation"
+
 ---
+# Yuyuan-Bart-139M, one model of [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM).
+The Yuyuan-Bart-400M is a biomedical generative language model jointly produced by Tsinghua University and International Digital Economy Academy.
+
+Paper: [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://arxiv.org/pdf/2204.03905.pdf)
+
+## Pretraining Corpora 
+We use PubMed abstracts as the pretraining corpora. The corpora contain about 41 GB of biomedical research paper abstracts on PubMed.
+
+## Pretraining Setup 
+We continuously pretrain large versions of BART for 120k steps with a batch size of 2560. We use the same vocabulary as BART to tokenize the texts. Although the input length limitation of BART is 1024, the tokenized PubMed abstracts rarely exceed 512. Therefore, for the sake of training efficiency, we truncate all the input texts to 512 maximum length. We mask 30% of the input tokens and the masked span length is determined by sampling from a Poisson distribution (λ = 3) as used in BART. We use a learning rate scheduler of 0.02 warm-up ratio and linear decay. The learning rate is set to 1e-4. We train the large version of BioBART(400M parameters) on 2 DGX with 16 40GB A100 GPUs for about 168 hours with the help of the open-resource framework DeepSpeed.
+
+
+
+## Usage
+```python
+from transformers import BartForConditionalGeneration, BartTokenizer
+tokenizer = BartTokenizer.from_pretrained('IDEA-CCNL/Yuyuan-Bart-400M')
+model = BartForConditionalGeneration.from_pretrained('IDEA-CCNL/Yuyuan-Bart-400M')
+
+text = 'Influenza is a <mask> disease.'
+input_ids = tokenizer([text], return_tensors="pt")['input_ids']
+model.eval()
+generated_ids = model.generate(
+    input_ids=input_ids,
+)
+preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids]
+print(preds)
+```
+
+## Citation
+If you find the resource is useful, please cite the following website in your paper.
+```
+@misc{BioBART,
+  title={BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model},
+  author={Hongyi Yuan and Zheng Yuan and Ruyi Gan and Jiaxing Zhang and Yutao Xie and Sheng Yu},
+  year={2022},
+  eprint={2204.03905},
+  archivePrefix={arXiv}
+}
+```