update

README.md

@@ -18,11 +18,18 @@ inference: false
 <br>
 
 <p align="center">
-        Qwen-VL <a href="https://modelscope.cn/models/qwen/Qwen-VL/summary">🤖 <a> | <a href="https://huggingface.co/Qwen/Qwen-VL">🤗</a>&nbsp ｜ Qwen-VL-Chat <a href="https://modelscope.cn/models/qwen/Qwen-VL-Chat/summary">🤖 <a>| <a href="https://huggingface.co/Qwen/Qwen-VL-Chat">🤗</a>&nbsp ｜ &nbsp<a href="https://modelscope.cn/studios/qwen/Qwen-VL-Chat-Demo/summary">Demo</a>&nbsp ｜ &nbsp<a href="https://github.com/QwenLM/Qwen-VL/blob/main/visual_memo.md">Report</a>&nbsp&nbsp | &nbsp&nbsp<a href="https://discord.gg/9bjvspyu">Discord</a>
+        Qwen-VL <a href="https://modelscope.cn/models/qwen/Qwen-VL/summary">🤖 <a> | <a href="https://huggingface.co/Qwen/Qwen-VL">🤗</a>&nbsp ｜ Qwen-VL-Chat <a href="https://modelscope.cn/models/qwen/Qwen-VL-Chat/summary">🤖 <a>| <a href="https://huggingface.co/Qwen/Qwen-VL-Chat">🤗</a>&nbsp ｜ &nbsp<a href="https://modelscope.cn/studios/qwen/Qwen-VL-Chat-Demo/summary">Demo</a>&nbsp ｜ &nbsp<a href="https://github.com/QwenLM/Qwen-VL/blob/master/visual_memo.md">Report</a>&nbsp&nbsp | &nbsp&nbsp<a href="https://discord.gg/9bjvspyu">Discord</a>
 
 </p>
 <br>
 
+**Qwen-VL** 是阿里云研发的大规模视觉语言模型（Large Vision Language Model, LVLM）。Qwen-VL 可以以图像、文本、检测框作为输入，并以文本和检测框作为输出。Qwen-VL 系列模型的特点包括：
+- **强大的性能**：在四大类多模态任务的标准英文测评中（Zero-shot Caption/VQA/DocVQA/Grounding）上，均取得同等通用模型大小下最好效果；
+- **多语言对话模型**：天然支持多语言对话，端到端支持图片里中英双语的长文本识别；
+- **多图交错对话**：支持多图输入和比较，指定图片问答，多图文学创作等；
+- **首个支持中文开放域定位的通用模型**：通过中文开放域语言表达进行检测框标注；
+- **细粒度识别和理解**：相比于目前其它开源LVLM使用的224分辨率，Qwen-VL是首个开源的448分辨率的LVLM模型。更高分辨率可以提升细粒度的文字识别、文档问答和检测框标注。
+
 **Qwen-VL** (Qwen Large Vision Language Model) is the visual multimodal version of the large model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-VL accepts image, text, and bounding box as inputs, outputs text and bounding box. The features of Qwen-VL include:
 - **Strong performance**: It significantly surpasses existing open-source Large Vision Language Models (LVLM) under similar scale settings on multiple English evaluation benchmarks (including Zero-shot caption, VQA, DocVQA, and Grounding).
 - **Multi-lingual LVLM support text recognization**: Qwen-VL naturally supports multi-lingual conversation, and it promotes end-to-end recognition of Chinese and English bi-lingual text in images.
@@ -30,13 +37,33 @@ inference: false
 - **First generalist model support grounding in Chinese**: Detecting bounding boxes through open-domain language expression in both Chinese and English.
 - **Fine-grained recognization and understanding**: Compared to the 224 resolution currently used by other open-source LVLM, the 448 resolution promotes fine-grained text recognition, document QA, and bounding box annotation.
 
+目前，我们提供了 Qwen-VL 系列的两个模型：
+- Qwen-VL: Qwen-VL 以 Qwen-7B 的预训练模型作为语言模型的初始化，并以 [Openclip ViT-bigG](https://github.com/mlfoundations/open_clip) 作为视觉编码器的初始化，中间加入单层随机初始化的 cross-attention，经过约1.5B的图文数据训练得到。最终图像输入分辨率为448。
+- Qwen-VL-Chat: 在 Qwen-VL 的基础上，我们使用对齐机制打造了基于大语言模型的视觉AI助手Qwen-VL-Chat，其训练数据涵盖了 QWen-7B 的纯文本 SFT 数据、开源 LVLM 的 SFT 数据、数据合成和人工标注的图文对齐数据。
+
+如果想了解更多关于模型的信息，请点击[链接](https://github.com/QwenLM/Qwen-VL/blob/master/visual_memo.md)查看我们的技术备忘录。
+
 We release two models of the Qwen-VL series:
 - Qwen-VL: The pre-trained LVLM model uses Qwen-7B as the initialization of the LLM, and [Openclip ViT-bigG](https://github.com/mlfoundations/open_clip) as the initialization of the visual encoder. And connects them with a randomly initialized cross-attention layer. Qwen-VL was trained on about 1.5B image-text paired data. The final image input resolution is 448.
 - Qwen-VL-Chat: A multimodal LLM-based AI assistant, which is trained with alignment techniques.
 
-For more details about Qwen-VL, please refer to our [technical memo](visual_memo.md).
+For more details about Qwen-VL, please refer to our [technical memo](https://github.com/QwenLM/Qwen-VL/blob/master/visual_memo.md).
+
+## 评测
 
-## Evaluation
+我们从两个角度评测了两个模型的能力：
+1. 在**英文标准 Benchmark** 上评测模型的基础任务能力。目前评测了四大类多模态任务：
+    - Zero-shot Caption: 评测模型在未见过数据集上的零样本图片描述能力；
+    - General VQA: 评测模型的通用问答能力，例如判断题、颜色、个数、类目等问答能力；
+    - Text-based VQA：评测模型对于图片中文字相关的识别/问答能力，例如文档问答、图表问答、文字问答等；
+    - Referring Expression Compression：评测模型给定物体描述画检测框的能力；
+
+2. **试金石 (TouchStone)**：为了评测模型整体的图文对话能力和人类对齐水平。我们为此构建了一个基于 GPT4 打分来评测 LVLM 模型的 Benchmark：TouchStone。在 TouchStone-v0.1 中：
+    - 评测基准总计涵盖 300+张图片、800+道题目、27个类别。包括基础属性问答、人物地标问答、影视作品问答、视觉推理、反事实推理、诗歌创作、故事写作，商品比较、图片解题等**尽可能广泛的类别**。
+    - 为了弥补目前 GPT4 无法直接读取图片的缺陷，我们给所有的带评测图片提供了**人工标注的充分详细描述**，并且将图片的详细描述、问题和模型的输出结果一起交给 GPT4 打分。
+    - 评测同时包含英文版本和中文版本。
+  
+评测结果如下：
 
 We evaluated the model's ability from two perspectives:
 1. **Standard Benchmarks**: We evaluate the model's basic task capabilities on four major categories of multimodal tasks:
@@ -58,13 +85,13 @@ Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has
     <img src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/radar.png" width="600"/>
 <p>
 
-### Zero-shot Caption & General VQA
+### Zero-shot Captioning & General VQA
 <table>
 <thead>
   <tr>
     <th rowspan="2">Model type</th>
     <th rowspan="2">Model</th>
-    <th colspan="2">Zero-shot Caption</th>
+    <th colspan="2">Zero-shot Captioning</th>
     <th colspan="5">General VQA</th>
   </tr>
   <tr>
@@ -79,7 +106,7 @@ Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has
 </thead>
 <tbody align="center">
   <tr>
-    <td rowspan="12">Generalist<br>Models</td>
+    <td rowspan="10">Generalist<br>Models</td>
     <td>Flamingo-9B</td>
     <td>-</td>
     <td>61.5</td>
@@ -166,10 +193,10 @@ Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has
     <td><b>78.8</b></td>
     <td><b>58.6</b></td>
     <td><b>59.3</b></td>
-    <td><b>67.1</b></td>
-    <td><b>34.3</b></td>
+    <td>67.1</td>
+    <td>35.2</td>
   </tr>
-  <tr>
+  <!-- <tr>
     <td>Qwen-VL (4-shot)</td>
     <td>-</td>
     <td>-</td>
@@ -178,18 +205,18 @@ Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has
     <td>-</td>
     <td>-</td>
     <td>39.1</td>
-  </tr>
+  </tr> -->
   <tr>
     <td>Qwen-VL-Chat</td>
-    <td>-</td>
-    <td>81.5</td>
-    <td>-</td>
-    <td>56.69</td>
-    <td>-</td>
-    <td>68.22</td>
-    <td>37.05</td>
-  </tr>
-  <tr>
+    <td>120.2</td>
+    <td>81.0</td>
+    <td>78.2</td>
+    <td>56.6</td>
+    <td>57.5</td>
+    <td><b>68.2</b></td>
+    <td><b>38.9</b></td>
+  </tr>
+  <!-- <tr>
     <td>Qwen-VL-Chat (4-shot)</td>
     <td>-</td>
     <td>-</td>
@@ -197,8 +224,8 @@ Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has
     <td>60.6</td>
     <td>-</td>
     <td>-</td>
-    <td>45.5</td>
-  </tr>
+    <td>44.45</td>
+  </tr> -->
   <tr>
     <td>Previous SOTA<br>(Per Task Fine-tuning)</td>
     <td>-</td>
@@ -213,10 +240,13 @@ Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has
 </tbody>
 </table>
 
+- 在 Zero-shot Caption 中，Qwen-VL 在 Flickr30K 数据集上取得了 **SOTA** 的结果，并在 Nocaps 数据集上取得了和 InstructBlip 可竞争的结果。
+- 在 General VQA 中，Qwen-VL 取得了 LVLM 模型同等量级和设定下 **SOTA** 的结果。
+
 - For zero-shot image captioning, Qwen-VL achieves the **SOTA** on Flickr30K and competitive results on Nocaps with InstructBlip.
 - For general VQA, Qwen-VL achieves the **SOTA** under the same generalist LVLM scale settings.
 
-### Text-based VQA (focuse on text understanding capabilities in images)
+### Text-oriented VQA (focuse on text understanding capabilities in images)
 
 <table>
 <thead>
@@ -284,6 +314,9 @@ Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has
 </tbody>
 </table>
 
+- 在文字相关的识别/问答评测上，取得了当前规模下通用 LVLM 达到的最好结果。
+- 分辨率对上述某几个评测非常重要，大部分 224 分辨率的开源 LVLM 模型无法完成以上评测，或只能通过切图的方式解决。Qwen-VL 将分辨率提升到 448，可以直接以端到端的方式进行以上评测。Qwen-VL 在很多任务上甚至超过了 1024 分辨率的 Pic2Struct-Large 模型。
+
 - In text-related recognition/QA evaluation, Qwen-VL achieves the SOTA under the generalist LVLM scale settings.
 - Resolution is important for several above evaluations. While most open-source LVLM models with 224 resolution are incapable of these evaluations or can only solve these by cutting images, Qwen-VL scales the resolution to 448 so that it can be evaluated end-to-end. Qwen-VL even outperforms Pic2Struct-Large models of 1024 resolution on some tasks.
 
@@ -392,22 +425,21 @@ Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has
     <td><b>83.12</b></td>
     <td>88.25</td>
     <td><b>77.21</b></td>
-    <td><b>85.58</b></td>
-    <td><b>85.48</b></td>
+    <td>85.58</td>
+    <td>85.48</td>
     <td>78.22</td>
   </tr>
   <tr>
     <td>Qwen-VL-7B-Chat</td>
-    <td><b>88.55</b></td>
+    <td>88.55</td>
     <td><b>92.27</b></td>
     <td>84.51</td>
     <td>82.82</td>
     <td><b>88.59</b></td>
+    <td>76.79</td>
+    <td><b>85.96</b></td>
+    <td><b>86.32</b></td>
     <td>-</td>
-    <td>-</td>
-    <td>-</td>
-    <td>-</td>
-  </tr>
   <tr>
     <td rowspan="3">Specialist SOTAs<br>(Specialist/Finetuned)</td>
     <td>G-DINO-L</td>
@@ -448,6 +480,11 @@ Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has
 </tbody>
 </table>
 
+- 在定位任务上，Qwen-VL 全面超过 Shikra-13B，取得了目前 Generalist LVLM 模型上在 Refcoco 上的 **SOTA**。
+- Qwen-VL 并没有在任何中文定位数据上训练过，但通过中文 Caption 数据和 英文 Grounding 数据的训练，可以 Zero-shot 泛化出中文 Grounding 能力。
+
+我们提供了以上**所有**评测脚本以供复现我们的实验结果。请阅读 [eval/EVALUATION.md](eval/EVALUATION.md) 了解更多信息。
+
 - Qwen-VL achieves the **SOTA** in all above referring expression comprehension benchmarks.
 - Qwen-VL has not been trained on any Chinese grounding data, but it can still generalize to the Chinese Grounding tasks in a zero-shot way by training Chinese Caption data and English Grounding data.
 
@@ -455,6 +492,8 @@ We provide all of the above evaluation scripts for reproducing our experimental
 
 ### Chat evaluation
 
+TouchStone 是一个基于 GPT4 打分来评测 LVLM 模型的图文对话能力和人类对齐水平的基准。它涵盖了 300+张图片、800+道题目、27个类别，包括基础属性、人物地标、视觉推理、诗歌创作、故事写作、商品比较、图片解题等**尽可能广泛的类别**。关于 TouchStone 的详细介绍，请参考[touchstone/README_CN.md](touchstone/README_CN.md)了解更多信息。
+
 TouchStone is a benchmark based on scoring with GPT4 to evaluate the abilities of the LVLM model on text-image dialogue and alignment levels with humans. It covers a total of 300+ images, 800+ questions, and 27 categories, such as attribute-based Q&A, celebrity recognition, writing poetry, summarizing multiple images, product comparison, math problem solving, etc. Please read [touchstone/README_CN.md](touchstone/README.md) for more information.
 
 #### English evaluation
@@ -476,17 +515,27 @@ TouchStone is a benchmark based on scoring with GPT4 to evaluate the abilities o
 | VisualGLM     | 247.1 |
 | Qwen-VL-Chat   | 401.2 |
 
+Qwen-VL-Chat 模型在中英文的对齐评测中均取得当前 LVLM 模型下的最好结果。
+
 Qwen-VL-Chat has achieved the best results in both Chinese and English alignment evaluation.
 
 ## Requirements
 
+* python 3.8及以上版本
+* pytorch 1.12及以上版本，推荐2.0及以上版本
+* 建议使用CUDA 11.4及以上（GPU用户需考虑此选项）
+
 * python 3.8 and above
 * pytorch 1.12 and above, 2.0 and above are recommended
 * CUDA 11.4 and above are recommended (this is for GPU users)
 
 ## Quickstart
 
-Below, we provide simple examples to show how to use Qwen-VL and Qwen-VL-Chat with 🤖 ModelScope and 🤗 Transformers.
+我们提供简单的示例来说明如何利用 🤗 Transformers 快速使用 Qwen-VL 和 Qwen-VL-Chat。
+
+在开始前，请确保你已经配置好环境并安装好相关的代码包。最重要的是，确保你满足上述要求，然后安装相关的依赖库。
+
+Below, we provide simple examples to show how to use Qwen-VL and Qwen-VL-Chat with 🤗 Transformers.
 
 Before running the code, make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.
 
@@ -494,7 +543,9 @@ Before running the code, make sure you have setup the environment and installed
 pip install -r requirements.txt
 ```
 
-Now you can start with ModelScope or Transformers. More usage aboue vision encoder, please refer to [FAQ](FAQ.md).
+接下来你可以开始使用Transformers来使用我们的模型。关于视觉模块的更多用法，请参考[教程](TUTORIAL.md)。
+
+Now you can start with Transformers. More usage aboue vision encoder, please refer to [tutorial](TUTORIAL_zh.md).
 
 #### 🤗 Transformers
 
@@ -544,14 +595,20 @@ else:
 
 ## FAQ
 
-If you meet problems, please refer to [FAQ](FAQ.md) and the issues first to search a solution before you launch a new issue.
+如遇到问题，敬请查阅 [FAQ](https://github.com/QwenLM/Qwen-VL/blob/master/FAQ_zh.md)以及issue区，如仍无法解决再提交issue。
+
+If you meet problems, please refer to [FAQ](https://github.com/QwenLM/Qwen-VL/blob/master/FAQ.md) and the issues first to search a solution before you launch a new issue.
 
 
 ## License Agreement
 
+研究人员与开发者可使用Qwen-VL和Qwen-VL-Chat或进行二次开发。我们同样允许商业使用，具体细节请查看[LICENSE](https://github.com/QwenLM/Qwen-VL/blob/master/LICENSE)。如需商用，请填写[问卷](https://dashscope.console.aliyun.com/openModelApply/qianwen)申请。
+
 Researchers and developers are free to use the codes and model weights of both Qwen-VL and Qwen-VL-Chat. We also allow their commercial use. Check our license at [LICENSE](LICENSE) for more details.
 
 ## Contact Us
 
+如果你想给我们的研发团队和产品团队留言，请通过邮件（[email protected]）联系我们。
+
 If you are interested to leave a message to either our research team or product team, feel free to send an email to [email protected].