update with IRR and new images

examples.py

@@ -36,3 +36,15 @@ def app():
     st.subheader("una coppia che passeggia sulla spiaggia al tramonto")
     st.markdown("*a couple walking on the beach at sunset*")
     st.image("static/img/examples/couple_3.jpeg")
+
+    st.markdown("### 2. Dresses")
+
+    col1, col2 = st.beta_columns(2)
+    col1.subheader("un vestito primavrile")
+    col1.markdown("*a dress for the spring*")
+    col1.image("static/img/examples/vestito1.png")
+
+    col2.subheader("un vestito autunnale")
+    col2.markdown("*a dress for the autumn*")
+    col2.image("static/img/examples/vestito_autunnale.png")
+

introduction.md

@@ -25,6 +25,9 @@ have the highest similarity with the text query.
 + *Image to Text*: This task is essentially a zero-shot image classification task. The user is asked for an image and for a set of captions/labels and CLIP
 is going to compute the similarity between the image and each label. The webapp is going to display a probability distribution over the captions.
 
++ *Examples and Applications*: This page showcases some interesting results we got from the model, we believe that there are 
+different applications that can start from here.
+
 # Novel Contributions
 
 The original CLIP model was trained on 400 million image-text pairs; this amount of data is not available for Italian. 
@@ -34,6 +37,11 @@ To get competitive results we followed three strategies:
   2. better augmentations;
   3. better training.
 
+For those interested, we have a [Comet](https://www.comet.ml/g8a9/clip-italian/reports/clip-italian-training-metrics) 
+that shows a **subset** of the experiments we run. Different hyper-parameters played a role in reducing the validation 
+loss. The optimizer gave us great performance and huge conversion speed, more data and augmentations helped a lot in generalizing,
+working on the training and on the loss gave us the final increase that you can see in the results.
+
 ## More and Better Data
 
 We eventually had to deal with the fact that we do not have the same data that OpenAI had during the training of CLIP.
@@ -65,6 +73,22 @@ a dataset with 700K translated captions.
 + [La Foto del Giorno](https://www.ilpost.it/foto-del-giorno/). This image-caption dataset is collected from [Il Post](https://www.ilpost.it/), a prominent Italian online newspaper. The collection contains almost 30K pairs: starting from early 2011, for each day, editors at Il Post pick several images picturing the most salient events in the world. Each photo comes along with an Italian caption.
 
 
+### A Note on Translations
+
+Instead of relying on open-source translators, we decided to use DeepL. **Translation quality** of the data was the main 
+reason of this choice. With the few images (wrt OpenAI) that we have, we cannot risk polluting our own data. CC is a great resource
+but the captions have to be handled accordingly. We translated 700K captions and we evaluated their quality:
+
+Two of us looked at a sample of 100 of the translations and rated them with scores from 1 to 4. 
+1: the sentence has lost is meaning or it's not possible to understand it; 2: it is possible to get the idea
+but there something wrong; 3: good, however a native speaker might complain about some translations; 4: good translation.
+
+The average score was of 3.8 and the two annotators had an inter-rater agreement - computed with [Gwet's AC1](https://bpspsychub.onlinelibrary.wiley.com/doi/full/10.1348/000711006X126600) using ordinal 
+weighting - of 0.86 (great agreement!). 
+
+We know that we annotated our own data; in the spirit of fairness we also share the annotations and the captions so
+that those interested can check the quality. The Google Sheet is [here](https://docs.google.com/spreadsheets/d/1m6TkcpJbmJlEygL7SXURIq2w8ZHuVvsmdEuCIH0VENk/edit?usp=sharing).
+
 ## Better Augmentations
 
 We knew that without a good augmentation strategy we could never get competitive results to a model trained on 400 million images. Therefor we implemented heavy augmentations to make the training more data efficient. They include random affine transformations and perspective changes, as well as occasional equalization and random changes to brightness, contrast, saturation and hue. We made sure to keep hue augmentations limited however to still give the model the ability to learn color definitions.  
@@ -116,6 +140,14 @@ We selected two different tasks:
 + image-retrieval 
 + zero-shot classification
 
+### Reproducibiliy
+
+Both experiments should be very easy to replicate, we share the two colab notebook we used to compute the two results
+
++ [Image Retrieval](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing)
++ [ImageNet Zero Shot Evaluation](https://colab.research.google.com/drive/1zfWeVWY79XXH63Ci-pk8xxx3Vu_RRgW-?usp=sharing)
+
+
 ### Image Retrieval
 
 This experiment is run against the MSCOCO-IT validation set (that we haven't used in training). Given in input
@@ -131,7 +163,6 @@ we use the MRR.
 It is true that we used MSCOCO-IT in training, and this might give us an advantage. However the original CLIP model was trained
 on 400million images (and some of them probably were from MSCOCO).
 
-You can find the colab to quickly rerun the experiments here: [Colab](https://colab.research.google.com/drive/1bLVwVKpAndpEDHqjzxVPr_9nGrSbuOQd?usp=sharing)
 
 ### Zero-shot image classification
 
@@ -146,13 +177,13 @@ To do this, we used DeepL to translate the image labels in ImageNet. We evaluate
 | Accuracy@10     |  **52.55**   | 42.91 |
 | Accuracy@100    |  **81.08**   | 67.11 |
 
-You can find the colab to quickly rerun the experiments here: [ImageNet Zero Shot Evaluation](https://colab.research.google.com/drive/1zfWeVWY79XXH63Ci-pk8xxx3Vu_RRgW-?usp=sharing)
-
 Our results confirm that CLIP-Italian is very competitive and beats mCLIP on the two different task
 we have been testing. Note, however, that our results are lower than those shown in the original OpenAI
 paper (see, [Radford et al., 2021](https://arxiv.org/abs/2103.00020)). However, considering that our results are in line with those obtained by mCLIP we think that 
 the translated image labels might have had an impact on the final scores.
 
+
+
 ## Qualitative Evaluation
 
 We hereby show some very interesting properties of the model. One is its ability to detect colors, 
@@ -199,6 +230,8 @@ Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018, July). [Conceptual capti
 
 Srinivasan, K., Raman, K., Chen, J., Bendersky, M., & Najork, M. (2021). [WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning](https://arxiv.org/pdf/2103.01913.pdf). arXiv preprint arXiv:2103.01913.
 
+Gwet, K. L. (2008). [Computing inter‐rater reliability and its variance in the presence of high agreement.](https://bpspsychub.onlinelibrary.wiley.com/doi/full/10.1348/000711006X126600) British Journal of Mathematical and Statistical Psychology, 61(1), 29-48.
+
 Reimers, N., & Gurevych, I. (2020, November). [Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation.](https://aclanthology.org/2020.emnlp-main.365/) In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 4512-4525).
 
 Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). [Learning Transferable Visual Models From Natural Language Supervision.](https://arxiv.org/abs/2103.00020) ICML.