bwang0911 commited on
Commit
166b7b5
1 Parent(s): db438d9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +136 -0
README.md CHANGED
@@ -1,3 +1,139 @@
1
  ---
 
 
 
 
 
 
 
 
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ tags:
3
+ - sentence-transformers
4
+ - feature-extraction
5
+ - sentence-similarity
6
+ language:
7
+ - es
8
+ - en
9
+ inference: false
10
  license: apache-2.0
11
  ---
12
+ <!-- TODO: add evaluation results here -->
13
+ <br><br>
14
+
15
+ <p align="center">
16
+ <img src="https://aeiljuispo.cloudimg.io/v7/https://cdn-uploads.huggingface.co/production/uploads/603763514de52ff951d89793/AFoybzd5lpBQXEBrQHuTt.png?w=200&h=200&f=face" alt="Finetuner logo: Finetuner helps you to create experiments in order to improve embeddings on search tasks. It accompanies you to deliver the last mile of performance-tuning for neural search applications." width="150px">
17
+ </p>
18
+
19
+
20
+ <p align="center">
21
+ <b>The text embedding set trained by <a href="https://jina.ai/"><b>Jina AI</b></a>.</b>
22
+ </p>
23
+
24
+
25
+ ## Intended Usage & Model Info
26
+
27
+ `jina-embeddings-v2-base-es` is a Spanish/English bilingual text **embedding model** supporting **8192 sequence length**.
28
+ It is based on a BERT architecture (JinaBERT) that supports the symmetric bidirectional variant of [ALiBi](https://arxiv.org/abs/2108.12409) to allow longer sequence length.
29
+ We have designed it for high performance in mono-lingual & cross-lingual applications and trained it specifically to support mixed Spanish-English input without bias.
30
+ Additionally, we provide the following embedding models:
31
+
32
+ - [`jina-embeddings-v2-small-en`](https://huggingface.co/jinaai/jina-embeddings-v2-small-en): 33 million parameters.
33
+ - [`jina-embeddings-v2-base-en`](https://huggingface.co/jinaai/jina-embeddings-v2-base-en): 137 million parameters.
34
+ - [`jina-embeddings-v2-base-zh`](): Chinese-English Bilingual embeddings (soon).
35
+ - [`jina-embeddings-v2-base-de`](): German-English Bilingual embeddings (soon).
36
+ - [`jina-embeddings-v2-base-es`](): Spanish-English Bilingual embeddings (soon) **(you are here)**.
37
+
38
+ ## Data & Parameters
39
+
40
+ Jina Embeddings V2 [technical report](https://arxiv.org/abs/2310.19923)
41
+
42
+ ## Usage
43
+
44
+ **<details><summary>Please apply mean pooling when integrating the model.</summary>**
45
+ <p>
46
+
47
+ ### Why mean pooling?
48
+
49
+ `mean poooling` takes all token embeddings from model output and averaging them at sentence/paragraph level.
50
+ It has been proved to be the most effective way to produce high-quality sentence embeddings.
51
+ We offer an `encode` function to deal with this.
52
+
53
+ However, if you would like to do it without using the default `encode` function:
54
+
55
+ ```python
56
+ import torch
57
+ import torch.nn.functional as F
58
+ from transformers import AutoTokenizer, AutoModel
59
+
60
+ def mean_pooling(model_output, attention_mask):
61
+ token_embeddings = model_output[0]
62
+ input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
63
+ return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
64
+
65
+ sentences = ['How is the weather today?', 'What is the current weather like today?']
66
+
67
+ tokenizer = AutoTokenizer.from_pretrained('jinaai/jina-embeddings-v2-base-es')
68
+ model = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-base-es', trust_remote_code=True)
69
+
70
+ encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
71
+
72
+ with torch.no_grad():
73
+ model_output = model(**encoded_input)
74
+
75
+ embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
76
+ embeddings = F.normalize(embeddings, p=2, dim=1)
77
+ ```
78
+
79
+ </p>
80
+ </details>
81
+
82
+ You can use Jina Embedding models directly from transformers package:
83
+ ```python
84
+ !pip install transformers
85
+ from transformers import AutoModel
86
+ from numpy.linalg import norm
87
+
88
+ cos_sim = lambda a,b: (a @ b.T) / (norm(a)*norm(b))
89
+ model = AutoModel.from_pretrained('jinaai/jina-embeddings-v2-base-es', trust_remote_code=True) # trust_remote_code is needed to use the encode method
90
+ embeddings = model.encode(['How is the weather today?', '¿Qué tiempo hace hoy?'])
91
+ print(cos_sim(embeddings[0], embeddings[1]))
92
+ ```
93
+
94
+ If you only want to handle shorter sequence, such as 2k, pass the `max_length` parameter to the `encode` function:
95
+
96
+ ```python
97
+ embeddings = model.encode(
98
+ ['Very long ... document'],
99
+ max_length=2048
100
+ )
101
+ ```
102
+
103
+ ## Fully-managed Embeddings Service
104
+
105
+ Alternatively, you can use Jina AI's [Embedding platform](https://jina.ai/embeddings/) for fully-managed access to Jina Embeddings models.
106
+
107
+ ## Use Jina Embeddings for RAG
108
+
109
+ According to the latest blog post from [LLamaIndex](https://blog.llamaindex.ai/boosting-rag-picking-the-best-embedding-reranker-models-42d079022e83),
110
+
111
+ > In summary, to achieve the peak performance in both hit rate and MRR, the combination of OpenAI or JinaAI-Base embeddings with the CohereRerank/bge-reranker-large reranker stands out.
112
+
113
+ <img src="https://miro.medium.com/v2/resize:fit:4800/format:webp/1*ZP2RVejCZovF3FDCg-Bx3A.png" width="780px">
114
+
115
+
116
+ ## Plans
117
+
118
+ 1. Bilingual embedding models supporting more European & Asian languages, including French, Italian and Japanese.
119
+ 2. Multimodal embedding models enable Multimodal RAG applications.
120
+ 3. High-performt rerankers.
121
+
122
+ ## Contact
123
+
124
+ Join our [Discord community](https://discord.jina.ai) and chat with other community members about ideas.
125
+
126
+ ## Citation
127
+
128
+ If you find Jina Embeddings useful in your research, please cite the following paper:
129
+
130
+ ```
131
+ @misc{günther2023jina,
132
+ title={Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents},
133
+ author={Michael Günther and Jackmin Ong and Isabelle Mohr and Alaeddine Abdessalem and Tanguy Abel and Mohammad Kalim Akram and Susana Guzman and Georgios Mastrapas and Saba Sturua and Bo Wang and Maximilian Werk and Nan Wang and Han Xiao},
134
+ year={2023},
135
+ eprint={2310.19923},
136
+ archivePrefix={arXiv},
137
+ primaryClass={cs.CL}
138
+ }
139
+ ```