Cross Modal Retrieval with Querybank Normalisation
Abstract
Profiting from large-scale training datasets, advances in neural architecture design and efficient inference, joint embeddings have become the dominant approach for tackling cross-modal retrieval. In this work we first show that, despite their effectiveness, state-of-the-art joint embeddings suffer significantly from the longstanding "hubness problem" in which a small number of gallery embeddings form the nearest neighbours of many queries. Drawing inspiration from the NLP literature, we formulate a simple but effective framework called Querybank Normalisation (QB-Norm) that re-normalises query similarities to account for hubs in the embedding space. QB-Norm improves retrieval performance without requiring retraining. Differently from prior work, we show that QB-Norm works effectively without concurrent access to any test set queries. Within the QB-Norm framework, we also propose a novel similarity normalisation method, the Dynamic Inverted Softmax, that is significantly more robust than existing approaches. We showcase QB-Norm across a range of cross modal retrieval models and benchmarks where it consistently enhances strong baselines beyond the state of the art. Code is available at https://vladbogo.github.io/QB-Norm/.
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- CLIPRerank: An Extremely Simple Method for Improving Ad-hoc Video Search (2024)
- FreestyleRet: Retrieving Images from Style-Diversified Queries (2023)
- PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models (2023)
- Do Vision and Language Encoders Represent the World Similarly? (2024)
- LeanVec: Search your vectors faster by making them fit (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
@davanstrien I noticed that the @librarian-bot didn't respond to the second request for recommendations. I suspect this happens because there was a response already.
If this is the case, I think ideally it should respond with a link to the other message. I would love to contribute to the project and tackle this case. I would really appreciate if you could point me to the repo, since I wasn't able to find it.
@davanstrien I noticed that the @librarian-bot didn't respond to the second request for recommendations. I suspect this happens because there was a response already.
If this is the case, I think ideally it should respond with a link to the other message. I would love to contribute to the project and tackle this case. I would really appreciate if you could point me to the repo, since I wasn't able to find it.
Most of the code is here: https://huggingface.co/spaces/librarian-bots/recommend_similar_papers
I would prefer to wait with this for a bit before getting a librarian-bot to respond with more comments since I prefer librarian-bot not to be too noisy on papers. We might add threading to paper comments soon, which would then make it possible to reply to a user in a separate thread. At that point, I think it would make sense to allow multiple requests to librarian-bot to account for new papers coming out + to give a user a better message about what's not working. I will add some clearer wording to the comments to make it clearer that librarian-bot will currently only make one comment per paper.
@davanstrien makes sense! Great to hear that there will be threading to paper comments soon. Keep up the good work!
P.S. if there are some public tasks/issues related to this project, please let me know.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper