For example, in semantic search, we index a corpus of documents, with each document containing valuable information on a specific topic. Due to the way embedding models work, those documents will need to be chunked, and similarity is determined by chunk-level comparisons to the input query vector. Then, these similar chunks are returned back to the user. By finding an effective chunking strategy, we can ensure our search results accurately capture the essence of the user’s query.

If our chunks are too small or too large, it may lead to imprecise search results or missed opportunities to surface relevant content. As a rule of thumb, if the chunk of text makes sense without the surrounding context to a human, it will make sense to the language model as well. Therefore, finding the optimal chunk size for the documents in the corpus is crucial to ensuring that the search results are accurate and relevant.

These models find semantically similar sentences within one language or across languages:

sentence-transformers/distiluse-base-multilingual-cased-v1: Multilingual knowledge distilled version of multilingual Universal Sentence Encoder. Supports 15 languages: Arabic, Chinese, Dutch, English, French, German, Italian, Korean, Polish, Portuguese, Russian, Spanish, Turkish.

Characteristics of Sentence Transformer (a.k.a bi-encoder) models:

Calculates a fixed-size vector representation (embedding) given texts, images, audio, or video.

Embedding calculation is often efficient, embedding similarity calculation is very fast.

Applicable for a wide range of tasks, such as semantic textual similarity, semantic search, clustering, classification, paraphrase mining, and more.

Often used as a first step in a two-step retrieval process, where a Cross-Encoder (a.k.a. reranker) model is used to re-rank the top-k results from the bi-encoder.

The issue with multilingual BERT (mBERT) as well as with XLM-RoBERTa is that those produce rather bad sentence representation out-of-the-box. Further, the vectors spaces between languages are not aligned, i.e., the sentences with the same content in different languages would be mapped to different locations in the vector space.

In my publication Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation I describe an easy approach to extend sentence embeddings to further languages.

Chien Vu also wrote a nice blog article on this technique: A complete guide to transfer learning from English to other Languages using Sentence Embeddings BERT Models

The two families of similarity

The first family is lexical similarity. This is where local libraries make light work of the problem. If I want to match names, organisations, cities, account references, or noisy OCR output, then normalisation, entity extraction, fuzzy matching, and character n-grams are often fast, cheap, and surprisingly effective.

The second family is semantic similarity. This is where embeddings start to matter. If two phrases are conceptually related but use different wording, fuzzy matching often falls apart. Embeddings give you a vector representation that lets you compare meaning rather than spelling.