{"id":12463,"date":"2022-01-05T07:00:05","date_gmt":"2022-01-05T06:00:05","guid":{"rendered":"https:\/\/sii.pl\/blog\/?p=12463"},"modified":"2023-06-14T14:18:37","modified_gmt":"2023-06-14T12:18:37","slug":"specialized-semantic-search-engine","status":"publish","type":"post","link":"https:\/\/sii.pl\/blog\/en\/specialized-semantic-search-engine\/","title":{"rendered":"Specialized Semantic Search Engine"},"content":{"rendered":"\n

This article shares the experience collected in constructing a PoC of a semantic search engine for Polish courts\u2019 rulings. The project\u2019s goal was to determine possible gains over a classical, lexical-search solution that our customer used to work with daily.<\/p>\n\n\n\n

The most challenging parts were to deal with a highly specific language of the documents and to build a text-similarity machine learning model without an annotated dataset.<\/p>\n\n\n\n

Lexical search methods<\/h2>\n\n\n\n

Lexical search methods are word frequency-based approaches, what means a set of documents is ranked according to the query\u2019s term frequency regardless of their proximity and context within the document. Hence it may produce many false matches<\/strong>. Let\u2019s assume a user is looking for: a park in Barcelona worth visiting<\/em>. The method will probably return the correct document containing Park Guel in Barcelona <\/em>is a place everyone must see. Still, it may as well answer with Taxis Park near the stadium of FC Barcelona<\/em>. Both documents are equally good from the algorithm\u2019s perspective because both include the terms park<\/em> and Barcelona<\/em>.<\/p>\n\n\n\n

However, only the first one corresponds to the actual intention of the user. Although it\u2019s an artificial example, it clearly states the limitation of lexical search.<\/p>\n\n\n\n

Modern semantic approaches aspire to outperform lexical methods and promise to capture the real intention and context of search queries. Unlike lexical techniques, they don\u2019t rely solely on keyword matching but exploit deep neural networks trained on massive corpora and adapted for text-similarity tasks. Hence, they are expected to produce more accurate results.<\/p>\n\n\n\n

How To Capture Text Similarity?<\/h2>\n\n\n\n

Transformer-based models reached human-level performance in many challenging text processing tasks<\/strong>. Mainly BERT-based (Devling et al., 2018<\/a>) models have been widely applied thanks to their finetuning capabilities. As they are pre-trained on massive corpora, one could expect them to immediately capture the concept of text similarity and produce text embeddings directly applicable for similarity tasks.<\/p>\n\n\n\n

Unfortunately, it appears that \u201craw\u201d text embeddings are outperformed even by a much simpler models like GloVe (Penningtion et al., 2014<\/a>) vectors averaged over all words! Luckily, further supervised fine-tuning may help to improve these disappointing results.<\/p>\n\n\n\n

There are plenty of annotated datasets (at least for English) that may help to fine-tune a pretrained model to a text-similarity task, but they may vary in their interpretation of text similarity. Some consider text similarity as a textual entailment, where the truth from a given sentence needs to follow the fact from the other.<\/p>\n\n\n\n

Alternatively, textual agreement might be expressed in terms of reading comprehension, where the task is to determine if a given text fragment answers the stated question. Another idea is to consider similarity as a specific duplication problem. The table below presents some of the available datasets and corresponding similarity tasks.<\/p>\n\n\n\n

\"Sample<\/a>
Tab. 1 Sample labeled datasets that might be used for text-similarity fine-tuning<\/figcaption><\/figure>\n\n\n\n

In the literature, text similarity models are often compared with the Semantic Text Similiarity Benchmark<\/a>. It defines similarity as relatedness: a score from 1 to 5, reflecting how much two sentences are related to each other in terms of their semantic, syntactic, and lexical properties. The model performance is expressed as a correlation coefficient between predictions and ground truths.<\/strong><\/p>\n\n\n\n

The following table presents the performance of GloVE<\/a>, BERT<\/a>, and SBERT<\/a> (a model trained for text-similarity) on the STS benchmark<\/a>. It\u2019s easy to notice that SBERT<\/a> overwhelms the \u201craw\u201d approaches, which proves the need for task-specific adaptation. It also shows that CLS-vectors are not the best choice\u2026<\/p>\n\n\n\n

\"Models\u2019<\/a>
Tab. 2 Models\u2019 performance comparison on the STSb<\/a><\/figcaption><\/figure>\n\n\n\n

Lack of Annotated Data<\/h2>\n\n\n\n

Some language domains are significantly different from the everyday language. Documents created by lawyers, scientists, pharmacists utilize distinctive vocabulary and contain lengthy and detailed sentences. Hence to capture the nuances of such domain-specific text-similarity, one would require a domain-specific dataset.<\/strong> Unfortunately, such appears rarely, especially for less common languages like Polish.<\/p>\n\n\n\n

Therefore, one of the most significant challenges in our project was the lack of annotated dataset for our problem of text-similarity. Ultimately, we managed to get 450 queries\/answer pairs that we could use for testing purposes and better understand the requirements, but we had to turn into unsupervised approaches for training.<\/p>\n\n\n\n

Domain Adaptation<\/h3>\n\n\n\n

At this point, one may ask, why not use general-purpose language models adapted to text-similarity tasks, e.g., using HerBERT (Mroczkowski et al., 2021<\/a>) (BERT<\/a> for Polish) finetuned on the STSb<\/a> dataset for Polish. It\u2019d work indeed, however as presented by Gururangan et al. (2020)<\/a>, adapting models to a particular domain leads to significant performance gains. Moreover Wang et al. (2021)<\/a> show that unsupervised approaches trained on the domain data outperform supervised generic models.<\/p>\n\n\n\n

Those findings were especially interesting, as the domain language of our problem was meaningfully different from everyday Polish. While performing the initial data analysis, we extracted the 10K most frequent lemmas from the judgment documents<\/strong> and compared them with the 10K most frequent lemmas from the National Corpus of Polish<\/a> (excluding stop words). We determined there is just 34% of overlap between those two sets!<\/p>\n\n\n\n

\"Most<\/a>
Tab. 3 Most common lemmas in both datasets and their respective positions<\/figcaption><\/figure>\n\n\n\n

The Table 3. presents the top 5 lemmas from the judgment\u2019s corpus and National Corpus of Polish their respective position in the other dataset. Moreover, we\u2019ve also noticed the judgments\u2019 sentences are significantly longer than everyday Polish, and more frequently written in the passive voice.<\/p>\n\n\n\n

Thus, considering the specificity of our domain\u2019s language and the effectiveness of unsupervised approaches shown in the aforementioned papers, we\u2019ve decided to turn our attention to unsupervised methods. Moreover, we had access to 350 000 courts\u2019 rulings making 8GB large corpus.<\/p>\n\n\n\n

Unsupervised Semantic Search<\/h2>\n\n\n\n

Building a semantic search engine is about constructing an efficient model for text similarity and scaling the solution to perform an effective search among thousands or millions of documents. Therefore, there is a need to make the right architectural decisions<\/strong>. We dealt with roughly 350 000 documents, which split into text passages led to 16 million of data points. Thus, we decided to apply a typical retrieve-rerank pattern often used in such systems. It means a two-step solution made of a quick but not very accurate method to retrieve the list of potential matches and a precise but slow technique to re-rank the retrieved candidates.<\/p>\n\n\n\n

We decided to rely on the Okapi BM25<\/a> algorithm from Elasticsearch with spaCy<\/a> Polish lemmatizer for the retrieve part and then rerank the top-K candidates with the semantic similarity model.<\/p>\n\n\n\n

Unfortunately, none of the described unsupervised methods could be directly applied to our task as we dealt with a highly asymmetric search problem. The models presented in this article were designed to work with sentences of similar (symmetric) length, and in such circumstances, they perform well. However, they provide disappointing results once the queries are significantly shorter than the answers (asymmetric). While collecting the requirements, we noticed that the users form short non-factoid questions and expect explanatory answers often span around several sentences.<\/p>\n\n\n\n

We experimentally determined that a single query made of several words requires a text passage of a hundred words on average to answer it. Hence, we had to mitigate the query\/answers asymmetry length somehow to make the models working.<\/p>\n\n\n\n

\"High-level<\/a>
Fig. 1 High-level overview of the system based on the retrieve-rerank pattern<\/figcaption><\/figure>\n\n\n\n

Our approach might be summarized as replacing the BM25<\/a> scores with semantic scores computed for the retrieved text passages. We designed a method that calculates semantic scores of the terms found by BM25<\/a> based on their context with respect to the query and aggregates such partial results to score the entire text passage. We used BERT<\/a> trained on our 8GB corpus as the MLM, and we finetuned it for text similarity using STSb<\/a> translated to Polish. However, instead of using the model to construct the embeddings, we predict the similarity score directly given the query and the contexts of found terms<\/strong>. The whole solution was deployed on Microsoft Azure.<\/p>\n\n\n\n

The results<\/h3>\n\n\n\n

To evaluate the solution\u2019s performance, we relied on the MRR (Mean Reciprocal Rank)<\/a> metric, which is widely used in evaluating information retrieval systems. At first, we focused on the test set of 450 documents and noticed a significant improvement provided by the semantic approach, especially when using the semantic similarity model pretrained on the domain-specific data.<\/p>\n\n\n\n

Next, we extended the test set with all documents from the courts\u2019 rulings corpus what made the search task more difficult as we expected to identify 450 documents from the test set among other 350 000 records<\/strong>. In such conditions, the semantic search presented its robustness once again and appeared to be twice as good as the previous lexical system used by the client (baseline). What might be worrying are low MRR<\/a> values for this experiment.<\/p>\n\n\n\n

However, this might be explained because some of the 350 000 documents were better matches for the constructed queries but were simply not labeled. Meanwhile, the MRR<\/a> was calculated only on the 450 labeled documents. Manual verification of the results confirmed this thesis.<\/p>\n\n\n\n

\"Performance<\/a>
Tab. 4 Performance evaluation of the built system and comparison to the baseline (previous lexical-search system)<\/figcaption><\/figure>\n\n\n\n

I believe that the presented MRR<\/a> values don\u2019t correspond to the real performance measure of the system, as the test set was too small and could be biased. Secondly, the extended test set was not fully annotated. However, in each case, the relative comparison of different methods shows the robustness of the semantic approach. What\u2019s most important from the business perspective is the utterly positive feedback from our customer after one month of using the elaborated solution.<\/p>\n\n\n\n

\"An<\/a>
Tab. 5 An example query and results produced by the previous lexical system and our solution<\/figcaption><\/figure>\n\n\n\n

For the lexical approach, the found keywords are in bold. The results were truncated for readability. I’m not a lawyer, so I’m not sure if the previous system’s result is correct. However, I have no doubts about the correctness of the second result. The text was translated from Polish with the help of Deepl<\/a>.<\/p>\n\n\n\n

Conclusions<\/h2>\n\n\n\n

Adapting pretrained masked language models to particular domains and then further finetuning on a text-similarity auxiliary dataset seems to yield surprisingly good results. Even though one can\u2019t directly use such methods in scenarios where the length of the query is much shorter than the length of the answer, it\u2019s relatively easy to construct an algorithm leveraging the text-similarity models and handling such asymmetric cases.<\/p>\n\n\n\n

Thanks to pragmatic architectural decisions and thoughtful research of the existing unsupervised text-similarity methods, we proved that domain-specific semantic search produces way better results than lexical approaches.<\/p>\n\n\n

\n \n
\n \n
\n
\n \n\n
<\/div>\n <\/div>\n
\n \n\n
<\/div>\n <\/div>\n
\n \n\n
<\/div>\n <\/div>\n
\n \n\n
<\/div>\n <\/div>\n
\n \n\n
<\/div>\n <\/div>\n <\/div>\n \n
\n
\n \n\n
<\/div>\n <\/div>\n
\n \n\n
<\/div>\n <\/div>\n
\n \n\n
<\/div>\n <\/div>\n
\n \n\n
<\/div>\n <\/div>\n
\n \n\n
<\/div>\n <\/div>\n <\/div>\n<\/div>\n \n\n
\n 4.8\/5 ( votes: 6) <\/div>\n <\/div>\n","protected":false},"excerpt":{"rendered":"

This article shares the experience collected in constructing a PoC of a semantic search engine for Polish courts\u2019 rulings. The … Continued<\/a><\/p>\n","protected":false},"author":276,"featured_media":12479,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_editorskit_title_hidden":false,"_editorskit_reading_time":0,"_editorskit_is_block_options_detached":false,"_editorskit_block_options_position":"{}","inline_featured_image":false,"footnotes":""},"categories":[1319],"tags":[1442,1501,1352,1348],"class_list":["post-12463","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-soft-development","tag-ai-en","tag-artifiical-intelligence-en","tag-data-science-en","tag-search-engine-en"],"acf":[],"aioseo_notices":[],"republish_history":[],"featured_media_url":"https:\/\/sii.pl\/blog\/wp-content\/uploads\/2022\/01\/Specialized-Semantic-Search-Engine.png","category_names":["Soft development"],"_links":{"self":[{"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/posts\/12463"}],"collection":[{"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/users\/276"}],"replies":[{"embeddable":true,"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/comments?post=12463"}],"version-history":[{"count":2,"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/posts\/12463\/revisions"}],"predecessor-version":[{"id":22288,"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/posts\/12463\/revisions\/22288"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/media\/12479"}],"wp:attachment":[{"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/media?parent=12463"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/categories?post=12463"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sii.pl\/blog\/en\/wp-json\/wp\/v2\/tags?post=12463"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}