Sease, profile picture
Uploaded bySease
7,476 views

Let's Build an Inverted Index: Introduction to Apache Lucene/Solr

The document is a seminar presentation about building an inverted index with Apache Lucene and Solr, covering the importance of open-source software in information retrieval. It outlines the roles of various data structures, the workings of Apache Lucene and Solr, and key concepts such as indexing, text analysis, and query operations. Additionally, it discusses the contributions to and benefits of open-source projects, as well as advanced filtering techniques in search functionalities.

Engineering◦
Related topics:
SEO and Search Engines•Information Retrieval•
In this document
Powered by AI

Presentation on Apache Lucene/Solr by Alessandro Benedetti and Andrea Gazzarini, highlighting their expertise in software engineering and information retrieval.

Overview of the Search Services team, their open-source culture, and expertise in Apache Lucene/Solr technologies.

Advantages of using open-source software include state-of-the-art technology, community support, and accessibility.

Services offered include training and consulting in open-source information retrieval technologies like Lucene/Solr.

Definition and scope of Information Retrieval (IR), emphasizing its processes related to searching documents and metadata.

Lucene is described as a high-performance library for scalable information retrieval, enabling search in applications.

Solr is a scalable search server based on Lucene, known for its reliability and extensive features for enterprise applications.

The Inverted Index is crucial for documents' search capabilities, as it maps content to its location in databases.

Documents in Lucene consist of indexed fields, with details on the structure and immutability of index segments.

Schema configurations dictate how an inverted index is constructed, defining fields and their attributes.

Field Types determine term generation within the index, and text analysis involves various processing elements like tokenizers.

Token filters, including word delimiters, stopword filters, and stemmers, improve search accuracy and index efficiency.

Synonym filters enhance recall in search results, allowing accurate retrieval of similar terms and phrases.

Introduction of Keep Word, N-Gram, and Phonetic Matching filters to improve search recall and precision.

Overview of field attributes affecting searches and hands-on exploration of schema.xml in Solr.

Discussion on how documents are indexed in Solr, including usage of Solr Cell framework and transaction log management.

Lucene assigns scores to results based on term frequency, document frequency, and BM25 scoring mechanisms.

Detailed explanation of BM25 scoring factors and the list of query parameters available in Solr.

Query parsing includes the use of various query filters that determine score calculations and result caching.

Overview of date query syntax for Solr and debugging options to troubleshoot query processing.

Descriptions of two master thesis projects focused on click models and search quality evaluation in software development.

Embed presentation

Downloaded 28 times
Seminars Let’s Build an Inverted Index: Introduction to Apache Lucene/Solr
 Alessandro Benedetti, Software Engineer
 
 Andrea Gazzarini, Software Engineer 28th November 2019
Seminars ▪ R&D Software Engineer ▪ Search Consultant ▪ Director ▪ Master Degree in Computer Science ▪ Apache Lucene/Solr Enthusiast ▪ Semantic, NLP, Machine Learning Technologies passionate ▪ Conference Speaker ▪ Beach Volleyball Player & Snowboarder Alessandro Benedetti
Seminars ▪ Software Engineer (1999-) ▪ “Hermit” Software Engineer (2010-) ▪ Java & Information Retrieval Passionate ▪ Apache Qpid (past) Committer ▪ Husband & Father ▪ Bass Player Andrea Gazzarini, “Gazza”
Seminars Search Services
 ● London Based - Italian made :) ● Open Source Enthusiasts ● Apache Lucene/Solr experts ! Community Contributors ● Active Researchers ● Hot Trends : Learning To Rank, 
 Document Similarity, Search Quality Evaluation,
 Relevancy Tuning
Seminars Who we are
Seminars Why should you use Open Source? • State of the Art / very valid technologies • Community Support • Vast Documentation • Code is accessible! • Customisable • Mostly free licensing
Seminars Why should you contribute to Open Source? • Share knowledge and ideas • Improve established technologies • Become part of a Community • Not only code - all your skills are relevant! • Be useful to the world
Seminars We only deal with Open Source Informational Retrieval … Revenue ? ● Trainings - Beginner/Intermediate/Advance/Ad Hoc for
 Information Retrieval, Apache Lucene/Solr, Search Relevance, Learning To Rank… ● Consulting - Open Source Software is ubiquitous/ Expertise ? Not really
 ! R&D Projects - Cheaper and more flexible for Companies using Open Source
 ! IR Projects - From the Client requirements collection till the Software delivery
Seminars Clients
Seminars Information Retrieval “Information retrieval (IR) is the activity of obtaining information system resources relevant to an information need from a collection of information resources. Searches can be based on full-text or other content-based indexing. Information retrieval is the science of searching for information in a document, searching for documents themselves, and also searching for metadata that describe data, and for databases of texts, images or sounds.” Wikipedia Information Need Corpus
Seminars Apache Lucene • http://lucene.apache.org • High-performance, scalable information retrieval software *library* • Enables search capabilities to your applications • Cohesive and simple interface, which hides a really complex world • Open Source: Apache Top Level Project
Seminars Apache Lucene - Brief History 2019 Lucene 8.3.0 (November)
Seminars Apache Solr • http://lucene.apache.org/solr • Highly reliable, scalable and fault tolerant search *server* • A Lucene “serverization” with a lot of additional features • All services are exposed through a HTTP (REST-Like interface) • Written in Java • Rich ecosystem for building enterprise-level applications (Plugins, Integrations, Clients) • Open Source: Apache Top Level Project “Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene™.”
Seminars
Seminars Apache Solr - Brief History Version 8.3.0 (November)2019
Seminars The Inverted Index The Inverted Index is the basic data structure used by Lucene to provide Search in a corpus of documents. From wikipedia : “In computer science, an inverted index (also referred to as postings file or inverted file) is an index data structure storing a mapping from content, such as words or numbers, to its locations in a database file, or in a document or a set of documents.”
Seminars The Lucene Document Document Field ValueField Name • Documents are the unit of information 
 for indexing and search. • A Document is a set of fields. • Each field has a name and a value.
Seminars The Lucene Inverted Index
Seminars The Lucene Inverted Index • Lucene directory (in memory, on disk, memory mapped) • Collection of immutable segments (fully working) • Each segment is composed by a set of binary files[1] [1] Lucene File Format Documentation Indexes evolve by: 1. Creating new segments for newly added documents. 2. Merging existing segments.
Seminars Schema Configuration • Per collection/index • Xml file • Define how the inverted Index will be built • Fields/Field Types definition
Seminars Schema Configuration • Define flexible expressions for groups of fields • Shared attributes for each field instance • Copy the source content to a destination field • Allow to run multiple analysis chains for the same content
Seminars Field Type • Define how the single terms (in the inverted index) will be generated out of the content Index Time Query Time Analysis chain executed when building the index Analysis chain executed when building the query
Seminars Text Analysis • Only text fields types (e.g. solr.TextField or subclasses) have a text analysis chain associated An analyzer can define • Zero or more CharFilter • One and only one Tokenizer • Zero or more TokenFilter
Seminars Char Filters • CharFilter is a component that pre-processes input characters. • CharFilters can be chained like Token Filters and placed in front of a Tokenizer. • CharFilters can add, change, or remove characters
 while preserving the original character offsets to support features like highlighting.
Seminars Tokenizers Tokenizers are responsible for breaking field data into lexical units, or tokens.[1] [1] https://lucene.apache.org/solr/guide/8_3/tokenizers.html
Seminars Token Filters Filters[1] examine a stream of tokens and keep them, transform them or discard them, 
 depending on the filter type being used. [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html
Seminars Word Delimiters Filter • Improve recall • Dedicated Filters: 
 solr.WordDelimiterGraphFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#word-delimiter-graph-filter Example: Default behavior. The whitespace tokenizer is used here to preserve non-alphanumeric characters. <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory"/> </analyzer>
 In: "hot-spot RoboBlaster/9000 100XL" Tokenizer to Filter: "hot-spot", "RoboBlaster/9000", "100XL" Out: "hot", "spot", "Robo", "Blaster", "9000", "100", "XL"
Seminars Stopword Filters • Reduce index size • Can improve precision (removing terms with low semantic value) • Can improve recall • Dedicated Filters: solr.StopFilterFactory, solr.ManagedStopFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#stop-filter Example: <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> </analyzer> In: "To be or what?" Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "what"(4) Out: "what"(4)
Seminars Stemmers • Improve Recall • Reduce index size • Dedicated Filters: solr.EnglishMinimalStemFilterFactory, solr.HunspellStemFilterFactory, solr.KStemFilterFactory,
 solr.PorterStemFilterFactory, solr.SnowballPorterFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#porter-stem-filter Example: <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.EnglishMinimalStemFilterFactory"/> </analyzer> In: "dogs cats" Tokenizer to Filter: "dogs", "cats" Out: "dog", "cat"
Seminars Synonym Filters[1/2] • Improve Recall • Dedicated Filters: solr.SynonymGraphFilterFactory • Index Time -> affect terms distributions, needs re-indexing • Query Time -> more flexible [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#synonym-graph-filter couch,sofa,divan teh => the huge,ginormous,humungous => large small => tiny,teeny,weeny
Seminars Synonym Filters[2/2] • Improve Recall • Dedicated Filters: 
 solr.SynonymGraphFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#synonym-graph-filter Example: <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" synonyms="mysynonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" synonyms="mysynonyms.txt"/> </analyzer>
 In: "teh small couch" Tokenizer to Filter: "teh"(1), "small"(2), "couch"(3) Out: "the"(1), "tiny"(2), "teeny"(2), "weeny"(2), "couch"(3), "sofa"(3), "divan"(3)
Seminars Keep Word Filter • Help in Entity tagging • Dedicated Filters: solr.KeepWordFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#keep-word-filter Example: <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.KeepWordFilterFactory" words="keepwords.txt" ignoreCase="true"/> </analyzer> In: "Happy, sad or funny" Tokenizer to Filter: "Happy", "sad", "or", "funny" Out: "Happy", "funny"
Seminars N-Gram Filtering • Improve Recall • Ideal for autocompletion • Dedicated Filters: solr.EdgeNGramFilterFactory, solr.NGramFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#edge-n-gram-filter <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="4"/> </analyzer>
Seminars Phonetic Matching • Improve Recall • Dedicated Filters: solr.BeiderMorseFilterFactory, solr.DaitchMokotoffSoundexFilterFactory, solr.DoubleMetaphoneFilterFactory, solr.PhoneticFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/phonetic-matching.html <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.BeiderMorseFilterFactory" nameType="GENERIC" ruleType="APPROX" concat="true" languageSet="auto"> </filter> </analyzer>
Seminars Common Grams Filter • Improve Precision • Useful for phrase queries [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#common-grams-filter Example: <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.CommonGramsFilterFactory" words="stopwords.txt" ignoreCase="true"/> </analyzer> In: "the Cat" Tokenizer to Filter: "the", "Cat" Out: "the_cat"
Seminars Field Attributes
Seminars Field Attributes
Seminars Solr Text Analysis - Hands On! • Analysis Screen from Solr Admin • Let’s explore the schema.xml
Seminars Indexing • Using the Solr Cell framework built on Apache Tika 
 for ingesting binary files or structured files such as Office, Word, PDF, and other proprietary formats.
 (Recommended for prototyping and exercise)
 • Uploading XML files by sending HTTP requests to the Solr server 
 from any environment where such requests can be generated.
 (Recommended for prototyping and exercise)
 • Writing a custom Java application to ingest data through Solr’s Java Client API 
 (which is described in more detail in Client APIs). 
 Using the Java API may be the best choice if you’re working with an application, 
 such as a Content Management System (CMS), that offers a Java API.

Seminars Indexing • Indexing is the procedure of building an index from the documents in input • Transaction Log (Rotating on hard commits) • Index built in memory • Soft commits(visibility) • Hard commits(durability) • openSearcher=true(visibility) • Auto commit • Merge policy
Seminars Lucene Score In order to measure the relevancy of a given result, Solr(Lucene) assigns it a “score” The formula behind the score computation is behind the scope of this course, however important things tha contribute to that formula are: • Term Frequency (TF): how many times a given term occurs within a single document • Document Frequency (DF): how many documents in the dataset contain a given term • TF/IDF: the ratio between the term frequency and the inverse document frequency (1/DF) • Field length: how many terms compose a field • Boosting: functions or in general things that boost the score computed for a given match. Boosting 
 can be applied at index time (deprecated now) or a query time Score values cannot be compared across queries, or even with the same query but with a different index.
Seminars ! Origin from Probabilistic Information Retrieval ! Default Similarity from Lucene 6.0 [1] ! 25th iteration in improving TF-IDF ! TF ! IDF ! Document(Field) Length ! Configuration parameters [1] LUCENE-6789 BM25 Term Scorer
Seminars BM25 Term Scorer - Inverse Document Frequency IDF Score
 has very similar behavior
Seminars BM25 Term Scorer - Term Frequency TF Score
 approaches
 asymptotically (k+1)
 
 k=1.2 in this example
Seminars BM25 Term Scorer - Document Length Document Length /
 Avg Document Length
 
 affects how fast we saturate TF score
Seminars Basic Search The list is not exhaustive and is not statically defined, because it depends on the query parser Some parameter (i.e. filters) accepts more than one value:
Seminars Queries Query • Regulated by Query Parsers • Calculates scores • Cached with results order preserved Filter Query • Regulated by Query Parsers • Does not calculate scores • Cached independently • Reusable q=field:value fq=field:value
Seminars Query Parsers • Main responsibility of the query parser is understand the input query syntax and build a Lucene query • This is the first component involved in the query execution chain • If it is not specified, then a default parser is used (Solr Standard Query Parser) • Solr comes with several available and ready-to-use query parsers • The query parameter “defType” defines the query parser that will be used in a request
Seminars Standard Query Parser Parameter Description q Defines a query using standard query syntax. This parameter is mandatory. q.op Specifies the default operator for query expressions, overriding the default operator specified in the Schema. Possible values are "AND" or "OR". df Specifies a default field, overriding the definition of a default field in the Schema. sow Split on whitespace: if set to false, whitespace-separated term sequences will be provided to text analysis in one shot, enabling proper function of analysis filters that operate over term sequences, e.g. multi-word synonyms and shingles. Defaults to true: text analysis is invoked separately for each individual whitespace-separated term.
Seminars Standard Query Parser • Phrase Search
 q=title:”a tale of two cities”
 • Wildcard Search
 q=title:c?ti*
 • Fuzzy Search
 q=title:cties~1 • Proximity Search 
 q=title:"tale cities"~2 • Range Search 
 downloads:[1000 TO 2000], author:{Ada TO Carmen} • Boosted Search
 q=tale of two cities^100 bunny • Constant Score Search
 AND subjects:(war stories)^=4 • Boolean Search
 (field1:term1) AND (field2:term1)
Seminars Date Queries Queries against fields using the TrieDateField type (typically range queries) should use the appropriate date syntax [1]: • timestamp:[* TO NOW] • createdate:[1976-03-06T23:59:59.999Z TO *] • createdate:[1995-12-31T23:59:59.999Z TO 2007-03-06T00:00:00Z] • pubdate:[NOW-1YEAR/DAY TO NOW/DAY+1DAY] • createdate:[1976-03-06T23:59:59.999Z TO 1976-03-06T23:59:59.999Z+1YEAR] • createdate:[1976-03-06T23:59:59.999Z/YEAR TO 1976-03-06T23:59:59.999Z] [1] https://en.wikipedia.org/wiki/ISO_8601 Timezone By default, all date math expressions are evaluated relative to the UTC TimeZone, but the TZ parameter can be specified to override this behaviour N.B. Independently of the locale Solr is executed, only ISO-8601 dates are supported in requests
Seminars Solr Query Debug - Hands On! • debug=query: return debug information about the query only. • debug=timing: return debug information about how long the query took to process. • debug=results: return debug information about the score results (also known as "explain").
Seminars Master Thesis: Click Models to Estimate Relevancy Ratings from Users Interactions Main responsibility of the candidate will be to:
 • learn basic concepts of Agile methodologies for software engineering
 • learn details of Search Quality Evaluation
 • grasp the fundamentals of click modelling, implicit and explicit relevancy feedback
 • design and implement the module in an existing Spring Boot REST service application 
 • benchmark the solution(s) through a careful quality/performance(times/ space) analysis
Seminars Master Thesis: Search Quality Evaluation for Continuous Integration Tools Main responsibility of the candidate will be to: 
 
 • learn basic concepts of Agile methodologies for software engineering • get familiar with Apache Lucene based search engines (Apache Solr/ Elasticsearch) • learn details of Search Quality Evaluation • grasp the fundamentals of Continuous Integration and Continuous Deployment through well established industry level technologies • design and implement plugins for Apache Jenkins, Atlassian Bamboo and JetBrains 
 TeamCity
Seminars

More Related Content

PDF
MySQL 8.0 EXPLAIN ANALYZE
byNorvald Ryeng
 
PDF
My first 90 days with ClickHouse.pdf
byAlkin Tezuysal
 
PDF
Spark Operator—Deploy, Manage and Monitor Spark clusters on Kubernetes
byDatabricks
 
PDF
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
byAltinity Ltd
 
PDF
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
PDF
Deep Dive into Building Streaming Applications with Apache Pulsar
byTimothy Spann
 
PDF
Apache Iceberg - A Table Format for Hige Analytic Datasets
byAlluxio, Inc.
 
PDF
Geospatial Indexing at Scale: The 15 Million QPS Redis Architecture Powering ...
byDaniel Hochman
 
MySQL 8.0 EXPLAIN ANALYZE
byNorvald Ryeng
 
My first 90 days with ClickHouse.pdf
byAlkin Tezuysal
 
Spark Operator—Deploy, Manage and Monitor Spark clusters on Kubernetes
byDatabricks
 
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
byAltinity Ltd
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
Deep Dive into Building Streaming Applications with Apache Pulsar
byTimothy Spann
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
byAlluxio, Inc.
 
Geospatial Indexing at Scale: The 15 Million QPS Redis Architecture Powering ...
byDaniel Hochman
 

What's hot

PPTX
Part1 of SQL Tuning Workshop - Understanding the Optimizer
byMaria Colgan
 
PDF
[pgday.Seoul 2022] PostgreSQL구조 - 윤성재
byPgDay.Seoul
 
PDF
Building large scale transactional data lake using apache hudi
byBill Liu
 
PDF
6 Nines: How Stripe keeps Kafka highly-available across the globe with Donny ...
byHostedbyConfluent
 
PDF
Building a Data Pipeline using Apache Airflow (on AWS / GCP)
byYohei Onishi
 
PDF
Elasticsearch From the Bottom Up
byfoundsearch
 
PPTX
Alfresco tuning part1
byLuis Cabaceira
 
PDF
How to Analyze and Tune MySQL Queries for Better Performance
byoysteing
 
PDF
Apache Flink internals
byKostas Tzoumas
 
PPTX
Apache Airflow overview
byNikolayGrishchenkov
 
PPTX
Prometheus and Grafana
byLhouceine OUHAMZA
 
PPTX
Data ingestion
bynitheeshe2
 
PDF
Apache Calcite Tutorial - BOSS 21
byStamatis Zampetakis
 
PPTX
Building a Virtual Data Lake with Apache Arrow
byDremio Corporation
 
PDF
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
byMydbops
 
PDF
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
byInfluxData
 
PPTX
Apache Flink Deep Dive
byDataWorks Summit
 
PPTX
Cassandra Lunch #88: Cadence
byAnant Corporation
 
PDF
Introduction to Apache Airflow
bymutt_data
 
PDF
Hive Bucketing in Apache Spark with Tejas Patil
byDatabricks
 
Part1 of SQL Tuning Workshop - Understanding the Optimizer
byMaria Colgan
 
[pgday.Seoul 2022] PostgreSQL구조 - 윤성재
byPgDay.Seoul
 
Building large scale transactional data lake using apache hudi
byBill Liu
 
6 Nines: How Stripe keeps Kafka highly-available across the globe with Donny ...
byHostedbyConfluent
 
Building a Data Pipeline using Apache Airflow (on AWS / GCP)
byYohei Onishi
 
Elasticsearch From the Bottom Up
byfoundsearch
 
Alfresco tuning part1
byLuis Cabaceira
 
How to Analyze and Tune MySQL Queries for Better Performance
byoysteing
 
Apache Flink internals
byKostas Tzoumas
 
Apache Airflow overview
byNikolayGrishchenkov
 
Prometheus and Grafana
byLhouceine OUHAMZA
 
Data ingestion
bynitheeshe2
 
Apache Calcite Tutorial - BOSS 21
byStamatis Zampetakis
 
Building a Virtual Data Lake with Apache Arrow
byDremio Corporation
 
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
byMydbops
 
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
byInfluxData
 
Apache Flink Deep Dive
byDataWorks Summit
 
Cassandra Lunch #88: Cadence
byAnant Corporation
 
Introduction to Apache Airflow
bymutt_data
 
Hive Bucketing in Apache Spark with Tejas Patil
byDatabricks
 

Similar to Let's Build an Inverted Index: Introduction to Apache Lucene/Solr

PPTX
Search Me: Using Lucene.Net
bygramana
 
PPT
Lucene Bootcamp -1
byGokulD
 
PDF
Lucene for Solr Developers
byErik Hatcher
 
PDF
Lucene for Solr Developers
byErik Hatcher
 
PPTX
Introduction to Apache Lucene/Solr
byRahul Jain
 
PPTX
Introduction to Lucene & Solr and Usecases
byRahul Jain
 
PPT
Lucene BootCamp
byGokulD
 
PDF
IR with lucene
byStelios Gorilas
 
PPTX
Apache Solr-Webinar
byEdureka!
 
PPTX
Lucene
byHarshit Agarwal
 
PPTX
Introduction to Information Retrieval using Lucene
byDeeKan3
 
PPTX
Introduction to Lucene and Solr - 1
byYI-CHING WU
 
PPTX
Intro to Apache Lucene and Solr
byGrant Ingersoll
 
PDF
Search Engine-Building with Lucene and Solr
byKai Chan
 
PDF
Search Engine-Building with Lucene and Solr, Part 1 (SoCal Code Camp LA 2013)
byKai Chan
 
PPTX
Apache lucene
byDr. Abhiram Gandhe
 
PDF
Lucene for Solr Developers
byErik Hatcher
 
PPTX
Illuminating Lucene.Net
byDean Thrasher
 
PPT
Introduction to Search Engines
byNitin Pande
 
PDF
Apache Solr crash course
byTommaso Teofili
 
Search Me: Using Lucene.Net
bygramana
 
Lucene Bootcamp -1
byGokulD
 
Lucene for Solr Developers
byErik Hatcher
 
Lucene for Solr Developers
byErik Hatcher
 
Introduction to Apache Lucene/Solr
byRahul Jain
 
Introduction to Lucene & Solr and Usecases
byRahul Jain
 
Lucene BootCamp
byGokulD
 
IR with lucene
byStelios Gorilas
 
Apache Solr-Webinar
byEdureka!
 
Lucene
byHarshit Agarwal
 
Introduction to Information Retrieval using Lucene
byDeeKan3
 
Introduction to Lucene and Solr - 1
byYI-CHING WU
 
Intro to Apache Lucene and Solr
byGrant Ingersoll
 
Search Engine-Building with Lucene and Solr
byKai Chan
 
Search Engine-Building with Lucene and Solr, Part 1 (SoCal Code Camp LA 2013)
byKai Chan
 
Apache lucene
byDr. Abhiram Gandhe
 
Lucene for Solr Developers
byErik Hatcher
 
Illuminating Lucene.Net
byDean Thrasher
 
Introduction to Search Engines
byNitin Pande
 
Apache Solr crash course
byTommaso Teofili
 

More from Sease

PPTX
Exploring Multilingual Embeddings for Italian Semantic Search: A Pretrained a...
bySease
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
bySease
 
PPTX
Hybrid Search with Apache Solr Reciprocal Rank Fusion
bySease
 
PPTX
Blazing-Fast Serverless MapReduce Indexer for Apache Solr
bySease
 
PPTX
From Natural Language to Structured Solr Queries using LLMs
bySease
 
PPTX
Hybrid Search With Apache Solr
bySease
 
PPTX
Multi Valued Vectors Lucene
bySease
 
PPTX
When SDMX meets AI-Leveraging Open Source LLMs To Make Official Statistics Mo...
bySease
 
PDF
How To Implement Your Online Search Quality Evaluation With Kibana
bySease
 
PDF
Introducing Multi Valued Vectors Fields in Apache Lucene
bySease
 
PPTX
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
bySease
 
PPTX
How does ChatGPT work: an Information Retrieval perspective
bySease
 
PDF
How To Implement Your Online Search Quality Evaluation With Kibana
bySease
 
PPTX
Neural Search Comes to Apache Solr
bySease
 
PPTX
Large Scale Indexing
bySease
 
PDF
Dense Retrieval with Apache Solr Neural Search.pdf
bySease
 
PDF
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
bySease
 
PDF
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdf
bySease
 
PPTX
How to cache your searches_ an open source implementation.pptx
bySease
 
PDF
Online Testing Learning to Rank with Solr Interleaving
bySease
 
Exploring Multilingual Embeddings for Italian Semantic Search: A Pretrained a...
bySease
 
Building Search Using OpenSearch: Limitations and Workarounds
bySease
 
Hybrid Search with Apache Solr Reciprocal Rank Fusion
bySease
 
Blazing-Fast Serverless MapReduce Indexer for Apache Solr
bySease
 
From Natural Language to Structured Solr Queries using LLMs
bySease
 
Hybrid Search With Apache Solr
bySease
 
Multi Valued Vectors Lucene
bySease
 
When SDMX meets AI-Leveraging Open Source LLMs To Make Official Statistics Mo...
bySease
 
How To Implement Your Online Search Quality Evaluation With Kibana
bySease
 
Introducing Multi Valued Vectors Fields in Apache Lucene
bySease
 
Stat-weight Improving the Estimator of Interleaved Methods Outcomes with Stat...
bySease
 
How does ChatGPT work: an Information Retrieval perspective
bySease
 
How To Implement Your Online Search Quality Evaluation With Kibana
bySease
 
Neural Search Comes to Apache Solr
bySease
 
Large Scale Indexing
bySease
 
Dense Retrieval with Apache Solr Neural Search.pdf
bySease
 
Neural Search Comes to Apache Solr_ Approximate Nearest Neighbor, BERT and Mo...
bySease
 
Word2Vec model to generate synonyms on the fly in Apache Lucene.pdf
bySease
 
How to cache your searches_ an open source implementation.pptx
bySease
 
Online Testing Learning to Rank with Solr Interleaving
bySease
 

Recently uploaded

PDF
Workshop practice theory course (Unit-1) By Varun Pratap Singh.pdf
byVarun Pratap Singh
 
PDF
Uses of LARSA - 4D and LUSAS IRICEN 14.10.25.pdf
byRajesh Prasad
 
PDF
Japanese Question Answering Datasets .
byNABLAS株式会社
 
PDF
Reciprocating pump presentation document
bydmsrnxjbpj
 
PPTX
William Mulholland and the St. Francis Dam Disaster
byBob Mayer
 
PPTX
33 KV SUBSTATION.pptx FROM LAYOUT TO PROTECTION
byPradipChanda5
 
PDF
Programming in Java- on operators types and usage
byPrabhaDevi33
 
PPTX
ATD.pptx traction distribution railway atd
bykhuranarahul599
 
PDF
HYDRAULIC TURBINES with question and answer.pdf
bysparunkumar1
 
PPT
Behavioral Modeling using Verilog and combinational or sequential circuit
bykirtipfraoa
 
PDF
Cloud Computing in IoT, Powering the Connected Future
bysrav69003
 
PDF
13 Malpractices In Bunkering Operations Seafarers Should Be Aware Of
byMahmoud Moghtaderi
 
PPTX
Planning and selection of equipment for different mining conditions. Equipmen...
bygugulothuakhil728
 
PPTX
DevFest Seattle 2025 - AI Native Design Patterns.pptx
bySujata Sridharan
 
PPTX
Meghon se Sakshatkar Google Cloud Workshop.pptx
byvanshikaprakash05
 
PPTX
Introduction to Relational Algebra Advance Database Part three.pptx
bywilliamsmareh
 
PDF
Modernize Your Data Platform with Snowflake & Stridely Solutions
byStridely Solutions
 
PDF
Fluid-Mechanics with question and answer.pdf
bysparunkumar1
 
PDF
Modularity In Lyra - A new studio's experiences building their prototype on U...
bypatrickcastillo41
 
PDF
Decarbonizing Healthcare: Exploring the Potential for Electrification with IESVE
byIES VE
 
Workshop practice theory course (Unit-1) By Varun Pratap Singh.pdf
byVarun Pratap Singh
 
Uses of LARSA - 4D and LUSAS IRICEN 14.10.25.pdf
byRajesh Prasad
 
Japanese Question Answering Datasets .
byNABLAS株式会社
 
Reciprocating pump presentation document
bydmsrnxjbpj
 
William Mulholland and the St. Francis Dam Disaster
byBob Mayer
 
33 KV SUBSTATION.pptx FROM LAYOUT TO PROTECTION
byPradipChanda5
 
Programming in Java- on operators types and usage
byPrabhaDevi33
 
ATD.pptx traction distribution railway atd
bykhuranarahul599
 
HYDRAULIC TURBINES with question and answer.pdf
bysparunkumar1
 
Behavioral Modeling using Verilog and combinational or sequential circuit
bykirtipfraoa
 
Cloud Computing in IoT, Powering the Connected Future
bysrav69003
 
13 Malpractices In Bunkering Operations Seafarers Should Be Aware Of
byMahmoud Moghtaderi
 
Planning and selection of equipment for different mining conditions. Equipmen...
bygugulothuakhil728
 
DevFest Seattle 2025 - AI Native Design Patterns.pptx
bySujata Sridharan
 
Meghon se Sakshatkar Google Cloud Workshop.pptx
byvanshikaprakash05
 
Introduction to Relational Algebra Advance Database Part three.pptx
bywilliamsmareh
 
Modernize Your Data Platform with Snowflake & Stridely Solutions
byStridely Solutions
 
Fluid-Mechanics with question and answer.pdf
bysparunkumar1
 
Modularity In Lyra - A new studio's experiences building their prototype on U...
bypatrickcastillo41
 
Decarbonizing Healthcare: Exploring the Potential for Electrification with IESVE
byIES VE
 

Let's Build an Inverted Index: Introduction to Apache Lucene/Solr

  • 1.
    Seminars Let’s Build anInverted Index: Introduction to Apache Lucene/Solr
 Alessandro Benedetti, Software Engineer
 
 Andrea Gazzarini, Software Engineer 28th November 2019
  • 2.
    Seminars ▪ R&D SoftwareEngineer ▪ Search Consultant ▪ Director ▪ Master Degree in Computer Science ▪ Apache Lucene/Solr Enthusiast ▪ Semantic, NLP, Machine Learning Technologies passionate ▪ Conference Speaker ▪ Beach Volleyball Player & Snowboarder Alessandro Benedetti
  • 3.
    Seminars ▪ Software Engineer(1999-) ▪ “Hermit” Software Engineer (2010-) ▪ Java & Information Retrieval Passionate ▪ Apache Qpid (past) Committer ▪ Husband & Father ▪ Bass Player Andrea Gazzarini, “Gazza”
  • 4.
    Seminars Search Services
 ● LondonBased - Italian made :) ● Open Source Enthusiasts ● Apache Lucene/Solr experts ! Community Contributors ● Active Researchers ● Hot Trends : Learning To Rank, 
 Document Similarity, Search Quality Evaluation,
 Relevancy Tuning
  • 5.
  • 6.
    Seminars Why should youuse Open Source? • State of the Art / very valid technologies • Community Support • Vast Documentation • Code is accessible! • Customisable • Mostly free licensing
  • 7.
    Seminars Why should youcontribute to Open Source? • Share knowledge and ideas • Improve established technologies • Become part of a Community • Not only code - all your skills are relevant! • Be useful to the world
  • 8.
    Seminars We only dealwith Open Source Informational Retrieval … Revenue ? ● Trainings - Beginner/Intermediate/Advance/Ad Hoc for
 Information Retrieval, Apache Lucene/Solr, Search Relevance, Learning To Rank… ● Consulting - Open Source Software is ubiquitous/ Expertise ? Not really
 ! R&D Projects - Cheaper and more flexible for Companies using Open Source
 ! IR Projects - From the Client requirements collection till the Software delivery
  • 9.
  • 10.
    Seminars Information Retrieval “Information retrieval(IR) is the activity of obtaining information system resources relevant to an information need from a collection of information resources. Searches can be based on full-text or other content-based indexing. Information retrieval is the science of searching for information in a document, searching for documents themselves, and also searching for metadata that describe data, and for databases of texts, images or sounds.” Wikipedia Information Need Corpus
  • 11.
    Seminars Apache Lucene • http://lucene.apache.org •High-performance, scalable information retrieval software *library* • Enables search capabilities to your applications • Cohesive and simple interface, which hides a really complex world • Open Source: Apache Top Level Project
  • 12.
    Seminars Apache Lucene -Brief History 2019 Lucene 8.3.0 (November)
  • 13.
    Seminars Apache Solr • http://lucene.apache.org/solr •Highly reliable, scalable and fault tolerant search *server* • A Lucene “serverization” with a lot of additional features • All services are exposed through a HTTP (REST-Like interface) • Written in Java • Rich ecosystem for building enterprise-level applications (Plugins, Integrations, Clients) • Open Source: Apache Top Level Project “Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene™.”
  • 14.
  • 15.
    Seminars Apache Solr -Brief History Version 8.3.0 (November)2019
  • 16.
    Seminars The Inverted Index TheInverted Index is the basic data structure used by Lucene to provide Search in a corpus of documents. From wikipedia : “In computer science, an inverted index (also referred to as postings file or inverted file) is an index data structure storing a mapping from content, such as words or numbers, to its locations in a database file, or in a document or a set of documents.”
  • 17.
    Seminars The Lucene Document Document FieldValueField Name • Documents are the unit of information 
 for indexing and search. • A Document is a set of fields. • Each field has a name and a value.
  • 18.
  • 19.
    Seminars The Lucene InvertedIndex • Lucene directory (in memory, on disk, memory mapped) • Collection of immutable segments (fully working) • Each segment is composed by a set of binary files[1] [1] Lucene File Format Documentation Indexes evolve by: 1. Creating new segments for newly added documents. 2. Merging existing segments.
  • 20.
    Seminars Schema Configuration • Percollection/index • Xml file • Define how the inverted Index will be built • Fields/Field Types definition
  • 21.
    Seminars Schema Configuration • Defineflexible expressions for groups of fields • Shared attributes for each field instance • Copy the source content to a destination field • Allow to run multiple analysis chains for the same content
  • 22.
    Seminars Field Type • Definehow the single terms (in the inverted index) will be generated out of the content Index Time Query Time Analysis chain executed when building the index Analysis chain executed when building the query
  • 23.
    Seminars Text Analysis • Onlytext fields types (e.g. solr.TextField or subclasses) have a text analysis chain associated An analyzer can define • Zero or more CharFilter • One and only one Tokenizer • Zero or more TokenFilter
  • 24.
    Seminars Char Filters • CharFilteris a component that pre-processes input characters. • CharFilters can be chained like Token Filters and placed in front of a Tokenizer. • CharFilters can add, change, or remove characters
 while preserving the original character offsets to support features like highlighting.
  • 25.
    Seminars Tokenizers Tokenizers are responsiblefor breaking field data into lexical units, or tokens.[1] [1] https://lucene.apache.org/solr/guide/8_3/tokenizers.html
  • 26.
    Seminars Token Filters Filters[1] examinea stream of tokens and keep them, transform them or discard them, 
 depending on the filter type being used. [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html
  • 27.
    Seminars Word Delimiters Filter •Improve recall • Dedicated Filters: 
 solr.WordDelimiterGraphFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#word-delimiter-graph-filter Example: Default behavior. The whitespace tokenizer is used here to preserve non-alphanumeric characters. <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.WordDelimiterGraphFilterFactory"/> </analyzer>
 In: "hot-spot RoboBlaster/9000 100XL" Tokenizer to Filter: "hot-spot", "RoboBlaster/9000", "100XL" Out: "hot", "spot", "Robo", "Blaster", "9000", "100", "XL"
  • 28.
    Seminars Stopword Filters • Reduceindex size • Can improve precision (removing terms with low semantic value) • Can improve recall • Dedicated Filters: solr.StopFilterFactory, solr.ManagedStopFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#stop-filter Example: <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.StopFilterFactory" words="stopwords.txt" ignoreCase="true"/> </analyzer> In: "To be or what?" Tokenizer to Filter: "To"(1), "be"(2), "or"(3), "what"(4) Out: "what"(4)
  • 29.
    Seminars Stemmers • Improve Recall •Reduce index size • Dedicated Filters: solr.EnglishMinimalStemFilterFactory, solr.HunspellStemFilterFactory, solr.KStemFilterFactory,
 solr.PorterStemFilterFactory, solr.SnowballPorterFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#porter-stem-filter Example: <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.EnglishMinimalStemFilterFactory"/> </analyzer> In: "dogs cats" Tokenizer to Filter: "dogs", "cats" Out: "dog", "cat"
  • 30.
    Seminars Synonym Filters[1/2] • ImproveRecall • Dedicated Filters: solr.SynonymGraphFilterFactory • Index Time -> affect terms distributions, needs re-indexing • Query Time -> more flexible [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#synonym-graph-filter couch,sofa,divan teh => the huge,ginormous,humungous => large small => tiny,teeny,weeny
  • 31.
    Seminars Synonym Filters[2/2] • ImproveRecall • Dedicated Filters: 
 solr.SynonymGraphFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#synonym-graph-filter Example: <analyzer type="index"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" synonyms="mysynonyms.txt"/> <filter class="solr.FlattenGraphFilterFactory"/> <!-- required on index analyzers after graph filters --> </analyzer> <analyzer type="query"> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.SynonymGraphFilterFactory" synonyms="mysynonyms.txt"/> </analyzer>
 In: "teh small couch" Tokenizer to Filter: "teh"(1), "small"(2), "couch"(3) Out: "the"(1), "tiny"(2), "teeny"(2), "weeny"(2), "couch"(3), "sofa"(3), "divan"(3)
  • 32.
    Seminars Keep Word Filter •Help in Entity tagging • Dedicated Filters: solr.KeepWordFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#keep-word-filter Example: <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.KeepWordFilterFactory" words="keepwords.txt" ignoreCase="true"/> </analyzer> In: "Happy, sad or funny" Tokenizer to Filter: "Happy", "sad", "or", "funny" Out: "Happy", "funny"
  • 33.
    Seminars N-Gram Filtering • ImproveRecall • Ideal for autocompletion • Dedicated Filters: solr.EdgeNGramFilterFactory, solr.NGramFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#edge-n-gram-filter <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="4"/> </analyzer>
  • 34.
    Seminars Phonetic Matching • ImproveRecall • Dedicated Filters: solr.BeiderMorseFilterFactory, solr.DaitchMokotoffSoundexFilterFactory, solr.DoubleMetaphoneFilterFactory, solr.PhoneticFilterFactory [1] https://lucene.apache.org/solr/guide/8_3/phonetic-matching.html <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.BeiderMorseFilterFactory" nameType="GENERIC" ruleType="APPROX" concat="true" languageSet="auto"> </filter> </analyzer>
  • 35.
    Seminars Common Grams Filter •Improve Precision • Useful for phrase queries [1] https://lucene.apache.org/solr/guide/8_3/filter-descriptions.html#common-grams-filter Example: <analyzer> <tokenizer class="solr.StandardTokenizerFactory"/> <filter class="solr.CommonGramsFilterFactory" words="stopwords.txt" ignoreCase="true"/> </analyzer> In: "the Cat" Tokenizer to Filter: "the", "Cat" Out: "the_cat"
  • 36.
  • 37.
  • 38.
    Seminars Solr Text Analysis- Hands On! • Analysis Screen from Solr Admin • Let’s explore the schema.xml
  • 39.
    Seminars Indexing • Using theSolr Cell framework built on Apache Tika 
 for ingesting binary files or structured files such as Office, Word, PDF, and other proprietary formats.
 (Recommended for prototyping and exercise)
 • Uploading XML files by sending HTTP requests to the Solr server 
 from any environment where such requests can be generated.
 (Recommended for prototyping and exercise)
 • Writing a custom Java application to ingest data through Solr’s Java Client API 
 (which is described in more detail in Client APIs). 
 Using the Java API may be the best choice if you’re working with an application, 
 such as a Content Management System (CMS), that offers a Java API.

  • 40.
    Seminars Indexing • Indexing isthe procedure of building an index from the documents in input • Transaction Log (Rotating on hard commits) • Index built in memory • Soft commits(visibility) • Hard commits(durability) • openSearcher=true(visibility) • Auto commit • Merge policy
  • 41.
    Seminars Lucene Score In orderto measure the relevancy of a given result, Solr(Lucene) assigns it a “score” The formula behind the score computation is behind the scope of this course, however important things tha contribute to that formula are: • Term Frequency (TF): how many times a given term occurs within a single document • Document Frequency (DF): how many documents in the dataset contain a given term • TF/IDF: the ratio between the term frequency and the inverse document frequency (1/DF) • Field length: how many terms compose a field • Boosting: functions or in general things that boost the score computed for a given match. Boosting 
 can be applied at index time (deprecated now) or a query time Score values cannot be compared across queries, or even with the same query but with a different index.
  • 42.
    Seminars ! Origin fromProbabilistic Information Retrieval ! Default Similarity from Lucene 6.0 [1] ! 25th iteration in improving TF-IDF ! TF ! IDF ! Document(Field) Length ! Configuration parameters [1] LUCENE-6789 BM25 Term Scorer
  • 43.
    Seminars BM25 Term Scorer- Inverse Document Frequency IDF Score
 has very similar behavior
  • 44.
    Seminars BM25 Term Scorer- Term Frequency TF Score
 approaches
 asymptotically (k+1)
 
 k=1.2 in this example
  • 45.
    Seminars BM25 Term Scorer- Document Length Document Length /
 Avg Document Length
 
 affects how fast we saturate TF score
  • 46.
    Seminars Basic Search The listis not exhaustive and is not statically defined, because it depends on the query parser Some parameter (i.e. filters) accepts more than one value:
  • 47.
    Seminars Queries Query • Regulated byQuery Parsers • Calculates scores • Cached with results order preserved Filter Query • Regulated by Query Parsers • Does not calculate scores • Cached independently • Reusable q=field:value fq=field:value
  • 48.
    Seminars Query Parsers • Mainresponsibility of the query parser is understand the input query syntax and build a Lucene query • This is the first component involved in the query execution chain • If it is not specified, then a default parser is used (Solr Standard Query Parser) • Solr comes with several available and ready-to-use query parsers • The query parameter “defType” defines the query parser that will be used in a request
  • 49.
    Seminars Standard Query Parser ParameterDescription q Defines a query using standard query syntax. This parameter is mandatory. q.op Specifies the default operator for query expressions, overriding the default operator specified in the Schema. Possible values are "AND" or "OR". df Specifies a default field, overriding the definition of a default field in the Schema. sow Split on whitespace: if set to false, whitespace-separated term sequences will be provided to text analysis in one shot, enabling proper function of analysis filters that operate over term sequences, e.g. multi-word synonyms and shingles. Defaults to true: text analysis is invoked separately for each individual whitespace-separated term.
  • 50.
    Seminars Standard Query Parser •Phrase Search
 q=title:”a tale of two cities”
 • Wildcard Search
 q=title:c?ti*
 • Fuzzy Search
 q=title:cties~1 • Proximity Search 
 q=title:"tale cities"~2 • Range Search 
 downloads:[1000 TO 2000], author:{Ada TO Carmen} • Boosted Search
 q=tale of two cities^100 bunny • Constant Score Search
 AND subjects:(war stories)^=4 • Boolean Search
 (field1:term1) AND (field2:term1)
  • 51.
    Seminars Date Queries Queries againstfields using the TrieDateField type (typically range queries) should use the appropriate date syntax [1]: • timestamp:[* TO NOW] • createdate:[1976-03-06T23:59:59.999Z TO *] • createdate:[1995-12-31T23:59:59.999Z TO 2007-03-06T00:00:00Z] • pubdate:[NOW-1YEAR/DAY TO NOW/DAY+1DAY] • createdate:[1976-03-06T23:59:59.999Z TO 1976-03-06T23:59:59.999Z+1YEAR] • createdate:[1976-03-06T23:59:59.999Z/YEAR TO 1976-03-06T23:59:59.999Z] [1] https://en.wikipedia.org/wiki/ISO_8601 Timezone By default, all date math expressions are evaluated relative to the UTC TimeZone, but the TZ parameter can be specified to override this behaviour N.B. Independently of the locale Solr is executed, only ISO-8601 dates are supported in requests
  • 52.
    Seminars Solr Query Debug- Hands On! • debug=query: return debug information about the query only. • debug=timing: return debug information about how long the query took to process. • debug=results: return debug information about the score results (also known as "explain").
  • 53.
    Seminars Master Thesis: Click Modelsto Estimate Relevancy Ratings from Users Interactions Main responsibility of the candidate will be to:
 • learn basic concepts of Agile methodologies for software engineering
 • learn details of Search Quality Evaluation
 • grasp the fundamentals of click modelling, implicit and explicit relevancy feedback
 • design and implement the module in an existing Spring Boot REST service application 
 • benchmark the solution(s) through a careful quality/performance(times/ space) analysis
  • 54.
    Seminars Master Thesis: Search QualityEvaluation for Continuous Integration Tools Main responsibility of the candidate will be to: 
 
 • learn basic concepts of Agile methodologies for software engineering • get familiar with Apache Lucene based search engines (Apache Solr/ Elasticsearch) • learn details of Search Quality Evaluation • grasp the fundamentals of Continuous Integration and Continuous Deployment through well established industry level technologies • design and implement plugins for Apache Jenkins, Atlassian Bamboo and JetBrains 
 TeamCity
  • 55.