Index Data in Distributed Lucene

Note

This feature is available in NCache Enterprise and Professional editions.

Distributed Lucene enables efficient searching by indexing the documents, where indexes need to be created and stored in a directory. The first step is to initialize the directory that stores indexes. Unlike Lucene.NET, NCacheDirectory is used for openeing a directory by specifying the cache name and index name.

Note

The default path for index directory is C:\ProgramData\ncache\lucene-index\{CACHE_NAME}\data for Windows, and /usr/share/ncache/lucene-index/{CACHE_NAME}/data for Linux.

Once the directory is initialized, IndexWriter is opened by providing the instance of NCacheDirectory and IndexWriterConfig. IndexWriter creates documents on the index with the same mechanism as in Lucene.NET using AddDocument method.

Note

For documents distribution among cache servers, NCache automatically adds a TextField with DocKey field and an auto-generated GUID value to a document if it is not specified by the user.

All the write operations performed on the indexes are synchronous in nature. So, they are only returned to the user on completion of the said operations. This adds data consistency and integrity to Distribtued Lucene.

Important

For any write operation on the writer, we need to call Commit in order to save it. Commit needs to be called, or else it won’t save the write operations.

Unlike Lucene.NET, you can open multiple writers on the same directory for parallel indexing. When the document is written, IndexWriter.Commit is called to persist the document and make it searchable. Once the IndexWriter is used for indexing the data, IndexWriter.Dispose needs to be called to free the resources in use.

Tip

You can get information about the completion of write operations at an instance using NCache provided Boolean property IndexWriter.OperationsCompleted.

Prerequisites

Enterprise:

Install-Package Lucene.Net.NCache

Professional:

Install-Package Lucene.Net.NCache.Professional
  • Make sure that your application is not using any native Lucene DLL/Reference.
  • To ensure the operation is fail-safe, it is recommended to handle any potential exceptions within your application, as explained in Handling Failures.
  • To handle any unseen exceptions, refer to the Troubleshooting section.

Indexing Data

The following example first opens NCacheDirectory by specifying cache name as LuceneCache and index name as ProductIndex. Then it opens an IndexWriter on the specified directory by specifying the analyzer that is used for analyzing data.

Warning

Index write operations are not allowed in case of a partial cluster.

// Specify the Distributed-Lucene cache name
string cache = "LuceneCache";

// Specify the index name to create the indexes
string indexName = "ProductIndex";

NCacheDirectory ncacheDirectory = null;
IndexWriter indexWriter = null;

try
{

    //Initializing
    // Create a directory and open it on the cache and the index path
    ncacheDirectory = NCacheDirectory.Open(cache, indexName);

    // Specify the analyzer used to analyze data
    Analyzer analyzer = new WhitespaceAnalyzer(LuceneVersion.LUCENE_48);

    // Create an indexWriterConfig which holds all the configurations to create an instance of the writer
    IndexWriterConfig config = new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer);

    // Create an instance of the writer
    indexWriter = new IndexWriter(ncacheDirectory, new IndexWriterConfig(LuceneVersion.LUCENE_48, _analyzer));

    // Indexing
    // Add the products information that is to be indexed
    Product[] products = FetchProductsFromDB();

    foreach (var prod in products)
    {
        // Create a document and add fields to it
        Document doc = new Document();
        doc.Add(new TextField("ProductID", prod.ProductID.ToString(), Field.Store.YES));
        doc.Add(new TextField("ProductName", prod.ProductName, Field.Store.NO));
        doc.Add(new TextField("Category", prod.Category, Field.Store.YES));
        doc.Add(new TextField("Description", prod.Description, Field.Store.YES));

        // Writer is created previously
        indexWriter.AddDocument(doc);
    }
    // Calling commit on the writer saves all the write operations
    indexWriter.Commit();

  if(indexWriter.OperationsCompleted)
  {
    // All writing operations are complete
    // Can proceed with other operations such as querying  
  }
}
catch (Exception ex)
{
    // Handle Lucene exceptions
}
finally
{
    // Dispose the indexWriter after indexing
    if (indexWriter != null) indexWriter.Dispose();

    // Dispose the ncacheDirectory
    if (ncacheDirectory != null) ncacheDirectory.Dispose();
}

Additional Resources

NCache provides a sample application for Distributed Lucene on GitHub.

See Also

Lucene Components and Overview
Configure Lucene Query Indexes
SQL Search in Cache
Search Cache with LINQ