Lucene搜索优化（Advanced Text Indexing with Lucene）
转载自：http://onjava.com/lpt/a/3273Lucene is a free text-indexing and -searching API written in Java.Toappreciate indexing techniques described later in this article, you need abasic understanding of Lucene's index structure.As I mentioned in the previousarticle in this series, a typical Lucene index is stored in a singledirectory in the filesystem on a hard disk.
The core elements of such an index are segments, documents, fields, andterms.Every index consists of one or more segments.Each segment containsone or more documents.Each document has one or more fields, and each fieldcontains one or more terms.Each term is a pair of Stringsrepresenting a field name and a value.A segment consists of a series offiles.The exact number of files that constitute each segment varies fromindex to index, and depends on the number of fields that the index contains.All files belonging to the same segment share a common prefix and differ in thesuffix.You can think of a segment as a sub-index, although each segment isnot a fully-independent index.
-rw-rw-r-- 1 otis otis 4 Nov 22 22:43 deletable-rw-rw-r-- 1 otis otis 1000000 Nov 22 22:43 _lfyc.f1-rw-rw-r-- 1 otis otis 1000000 Nov 22 22:43 _lfyc.f2-rw-rw-r-- 1 otis otis 31030502 Nov 22 22:28 _lfyc.fdt-rw-rw-r-- 1 otis otis 8000000 Nov 22 22:28 _lfyc.fdx-rw-rw-r-- 1 otis otis 16 Nov 22 22:28 _lfyc.fnm-rw-rw-r-- 1 otis otis 1253701335 Nov 22 22:43 _lfyc.frq-rw-rw-r-- 1 otis otis 1871279328 Nov 22 22:43 _lfyc.prx-rw-rw-r-- 1 otis otis 14122 Nov 22 22:43 _lfyc.tii-rw-rw-r-- 1 otis otis 1082950 Nov 22 22:43 _lfyc.tis-rw-rw-r-- 1 otis otis 18 Nov 22 22:43 segments Example 1: An index consisting of a single segment.
Note that all files that belong to this segment start with a common prefix:_lfyc.Because this index contains two fields, you will noticetwo files with the fN suffix, where N is a number.Ifthis index had three fields, a file named _lfyc.f3 would also bepresent in the index directory.
The number of segments in an index is fixed once the index is fully built,but it varies while indexing is in progress.Lucene adds segments as newdocuments are added to the index, and merges segments every so often.In thenext section we will learn how to control creation and merging of segments inorder to improve indexing speed.
For more information about the files that make up a Lucene index, please seethe File Formats document on Lucene's web site.You can find the URL in the Reference section at the end of this article.
Indexing Speed Factors
The previous article demonstrated how to index text using theLuceneIndexExample class.Because the example was so basic, therewas no need to think about speed.If you are using Lucene in a non-trivialapplication, you will want to ensure optimal indexing performance.Thebottleneck of a typical text-indexing application is the process of writingindex files onto a disk.Therefore, we need to instruct Lucene to be smartabout adding and merging segments while indexing documents.
When new documents are added to a Lucene index, they are initially stored inmemory instead of being immediately written to the disk.This is done forperformance reasons.The simplest way to improve Lucene's indexing performanceis to adjust the value of IndexWriter's mergeFactorinstance variable.This value tells Lucene how many documents to store inmemory before writing them to the disk, as well as how often to merge multiplesegments together.With the default value of 10, Lucene will store 10documents in memory before writing them to a single segment on the disk.ThemergeFactor value of 10 also means that once the number ofsegments on the disk has reached the power of 10, Lucene will merge thesesegments into a single segment.(There is a small exception to this rule,which I shall explain shortly.)
For instance, if we set mergeFactor to 10, a new segment willbe created on the disk for every 10 documents added to the index.When the10th segment of size 10 is added, all 10 will be merged into a single segmentof size 100.When 10 such segments of size 100 have been added, they will bemerged into a single segment containing 1000 documents, and so on.Therefore,at any time, there will be no more than 9 segments in each power of 10 indexsize.
The exception noted earlier has to do with another IndexWriterinstance variable: maxMergeDocs.While merging segments, Lucenewill ensure that no segment with more than maxMergeDocs iscreated.For instance, if we set maxMergeDocs to 1000, when weadd the 10,000th document, instead of merging multiple segments into a singlesegment of size 10,000, Lucene will create a 10th segment of size 1000, andkeep adding segments of size 1000 for every 1000 documents added.
The default value of maxMergeDocs isInteger#MAX_VALUE.In my experience, one rarely needs to changethis value.
Now that I have explained how mergeFactor andmaxMergeDocs work, you can see that using a higher value formergeFactor will cause Lucene to use more RAM, but will let Lucenewrite data to disk less frequently, which will speed up the indexing process.A smaller mergeFactor will use less memory and will cause theindex to be updated more frequently, which will make it more up-to-date, butwill also slow down the indexing process.Similarly, a largermaxMergeDocs is better suited for batch indexing, and a smallermaxMergeDocs is better for more interactive indexing.
To get a better feel for how different values of mergeFactorand maxMergeDocs affect indexing speed, take a look at theIndexTuningDemo class below.This class takes three arguments onthe command line: the total number of documents to add to the index, the valueto use for mergeFactor, and the value to use formaxMergeDocs.All three arguments must be specified, must beintegers, and must be in this order.In order to keep the code short andclean, there are no checks for improper usage.
import org.apache.lucene.index.IndexWriter;import org.apache.lucene.analysis.Analyzer;import org.apache.lucene.analysis.StopAnalyzer;import org.apache.lucene.document.Document;import org.apache.lucene.document.Field;/** * Creates an index called 'index' in a temporary directory. * The number of documents to add to this index, the mergeFactor and * the maxMergeDocs must be specified on the command line * in that order - this class expects to be called correctly. ** Note: before running this for the first time, manually create the * directory called 'index' in your temporary directory. */public class IndexTuningDemo{ public static void main(String[] args) throws Exception { int docsInIndex= Integer.parseInt(args); // create an index called 'index' in a temporary directory String indexDir = System.getProperty("java.io.tmpdir", "tmp") + System.getProperty("file.separator") + "index"; Analyzer analyzer = new StopAnalyzer(); IndexWriter writer = new IndexWriter(indexDir, analyzer, true); // set variables that affect speed of indexing writer.mergeFactor = Integer.parseInt(args); writer.maxMergeDocs= Integer.parseInt(args); long startTime = System.currentTimeMillis(); for (int i = 0; i < docsInIndex; i++) { Document doc = new Document(); doc.add(Field.Text("fieldname", "Bibamus, moriendum est")); writer.addDocument(doc); } writer.close(); long stopTime = System.currentTimeMillis(); System.out.println("Total time: " + (stopTime - startTime) + " ms"); }} Here are some results:
prompt> time java IndexTuningDemo 100000 10 1000000Total time: 410092 msreal 6m51.801suser 5m30.000ssys 0m45.280sprompt> time java IndexTuningDemo 100000 1000 100000Total time: 249791 msreal 4m11.470suser 3m46.330ssys 0m3.660s As you can see, both invocations created an index with 100,000 documents,but the first one took much longer to complete.That is because it used thedefault mergeFactor of 10, which caused Lucene to write documentsto the disk more often than the mergeFactor of 1000 used in thesecond invocation.
Note that while these two variables can help improve indexing performance,they also affect the number of file descriptors that Lucene uses, and cantherefore cause the "Too many open files" exception.If you get this error, youshould first see if you can optimize the index, as will be described shortly.Optimization may help indexes that contain more than one segment.Ifoptimizing the index does not solve the problem, you could try increasing themaximum number of open files allowed on your computer.This is usually done atthe operating-system level and varies from OS to OS.If you are using Luceneon a computer that uses a flavor of the UNIX OS, you can see themaximum number of open files allowed from the command line.
Under bash, you can see the current settings with the built-inulimit command:
prompt> ulimit -n Under tcsh, the equivalent is:
prompt> limit descriptors To change the value under bash, use this:
prompt> ulimit -n <max number of open files here> Under tcsh, use the following:
prompt> limit descriptors <max number of open files here> To estimate a setting for the maximum number of open files allowed whileindexing, keep in mind that the maximum number of files Lucene will open is(1 + mergeFactor) * FilesPerSegment.
For instance, with a default mergeFactor of 10 and an index of1 million documents, Lucene will require 110 open files on an unoptimizedindex.When IndexWrite's optimize() method iscalled, all segments are merged into a single segment, which minimizes thenumber of open files that Lucene needs.
<!-- CS_PAGE_INDEX-->
<!-- CS_PAGE_BREAK-->
<!-- CS_PAGE_INDEX-->
In-Memory Indexing
In the previous section, I mentioned that new documents added to an indexare stored in memory before being written to the disk. You also saw how tocontrol the rate at which this is done via IndexWriter's instancevariables.The Lucene distribution contains the RAMDirectoryclass, which gives even more control over this process.This class implementsthe Directory interface, just like FSDirectory does,but stores indexed documents in memory, while FSDirectory storesthem on disk.
Because RAMDirectory does not write anything to the disk, itis faster than FSDirectory.However, since computers usually comewith less RAM than hard disk space, RAMDirectory is not suitablefor very large indices.
The MemoryVsDisk class demonstrates how to useRAMDirectory as an in-memory buffer in order to improve theindexing speed.
import org.apache.lucene.index.IndexWriter;import org.apache.lucene.analysis.Analyzer;import org.apache.lucene.analysis.StopAnalyzer;import org.apache.lucene.document.Document;import org.apache.lucene.document.Field;import org.apache.lucene.store.Directory;import org.apache.lucene.store.RAMDirectory;import java.io.IOException;/** * Creates an index called 'index' in a temporary directory. * The number of documents to add to this index, the mergeFactor and * the maxMergeDocs must be specified on the command line * in that order - this class expects to be called correctly. * Additionally, if the fourth command line argument is '-r' this * class will first index all documents in RAMDirectory before * flushing them to the disk in the end.To make this class use the * regular FSDirectory use '-f' as the fourth command line argument. ** Note: before running this for the first time, manually create the * directory called 'index' in your temporary directory. */public class MemoryVsDisk{ public static void main(String[] args) throws Exception { int docsInIndex= Integer.parseInt(args); // create an index called 'index' in a temporary directory String indexDir = System.getProperty("java.io.tmpdir", "tmp") + System.getProperty("file.separator") + "index"; Analyzer analyzer= new StopAnalyzer(); long startTime = System.currentTimeMillis(); if ("-r".equalsIgnoreCase(args)) { // if -r argument was specified, use RAMDirectory RAMDirectory ramDir = new RAMDirectory(); IndexWriterramWriter = new IndexWriter(ramDir, analyzer, true); addDocs(ramWriter, docsInIndex); IndexWriter fsWriter = new IndexWriter(indexDir, analyzer, true); fsWriter.addIndexes(new Directory[] { ramDir }); ramWriter.close(); fsWriter.close(); } else { // create an index using FSDirectory IndexWriter fsWriter= new IndexWriter(indexDir, analyzer, true); fsWriter.mergeFactor= Integer.parseInt(args); fsWriter.maxMergeDocs = Integer.parseInt(args); addDocs(fsWriter, docsInIndex); fsWriter.close(); } long stopTime = System.currentTimeMillis(); System.out.println("Total time: " + (stopTime - startTime) + " ms"); } private static void addDocs(IndexWriter writer, int docsInIndex) throws IOException { for (int i = 0; i < docsInIndex; i++) { Document doc = new Document(); doc.add(Field.Text("fieldname", "Bibamus, moriendum est")); writer.addDocument(doc); } }} To create an index with 10,000 documents and only use FSDirectory, use this:
prompt> time java MemoryVsDisk 10000 10 100000 -fTotal time: 41380 msreal 0m42.739suser 0m36.750ssys 0m4.180s To create the index of the same size but do it faster, withRAMDirectory, call MemoryVsDisk as follows:
prompt>time java MemoryVsDisk 10000 10 100000 -rTotal time: 27325 msreal 0m28.695suser 0m27.920ssys 0m0.610s However, note that you can achieve the same, or even better, performance bychoosing a more suitable value for mergeFactor:
prompt> time java MemoryVsDisk 10000 1000 100000 -fTotal time: 24724 msreal 0m26.108suser 0m25.280ssys 0m0.620s Be careful, however, when tuning mergeFactor.A value thatrequires more memory than your JVM can access may cause thejava.lang.OutOfMemoryError error.
Finally, do not forget that you can greatly influence the performance ofany Java application by giving the JVM more memory to work with:
prompt> time java -Xmx300MB -Xms200MB MemoryVsDisk 10000 10 100000 -rTotal time: 15166 msreal 0m17.311suser 0m15.400ssys 0m1.590sMerging Indices
If you want to improve indexing performance with Lucene, and manipulatingIndexWriter's mergeFactor andmaxMergeDocs prove insufficient, you can useRAMDirectory to create in-memory indices.You could create amulti-threaded indexing application that uses multipleRAMDirectory-based indices in parallel, one in each thread, andmerges them into a single index on the disk using IndexWriter'saddIndexes(Directory[]) method.Taking this idea further, asophisticated indexing application could even create in-memory indices onmultiple computers in parallel.To make full use of this approach, one needsto ensure that the thread that performs the actual indexing on the disk isnever idle, as that translates to wasted time.
Indexing in Multi-Threaded Environments
While multiple threads or processes can search (i.e. read) a single Luceneindex simultaneously, only a single thread or process is allowed to modify(write) an index at a time.If your indexing application uses multipleindexing threads that are adding documents to the same index, you mustserialize their calls to the IndexWriter.addDocument(Document)method.Leaving these calls unserialized may cause threads to get in eachother's way and modify the index in unwanted ways, causing Lucene to throwexceptions.In addition, to prevent misuse, Lucene uses file-based locks inorder to stop multiple threads or processes from creatingIndexWriters with the same index directory at the same time.
For instance, this code:
import org.apache.lucene.index.IndexWriter;import org.apache.lucene.analysis.Analyzer;import org.apache.lucene.analysis.StopAnalyzer;/** * Demonstrates how Lucene uses locks to prevent multiple processes from * writing to the same index at the same time. * Note: before running this for the first time, manually create the * directory called 'index' in your temporary directory. */public class DoubleTrouble{ public static void main(String[] args) throws Exception { // create an index called 'index' in a temporary directory String indexDir = System.getProperty("java.io.tmpdir", "tmp") + System.getProperty("file.separator") + "index"; Analyzer analyzer = new StopAnalyzer(); IndexWriter firstWriter= new IndexWriter(indexDir, analyzer, true); // the following line will cause an exception IndexWriter secondWriter = new IndexWriter(indexDir, analyzer, false); // the following two lines will never even be reached firstWriter.close(); secondWriter.close(); }} will cause the following exception:
Exception in thread "main" java.io.IOException: \ Index locked for write: Lock@/tmp/index/write.lock at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:145) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:122) at DoubleTrouble.main(DoubleTrouble.java:23)Optimizing Indices
I have mentioned index optimization a few times in this article, but I havenot yet explained it.To optimize an index, one has to calloptimize() on an IndexWriter instance.When thishappens, all in-memory documents are flushed to the disk and all index segmentsare merged into a single segment, reducing the number of files that make up theindex.However, optimizing an index does not help improve indexingperformance.As a matter of fact, optimizing an index during the indexingprocess will only slow things down.Despite this, optimizing may sometimes benecessary in order to keep the number of open files under control.Forinstance, optimizing an index during the indexing process may be needed insituations where searching and indexing happen concurrently, since bothprocesses keep their own set of open files.A good rule of thumb is that ifmore documents will be added to the index soon, you should avoid callingoptimize().If, on the other hand, you know that the index willnot be modified for a while, and the index will only be searched, you shouldoptimize it.That will reduce the number of segments (files on the disk), andconsequently improve search performance--the fewer files Lucene has to openwhile searching, the faster the search.
To illustrate the effect of optimizing an index, we can use theIndexOptimizeDemo class:
import org.apache.lucene.index.IndexWriter;import org.apache.lucene.analysis.Analyzer;import org.apache.lucene.analysis.StopAnalyzer;import org.apache.lucene.document.Document;import org.apache.lucene.document.Field;/** * Creates an index called 'index' in a temporary directory. * If you want the index to optimize the index at the end use '-o' * command line argument.If you do not want to optimize the index * at the end use any other value for the command line argument. * This class expects to be called correctly. ** Note: before running this for the first time, manually create the * directory called 'index' in your temporary directory. */public class IndexOptimizeDemo{ public static void main(String[] args) throws Exception { // create an index called 'index' in a temporary directory String indexDir = System.getProperty("java.io.tmpdir", "tmp") + System.getProperty("file.separator") + "index"; Analyzer analyzer = new StopAnalyzer(); IndexWriter writer = new IndexWriter(indexDir, analyzer, true); for (int i = 0; i < 15; i++) { Document doc = new Document(); doc.add(Field.Text("fieldname", "Bibamus, moriendum est")); writer.addDocument(doc); } if ("-o".equalsIgnoreCase(args)) { System.out.println("Optimizing the index..."); writer.optimize(); } writer.close(); }} As you can see from the class Javadoc and code, the created index will beoptimized only if -o command line argument is used.To create anunoptimized index with this class, use this:
prompt> java IndexOptimizeDemo -n-rw-rw-r-- 1 otis otis 10 Feb 18 23:50 _a.f1-rw-rw-r-- 1 otis otis 260 Feb 18 23:50 _a.fdt-rw-rw-r-- 1 otis otis 80 Feb 18 23:50 _a.fdx-rw-rw-r-- 1 otis otis 14 Feb 18 23:50 _a.fnm-rw-rw-r-- 1 otis otis 30 Feb 18 23:50 _a.frq-rw-rw-r-- 1 otis otis 30 Feb 18 23:50 _a.prx-rw-rw-r-- 1 otis otis 11 Feb 18 23:50 _a.tii-rw-rw-r-- 1 otis otis 41 Feb 18 23:50 _a.tis-rw-rw-r-- 1 otis otis 4 Feb 18 23:50 deletable-rw-rw-r-- 1 otis otis 5 Feb 18 23:50 _g.f1-rw-rw-r-- 1 otis otis 130 Feb 18 23:50 _g.fdt-rw-rw-r-- 1 otis otis 40 Feb 18 23:50 _g.fdx-rw-rw-r-- 1 otis otis 14 Feb 18 23:50 _g.fnm-rw-rw-r-- 1 otis otis 15 Feb 18 23:50 _g.frq-rw-rw-r-- 1 otis otis 15 Feb 18 23:50 _g.prx-rw-rw-r-- 1 otis otis 11 Feb 18 23:50 _g.tii-rw-rw-r-- 1 otis otis 41 Feb 18 23:50 _g.tis-rw-rw-r-- 1 otis otis 22 Feb 18 23:50 segments Example 2: An unoptimized index usually contains more than onesegment.
This index contains two segments.To create a fully-optimized index, callthis class with -o command line argument:
prompt> java IndexOptimizeDemo -o-rw-rw-r-- 1 otis otis 4 Feb 18 23:50 deletable-rw-rw-r-- 1 otis otis 15 Feb 18 23:50 _h.f1-rw-rw-r-- 1 otis otis 390 Feb 18 23:50 _h.fdt-rw-rw-r-- 1 otis otis 120 Feb 18 23:50 _h.fdx-rw-rw-r-- 1 otis otis 14 Feb 18 23:50 _h.fnm-rw-rw-r-- 1 otis otis 45 Feb 18 23:50 _h.frq-rw-rw-r-- 1 otis otis 45 Feb 18 23:50 _h.prx-rw-rw-r-- 1 otis otis 11 Feb 18 23:50 _h.tii-rw-rw-r-- 1 otis otis 41 Feb 18 23:50 _h.tis-rw-rw-r-- 1 otis otis 15 Feb 18 23:50 segments Example 3: A fully-optimized index contains only a singlesegment.
Conclusion
This article has discussed the basic structure of a Lucene index and hasdemonstrated a few techniques for improving indexing performance.You alsolearned about potential problems with indexing in multi-threaded environments,about what it means to optimize an index, and how this affects indexing.Thisknowledge should allow you to gain more control over Lucene's indexing processto improve its performance.The next article will examine Lucene's text-searching capabilities.
页:
[1]