Set of nightly Lucene benchmarks on sparse and dense documents Use, just like the rest of Lucene's index parts.Īlong with these Lucene improvements we've added With all these improvements, and I'm sure many more to come, weįinally get Lucene's doc values to a point where only pay for what you Performance beyond where we were previously Use cases tested by Lucene's existing nightly benchmarks, and in some These changes brought back much of our search performance on the dense So we are free to make major changes to the index formatĪbstraction for the common single-valued numerics and binary cases Hard to improve our default codec to take advantage of the more Performance gains as well, since the more restrictive API gives codecs See the size of your index suddenly grow 10-fold! But even denseĬases, where most documents have a value for the field, should see Lucene's non-sparse encoding of such fields has been particularly Sparse cases, where not all documents have a value for each doc valuesįield, should especially benefit from this change. Since this was aįirst as the existing Lucene codec had to use temporary silly wrapperĬlasses to translate its random-access API into an iterator API. That change was already massive enough that we decided to break outĪll such codec improvements to future issues. Other optimizations, like our postings implementations do. Restrictive access pattern than an arbitrary random access API, thisĬhange gives codecs more freedom to use aggressive compression and Previous random-access API to an iterator API instead. Switching out how doc values are accessed at search time from the , which was "simply" a low-level raw plumbing change using expressions to combine multiple signals into a score, or for sorting,įield/document holding index-time scoring signals (the field's length Row-stride fashion, and are therefore relatively slow to access.ĭoc values can be used to hold scoring signals, e.g. This is in contrast to Lucene's stored documentįields, which store all field values for one document together in a Quite fast to access at search time, since they are storedĬolumn-stride such that only the value for that one field needs to beĭecoded per hit. Multi-valued) and binary data blobs per document. Store numerics (single- or multi-valued), sorted keywords (single or These changes will be in Lucene's next major release (7.0) and will likely not be back-ported to any 6.x release, so it will be some time until Elasticsearch exposes this.ĭoc values are Lucene's column-stride field value storage, letting you These changes fix doc values so you only pay for what you use, just like all other parts of the Lucene index. Recently we've made some big changes to Apache Lucene around how doc values are indexed and searched, including new nightly benchmarks to measure our progress, based on the New York City taxi ride data corpus.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |