Grant Ingersoll wrote:
>
> Some randomly pieced together thoughts (I may not even
be fully awake
> yet  so feel
free to tell me I'm not understanding this correctly)
>
> My first thought was how is this different from just
having a binary
> field, but if I understand correctly it is to be stored
in a separate file?
>
> Now you are proposing a faster storage mechanism for
them, essentially,
> since they are to be stored separately from the
Documents themselves?
> But the other key is they are all stored next to each
other, right, so
> the scan is a lot faster?
>
Yes, scanning and skipping would be much faster, comparable
to a posting
list. In fact, what I'm proposing is a new kind of posting
list. Since
you mentioned the magic term "flexible indexing"
already ;), let's take
a look at
http://wiki.apache.org/lucene-java/FlexibleIndexing.
Here 4
kinds of posting lists are proposed:
a. <doc>+
b. <doc, boost>+
c. <doc, freq, <position>+ >+
d. <doc, freq, <position, boost>+ >+
Today, we have c. and d. already. c. is the original Lucene
format, and
d. can be achieved by storing the boost as a payload.
The new format I'm proposing actually covers a. and b. If
you don't
store a payload it's basically a binary posting list without
freq and
positions (a.). If you store the boost as a payload, then
you have b.
> I think one of the questions that will come up from
users is when should
> I use addMetadata and when should I use addField? Why
make the
> distinction to the user? Fields have always
represented metadata, all
I'd like to make a distinction because IMO these are two
different use
cases. Not necessarily in terms of functionality, but in
terms of
performance. You are right, you can store everything today
as stored
fields, but if you want to use e. g. a stored value for
scoring, then
performance is terrible. This is simply the nature of the
store - it is
optimized for returning all stored fields for a document.
Even a
FieldSelector doesn't help you too much, unless the docs
contain very
big fields that you don't want to return. The reason is that
two random
I/Os are necessary to find the stored fields of a document.
Then only
sequential I/O has to be performed. And the overhead of
loading e. g.
10KB instead of 2KB is not big, much less than two random
I/Os, I believe.
Payloads are also much better in terms of cache utilization.
Since they
are stored next to each other, and if accessed frequently
(in every
search), then it's very likely that big portions of that
posting list
will be in the cache.
So the answer to the question when to use a stored field and
when to use
a payload should be: use payloads when you access the data
during query
evaluation/scoring, use stored fields when you need the data
to
construct a search result from a hit.
> fields, right? Perhaps in this way, if users were
willing to commit to
> fixed length fields for the first level, we could also
make field
> updating of these types of fields possible w/o having
to reindex?????
>
Yes I was thinking the same. Just like norms.
------------------------------------------------------------
---------
To unsubscribe, e-mail: java-dev-unsubscribe lucene.apache.org
For additional commands, e-mail: java-dev-helplucene.apache.org
|
