Created: (SOLR-236) Field collapsing

    [ https://issues.apache.org/jira/browse/SO
LR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:
comment-tabpanel#action_12501582 ] 

Yonik Seeley commented on SOLR-236:
-----------------------------------

Oh I see... the modified sort is *just* to build the
filter.

The building-the-filter part is a problem though... asking
for *all* matching docs in sorted order isn't that
scalable.
If we get the interface right though, more efficient
implementations can follow.
For that reason, it might be good for implementatin details
like "collapseCache" to be private.

> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.2
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version (1.2)
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

    [ https://issues.apache.org/jira/browse/SO
LR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:
comment-tabpanel#action_12501583 ] 

Emmanuel Keller commented on SOLR-236:
--------------------------------------

Correct, except that collapse result is only used as filter
to the final result to hide collapsed documents.

P.S.: Sorry, if my answers are a little short, I am not
perfectly fluent in english.

> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.2
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version (1.2)
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

    [ https://issues.apache.org/jira/browse/SO
LR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:
comment-tabpanel#action_12503125 ] 

Ryan McKinley commented on SOLR-236:
------------------------------------

Any thoughts on what the faceting semantics for field
collapsing should be?

That is, should faceting apply to the collapsed results or
the pre-collapsed results?  

I think the pre-collapsed results.

> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.2
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version (1.2)
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

    [ https://issues.apache.org/jira/browse/SO
LR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:
comment-tabpanel#action_12503131 ] 

Yonik Seeley commented on SOLR-236:
-----------------------------------

Yes, it seems like faceting should be for pre-collapsed.

> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.2
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version (1.2)
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

    [ https://issues.apache.org/jira/browse/SO
LR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:
comment-tabpanel#action_12503162 ] 

Emmanuel Keller commented on SOLR-236:
--------------------------------------

Do we have to make a choice ? Both behaviors are
interesting. 
What about a new parameter like collapse.facet=[pre|post] ?



> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.2
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version (1.2)
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

    [ https://issues.apache.org/jira/browse/SO
LR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:
comment-tabpanel#action_12503185 ] 

Yonik Seeley commented on SOLR-236:
-----------------------------------

We facet on the complete set of documents matching a query,
even when the user only requests the top 10 matches.  It
seems we should do the same here.  The set of documents is
the same, the only difference is what "top"
documents are returned.

> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.2
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version (1.2)
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

     [ 
https://issues.apache.org/jira/browse/SOLR-236?page=com.atla
ssian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Keller updated SOLR-236:
---------------------------------

    Attachment: SOLR-236-FieldCollapsing.patch

New release:
- Fieldcollapsing added on DisMaxRequestHandler
- Types are correctly handled on collapsed field

> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.2
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version (1.2)
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

     [ 
https://issues.apache.org/jira/browse/SOLR-236?page=com.atla
ssian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Keller updated SOLR-236:
---------------------------------

    Attachment:     (was: SOLR-236-FieldCollapsing.patch)

> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.2
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version (1.2)
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

     [ 
https://issues.apache.org/jira/browse/SOLR-236?page=com.atla
ssian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Emmanuel Keller updated SOLR-236:
---------------------------------

          Description: 
This patch include a new feature called "Field
collapsing".

"Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99

The implementation add 3 new query parameters (SolrParams):
"collapse.field" to choose the field used to group
results
"collapse.type" normal (default value) or
adjacent
"collapse.max" to select how many continuous
results are allowed before collapsing

TODO (in progress):
- More documentation (on source code)
- Test cases

Two patches:
- "field_collapsing.patch" for current development
version
- "field_collapsing_1.1.0.patch" for Solr-1.1.0


P.S.: Feedback and misspelling correction are welcome 

  was:
This patch include a new feature called "Field
collapsing".

"Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99

The implementation add 3 new query parameters (SolrParams):
"collapse.field" to choose the field used to group
results
"collapse.type" normal (default value) or
adjacent
"collapse.max" to select how many continuous
results are allowed before collapsing

TODO (in progress):
- More documentation (on source code)
- Test cases

Two patches:
- "field_collapsing.patch" for current development
version (1.2)
- "field_collapsing_1.1.0.patch" for Solr-1.1.0


P.S.: Feedback and misspelling correction are welcome 

    Affects Version/s:     (was: 1.2)
                       1.3

> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.3
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a given field to a single entry in the
result set. Site collapsing is a special case of this, where
all results for a given web site is collapsed into one or
two entries in the result set, typically with an associated
"more documents from this site" link. See also
Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.

Having worked on a number of customer implementations
regarding this
feature I can say that the number one requirement is for the
facet
counts to be accurate post collapsing.  It all comes down to
the user
experience.  For example, if I run a query that get
collapsed and has a
facet count for the non-collapsed value then when I click on
that facet
for refinement the number of hits in my subsequent query
will not match
the number of hits displayed by that facet count.  Ie if it
says there
are 10 docs in my result set of type x then when I click on
type x I
expect to get back 10 hits.  Further, I could easily end up
with a
result set with 15 total hits but a facet count hat says
there are 200
results of type x which is very disconcerting from a user
perspective. 

I agree that there are times when pre-faceting is also good,
but
post-faceting has always been a rather hard requirement for
most
ecommerce/data discovery sites.

- will

-----Original Message-----
From: Emmanuel Keller (JIRA) [mailto:jiraapache.org] 
Sent: Sunday, June 10, 2007 7:33 AM
To: solr-devlucene.apache.org
Subject: [jira] Commented: (SOLR-236) Field collapsing


    [
https://issues.apache.org/jira/browse/SO
LR-236?page=com.atlassian.jira.p
lugin.system.issuetabpanels:comment-tabpanel#action_12503162
] 

Emmanuel Keller commented on SOLR-236:
--------------------------------------

Do we have to make a choice ? Both behaviors are
interesting. 
What about a new parameter like collapse.facet=[pre|post] ?



> Field collapsing
> ----------------
>
>                 Key: SOLR-236
>                 URL: https:
//issues.apache.org/jira/browse/SOLR-236
>             Project: Solr
>          Issue Type: New Feature
>          Components: search
>    Affects Versions: 1.2
>            Reporter: Emmanuel Keller
>         Attachments: field_collapsing_1.1.0.patch,
SOLR-236-FieldCollapsing.patch,
SOLR-236-FieldCollapsing.patch
>
>
> This patch include a new feature called "Field
collapsing".
> "Used in order to collapse a group of results with
similar value for a
given field to a single entry in the result set. Site
collapsing is a
special case of this, where all results for a given web site
is
collapsed into one or two entries in the result set,
typically with an
associated "more documents from this site" link.
See also Duplicate
detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=2
99
> The implementation add 3 new query parameters
(SolrParams):
> "collapse.field" to choose the field used to
group results
> "collapse.type" normal (default value) or
adjacent
> "collapse.max" to select how many continuous
results are allowed
before collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current
development version (1.2)
> - "field_collapsing_1.1.0.patch" for
Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue
online.