Proposal for implementing Full Text Search

Hi,
I would like to work on implementing full text search in
Bricolage.
This is a rough draft of the proposal i drew up.
Please give me your comments.

Proposal: Add full text-search support to Bricolage.

          The tools will broadly include:
        * DBI- a database interface for Perl
        * MASON::HTML- A Perl-based web programming
language. It allows Perl
to be embedded in HTML.
        * HTML::Template- A Perl module to use HTML
Templates from CGI
scripts

Synopsis:

    The aim of this project is to include full text-search
support to the
content mangement system, Bricolage.
    Bricolage already has search capabilities. A full
text-search will be
integrated to the already existent
    search API.

    To implement this four major requirements will have to
be fulfilled:

    A tsearch index will have to be generated for the data
placed in each
table.
    An additional field will have to be added holding the
tsearch index
corresponding
    to the data it holds. A function to generate the tsearch
index and add
it to
    each table will have to be written

    Next the search API will have to be modified in such a
manner that it
recognizes
    the tsearch index column that has been added to each
table so that the
complete
    text that is to be searched for can be recognized within
any table
simply using
    the tsearch index.

    A UI widget already exists in Bricolage, It has to be
modified to allow
for
    display of the full-text search results. The UI widget
will have to be
adapted
    to the modified search API

    Since Bricolage currently boasts an array of SOAP
integrated APIs, the
modified
    search API, or rather the full-text search API, will
naturally be
integrated too.
    So the API should be further appended to hold functions
to code the
users' search
    request into XML and send it across to the server and a
function that
recieves the
    result from the server and decodes it before displaying
it the user.

    The project will support only postgresql 8.3 upwards, as
the full text
search is
    natively supported only by postgresql 8.3 upwards. So
there will have to
be a provision
    to disable the full text search when lower versions are
used.

-- 
Cheers,
Unni

Hi!

On Wed, Mar 26, 2008 at 11:06 PM, Unni <kpnunnigmail.com> wrote:
> Hi,
>  I would like to work on implementing full text search
in Bricolage.
>  This is a rough draft of the proposal i drew up.
>  Please give me your comments.
>

Looks good to me, guys any other suggestion ? Especially,
soap api
part, I dont have much idea about bric_soap

regards
VK
-- 
The hidden harmony is better than the obvious!!

On Mar 26, 2008, at 16:06, Unni wrote:

> I would like to work on implementing full text search
in Bricolage.
> This is a rough draft of the proposal i drew up.
> Please give me your comments.

Many thanks for your proposal, Unni. We really appreciate
it. Quick  
question: Do you have any exper

> Proposal: Add full text-search support to Bricolage.
>
>          The tools will broadly include:
>        * DBI- a database interface for Perl
>        * MASON::HTML- A Perl-based web programming
language. It  
> allows Perl
> to be embedded in HTML.
>        * HTML::Template- A Perl module to use HTML
Templates from CGI
> scripts

What's this bit about? It sounds like some of the things
that  
Bricolage is built on, though I don't know how they relate
to full- 
text search. Or is this just a side-effect of the TPF
proposal form?

> Synopsis:
>
>    The aim of this project is to include full
text-search support to  
> the
> content mangement system, Bricolage.
>    Bricolage already has search capabilities. A full
text-search  
> will be
> integrated to the already existent
>    search API.

Good synopsis.

>    To implement this four major requirements will have
to be  
> fulfilled:
>
>    A tsearch index will have to be generated for the
data placed in  
> each
> table.
>    An additional field will have to be added holding
the tsearch index
> corresponding
>    to the data it holds. A function to generate the
tsearch index  
> and add
> it to
>    each table will have to be written

A trigger function, yes?

To what objects in Bricolage do you intend to add full-text
search  
support?

>    Next the search API will have to be modified in such
a manner  
> that it
> recognizes
>    the tsearch index column that has been added to each
table so  
> that the
> complete
>    text that is to be searched for can be recognized
within any table
> simply using
>    the tsearch index.

Probably a new method.

>    A UI widget already exists in Bricolage, It has to
be modified to  
> allow
> for
>    display of the full-text search results. The UI
widget will have  
> to be
> adapted
>    to the modified search API

What do these modifications entail? Hints:

  * Result context display
  * Form field(s)
  * Form callback (think controller)

>    Since Bricolage currently boasts an array of SOAP
integrated  
> APIs, the
> modified
>    search API, or rather the full-text search API, will
naturally be
> integrated too.
>    So the API should be further appended to hold
functions to code the
> users' search
>    request into XML and send it across to the server
and a function  
> that
> recieves the
>    result from the server and decodes it before
displaying it the  
> user.

Okay, good.

>    The project will support only postgresql 8.3
upwards, as the full  
> text
> search is
>    natively supported only by postgresql 8.3 upwards.
So there will  
> have to
> be a provision
>    to disable the full text search when lower versions
are used.

The installer code needs to be modified to detect this the
RDBMS and  
version in order to set this at install time, but users
should be able  
to change the setting later should they upgrade the
database.

Which reminds me: there also needs to be a migration script
that users  
can run to populate the tsearch index column, so that
existing  
Bricolage installations can get the full-text support when
they  
upgrade. Make sense?

This looks good, Unni. If you can just address the points I
raise  
above, I think you'll be good to go!

Best,

David

Hi all,
>
      I'll just mention the changes i made in my proposal.

I intend to begin by adding full text search for objects
like stories and
media.

The generation of tsearch index and population of the tables
will be
performed
by a row-level trigger fuction.

Anew method is to be added to the search API so that it may
recognize and
utilize the tsearch index that has been added.

The UI will be modified so as to facilitate display of full
text search
result. A callback will be written to use the modified API

The recognition of the postgresql version will  be  done  by
 modifying
the  installer code  so as to enable/disable  full text 
search  at
install  time. This setting can be modified later on and a
migration script
will be writtenwhich can be run to enable full text search
on existent
upgraded versions.

Do let me know what you think.



> Cheers,
>
>
Unni

>
> >
> >
> > cheers,
> > Unni
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Mar 28, 2008 at 7:37 AM, David E. Wheeler
<davidkineticode.com>
> > wrote:
> >
> > > On Mar 26, 2008, at 16:06, Unni wrote:
> > >
> > > > I would like to work on implementing
full text search in Bricolage.
> > > > This is a rough draft of the proposal i
drew up.
> > > > Please give me your comments.
> > >
> > > Many thanks for your proposal, Unni. We
really appreciate it. Quick
> > > question: Do you have any exper
> >
> >
> >
> >
> > >
> > >
> > > > Proposal: Add full text-search support
to Bricolage.
> > > >
> > > >          The tools will broadly
include:
> > > >        * DBI- a database interface for
Perl
> > > >        * MASON::HTML- A Perl-based web
programming language. It
> > > > allows Perl
> > > > to be embedded in HTML.
> > > >        * HTML::Template- A Perl module
to use HTML Templates from
> > > CGI
> > > > scripts
> > >
> > > What's this bit about? It sounds like some of
the things that
> > > Bricolage is built on, though I don't know
how they relate to full-
> > > text search. Or is this just a side-effect of
the TPF proposal form?
> > >
> > > > Synopsis:
> > > >
> > > >    The aim of this project is to include
full text-search support to
> > > > the
> > > > content mangement system, Bricolage.
> > > >    Bricolage already has search
capabilities. A full text-search
> > > > will be
> > > > integrated to the already existent
> > > >    search API.
> > >
> > > Good synopsis.
> > >
> > > >    To implement this four major
requirements will have to be
> > > > fulfilled:
> > > >
> > > >    A tsearch index will have to be
generated for the data placed in
> > > > each
> > > > table.
> > > >    An additional field will have to be
added holding the tsearch
> > > index
> > > > corresponding
> > > >    to the data it holds. A function to
generate the tsearch index
> > > > and add
> > > > it to
> > > >    each table will have to be written
> > >
> > > A trigger function, yes?
> > >
> > > To what objects in Bricolage do you intend to
add full-text search
> > > support?
> > >
> > > >    Next the search API will have to be
modified in such a manner
> > > > that it
> > > > recognizes
> > > >    the tsearch index column that has
been added to each table so
> > > > that the
> > > > complete
> > > >    text that is to be searched for can
be recognized within any
> > > table
> > > > simply using
> > > >    the tsearch index.
> > >
> > > Probably a new method.
> > >
> > > >    A UI widget already exists in
Bricolage, It has to be modified to
> > > > allow
> > > > for
> > > >    display of the full-text search
results. The UI widget will have
> > > > to be
> > > > adapted
> > > >    to the modified search API
> > >
> > > What do these modifications entail? Hints:
> > >
> > >  * Result context display
> > >  * Form field(s)
> > >  * Form callback (think controller)
> > >
> > > >    Since Bricolage currently boasts an
array of SOAP integrated
> > > > APIs, the
> > > > modified
> > > >    search API, or rather the full-text
search API, will naturally be
> > > > integrated too.
> > > >    So the API should be further appended
to hold functions to code
> > > the
> > > > users' search
> > > >    request into XML and send it across
to the server and a function
> > > > that
> > > > recieves the
> > > >    result from the server and decodes it
before displaying it the
> > > > user.
> > >
> > > Okay, good.
> > >
> > > >    The project will support only
postgresql 8.3 upwards, as the full
> > > > text
> > > > search is
> > > >    natively supported only by postgresql
8.3 upwards. So there will
> > > > have to
> > > > be a provision
> > > >    to disable the full text search when
lower versions are used.
> > >
> > > The installer code needs to be modified to
detect this the RDBMS and
> > > version in order to set this at install time,
but users should be able
> > > to change the setting later should they
upgrade the database.
> > >
> > > Which reminds me: there also needs to be a
migration script that users
> > > can run to populate the tsearch index column,
so that existing
> > > Bricolage installations can get the full-text
support when they
> > > upgrade. Make sense?
> > >
> > > This looks good, Unni. If you can just
address the points I raise
> > > above, I think you'll be good to go!
> > >
> > > Best,
> > >
> > > David
> > >
> > >
> >
> >
> > --
> > Cheers,
> > Unni
>
>
>
>
> --
> Cheers,
> Unni




-- 
Cheers,
Unni

In the  revised  proposal  please read the proposal
statement as:


Proposal: Add full text-search support to Bricolage.

      Postgresql versions 8.3 and above have built-in full
text search
support.
      Support for this utility will be added to Bricolage.


>
>
>
>
> cheers,
> Unni
>
>
>
>
>
>
>
>
>
>
> On Fri, Mar 28, 2008 at 7:37 AM, David E. Wheeler
<davidkineticode.com>
> wrote:
>
> > On Mar 26, 2008, at 16:06, Unni wrote:
> >
> > > I would like to work on implementing full
text search in Bricolage.
> > > This is a rough draft of the proposal i drew
up.
> > > Please give me your comments.
> >
> > Many thanks for your proposal, Unni. We really
appreciate it. Quick
> > question: Do you have any exper
>
>
>
>
> >
> >
> > > Proposal: Add full text-search support to
Bricolage.
> > >
> > >          The tools will broadly include:
> > >        * DBI- a database interface for Perl
> > >        * MASON::HTML- A Perl-based web
programming language. It
> > > allows Perl
> > > to be embedded in HTML.
> > >        * HTML::Template- A Perl module to use
HTML Templates from CGI
> > > scripts
> >
> > What's this bit about? It sounds like some of the
things that
> > Bricolage is built on, though I don't know how
they relate to full-
> > text search. Or is this just a side-effect of the
TPF proposal form?
> >
> > > Synopsis:
> > >
> > >    The aim of this project is to include full
text-search support to
> > > the
> > > content mangement system, Bricolage.
> > >    Bricolage already has search capabilities.
A full text-search
> > > will be
> > > integrated to the already existent
> > >    search API.
> >
> > Good synopsis.
> >
> > >    To implement this four major requirements
will have to be
> > > fulfilled:
> > >
> > >    A tsearch index will have to be generated
for the data placed in
> > > each
> > > table.
> > >    An additional field will have to be added
holding the tsearch index
> > > corresponding
> > >    to the data it holds. A function to
generate the tsearch index
> > > and add
> > > it to
> > >    each table will have to be written
> >
> > A trigger function, yes?
> >
> > To what objects in Bricolage do you intend to add
full-text search
> > support?
> >
> > >    Next the search API will have to be
modified in such a manner
> > > that it
> > > recognizes
> > >    the tsearch index column that has been
added to each table so
> > > that the
> > > complete
> > >    text that is to be searched for can be
recognized within any table
> > > simply using
> > >    the tsearch index.
> >
> > Probably a new method.
> >
> > >    A UI widget already exists in Bricolage,
It has to be modified to
> > > allow
> > > for
> > >    display of the full-text search results.
The UI widget will have
> > > to be
> > > adapted
> > >    to the modified search API
> >
> > What do these modifications entail? Hints:
> >
> >  * Result context display
> >  * Form field(s)
> >  * Form callback (think controller)
> >
> > >    Since Bricolage currently boasts an array
of SOAP integrated
> > > APIs, the
> > > modified
> > >    search API, or rather the full-text search
API, will naturally be
> > > integrated too.
> > >    So the API should be further appended to
hold functions to code the
> > > users' search
> > >    request into XML and send it across to the
server and a function
> > > that
> > > recieves the
> > >    result from the server and decodes it
before displaying it the
> > > user.
> >
> > Okay, good.
> >
> > >    The project will support only postgresql
8.3 upwards, as the full
> > > text
> > > search is
> > >    natively supported only by postgresql 8.3
upwards. So there will
> > > have to
> > > be a provision
> > >    to disable the full text search when lower
versions are used.
> >
> > The installer code needs to be modified to detect
this the RDBMS and
> > version in order to set this at install time, but
users should be able
> > to change the setting later should they upgrade
the database.
> >
> > Which reminds me: there also needs to be a
migration script that users
> > can run to populate the tsearch index column, so
that existing
> > Bricolage installations can get the full-text
support when they
> > upgrade. Make sense?
> >
> > This looks good, Unni. If you can just address the
points I raise
> > above, I think you'll be good to go!
> >
> > Best,
> >
> > David
> >
> >
>
>
> --
> Cheers,
> Unni




-- 
Cheers,
Unni

On Mar 29, 2008, at 01:10, Unni wrote:

>      I'll just mention the changes i made in my
proposal.

Thanks. I still don't know what the bits about DBI, Mason,
and  
HTML::Template have to do with it (and it's HTML::Mason, not
 
MASON::HTML). Can you elaborate on that?

> I intend to begin by adding full text search for
objects like  
> stories and
> media.

The proposal says "related media." You'll want to
change that to just  
"media." You might as well include templates, too,
as they use all the  
same interface elements as stories and media. It won't take
much more  
work for templates.

> The generation of tsearch index and population of the
tables will be
> performed
> by a row-level trigger fuction.

Okay.

> Anew method is to be added to the search API so that it
may  
> recognize and
> utilize the tsearch index that has been added.

Okay.

> The UI will be modified so as to facilitate display of
full text  
> search
> result. A callback will be written to use the modified
API

Okay. Have you looked at the Bricolage source code at all,
yet? Do you  
know what you're really getting into here? Also, I mentioned
a few of  
the details in my last post. I suggest that you integrate
them.

> The recognition of the postgresql version will  be 
done  by   
> modifying
> the  installer code  so as to enable/disable  full text
 search  at
> install  time. This setting can be modified later on
and a migration  
> script
> will be writtenwhich can be run to enable full text
search on existent
> upgraded versions.

Great. How will it work, BTW? Any ideas?

> Do let me know what you think.

Best,

David

PS: Please subscribe to the users list so that I don't have
to keep  
approving your posts!

Best,

David

On Mar 30, 2008, at 06:43, Unni wrote:

> I've made a few small changes.

Thanks Unni, looks great.

> The installer code will
> 	have to be modified to recognize the RDBMS and the
version, so that  
> we can disable the
> 	full text search option for lower versions at install
time.

Just to be clear, the installer already detects these
things. The bit  
you'll have to do to plug it into the configuration will
actually be  
quite simple.

> The users can modify this
> 	setting if they upgrade. A migration script will be
written which  
> can be run to generate
> 	the tsearch index and populate the relevant tables so
that the  
> users can enable full
> 	text search once they upgrade.

Just as an implementation note, not important for this
proposal, but  
important for developing a full project plan if/when your
project is  
accepted, is to think about the format to use for that
column. For  
stories and media in particular, data will have to be pulled
from a  
bunch of different tables to populate the index.

> The script will function in a similar manner to the
trigger
> 	function

Ideally it would just use the trigger function directly,
eh?

Anyway, those are just nits, not really important for this
proposal,  
but for your project plan. So it looks great as it is right
now. Thanks!

Best,

David