Creating a template with search fields

Hello,

 I would like to create an svnwiki page template that has
some fields
whose content can be searched with the svnwiki search
function. For
example, something like the following:


   First Name:  
   Last Name:

   Content:
   Description of your modification:
   ...


The above example is basically the same as the standard
template,
except that it has two additional fields, First Name and
Last
Name. Once the page is created, I want to be able to search
for text
ONLY in those fields. For example, I want to be able to run
a search
for all wiki pages that contain the string "Ivan"
within the field
First Name, instead of the standard full-text search. What
is the best
way to go about doing this? Thanks,

    -Ivan

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Inglés
/ Está bien escribir en español.

Let the repository be with you.

>  I would like to create an svnwiki page template that
has some
>  fields whose content can be searched with the svnwiki
search
>  function. For example, something like the following:
>
>    First Name:  
>    Last Name:
> 
>    Content:
>    Description of your modification:
>    ...
> 
> The above example is basically the same as the standard
template,
> except that it has two additional fields, First Name
and Last
> Name. Once the page is created, I want to be able to
search for text
> ONLY in those fields. For example, I want to be able to
run a search
> for all wiki pages that contain the string
"Ivan" within the field
> First Name, instead of the standard full-text search.
What is the best
> way to go about doing this?

Hmm.

I suppose there are two ways to do it:

1. Do exactly what you described.  Create an Svnwiki
extension that:

   1. defines a tag such as <personData/>, which gets
expanded to
   “First Name: $firstname, Last name: $lastname”, where
the
   $firstname and $lastname are the values of a Subversion
property in
   the file.

   2. (optional) Also have the extension define two fields
that get
   included in the edit form for the file and then, when the
file is
   saved, receives the input and sets the properties.

   3. Defines fields to show in the search form and filters
the
   results based on those fields.

   Right now we'd need to modify Svnwiki to make it possible
to do #2
   and #3 and to add something to the documentation for how
you'd do
   #1, but I would be willing to do this if you decide to
follow this
   route.

2. Do something different to what you described, but which
may be
   easier to implement and work as well or even better for
what you
   want.  If you want to explore this, you should probably
describe
   what you need better to us so we can find other ways in
which
   existing Svnwiki functionality may match what you want.


   For example, perhaps using a lastname/firstname/page-name
structure
   for the contents of your wiki will suffice?  Or, even if
it
   doesn't, perhaps it's easy to just extend this idea
somewhat,
   requiring less work than #1 would require.  

Cheers,

Alejo.
http://azul.freaks-uni
dos.net/

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Ingl&amp;#233;s
/ Est&amp;#225; bien escribir en espa&amp;#241;ol.

Let the repository be with you.

Hi Alejo,

  I am of course willing to contribute effort towards
extending the
search form, if you tell me where I need to look to figure
out how to
extend the code.

  Here is what I am trying to do: in the research lab where
I work, we
have a bunch of experimental data (lots of very large files
sitting in
a number of directories) on a server that is accessible by
all members
of the lab. We want to have some meta-data associated with
each set of
files, that includes fixed pre-determined fields, such as
author,
date, animal species, chemicals used, etc., as well as some
unstructured information, such as email discussions, etc.,
that would
be included in a wiki form. So basically each file or a set
of files
gets its own wiki page and a small number of database-like
information
fields.

  When a user searches the wiki pages, I want them to be
able to
search _either_ the fixed fields, _or_ the free-form wiki
content. So
a search could be "AnimalSpecies: Aplysia", which
would search only
the AnimalSpecies field, or it could be just
"Aplysia", which would
search the entire content of each wiki page for that string.

  Your solution #2 might work, but if I understand
correctly, the
search would still be a full-text search, no? I can think of
more
detailed examples, if necessary. 


         -Ivan


Alejandro Forero Cuervo <azulfreaks-unidos.net>
writes:

> I suppose there are two ways to do it:
>
> 1. Do exactly what you described.  Create an Svnwiki
extension that:
>
>    1. defines a tag such as <personData/>, which
gets expanded to
>    “First Name: $firstname, Last name: $lastname”,
where the
>    $firstname and $lastname are the values of a
Subversion property in
>    the file.
>
>    2. (optional) Also have the extension define two
fields that get
>    included in the edit form for the file and then,
when the file is
>    saved, receives the input and sets the properties.
>
>    3. Defines fields to show in the search form and
filters the
>    results based on those fields.
>
>    Right now we'd need to modify Svnwiki to make it
possible to do #2
>    and #3 and to add something to the documentation for
how you'd do
>    #1, but I would be willing to do this if you decide
to follow this
>    route.
>
> 2. Do something different to what you described, but
which may be
>    easier to implement and work as well or even better
for what you
>    want.  If you want to explore this, you should
probably describe
>    what you need better to us so we can find other ways
in which
>    existing Svnwiki functionality may match what you
want. 
>
>    For example, perhaps using a
lastname/firstname/page-name structure
>    for the contents of your wiki will suffice?  Or,
even if it
>    doesn't, perhaps it's easy to just extend this idea
somewhat,
>    requiring less work than #1 would require.  
>
> Cheers,
>
> Alejo.
> http://azul.freaks-uni
dos.net/

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Ingl&amp;#233;s
/ Est&amp;#225; bien escribir en espa&amp;#241;ol.

Let the repository be with you.

>   I am of course willing to contribute effort towards
extending the
> search form, if you tell me where I need to look to
figure out how to
> extend the code.

Well, all the searching is done in src/estraier.scm.  Near
the bottom
is the estraier-search-form procedure, that returns the HTML
for the
search form.  Searches are handled by the estraier-search
procedure,
which opens the estraier database, sets the condition and
returns a
stream with one snippet for each search result.  I suspect
you'd have
to tweak those, but I don't know if the Estraier API
supports what we
need to do.  If it doesn't, it would be messy: I guess you'd
have to
add a filter (and probably a function that decodes the query
string to
see what criteria to filter on and make sure that criteria
doesn't get
passed to Estraier's est-cond-set-phrase).

> Here is what I am trying to do: in the research lab
where I work, we
> have a bunch of experimental data (lots of very large
files sitting
> in a number of directories) on a server that is
accessible by all
> members of the lab. We want to have some meta-data
associated with
> each set of files, that includes fixed pre-determined
fields, such
> as author, date, animal species, chemicals used, etc.,
as well as
> some unstructured information, such as email
discussions, etc., that
> would be included in a wiki form. So basically each
file or a set of
> files gets its own wiki page and a small number of
database-like
> information fields.

Another idea that just occurred to me today while I was
showering was:
how about doing this with tags?  You could maybe tag pages
like
'firstnameIvan' and 'lastnameRaikov' (or, even simpler, just
'ivan'
and 'raikov' or 'ivanraikov'?) and then just use tags to get
to them?

Here are some examples of the current functionality we have
for tags:

- Showing related tags in a page: see the bottom of:
  http://wiki.freaks-unidos.net/weblogs/azul/firefox-
ssh-tunnel
  Click 'sysadmin'

- Files in a tag:
  htt
p://wiki.freaks-unidos.net/xsvnwiki-tags/linux
  Don't forget to tlick 'Show related tags'. 

What do you think?  Do you think this would work for what
you want?
If not, do you think this idea could be improved to work for
what you
want?

Incidentally, I was improving the functionality for tagging
last
weekend (haven't yet commited my changes).

Regardless of what you decide to do, I think we should make
it
possible to search within a given tag...  Hmm.

> Your solution #2 might work, but if I understand
correctly, the
> search would still be a full-text search, no? I can
think of more
> detailed examples, if necessary. 

Yes, the search would still be a full text search.

Alejo.
http://azul.freaks-uni
dos.net/

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Inglés
/ Está bien escribir en español.

Let the repository be with you.

  Okay, I will look through the Hyper Estraier API
documentation and
see if I can figure out how the search interface should be
done. As
for tags, what are the limitations on their names? Could you
have tags
with phrases such as name:Ivan+Raikov? And would you be able
to
actually search for all tags that match a certain
substring?

  One idea I had was perhaps to have a Keywords page
associated with
each page in the wiki, just like there is a Discussion page.
Then
users could something like the following in the keywords
page:

FirstName: Ivan
LastName: Raikov

Would it be possible to make Hyper Estraier to search for
all lines
prefixed with FirstName: (for example), and then to search
within
those search results? The downside would be that once the
keywords
page is found, the user would have to click one more time to
get to
the actual page, but perhaps this could be solved with a
simple
redirect. Thanks for all your help with this,

      -Ivan


Alejandro Forero Cuervo <azulfreaks-unidos.net>
writes:

> Well, all the searching is done in src/estraier.scm. 
Near the bottom
> is the estraier-search-form procedure, that returns the
HTML for the
> search form.  Searches are handled by the
estraier-search procedure,
> which opens the estraier database, sets the condition
and returns a
> stream with one snippet for each search result.  I
suspect you'd have
> to tweak those, but I don't know if the Estraier API
supports what we
> need to do.  If it doesn't, it would be messy: I guess
you'd have to
> add a filter (and probably a function that decodes the
query string to
> see what criteria to filter on and make sure that
criteria doesn't get
> passed to Estraier's est-cond-set-phrase).
>
> Another idea that just occurred to me today while I was
showering was:
> how about doing this with tags?  You could maybe tag
pages like
> 'firstnameIvan' and 'lastnameRaikov' (or, even simpler,
just 'ivan'
> and 'raikov' or 'ivanraikov'?) and then just use tags
to get to them?
>
> Here are some examples of the current functionality we
have for tags:
>
> - Showing related tags in a page: see the bottom of:
>   http://wiki.freaks-unidos.net/weblogs/azul/firefox-
ssh-tunnel
>   Click 'sysadmin'
>
> - Files in a tag:
>   htt
p://wiki.freaks-unidos.net/xsvnwiki-tags/linux
>   Don't forget to tlick 'Show related tags'. 
>
> What do you think?  Do you think this would work for
what you want?
> If not, do you think this idea could be improved to
work for what you
> want?

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Ingl&amp;#233;s
/ Est&amp;#225; bien escribir en espa&amp;#241;ol.

Let the repository be with you.

> Okay, I will look through the Hyper Estraier API
documentation and
> see if I can figure out how the search interface should
be done.

Excellent!

> As for tags, what are the limitations on their names?
Could you have
> tags with phrases such as name:Ivan+Raikov?

Well, the algorithm for tags will split the list of tags in
spaces and
then, for every tag, will remove non us-ascii characters and
lowercase
things.  So you'd end up with nameivanraikov.  We thought
this would
make the system easier to use.

> And would you be able to actually search for all tags
that match a
> certain substring?

Hmm, not right now but I guess this really is something I
should
implement: registering the tags in the eastraier DB to make
them
searchable.

> One idea I had was perhaps to have a Keywords page
associated with
> each page in the wiki, just like there is a Discussion
page. Then
> users could something like the following in the
keywords page:
> 
> FirstName: Ivan
> LastName: Raikov
> 
> Would it be possible to make Hyper Estraier to search
for all lines
> prefixed with FirstName: (for example), and then to
search within
> those search results?  The downside would be that once
the keywords
> page is found, the user would have to click one more
time to get to
> the actual page, but perhaps this could be solved with
a simple
> redirect.

Hmm, I guess this would work.  I don't like the fact that
you'd be
encoding all this metadata as another file's contents.  In
this case
I'd probably recommend that the properties be encoded as
specific
Subversion properties (and then we extend our search
implementation to
make it possible to use that).

BTW, here is one video that I watched a few days ago which I
just
realized that you could be potentially interested on
watching:

  http://video.google.com/videoplay?docid=2159021324062
223592

Thanks.

Alejo.
http://azul.freaks-uni
dos.net/

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Inglés
/ Está bien escribir en español.

Let the repository be with you.

   Well, I looked at the Hyper Estraier documentation, and I
think that
if each wiki page has an associated keywords page (with an
attribute
like svnwiki:keywords-page), then one could use the regular
expression search feature of Hyper Estraier to search the
contents of
a particular keyword field in all keywords pages. What do
you think
about the following code:


(define (estraier-kw-search-form env)
  (if *have-estraier*
    (let-from-environment env (path-in path)
      (html-stream
        ((div id "search")
         ((form action (get-props-parents-first
"svnwiki:application-url" path-in path
"") method "get")
          ((input type "hidden" name
"action" value "search"))
	  ((label for "keyword-field") "Keyword:
") 
          ((input type "text" name "kw"
id "keyword-field"))
	  ((label for "search-field") "Value:
") 
          ((input type "text" name "q"
id "search-field"))
          (br)
          ((input type "submit" value
"Search" id "search-button"))))))
    stream-null))


(define (estraier-kw-search env)
  (if *have-estraier*
    (let-from-environment env (data user-input)
      (let ((condition (est-cond-new)))
        ;; only search pages with attribute
keywords-page=yes
	(est-cond-add-attr condition  "keywords-page STREQ
yes") 
	(est-cond-set-phrase condition (string-append 
					"[RX] " ;; regex search
					"^" (stream->string (user-input 'kw
stream-null))
					":[ t]*" (stream->string (user-input 'q
stream-null))))
        (let* ((db (est-db-open (svnwiki-make-pathname data
"estraier-db") 0))
               (hints (cbmapopenex 32))
               (results (vector->stream (est-db-search db
condition hints)))
               (words (est-hints-to-words hints)))
          (stream-map
            (let* ((encoding (svnwiki-encoding-for-file
env))
                   (conv-func
                     (if (string=? encoding
"utf-8")
                       identity
                       (let ((conv (iconv-open encoding
"utf-8")))
                         (cut iconv conv <>)))))
              (lambda (id)
                (let ((doc (est-db-get-doc db id 0)))
                  (list
                    (est-doc-attr doc "uri")
                    (est-doc-attr doc "title")
                    (conv-func (est-doc-make-snippet doc
words 480 96 96))))))
            results))))
    stream-null))

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Ingl&amp;#233;s
/ Est&amp;#225; bien escribir en espa&amp;#241;ol.

Let the repository be with you.

  What is the disadvantage of putting the metadata in a
file? I
thought that was an easy way to go around Hyper Estraier's
limitation
of full-text search only. My problem with encoding the
metadata as
Subversion properties is that the search routines would have
to be
modified every time a new property is added. But perhaps I
do not
understand all the complexities involved in adding a
keywords page. I
will watch the video, thanks.

     -Ivan
  

Alejandro Forero Cuervo <azulfreaks-unidos.net>
writes:

>
> Hmm, I guess this would work.  I don't like the fact
that you'd be
> encoding all this metadata as another file's contents. 
In this case
> I'd probably recommend that the properties be encoded
as specific
> Subversion properties (and then we extend our search
implementation to
> make it possible to use that).
>
> BTW, here is one video that I watched a few days ago
which I just
> realized that you could be potentially interested on
watching:
>
>   http://video.google.com/videoplay?docid=2159021324062
223592
>

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Ingl&amp;#233;s
/ Est&amp;#225; bien escribir en espa&amp;#241;ol.

Let the repository be with you.

> What is the disadvantage of putting the metadata in a
file? I
> thought that was an easy way to go around Hyper
Estraier's
> limitation of full-text search only.

Well, I find it easier to work with programatically.  For
example, you
can very easily set a property on a large number of files
from the
shell or get the list of files with a certain property.  Of
course,
you can also do that if you store the data in a 'keywords'
file (or a
'keywords-$FILE' file), but not so easily, I'd think.

> My problem with encoding the metadata as Subversion
properties is
> that the search routines would have to be modified
every time a new
> property is added.

I would think the code would look at one given property,
say
'svnwiki:search:properties', with a list of the names of
the
properties (eg. 'firstname lastname currentstate').  It
would then use
Subversion properties with those names.

Alejo.
http://azul.freaks-uni
dos.net/

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Inglés
/ Está bien escribir en español.

Let the repository be with you.

> Good idea, and what you say about setting a property on
multiple
> files makes sense.  I will then create a version of the
search form
> that looks at properties instead of keywords pages.

I took a look at the 'attributes' concept in Estraier
earlier this
morning and it seemed to do precissely what we need.  I
think it will
allow us to very cleanly implement what you need.  So I
believe these
are our options:

1. We register the tags (from svnwiki:tags + [[tags:...]])
as an
"svnwiki:tags" Estraier attribute for the files.

2. We register, for each wiki file, one Estraier attribute
for each
Subversion property set on it from a list of properties in
the value
of the Subversion property 'svnwiki:search-attributes' (so
you'd set
'svnwiki:search-attributes' to something like 'firstname
lastname
country' in the root of the wiki).

Both would suffice to allow users to search by
tags/keywords.  I would
recommend #1 as I've had a very good experience with
folksonomies /
colaborative tagging in general but I will probably end up
doing both
of the above, so it is entirely up to you to decide what to
use. 

The only drawback I can think of is that the search that end
users
would have to enter to search by tags/keywords could be
more
intuitive, but I don't think we should attempt to
parse/decode the
search queries instead of passing them directly to
Estraier.

Alejo.
http://azul.freaks-uni
dos.net/

_______________________________________________
Svnwiki mailing list
Svnwikilistas.el-directorio.org
http://listas.el-directorio.org/cgi-bin/mailman/
listinfo/svnwiki

* We prefer english for this list / it's ok to write in
Spanish.
* El idioma preferido en la lista es el Inglés
/ Está bien escribir en español.

Let the repository be with you.