making lenya more data-friendly...

hi everyone!


currently, lenya is heavily document-centric and assumes
"one file per 
webpage".

this paradigm makes it quite hard to integrate database
queries or 
aggregate data from multiple data-centric documents. such
tasks are 
currently done directly on the cocoon level, or with custom
lenya 
resource-types. in either case, you lose the benefits of
lenya and are 
programming on the "bare metal" of web
applications ;)


i would like to propose some new infrastructure to ease the
integration 
and aggregation of data sources:


1. an xslt resource type module.
    it would be nice to handle xslts as documents, edit,
download and
    upload them via the gui, and benefit from revision
control.

2. a "stylesheet" metadata field for each lenya
document.
    it allows to apply a custom stylesheet to a single
document (in
    addition to any stylesheets the resource type may apply
already).

3. an aggregation resouce type module.
    it contains xml that defines which sources should be
queried and
    aggregated, like so:
    <aggregate>
       <part
src="lenyadoc://somerawcontent"/>
       <part
src="cocoon://modules/xhtml/someprocessedcontent"/
>
       <part
src="cocoon://modules/xquery/somedatabasestuff"/&g
t;
    </aggregate>
    this file is then parsed, the sources resolved, the
content
    aggregated and passed to a custom stylesheet as defined
in (2).

4. a couple of data-centric resource type modules as
examples, such as:
    * an xpath-query doctype that allows to filter arbitrary
data out of
      xml data documents.
    * an xquery doctype that talks to xml databases
    * an esql doctype that handles embedded sql queries
    documents of this type would contain a query, and return
the results
    when processed. that way, you can edit and develop your
queries with
    revision control and without breaking the
"live" version while you
    tinker in "authoring".

do you think this might be generally useful?
can you spot problems, or suggest improvements?
your comments are very welcome.


*.*

implementation issues:

some of these points already sit on my disk in 
so-far-pre-alpha-i-don't-have-a number-for-it versions:

(1) is basically the resource doctype, with an additional
xslt rng for 
validation and a special mime-type
"application/xml+xslt" so that xslts 
can be easily found among the other resources.

(2) is a single xconf with a custom metadata set.

(3) is an xsp that does not quite work yet, because some
resource type 
modules don't have matchers for absolute addressing (i begin
to 
understand andreas' feelings about over-use of
{page-envelope}). 
should probably become a custom generator. we need to define
the 
contract of resource type modules with the outside world
more explicitly.
(at first i had this totally brilliant idea to store a
complete sitemap 
with an aggregator pipeline as a lenya document and mount it
dynamically 
into the module sitemap. which of course does not work.
completely 
erratic behaviour due to caching issues. i have written off
the two days 
i spent on that and put myself in detention to write a
hundred times 
"cocoon sitemap is not a programming language!".)

for (4), there is a prototype that talks to an exist-db
instance. it 
works but requires a custom handler on the other end,
because cocoon 
can't talk to plain REST services due to cinclude
limitations... :(


regards,


jörn





-- 
"Open source takes the bullshit out of software."
	- Charles Ferguson on TechnologyReview.com

--
Jörn Nettingsmeier, EDV-Administrator
Institut für Politikwissenschaft
Universität Duisburg-Essen, Standort Duisburg
Mail: pol-adminuni-due.de, Telefon: 0203/379-2736

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribelenya.apache.org
For additional commands, e-mail: dev-helplenya.apache.org

On Tue, 2006-09-26 at 11:35 +0200, Jörn Nettingsmeier
wrote:
> hi everyone!
> 

Hi Jörn,

I am developing what you describe in Forrest and it would be
quite easy
to integrate into Lenya. Have a look at
http://forrest.a
pache.org/pluginDocs/plugins_0_80/org.apache.forrest.plugin.
internal.dispatcher/

salu2

> 
> currently, lenya is heavily document-centric and
assumes "one file per 
> webpage".
> 
> this paradigm makes it quite hard to integrate database
queries or 
> aggregate data from multiple data-centric documents.
such tasks are 
> currently done directly on the cocoon level, or with
custom lenya 
> resource-types. in either case, you lose the benefits
of lenya and are 
> programming on the "bare metal" of web
applications ;)
> 
> 
> i would like to propose some new infrastructure to ease
the integration 
> and aggregation of data sources:
> 
> 
> 1. an xslt resource type module.
>     it would be nice to handle xslts as documents,
edit, download and
>     upload them via the gui, and benefit from revision
control.
> 
> 2. a "stylesheet" metadata field for each
lenya document.
>     it allows to apply a custom stylesheet to a single
document (in
>     addition to any stylesheets the resource type may
apply already).
> 
> 3. an aggregation resouce type module.
>     it contains xml that defines which sources should
be queried and
>     aggregated, like so:
>     <aggregate>
>        <part
src="lenyadoc://somerawcontent"/>
>        <part
src="cocoon://modules/xhtml/someprocessedcontent"/
>
>        <part
src="cocoon://modules/xquery/somedatabasestuff"/&g
t;
>     </aggregate>
>     this file is then parsed, the sources resolved, the
content
>     aggregated and passed to a custom stylesheet as
defined in (2).
> 
> 4. a couple of data-centric resource type modules as
examples, such as:
>     * an xpath-query doctype that allows to filter
arbitrary data out of
>       xml data documents.
>     * an xquery doctype that talks to xml databases
>     * an esql doctype that handles embedded sql queries
>     documents of this type would contain a query, and
return the results
>     when processed. that way, you can edit and develop
your queries with
>     revision control and without breaking the
"live" version while you
>     tinker in "authoring".
> 
> do you think this might be generally useful?
> can you spot problems, or suggest improvements?
> your comments are very welcome.
> 
> 
> *.*
> 
> implementation issues:
> 
> some of these points already sit on my disk in 
> so-far-pre-alpha-i-don't-have-a number-for-it versions:
> 
> (1) is basically the resource doctype, with an
additional xslt rng for 
> validation and a special mime-type
"application/xml+xslt" so that xslts 
> can be easily found among the other resources.
> 
> (2) is a single xconf with a custom metadata set.
> 
> (3) is an xsp that does not quite work yet, because
some resource type 
> modules don't have matchers for absolute addressing (i
begin to 
> understand andreas' feelings about over-use of
{page-envelope}). 
> should probably become a custom generator. we need to
define the 
> contract of resource type modules with the outside
world more explicitly.
> (at first i had this totally brilliant idea to store a
complete sitemap 
> with an aggregator pipeline as a lenya document and
mount it dynamically 
> into the module sitemap. which of course does not work.
completely 
> erratic behaviour due to caching issues. i have written
off the two days 
> i spent on that and put myself in detention to write a
hundred times 
> "cocoon sitemap is not a programming
language!".)
> 
> for (4), there is a prototype that talks to an exist-db
instance. it 
> works but requires a custom handler on the other end,
because cocoon 
> can't talk to plain REST services due to cinclude
limitations... :(
> 
> 
> regards,
> 
> 
> jörn
> 
> 
> 
> 

-- 
Thorsten Scherler
COO Spain
Wyona Inc.  -  Open Source Content Management  -  Apache
Lenya
http://www.wyona.com     
             http://lenya.apache.org
thorsten.scherlerwyona.com                thorstenapache.org


------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribelenya.apache.org
For additional commands, e-mail: dev-helplenya.apache.org

Thorsten Scherler wrote:
> On Tue, 2006-09-26 at 11:35 +0200, Jörn Nettingsmeier
wrote:
>> hi everyone!
>>
> 
> Hi Jörn,
> 
> I am developing what you describe in Forrest and it
would be quite easy
> to integrate into Lenya. Have a look at
> http://forrest.a
pache.org/pluginDocs/plugins_0_80/org.apache.forrest.plugin.
internal.dispatcher/

hmm. it looks very interesting, but unfortunately i know
nothing about 
forrest. i wonder: basically we need resource type modules,
which is a 
lenya-specific concept, right? so we need custom code there.

perhaps we could steal the dispatcher engine for the
proposed aggregator 
doctype (what i currently do with an xsp).

the template-style behaviour looks nice, better than my
"aggregate and 
then sort out the mess in a stylesheet".
can you post an example document that uses this dispatcher
plugin?


aside, what i don't like (for lenya) are the "css
contracts". a generic 
cms simply needs to stay out of the way and let the users do
their own css.
the "naming conventions" stuff does not convince
me. tossing semantics 
into css classes is the wrong way imho. i tried it once, and
it's 
painful in the long run. but that's just my personal
opinion.

-- 
"Open source takes the bullshit out of software."
	- Charles Ferguson on TechnologyReview.com

--
Jörn Nettingsmeier, EDV-Administrator
Institut für Politikwissenschaft
Universität Duisburg-Essen, Standort Duisburg
Mail: pol-adminuni-due.de, Telefon: 0203/379-2736

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribelenya.apache.org
For additional commands, e-mail: dev-helplenya.apache.org

Jörn Nettingsmeier wrote:
> [...]
> 
> 
> aside, what i don't like (for lenya) are the "css
contracts". a generic 
> cms simply needs to stay out of the way and let the
users do their own css.
> the "naming conventions" stuff does not
convince me. tossing semantics 
> into css classes is the wrong way imho. i tried it
once, and it's 
> painful in the long run. but that's just my personal
opinion.
> 

Why don't you upload your css as a resource, then reference
the css in 
your content in the head, then alter your page2xhtml.xsl to
pass 
elements in the head to the result document (if this doesn't
happen 
already, not sure).

--Doug

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribelenya.apache.org
For additional commands, e-mail: dev-helplenya.apache.org

Doug Chestnut wrote:
> 
> 
> Jörn Nettingsmeier wrote:
>> [...]
>>
>>
>> aside, what i don't like (for lenya) are the
"css contracts". a 
>> generic cms simply needs to stay out of the way and
let the users do 
>> their own css.
>> the "naming conventions" stuff does not
convince me. tossing semantics 
>> into css classes is the wrong way imho. i tried it
once, and it's 
>> painful in the long run. but that's just my
personal opinion.
>>
> 
> Why don't you upload your css as a resource, then
reference the css in 
> your content in the head, then alter your
page2xhtml.xsl to pass 
> elements in the head to the result document (if this
doesn't happen 
> already, not sure).

nice idea. i think i'm going to do that in my current
project! thanks!

-- 
"Open source takes the bullshit out of software."
	- Charles Ferguson on TechnologyReview.com

--
Jörn Nettingsmeier, EDV-Administrator
Institut für Politikwissenschaft
Universität Duisburg-Essen, Standort Duisburg
Mail: pol-adminuni-due.de, Telefon: 0203/379-2736

------------------------------------------------------------
---------
To unsubscribe, e-mail: dev-unsubscribelenya.apache.org
For additional commands, e-mail: dev-helplenya.apache.org