Proposal for new automation architecture

Philippe Bossut wrote:
>
>
> John Anderson wrote:
>> Having written my first functional test yesterday I
have some 
>> thoughts. The biggest problem I encountered when
trying to write and 
>> debug tests is navigating all the layers:
>>
>> my test <-> CATS <-> CPIA Script
<-> Chandler
>>
>> Fortunately I'm very familiar with Chandler,
somewhat familiar with 
>> CPIA Script and CATS is small enough to grock
without much effort. 
>> However, I suspect most developers would find all
the layers daunting 
>> and trying to debug things would only make it
worse. 
> Agree with that.
>> I think it would be preferable to make the small
changes necessary to 
>> CPIA Script to make it appropriate for testing
instead of adding 
>> another layer, e.g. CATS.
> Improving CPIA Script to make scripting easier is
indeed a good idea. 
> I don't think it will replace entirely a test harness
though like CATS 
> or, better, OAF (proposed by Mikeal). There's a lot of
test functions 
> (batch, log, data gathering and stats) that have no
place in a 
> Chandler level scripting language. John, I suggest you
read Mikeal 
> proposal 
> (http://wiki.osafoundation.org/bin/view/Pr
ojects/OpenAutomationFramework) 
> first. Keep in mind also that Mikeal is trying to solve
a problem that 
> includes Chandler and Cosmo.
>> Similarly, I think it's preferable to modify
Chandler to eliminate 
>> some of CPIA Script.
> What alternative to CPIA scripting do you propose? No
scripting at 
> all? Another script mechanism? Leverage an existing
one?
I may have left you with some incorrect impressions. I read
Mikeal's 
proposal, and agree it's addressing areas like batch, log,
data 
gathering and stats, that have no place in Chandler
scripting. As a 
writer of a functional test, however, I'm hoping that I
don't need to 
know much about all that just to add another functional test
to the 
build cycle. I'm more concerned about how hard it's going to
be for 
people to write and debug new functional tests. This is an
area I think 
we could make some improvements, which would lead to better
tests, 
quicker release cycles, and higher quality software.

I'm not proposing an alternative to CPIA script, just
improvements to 
it. For example, with some minor changes to the way blocks
and events 
are named and looked up we could probably eliminate much of
the 
differences in how you access these Items in CPIA script and
CPIA. Since 
CPIA and CPIA script are both just Python running in
Chandler, there 
should be much that is in common.

>
> Cheers,
> - Philippe
>
_______________________________________________
Cosmo mailing list
Cosmoosafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo

Mikeal Rogers wrote:
>
> On Feb 10, 2006, at 10:04 AM, Philippe Bossut wrote:
>
>>
>>
>> John Anderson wrote:
>>> Having written my first functional test
yesterday I have some 
>>> thoughts. The biggest problem I encountered
when trying to write and 
>>> debug tests is navigating all the layers:
>>>
>>> my test <-> CATS <-> CPIA Script
<-> Chandler
>>>
>>> Fortunately I'm very familiar with Chandler,
somewhat familiar with 
>>> CPIA Script and CATS is small enough to grock
without much effort. 
>>> However, I suspect most developers would find
all the layers 
>>> daunting and trying to debug things would only
make it worse.
>> Agree with that.
>
> One of the requirements is that the system be easy to
use. Obviously 
> there is another layer of complexity over what we do
with CATS but it 
> is still designed to be very easy for someone to pick
up and start 
> writing scripts and to see legible output. Part of the
deliverables 
> for the first version of this framework will be;
>
> -Command line python wrapper (much like do_tests, a
script is imported 
> and output is generated that is legible using a set of
default 
> parameters for the framework)
Yes, we've all been using variation of this for unit tests
and 
functional tests and find it useful. Also, when you get a
particularly 
tricky functional test failing somewhere deep in Chandler,
were a 
traceback isn't enough to diagnose the problem, it's often
handy to 
track it down in a debugger. So you might set things up so
you can 
attach in wing, e.g. include wingdbstub.py.
> -Sufficient Documentation ("Writting Chandler
automation in 10 
> minutes" style doc, extended OAF documentation for
developers who wish 
> to use non-default features in the system, and maybe
most importantly 
> GOOD documentation for the chandler test library that
can facilitate 
> both easy test script authoring and developer
improvements to the 
> chandler testing library itself.
I think much of what makes writing functional tests
difficult has little 
to do with your proposed framework, and more to do with how
you access 
the pieces of Chandler, do menu commands, click on buttons,
etc., i.e. 
the stuff that is mostly in CPIA script and CPIA.
>
> The output can be very customized using this framework,
but the 
> default output will be humanly legible and go directly
to a file.
>
> Also, a -debug flag can be set, which sets all output
in the framework 
> to be processed as it comes in to the output object.
This is no good 
> for performance tests but will make debugging issues
worlds easier 
> than in CATS.
>
> To finish up, many of the extra layers that developer
might find 
> "daunting" will be transparent in the
implementation, but the output 
> that developers depend on (such as a tracebacks in the
log if a 
> failure occures) are made easy and reliable by this
abstraction.
>
> I hope this alleviates your concerns.
>
> -Mikeal
>
>>> I think it would be preferable to make the
small changes necessary 
>>> to CPIA Script to make it appropriate for
testing instead of adding 
>>> another layer, e.g. CATS.
>> Improving CPIA Script to make scripting easier is
indeed a good idea. 
>> I don't think it will replace entirely a test
harness though like 
>> CATS or, better, OAF (proposed by Mikeal). There's
a lot of test 
>> functions (batch, log, data gathering and stats)
that have no place 
>> in a Chandler level scripting language. John, I
suggest you read 
>> Mikeal proposal 
>> (http://wiki.osafoundation.org/bin/view/Pr
ojects/OpenAutomationFramework) 
>> first. Keep in mind also that Mikeal is trying to
solve a problem 
>> that includes Chandler and Cosmo.
>>> Similarly, I think it's preferable to modify
Chandler to eliminate 
>>> some of CPIA Script.
>> What alternative to CPIA scripting do you propose?
No scripting at 
>> all? Another script mechanism? Leverage an existing
one?
>>
>> Cheers,
>> - Philippe
>>
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>
>> Open Source Applications Foundation "Dev"
mailing list
>> h
ttp://lists.osafoundation.org/mailman/listinfo/dev
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev"
mailing list
> h
ttp://lists.osafoundation.org/mailman/listinfo/dev
_______________________________________________
Cosmo mailing list
Cosmoosafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo