On Feb 10, 2006, at 11:20 AM, John Anderson wrote:
>
>
> Mikeal Rogers wrote:
>>
>> On Feb 10, 2006, at 10:04 AM, Philippe Bossut
wrote:
>>
>>>
>>>
>>> John Anderson wrote:
>>>> Having written my first functional test
yesterday I have some
>>>> thoughts. The biggest problem I encountered
when trying to write
>>>> and debug tests is navigating all the
layers:
>>>>
>>>> my test <-> CATS <-> CPIA
Script <-> Chandler
>>>>
>>>> Fortunately I'm very familiar with
Chandler, somewhat familiar
>>>> with CPIA Script and CATS is small enough
to grock without much
>>>> effort. However, I suspect most developers
would find all the
>>>> layers daunting and trying to debug things
would only make it
>>>> worse.
>>> Agree with that.
>>
>> One of the requirements is that the system be easy
to use.
>> Obviously there is another layer of complexity over
what we do
>> with CATS but it is still designed to be very easy
for someone to
>> pick up and start writing scripts and to see
legible output. Part
>> of the deliverables for the first version of this
framework will be;
>>
>> -Command line python wrapper (much like do_tests, a
script is
>> imported and output is generated that is legible
using a set of
>> default parameters for the framework)
> Yes, we've all been using variation of this for unit
tests and
> functional tests and find it useful. Also, when you get
a
> particularly tricky functional test failing somewhere
deep in
> Chandler, were a traceback isn't enough to diagnose the
problem,
> it's often handy to track it down in a debugger. So you
might set
> things up so you can attach in wing, e.g. include
wingdbstub.py.
This is probably a 2.0 feature. But, this framework has many
advantages for debugging test cases.
For example: Although the framework does all the reporting
for you,
it uses methods in the TestCase class which you inherit from
to
define the class for your own test. This means that in
debugging you
could write custom code and push additional reporting into
the output
object using same method the framework uses. The current
CATS design
requires you write custom code in the Logger object to do
this, and
the logger object has a bunch of logic to do fuzzy guessing
as to
which state the test is in. The idea is to have the
framework do most
of the heavy lifting for you but not keep you in such a
tight box
that you can't extend it without hacking at the framework
itself.
>> -Sufficient Documentation ("Writting Chandler
automation in 10
>> minutes" style doc, extended OAF documentation
for developers who
>> wish to use non-default features in the system, and
maybe most
>> importantly GOOD documentation for the chandler
test library that
>> can facilitate both easy test script authoring and
developer
>> improvements to the chandler testing library
itself.
> I think much of what makes writing functional tests
difficult has
> little to do with your proposed framework, and more to
do with how
> you access the pieces of Chandler, do menu commands,
click on
> buttons, etc., i.e. the stuff that is mostly in CPIA
script and CPIA.
This is an issue that will be handled outside the framework.
Again,
the _framework_ is not Chandler specific.
All of the QAUITestAppLib will need to be re-written to work
with
OAF, during this rewrite we will be taking care of many of
the
current issues developers have with it. I don't know if we
can
replace CPIA, after the framework is signed off on my next
task is to
come up with a plan to overhaul QAUITestAppLib.
At the very least we intend to make the new library more
modular and
reuse code as much as possible. Making it easier for
developers to
alter/extend the functionality inside of inherited objects
in the
test scripts is a priority, as is making it easier for
developers and
QA to extend the library to cover new functionality in
Chandler.
The reason for all of the layers in the design is to make
the
framework non-application specific and extendable. The
implementation
will make these layers mostly transparent to the user. In
the same
way we can layer the new Chandler QAUITestAppLib to be
non-CPIA
specific in case you plan on implementing something better
in the
future.
>>
>> The output can be very customized using this
framework, but the
>> default output will be humanly legible and go
directly to a file.
>>
>> Also, a -debug flag can be set, which sets all
output in the
>> framework to be processed as it comes in to the
output object.
>> This is no good for performance tests but will make
debugging
>> issues worlds easier than in CATS.
>>
>> To finish up, many of the extra layers that
developer might find
>> "daunting" will be transparent in the
implementation, but the
>> output that developers depend on (such as a
tracebacks in the log
>> if a failure occures) are made easy and reliable by
this abstraction.
>>
>> I hope this alleviates your concerns.
>>
>> -Mikeal
>>
>>>> I think it would be preferable to make the
small changes
>>>> necessary to CPIA Script to make it
appropriate for testing
>>>> instead of adding another layer, e.g. CATS.
>>> Improving CPIA Script to make scripting easier
is indeed a good
>>> idea. I don't think it will replace entirely a
test harness
>>> though like CATS or, better, OAF (proposed by
Mikeal). There's a
>>> lot of test functions (batch, log, data
gathering and stats) that
>>> have no place in a Chandler level scripting
language. John, I
>>> suggest you read Mikeal proposal (http://wiki.osafoundat
ion.org/
>>> bin/view/Projects/OpenAutomationFramework)
first. Keep in mind
>>> also that Mikeal is trying to solve a problem
that includes
>>> Chandler and Cosmo.
>>>> Similarly, I think it's preferable to
modify Chandler to
>>>> eliminate some of CPIA Script.
>>> What alternative to CPIA scripting do you
propose? No scripting
>>> at all? Another script mechanism? Leverage an
existing one?
>>>
>>> Cheers,
>>> - Philippe
>>>
>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>>
>>> Open Source Applications Foundation
"Dev" mailing list
>>> h
ttp://lists.osafoundation.org/mailman/listinfo/dev
>>
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>
>> Open Source Applications Foundation "Dev"
mailing list
>> h
ttp://lists.osafoundation.org/mailman/listinfo/dev
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev"
mailing list
> h
ttp://lists.osafoundation.org/mailman/listinfo/dev
_______________________________________________
Cosmo mailing list
Cosmo osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo
|