QA Test Scripts

Aparna, Adam, Mikeal et al,

I found a bug before that was so glaring that I was
convinced it  
must've not been in the test scripts. I have since realized
that it's  
due to a weird interaction of things in my local environment
so I  
never bugged you guys about it.

But the general question remains, what is the best way to
ensure that  
certain scenarios, features, make it into the QA Test
scripts? Are  
the scripts publicly available somewhere? Can I edit them or
at least  
add a comment? Or should file a bug? Or ________?


Thanks,

Bobby
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

Since I am writing the initial Cosmo QA Test scripts in our
new  
framework this week.. probably all of the methods you stated
will  
find their way to my lap so I can incorporate the
functionality.  
After the first round of them is written they will be
committed to  
the Cosmo repo so you can go add/edit them etc. I will send
out an  
email with some direction as soon as that is ready. The hope
is to  
have a large chunk of the UI automation done and ready for
everyone  
to use by 0.6.1.

Adam


On Jan 30, 2007, at 3:29 PM, Bobby Rullo wrote:

> Aparna, Adam, Mikeal et al,
>
> I found a bug before that was so glaring that I was
convinced it  
> must've not been in the test scripts. I have since
realized that  
> it's due to a weird interaction of things in my local
environment  
> so I never bugged you guys about it.
>
> But the general question remains, what is the best way
to ensure  
> that certain scenarios, features, make it into the QA
Test scripts?  
> Are the scripts publicly available somewhere? Can I
edit them or at  
> least add a comment? Or should file a bug? Or
________?
>
>
> Thanks,
>
> Bobby
> _______________________________________________
> cosmo-dev mailing list
> cosmo-devlists.osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

What about non-automatable stuff?

Like "verify feature x doesn't look wack when doing
___"

Bobby


On Jan 30, 2007, at 3:37 PM, Adam Christian wrote:

> Since I am writing the initial Cosmo QA Test scripts in
our new  
> framework this week.. probably all of the methods you
stated will  
> find their way to my lap so I can incorporate the
functionality.  
> After the first round of them is written they will be
committed to  
> the Cosmo repo so you can go add/edit them etc. I will
send out an  
> email with some direction as soon as that is ready. The
hope is to  
> have a large chunk of the UI automation done and ready
for everyone  
> to use by 0.6.1.
>
> Adam
>
>
> On Jan 30, 2007, at 3:29 PM, Bobby Rullo wrote:
>
>> Aparna, Adam, Mikeal et al,
>>
>> I found a bug before that was so glaring that I was
convinced it  
>> must've not been in the test scripts. I have since
realized that  
>> it's due to a weird interaction of things in my
local environment  
>> so I never bugged you guys about it.
>>
>> But the general question remains, what is the best
way to ensure  
>> that certain scenarios, features, make it into the
QA Test  
>> scripts? Are the scripts publicly available
somewhere? Can I edit  
>> them or at least add a comment? Or should file a
bug? Or ________?
>>
>>
>> Thanks,
>>
>> Bobby
>> _______________________________________________
>> cosmo-dev mailing list
>> cosmo-devlists.osafoundation.org
>> http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev
>

_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

Probably the best thing for manual testing is to get a
checklist  
going on a wiki page so people can add those details.

Mikeal, Aparna?

Adam

On Jan 30, 2007, at 3:48 PM, Bobby Rullo wrote:

> What about non-automatable stuff?
>
> Like "verify feature x doesn't look wack when
doing ___"
>
> Bobby
>
>
> On Jan 30, 2007, at 3:37 PM, Adam Christian wrote:
>
>> Since I am writing the initial Cosmo QA Test
scripts in our new  
>> framework this week.. probably all of the methods
you stated will  
>> find their way to my lap so I can incorporate the
functionality.  
>> After the first round of them is written they will
be committed to  
>> the Cosmo repo so you can go add/edit them etc. I
will send out an  
>> email with some direction as soon as that is ready.
The hope is to  
>> have a large chunk of the UI automation done and
ready for  
>> everyone to use by 0.6.1.
>>
>> Adam
>>
>>
>> On Jan 30, 2007, at 3:29 PM, Bobby Rullo wrote:
>>
>>> Aparna, Adam, Mikeal et al,
>>>
>>> I found a bug before that was so glaring that I
was convinced it  
>>> must've not been in the test scripts. I have
since realized that  
>>> it's due to a weird interaction of things in my
local environment  
>>> so I never bugged you guys about it.
>>>
>>> But the general question remains, what is the
best way to ensure  
>>> that certain scenarios, features, make it into
the QA Test  
>>> scripts? Are the scripts publicly available
somewhere? Can I edit  
>>> them or at least add a comment? Or should file
a bug? Or ________?
>>>
>>>
>>> Thanks,
>>>
>>> Bobby
>>>
_______________________________________________
>>> cosmo-dev mailing list
>>> cosmo-devlists.osafoundation.org
>>> http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev
>>
>

_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

Since I am writing the initial Cosmo QA Test scripts in our
new  
framework this week.. probably all of the methods you stated
will  
find their way to my lap so I can incorporate the
functionality.  
After the first round of them is written they will be
committed to  
the Cosmo repo so you can go add/edit them etc. I will send
out an  
email with some direction as soon as that is ready. The hope
is to  
have a large chunk of the UI automation done and ready for
everyone  
to use by 0.6.1.

Adam


On Jan 30, 2007, at 3:29 PM, Bobby Rullo wrote:

> Aparna, Adam, Mikeal et al,
>
> I found a bug before that was so glaring that I was
convinced it  
> must've not been in the test scripts. I have since
realized that  
> it's due to a weird interaction of things in my local
environment  
> so I never bugged you guys about it.
>
> But the general question remains, what is the best way
to ensure  
> that certain scenarios, features, make it into the QA
Test scripts?  
> Are the scripts publicly available somewhere? Can I
edit them or at  
> least add a comment? Or should file a bug? Or
________?
>
>
> Thanks,
>
> Bobby
> _______________________________________________
> cosmo-dev mailing list
> cosmo-devlists.osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

What about non-automatable stuff?

Like "verify feature x doesn't look wack when doing
___"

Bobby


On Jan 30, 2007, at 3:37 PM, Adam Christian wrote:

> Since I am writing the initial Cosmo QA Test scripts in
our new  
> framework this week.. probably all of the methods you
stated will  
> find their way to my lap so I can incorporate the
functionality.  
> After the first round of them is written they will be
committed to  
> the Cosmo repo so you can go add/edit them etc. I will
send out an  
> email with some direction as soon as that is ready. The
hope is to  
> have a large chunk of the UI automation done and ready
for everyone  
> to use by 0.6.1.
>
> Adam
>
>
> On Jan 30, 2007, at 3:29 PM, Bobby Rullo wrote:
>
>> Aparna, Adam, Mikeal et al,
>>
>> I found a bug before that was so glaring that I was
convinced it  
>> must've not been in the test scripts. I have since
realized that  
>> it's due to a weird interaction of things in my
local environment  
>> so I never bugged you guys about it.
>>
>> But the general question remains, what is the best
way to ensure  
>> that certain scenarios, features, make it into the
QA Test  
>> scripts? Are the scripts publicly available
somewhere? Can I edit  
>> them or at least add a comment? Or should file a
bug? Or ________?
>>
>>
>> Thanks,
>>
>> Bobby
>> _______________________________________________
>> cosmo-dev mailing list
>> cosmo-devlists.osafoundation.org
>> http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev
>

_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

Probably the best thing for manual testing is to get a
checklist  
going on a wiki page so people can add those details.

Mikeal, Aparna?

Adam

On Jan 30, 2007, at 3:48 PM, Bobby Rullo wrote:

> What about non-automatable stuff?
>
> Like "verify feature x doesn't look wack when
doing ___"
>
> Bobby
>
>
> On Jan 30, 2007, at 3:37 PM, Adam Christian wrote:
>
>> Since I am writing the initial Cosmo QA Test
scripts in our new  
>> framework this week.. probably all of the methods
you stated will  
>> find their way to my lap so I can incorporate the
functionality.  
>> After the first round of them is written they will
be committed to  
>> the Cosmo repo so you can go add/edit them etc. I
will send out an  
>> email with some direction as soon as that is ready.
The hope is to  
>> have a large chunk of the UI automation done and
ready for  
>> everyone to use by 0.6.1.
>>
>> Adam
>>
>>
>> On Jan 30, 2007, at 3:29 PM, Bobby Rullo wrote:
>>
>>> Aparna, Adam, Mikeal et al,
>>>
>>> I found a bug before that was so glaring that I
was convinced it  
>>> must've not been in the test scripts. I have
since realized that  
>>> it's due to a weird interaction of things in my
local environment  
>>> so I never bugged you guys about it.
>>>
>>> But the general question remains, what is the
best way to ensure  
>>> that certain scenarios, features, make it into
the QA Test  
>>> scripts? Are the scripts publicly available
somewhere? Can I edit  
>>> them or at least add a comment? Or should file
a bug? Or ________?
>>>
>>>
>>> Thanks,
>>>
>>> Bobby
>>>
_______________________________________________
>>> cosmo-dev mailing list
>>> cosmo-devlists.osafoundation.org
>>> http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev
>>
>

_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

All the non-UI test cases are here;

http://svn.osafoundation.org/server/cosmo/tr
unk/src/test/functional/ 
protocol/

You need to install py.test, after doing that you just run
the  
py.test command line utility and they auto-magically all
run.

The UI test cases are being automated with windmill and
should be  
checked in this week or next week here;

http://svn.osafoundation.org/server/cosmo/tr
unk/src/test/functional/ 
windmill/

There shouldn't be anything in the UI that we can't
automate,  
although it's doubtful that we would ever have time to
create a test  
for every possible verification. If a div slides over a
couple pixes  
we can find a way to catch that, but we probably don't have
time to  
write automated tests to verify that every single pixel is
in alignment.

During the next release we'll be writing up testplans for
everything  
that we currently test, and we'll be adding more as new
features come  
in.

The ideal process is as follows.

- A feature is specified and design and dev writes some
initial docs
- QA writes a test plan
- The feature drops into the code base and QA does
verification and  
logs bugs
- QA automates 90% of the feature

-Mikeal

PS. We intend on writing a tool to keep track of test plans
and  
results and have a more interactive checkpoint testing
process where  
we mark manual tests off of the list. Ideally the automation
system  
will interface with this as well to check off the automated
tests  
that appear in the test plans. This also would give us an
easy way to  
know how much automation we have at any given time. No, I
don't have  
a date -- sometime after Preview 

_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

I'd suggest looking at Dan's test specs/plans for Chandler
as a  
template for those pages.  The triage spec <http:// 
svn.osafoundation.org/docs/trunk/testspecs/rel0_7/Triage-0.7
- 
testspec.html> in particular looks like it records manual
steps (see  
TC0051 and following).     Having these available would help
 
developers run some of these themselves before checking in.

Ted


On Jan 30, 2007, at 3:54 PM, Adam Christian wrote:

> Probably the best thing for manual testing is to get a
checklist  
> going on a wiki page so people can add those details.
>
> Mikeal, Aparna?
>
> Adam
>
> On Jan 30, 2007, at 3:48 PM, Bobby Rullo wrote:
>
>> What about non-automatable stuff?
>>
>> Like "verify feature x doesn't look wack when
doing ___"
>>
>> Bobby
>>
>>
>> On Jan 30, 2007, at 3:37 PM, Adam Christian wrote:
>>
>>> Since I am writing the initial Cosmo QA Test
scripts in our new  
>>> framework this week.. probably all of the
methods you stated will  
>>> find their way to my lap so I can incorporate
the functionality.  
>>> After the first round of them is written they
will be committed  
>>> to the Cosmo repo so you can go add/edit them
etc. I will send  
>>> out an email with some direction as soon as
that is ready. The  
>>> hope is to have a large chunk of the UI
automation done and ready  
>>> for everyone to use by 0.6.1.
>>>
>>> Adam
>>>
>>>
>>> On Jan 30, 2007, at 3:29 PM, Bobby Rullo
wrote:
>>>
>>>> Aparna, Adam, Mikeal et al,
>>>>
>>>> I found a bug before that was so glaring
that I was convinced it  
>>>> must've not been in the test scripts. I
have since realized that  
>>>> it's due to a weird interaction of things
in my local  
>>>> environment so I never bugged you guys
about it.
>>>>
>>>> But the general question remains, what is
the best way to ensure  
>>>> that certain scenarios, features, make it
into the QA Test  
>>>> scripts? Are the scripts publicly available
somewhere? Can I  
>>>> edit them or at least add a comment? Or
should file a bug? Or  
>>>> ________?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Bobby
>>>>
_______________________________________________
>>>> cosmo-dev mailing list
>>>> cosmo-devlists.osafoundation.org
>>>> http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev
>>>
>>
>
> _______________________________________________
> cosmo-dev mailing list
> cosmo-devlists.osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev

On 1/30/07, Mikeal Rogers <mikealosafoundation.org>
wrote:
> All the non-UI test cases are here;
>
> http://svn.osafoundation.org/server/cosmo/tr
unk/src/test/functional/
> protocol/
>
> You need to install py.test, after doing that you just
run the
> py.test command line utility and they auto-magically
all run.

can you put this info on the cosmo dev home wiki page?
there's an
obvious place with red placeholder text. also, could you rm
anything
from src/test/functional that is obsolete? thanks.
_______________________________________________
cosmo-dev mailing list
cosmo-devlists.osafoundation.org
http://lists.osafoundation.org/mailman/listinfo/cosmo-d
ev