Focus - AT vs. mouse/keyboard manipulation

(cross-posting to metacity-devel, because I don't know how
to fix this
and I suspect the metacity hackers will)

Hi Zack;

I think you're encountering several issues here.  With
regard to the
Action interface, items which aren't posted to the screen
may still be
activated.  For menu items this probably isn't really a
problem, with
respect to the desired effect for end users.

For dialogs, I agree that there's a problem.  The
difficulty you are
seeing is probably a result of "focus stealing
prevention" which was
added to the metacity window manager (in the gnome-2.14
timeframe I
think).  This behavioral change was intended to prevent
applications
from "stealing" focus except in response to a
direct user action. 
However, I believe that the AT-SPI Actions should probably
be treated as
"user" actions even though they are technically
'programmatic' ones.  I
don't know whether new metacity or window-manager API would
be required
to make this work, or not.  For this reason I am
cross-posting your
message.

Metacity folks: if an end-user of an assistive technology
such as a
screen reader or onscreen keyboard invokes the
"activate" action on a
button which opens a dialog, the desired behavior is for
that dialog to
take focus, since it is being posted in response to an
end-user action
(albeit indirectly).  I am happy to annotate the AT-SPI docs
to warn
clients of the API that they should not use the
Action:doAction API
except in response to direct end-user requests.

Zack, a specific test case that we can use for diagnosis and
discussion
would help.  Do you have one in mind?

Best regards,

Bill

On Wed, 2006-08-09 at 21:57, Zack Cerza wrote:
> Hi,
> 
> If I do something that results in a dialog popping up
via either the 
> mouse or the keyboard, that dialog has focus. If I open
the dialog via 
> AT-SPI's AccessibleAction interfaces (e.g. 'click'
and/or 'activate'), 
> the dialog does not have focus. Now, it seems that this
would be 
> considered a bug, but even if it isn't, I can't seem
to find an 
> alternative way to ensure that the dialog is given
focus.
> 
> I did try using AccessibleComponent_grabFocus() on
Accessibles which 
> implement that interface, but ran into problems; for
the vast majority 
> of AccessibleComponents, the function returns FALSE,
indicating that it 
> was unsuccessful (which it was). Why would that
function always fail?
> 
> I did find that most (if not all) Accessibles with an 
> AccessibleEditableText interface can successfully
grabFocus, but even 
> then, the window or dialog they're in must be focused;
they will not 
> grab focus from another window.
> 
> So, my question is: How can an AT ensure that the same
windows and UI 
> controls that have focus after a given user action
(i.e. with just the 
> keyboard and/or mouse) is performed have focus when the
same workflow is 
> performed by the AT?
> 
> Thanks,
> Zack
> _______________________________________________
> Gnome-accessibility-devel mailing list
> Gnome-accessibility-develgnome.org
> http://mail.gnome.org/mailman/listinfo/gnome-acc
essibility-devel

_______________________________________________
metacity-devel-list mailing list
metacity-devel-listgnome.org
http://mail.gnome.org/mailman/listinfo/metacity-devel-
list

Hi Bill,

Test case is available here -
http
://bugzilla.gnome.org/show_bug.cgi?id=347481

Thanks
Nagappan

Bill Haneman wrote:
> (cross-posting to metacity-devel, because I don't know
how to fix this
> and I suspect the metacity hackers will)
>
> Hi Zack;
>
> I think you're encountering several issues here.  With
regard to the
> Action interface, items which aren't posted to the
screen may still be
> activated.  For menu items this probably isn't really
a problem, with
> respect to the desired effect for end users.
>
> For dialogs, I agree that there's a problem.  The
difficulty you are
> seeing is probably a result of "focus stealing
prevention" which was
> added to the metacity window manager (in the gnome-2.14
timeframe I
> think).  This behavioral change was intended to prevent
applications
> from "stealing" focus except in response to
a direct user action. 
> However, I believe that the AT-SPI Actions should
probably be treated as
> "user" actions even though they are
technically 'programmatic' ones.  I
> don't know whether new metacity or window-manager API
would be required
> to make this work, or not.  For this reason I am
cross-posting your
> message.
>
> Metacity folks: if an end-user of an assistive
technology such as a
> screen reader or onscreen keyboard invokes the
"activate" action on a
> button which opens a dialog, the desired behavior is
for that dialog to
> take focus, since it is being posted in response to an
end-user action
> (albeit indirectly).  I am happy to annotate the AT-SPI
docs to warn
> clients of the API that they should not use the
Action:doAction API
> except in response to direct end-user requests.
>
> Zack, a specific test case that we can use for
diagnosis and discussion
> would help.  Do you have one in mind?
>
> Best regards,
>
> Bill
>
> On Wed, 2006-08-09 at 21:57, Zack Cerza wrote:
>   
>> Hi,
>>
>> If I do something that results in a dialog popping
up via either the 
>> mouse or the keyboard, that dialog has focus. If I
open the dialog via 
>> AT-SPI's AccessibleAction interfaces (e.g.
'click' and/or 'activate'), 
>> the dialog does not have focus. Now, it seems that
this would be 
>> considered a bug, but even if it isn't, I can't
seem to find an 
>> alternative way to ensure that the dialog is given
focus.
>>
>> I did try using AccessibleComponent_grabFocus() on
Accessibles which 
>> implement that interface, but ran into problems;
for the vast majority 
>> of AccessibleComponents, the function returns
FALSE, indicating that it 
>> was unsuccessful (which it was). Why would that
function always fail?
>>
>> I did find that most (if not all) Accessibles with
an 
>> AccessibleEditableText interface can successfully
grabFocus, but even 
>> then, the window or dialog they're in must be
focused; they will not 
>> grab focus from another window.
>>
>> So, my question is: How can an AT ensure that the
same windows and UI 
>> controls that have focus after a given user action
(i.e. with just the 
>> keyboard and/or mouse) is performed have focus when
the same workflow is 
>> performed by the AT?
>>
>> Thanks,
>> Zack
>> _______________________________________________
>> Gnome-accessibility-devel mailing list
>> Gnome-accessibility-develgnome.org
>> http://mail.gnome.org/mailman/listinfo/gnome-acc
essibility-devel
>>     
>
> _______________________________________________
> Gnome-accessibility-devel mailing list
> Gnome-accessibility-develgnome.org
> http://mail.gnome.org/mailman/listinfo/gnome-acc
essibility-devel
>   

-- 
Nagappan A <anagappannovell.com>
Novell Software Development (I) Pvt. Ltd.
Linux Desktop Testing Project - http://ldtp.freedesktop.o
rg
http://nagappanal.blo
gspot.com/

Novell, Inc.
SUSE® Linux Enterprise 10
Your Linux is ready™
http://www.novell.com/lin
ux

_______________________________________________
metacity-devel-list mailing list
metacity-devel-listgnome.org
http://mail.gnome.org/mailman/listinfo/metacity-devel-
list

On 8/14/06, Bill Haneman <Bill.Hanemansun.com> wrote:
> (cross-posting to metacity-devel, because I don't know
how to fix this
> and I suspect the metacity hackers will)
>
> Hi Zack;
>
> I think you're encountering several issues here.  With
regard to the
> Action interface, items which aren't posted to the
screen may still be
> activated.  For menu items this probably isn't really
a problem, with
> respect to the desired effect for end users.
>
> For dialogs, I agree that there's a problem.  The
difficulty you are
> seeing is probably a result of "focus stealing
prevention" which was
> added to the metacity window manager (in the gnome-2.14
timeframe I
> think).  This behavioral change was intended to prevent
applications
> from "stealing" focus except in response to
a direct user action.

This is a pretty good description of the behavior, but there
are some
pedantic clarifications I'd like to make (feel free to skip
this
paragraph if you just want to solve your problem ASAP
instead of
getting the verbose lecture):  The behavioral change was
also intended
to answer the question "how do we handle the fact that
user actions
are often far from instantaneous, and that users will
continue issuing
more actions instead of waiting for the first to be
processed?"
Indirect or non-user actions (e.g. Telsa sending an IM to
Alan just as
he's about to type his root password into some window,
resulting in
Alan accidentally IM'ing his root password back) was
definitely a big
part of the consideration, but that was actually considered
a special
case with a separate hint and handling for it.  It's just
that, as a
side-effect of the particular implementation we chose, a
broad range
of indirect interactions (e.g. the IM case) automatically
worked
without the additional hints.  But you are right that the
AT-SPI
actions are being lumped into the latter case because you
haven't
provided the WM sufficient hints to do otherwise.  This is
actually
very similar to another case:  single-instance application
launches
(where the second instance would request the first to open a
window
for it, and then the second instance would shut itself down)
suffered
from the same problem until they fixed the hints provided to
the WM.
Oh, and the other pedantic clarification -- this was
introduced in the
gnome-2.10 timeframe (well, developed during the gnome-2.6
timeframe,
added in gnome-2.8, removed at the last minute because of
insufficient
support for the new spec, and then finally landed in
gnome-2.10).

> However, I believe that the AT-SPI Actions should
probably be treated as
> "user" actions even though they are
technically 'programmatic' ones.  I
> don't know whether new metacity or window-manager API
would be required
> to make this work, or not.  For this reason I am
cross-posting your
> message.

No additional window manager API is required; it long since
added all
API you need -- you just need to use it (see below).  You
may
need/want new AT-SPI API, though.

> Metacity folks: if an end-user of an assistive
technology such as a
> screen reader or onscreen keyboard invokes the
"activate" action on a
> button which opens a dialog, the desired behavior is
for that dialog to
> take focus, since it is being posted in response to an
end-user action
> (albeit indirectly).

If it's a response to a direct user action (e.g. mouse
button press on
a button of the onscreen keyboard or such), then you have
the
timestamp of that user interaction.  If you just forward it
to the
other process, and have the other process call
gdk_x11_window_set_user_time(gdk_window, timestamp) on the
relevant
window, then you're good to go.  (If you want to do this on
a widget
that hasn't been shown yet, you may be unable to obtain a
gdk_window
unless you first call gtk_widget_realize())

> I am happy to annotate the AT-SPI docs to warn clients
of the API that they
> should not use the Action:doAction API except in
response to direct end-user
> requests.

If it's not in response to a direct end-user request, just
don't send
a timestamp and do no updating of timestamps on the other
end.  Then
metacity will automatically treat it correctly.

Hope that helps,
Elijah
_______________________________________________
metacity-devel-list mailing list
metacity-devel-listgnome.org
http://mail.gnome.org/mailman/listinfo/metacity-devel-
list

On Wed, 2006-08-16 at 19:45, Elijah Newren wrote:
...

Hi Elijah:

I appreciate the underlying logic behind the focus-stealing
prevention,
and in particular the oft-cited case where focus moves out
of a password
field during authentication.

It does however cause some serious problems for AT-SPI.

We basically cannot transfer timestamps with our doAction
requests,
API/ABI-compatibly.

The best we can do, I think, is have the ATK implementation
code (in
libgail) call gdk_x11_window_set_user_time() on the relevant
toplevel. 
However, we have to get an appropriate timestamp from
somewhere, and
unfortunately it cannot be gotten from AT-SPI.  So libgail
has to
somehow obtain a timestamp for "the current
time" when A(TK receives the
request. Unfortunately the canonical CURRENT_TIME macro in
X11 is just
'0' which fails to solve the problem, I think.  Would the
value returned
from gdk_x11_get_user_time() do the trick?

Bill

> This is a pretty good description of the behavior, but
there are some
> pedantic clarifications I'd like to make (feel free to
skip this
> paragraph if you just want to solve your problem ASAP
instead of
> getting the verbose lecture):  The behavioral change
was also intended
> to answer the question "how do we handle the fact
that user actions
> are often far from instantaneous, and that users will
continue issuing
> more actions instead of waiting for the first to be
processed?"
> Indirect or non-user actions (e.g. Telsa sending an IM
to Alan just as
> he's about to type his root password into some window,
resulting in
> Alan accidentally IM'ing his root password back) was
definitely a big
> part of the consideration, but that was actually
considered a special
> case with a separate hint and handling for it.  It's
just that, as a
> side-effect of the particular implementation we chose,
a broad range
> of indirect interactions (e.g. the IM case)
automatically worked
> without the additional hints.  But you are right that
the AT-SPI
> actions are being lumped into the latter case because
you haven't
> provided the WM sufficient hints to do otherwise.  This
is actually
> very similar to another case:  single-instance
application launches
> (where the second instance would request the first to
open a window
> for it, and then the second instance would shut itself
down) suffered
> from the same problem until they fixed the hints
provided to the WM.
> Oh, and the other pedantic clarification -- this was
introduced in the
> gnome-2.10 timeframe (well, developed during the
gnome-2.6 timeframe,
> added in gnome-2.8, removed at the last minute because
of insufficient
> support for the new spec, and then finally landed in
gnome-2.10).
> 
> > However, I believe that the AT-SPI Actions should
probably be treated as
> > "user" actions even though they are
technically 'programmatic' ones.  I
> > don't know whether new metacity or window-manager
API would be required
> > to make this work, or not.  For this reason I am
cross-posting your
> > message.
> 
> No additional window manager API is required; it long
since added all
> API you need -- you just need to use it (see below). 
You may
> need/want new AT-SPI API, though.
> 
> > Metacity folks: if an end-user of an assistive
technology such as a
> > screen reader or onscreen keyboard invokes the
"activate" action on a
> > button which opens a dialog, the desired behavior
is for that dialog to
> > take focus, since it is being posted in response
to an end-user action
> > (albeit indirectly).
> 
> If it's a response to a direct user action (e.g. mouse
button press on
> a button of the onscreen keyboard or such), then you
have the
> timestamp of that user interaction.  If you just
forward it to the
> other process, and have the other process call
> gdk_x11_window_set_user_time(gdk_window, timestamp) on
the relevant
> window, then you're good to go.  (If you want to do
this on a widget
> that hasn't been shown yet, you may be unable to
obtain a gdk_window
> unless you first call gtk_widget_realize())
> 
> > I am happy to annotate the AT-SPI docs to warn
clients of the API that they
> > should not use the Action:doAction API except in
response to direct end-user
> > requests.
> 
> If it's not in response to a direct end-user request,
just don't send
> a timestamp and do no updating of timestamps on the
other end.  Then
> metacity will automatically treat it correctly.
> 
> Hope that helps,
> Elijah
> _______________________________________________
> Gnome-accessibility-devel mailing list
> Gnome-accessibility-develgnome.org
> http://mail.gnome.org/mailman/listinfo/gnome-acc
essibility-devel

_______________________________________________
metacity-devel-list mailing list
metacity-devel-listgnome.org
http://mail.gnome.org/mailman/listinfo/metacity-devel-
list

On 8/17/06, Bill Haneman <Bill.Hanemansun.com> wrote:
> On Wed, 2006-08-16 at 19:45, Elijah Newren wrote:
> ...
>
> Hi Elijah:
>
> I appreciate the underlying logic behind the
focus-stealing prevention,
> and in particular the oft-cited case where focus moves
out of a password
> field during authentication.
>
> It does however cause some serious problems for AT-SPI.
>
> We basically cannot transfer timestamps with our
doAction requests,
> API/ABI-compatibly.

Ouch.  That doesn't sound good.

> The best we can do, I think, is have the ATK
implementation code (in
> libgail) call gdk_x11_window_set_user_time() on the
relevant toplevel.
> However, we have to get an appropriate timestamp from
somewhere, and
> unfortunately it cannot be gotten from AT-SPI.  So
libgail has to
> somehow obtain a timestamp for "the current
time" when A(TK receives the
> request. Unfortunately the canonical CURRENT_TIME macro
in X11 is just
> '0' which fails to solve the problem, I think.

Yes, that's correct.  GDK_CURRENT_TIME/CurrentTime (gdk
& X names for
the same thing) are not useful in cases like this.

Bit of trivia: In fact, since 0 is the only known invalid
timestamp,
the spec makes use of this to say that it should be treated
as a
specialized "please do not focus me at all"
hint.  So, this is quite
the opposite of what you want.

Short rant: CurrentTime is also buggy in a wide range of
other cases
(even when the timestamp is meant for a message to the
Xserver instead
of to the WM), as its use leads to subtle and nasty race
conditions.
I'd be willing to guess that most uses of it are wrong in
application/toolkit/window manager codes.

> Would the value returned from gdk_x11_get_user_time()
do the trick?

Short answer: No, it would result in no change whatsoever
from the
current behavior, since this is precisely what gtk+ already
does.

Long (explanatory) answer: gdk_x11_get_user_time() merely
returns
display_x11->user_time, which stores the last time gtk+
was able to
detect that the user interacted with that particular
process.
gdk_x11_window_set_user_time() exists to allow app authors
to update
the timestamp on a window when there's some kind of special
user
interaction with an application that gtk+ has no way of
being aware
of.  (Examples of "special interaction":
single-instance applications
forwarding requests to other instances to open a window,
metacity
handling a keybinding and forwarding it to the gnome-panel
to have it
show a "Run application" dialog, and apparently
AT-SPI/ATK...).
gdk_x11_window_set_user_time() happens to also update
display_x11->user_time as a side effect.

To fix this:

We need some way of getting the timestamp from the sender
process to
metacity.  You actually don't need to request the
application author
to supply it; you could just call
gdk_x11_display_get_user_time().
But we do need to either get it sent to the other process,
or get it
sent to the WM in some manner that also lets the WM know
precisely
which window (or windows) it will be appilcable for.  The
latter is
pretty problematic.  It means we need to extend the spec so
things
will work with other window managers as well (e.g. KWin). 
But there's
an even bigger question of whether the latter option is even
feasible;
does the sending process really know which windows will be
launched in
response to an action?

We may need to dream up some creative ways of trying to get
the
timestamp from the sender to the receiver inside AT-SPI
(e.g. set a
property on certain windows saying they can receive
additional or
modified messages, and have AT-SPI detect such, and then
have AT-SPI
send additional or modified messages)


Hope that helps,
Elijah
_______________________________________________
metacity-devel-list mailing list
metacity-devel-listgnome.org
http://mail.gnome.org/mailman/listinfo/metacity-devel-
list

On Thu, 2006-08-17 at 18:06, Elijah Newren wrote:
...
> Yes, that's correct.  GDK_CURRENT_TIME/CurrentTime
(gdk & X names for
> the same thing) are not useful in cases like this.

> ...
> > Would the value returned from
gdk_x11_get_user_time() do the trick?
> 
> Short answer: No, it would result in no change
whatsoever from the
> current behavior, since this is precisely what gtk+
already does.

Hmm, thinking sideways a bit...

is there some event we could explicitly fire (inside either
atk-bridge
or libgail -i.e. within the application's process space)
which would
have no real effect on the running app but which would
result in the
Xserver updating its user-time stamp?

Seems to me that's what we want - to tell the application
"hey, the user
just interacted with you!".

Bill

_______________________________________________
metacity-devel-list mailing list
metacity-devel-listgnome.org
http://mail.gnome.org/mailman/listinfo/metacity-devel-
list

On 8/17/06, Bill Haneman <Bill.Hanemansun.com> wrote:
> Hmm, thinking sideways a bit...

Always welcome.  

> is there some event we could explicitly fire (inside
either atk-bridge
> or libgail -i.e. within the application's process
space) which would

which process?  (The process that actually received the user
interaction event (which I called the "sender"
in my previous email),
or the one that will be handling the user interaction event
by opening
a window (the "receiver")?)

> have no real effect on the running app but which would
result in the
> Xserver updating its user-time stamp?

Since I don't know which process you are referring to,
here's some
guesses at what you're trying to get at:

Do you mean having the "receiver" ping the
Xserver for a timestamp and
then updating its user-time property accordingly?  That
wouldn't work,
as not only would it break the non-user-interaction cases,
it has all
kinds of nasty race conditions.

Or are you thinking in terms of having the
"sender" somehow update the
user-time of the process it is trying to send a message to? 
Something
like that may be tenable, though we'd have to be careful
about
how/when it got updated relative to receiving the do_process
message.

> Seems to me that's what we want - to tell the
application "hey, the user
> just interacted with you!".

No, not at all.  Messages and assumptions like this is
exactly what
caused the huge focus-on-map security & other bug mess
in the first
place.  We want to tell the application "hey, the user
interacted with
you at time <x>" which is a world different than
"hey, the user 'just'
interacted with you".  'just' carries no useful
information with it
and results in all kinds of bugs.

Hope that helps,
Elijah
_______________________________________________
metacity-devel-list mailing list
metacity-devel-listgnome.org
http://mail.gnome.org/mailman/listinfo/metacity-devel-
list

"Just interacted" in this case must, I'm
afraid, mean "now", i.e. "on
receipt".

My intention below is for the _receiver_ to generate an
event which
causes the Xserver to update its internal timestamp.

I know you don't like it, but I think this is really what
is required. 
Yes, I know there are possible race conditions, but there
are race
conditions in X as well, it's just that you're dealing
with a single
event/keyboard-focus stream in the X case.  Whereas we
actually DO need
to programmatically move focus from outside.

The problem with the whole focus-stealing-prevention design
as it now
stands is that it doesn't account for the very case we
need, i.e.
"remote agent activates process via something other
than an X event".

Bill

On Thu, 2006-08-17 at 18:33, Elijah Newren wrote:
> On 8/17/06, Bill Haneman <Bill.Hanemansun.com> wrote:
> > Hmm, thinking sideways a bit...
> 
> Always welcome.  
> 
> > is there some event we could explicitly fire
(inside either atk-bridge
> > or libgail -i.e. within the application's process
space) which would
> 
> which process?  (The process that actually received the
user
> interaction event (which I called the
"sender" in my previous email),
> or the one that will be handling the user interaction
event by opening
> a window (the "receiver")?)
> 
> > have no real effect on the running app but which
would result in the
> > Xserver updating its user-time stamp?
> 
> Since I don't know which process you are referring to,
here's some
> guesses at what you're trying to get at:
> 
> Do you mean having the "receiver" ping the
Xserver for a timestamp and
> then updating its user-time property accordingly?  That
wouldn't work,
> as not only would it break the non-user-interaction
cases, it has all
> kinds of nasty race conditions.
> 
> Or are you thinking in terms of having the
"sender" somehow update the
> user-time of the process it is trying to send a message
to?  Something
> like that may be tenable, though we'd have to be
careful about
> how/when it got updated relative to receiving the
do_process message.
> 
> > Seems to me that's what we want - to tell the
application "hey, the user
> > just interacted with you!".
> 
> No, not at all.  Messages and assumptions like this is
exactly what
> caused the huge focus-on-map security & other bug
mess in the first
> place.  We want to tell the application "hey, the
user interacted with
> you at time <x>" which is a world different
than "hey, the user 'just'
> interacted with you".  'just' carries no useful
information with it
> and results in all kinds of bugs.
> 
> Hope that helps,
> Elijah

_______________________________________________
metacity-devel-list mailing list
metacity-devel-listgnome.org
http://mail.gnome.org/mailman/listinfo/metacity-devel-
list

BTW, I should mention that I'll be offline until Monday, so
I guess
we'll need to continue our various discussions then.

Best regards,

Bill

_______________________________________________
metacity-devel-list mailing list
metacity-devel-listgnome.org
http://mail.gnome.org/mailman/listinfo/metacity-devel-
list