KHTMLPart + DOM: how to "activate" node/emulate mouse click

Hi all,

Playing with KHTMLPart and DOM in KDE3.5. The goal is:
- Load a web page into KHTMLPart (done);
- Obtain and parse DOM tree to find a DOM::Node with
speficic image, which 
contains the link (done);
- Activate this link. Here's the problem: setActiveNode()
only sets the focus 
on this node. Is there a way to emulate user behavior like
"User clicked the 
mouse button on this DOM::Node"?

I don't want to do it by parsing the appropriate link href
and get()ting it 
myself - in this case it actually won't work because the
link connected to 
JavaScript onclick() event. Also I'd prefer more universal
solution, which 
would also work with generic links, form buttons etc.

So far the only way I tnink about is to obtain node region,
translate it to 
KHTMLView, and emulate a mouse click by sending a
QMouseEvent. Not sure it 
gonna work, and it looks quite ugly. Are there easier ways?

-- 
With best regards, George.
http://www.kchmviewer.net - the first CHM files viewer for Qt/KDE.
http://www.transientmail
.com - a temporary spam-safe e-mail address.
 
>> Visit 
http://mail.kde.org/mailman/listinfo/kde-devel#unsub to
unsubscribe <<

On Monday 05 Mar 2007 2:18:26 pm George Yunaev wrote:
Hi George,

> Hi all,
>
> Playing with KHTMLPart and DOM in KDE3.5. The goal is:
> - Load a web page into KHTMLPart (done);
> - Obtain and parse DOM tree to find a DOM::Node with
speficic image, which
> contains the link (done);
> - Activate this link. Here's the problem:
setActiveNode() only sets the
> focus on this node. Is there a way to emulate user
behavior like "User
> clicked the mouse button on this DOM::Node"?
Yes there is a method, read on..
> I don't want to do it by parsing the appropriate link
href and get()ting it
> myself - in this case it actually won't work because
the link connected to
> JavaScript onclick() event. Also I'd prefer more
universal solution, which
> would also work with generic links, form buttons etc.
>
> So far the only way I tnink about is to obtain node
region, translate it to
> KHTMLView, and emulate a mouse click by sending a
QMouseEvent. Not sure it
> gonna work, and it looks quite ugly. Are there easier
ways?

As I understand your problem, you we have three issues

1) Clicking on simple links (where it href's a simple url).
in that case, a 
simple openURL will do
2) Clicking on a URL which href's # (itself) but has an
"onclick" event 
associated with it (in which case you need to run the script
with 
executeScript())
3) Clicking on an image (the property being the alt tag here
instead of the 
link name.) Cases (1) and (2) are handled in the same
manner

here's some code, which will help you out. 
(it's shamelessly copy-pasted from my KWebTest)

//m_part is the pointer to the khtml part:
//linkNameLineEdit is a simple line edit that contains
// EITHER the name of a link OR the "name" of an
image (ALT tag)
// This code simulates clicking on a link (image/text)
// WARNING: it is assumed all that the link/images names are
unique. 
//JsRunner is a class i wrote to simulate all JavaScript
(and later ALL) user 
events on a web page

void JsRunner::executeLinks()
{
  
  repeat = true;
  if(!m_part)
  {

    return;
  }
  
  
  DOM::HTMLDocument * doc = new
DOM::HTMLDocument(m_part->htmlDocument());
  DOM::HTMLCollection  collection = doc->links();

  
  for(unsigned long i = 0; i< collection.length(); i ++)
  {
    DOM::Node node = collection.item( i);
    DOM:OMString
domStringName = node.nodeName();
    QString stringName = domStringName.string();
    DOM::HTMLLinkElement * htmlLinkEle =
(DOM::HTMLLinkElement*)&(node);

   
if(htmlLinkEle->href().string().contains("#"))
    {
	//if the link refers to itself, get it's onclick event
      QString jText = 
htmlLinkEle->getAttribute(DOM:OMString
(QString("onclick"))).string();
    }
    exploreDomNode(node, text);
  }
  DOM::HTMLCollection  collectionImg = doc->images();
  
  for(unsigned long i = 0; i< collectionImg.length(); i
++)
  {
    DOM::Node nodeImg = collection.item( i);
    if(nodeImg.isNull())
    {	
  
      continue;
    }
    DOM::Node actualNode = nodeImg.firstChild();
    
    if(actualNode.nodeName().string()=="IMG")
    {
      DOM::Element targetElement(actualNode);
      
      
if(targetElement.getAttribute("title").string()==l
inkNameLineEdit->text())
      {
        DOM::Element nodeImgEle(nodeImg);
        if(nodeImgEle.hasAttribute("onclick"))
	
	//This is how you execute the a JS on your page
         
m_part->executeScript(nodeImgEle.getAttribute("oncli
ck").string());
      }
    }
  }
}

void JsRunner::exploreDomNode( DOM::Node node)
{
  if(node.isNull())
  {
    return;
  }
  repeat = false;
  DOM::NodeList childNodeList = node.childNodes();
  if(childNodeList.length()==0)
    return;
  for( unsigned long i = 0 ; i < childNodeList.length() ;
i ++)
  {
    QString childString;
    DOM::Node childNode = childNodeList.item(i);
    if(childNode.nodeValue().string() ==
linkNameLineEdit->text())
    {
      DOM::HTMLLinkElement * htmlLinkEle =
(DOM::HTMLLinkElement*)&(node);
      QString jText = 
htmlLinkEle->getAttribute(DOM:OMString
(QString("onclick"))).string();
      if(jText!= QString::null)
      {
        m_part->executeScript( jText);
      }
      else
      {
        QString href = htmlLinkEle->href().string();
        m_part->openURL(href);
      }
      return;
    }
    else
    {
      repeat = true;
    }
    exploreDomNode(childNode, text);
  }
}




 
>> Visit 
http://mail.kde.org/mailman/listinfo/kde-devel#unsub to
unsubscribe <<

Hi Manish and all,

I've found the way to emulate click on a node itself. It is
simpler, and 
generates an actual click without needing to parse the node
content, or 
without messing with Qt or DOM internals. So far was tested
on images, 
anchors and buttons - seems to work everywhere. Here is the
code:

void KDMainWindow::emulateMouseClick( DOM::Node pNode )
{
	QRect nr = pNode.getRect();
	
	// Make the click in the middle of pic/link/widget
	QPoint coords = QPoint( nr.x() + nr.width() / 2, nr.y() +
nr.height() / 2 );
	
	// Convert to viewport
	coords = m_browser->view()->contentsToViewport(
coords );
			
	// Create the press/release event. Note that postEvent
takes care of
        // pointers, so no need to delete them.
	QEvent * bp = new QMouseEvent ( QEvent::MouseButtonPress,
coords,
                           Qt::LeftButton, Qt::LeftButton
);
	QEvent * br = new QMouseEvent ( QEvent::MouseButtonRelease,
coords, 
                           Qt::LeftButton, Qt::LeftButton
);
	
	// Post events to viewport!!! not to ScrollView!
	QApplication::postEvent(
m_browser->view()->viewport(), bp );
	QApplication::postEvent(
m_browser->view()->viewport(), br );
}

Obviously it could be enchanced to emulate double or right
clicks, or even 
dragging (read: testing AJAX apps like Google maps)

By the way, an attempt to post this event to a widget
created from QObject but 
not from QWidget (for example, KHTMLPart) results in
segfault in 
isActiveWidget(), as Qt does not check that the recipient is
a widget.

2Manish: if you tell me your KWebTest is integrated (or
going to be) with Perl 
to provide scripting for QA, I'd say that we're developing
the same 
application, and there is little reason to develop two
different, so it might 
be reasonable to combine the forces?

2Others: we need such a web test framework, because there
seems to be no other  
way to automate testing of AJAX apps, which become more and
more popular.

> On Monday 05 Mar 2007 2:18:26 pm George Yunaev wrote:
> Hi George,
>
> > Hi all,
> >
> > Playing with KHTMLPart and DOM in KDE3.5. The goal
is:
> > - Load a web page into KHTMLPart (done);
> > - Obtain and parse DOM tree to find a DOM::Node
with speficic image,
> > which contains the link (done);
> > - Activate this link. Here's the problem:
setActiveNode() only sets the
> > focus on this node. Is there a way to emulate user
behavior like "User
> > clicked the mouse button on this DOM::Node"?
>
> Yes there is a method, read on..
>
> > I don't want to do it by parsing the appropriate
link href and get()ting
> > it myself - in this case it actually won't work
because the link
> > connected to JavaScript onclick() event. Also I'd
prefer more universal
> > solution, which would also work with generic
links, form buttons etc.
> >
> > So far the only way I tnink about is to obtain
node region, translate it
> > to KHTMLView, and emulate a mouse click by sending
a QMouseEvent. Not
> > sure it gonna work, and it looks quite ugly. Are
there easier ways?
>
> As I understand your problem, you we have three issues
>
> 1) Clicking on simple links (where it href's a simple
url). in that case, a
> simple openURL will do
> 2) Clicking on a URL which href's # (itself) but has an
"onclick" event
> associated with it (in which case you need to run the
script with
> executeScript())
> 3) Clicking on an image (the property being the alt tag
here instead of the
> link name.) Cases (1) and (2) are handled in the same
manner
>
> here's some code, which will help you out.
> (it's shamelessly copy-pasted from my KWebTest)
>
> //m_part is the pointer to the khtml part:
> //linkNameLineEdit is a simple line edit that contains
> // EITHER the name of a link OR the "name" of
an image (ALT tag)
> // This code simulates clicking on a link (image/text)
> // WARNING: it is assumed all that the link/images
names are unique.
> //JsRunner is a class i wrote to simulate all
JavaScript (and later ALL)
> user events on a web page
>
> void JsRunner::executeLinks()
> {
>
>   repeat = true;
>   if(!m_part)
>   {
>
>     return;
>   }
>
>
>   DOM::HTMLDocument * doc = new
DOM::HTMLDocument(m_part->htmlDocument());
>   DOM::HTMLCollection  collection = doc->links();
>
>
>   for(unsigned long i = 0; i< collection.length(); i
++)
>   {
>     DOM::Node node = collection.item( i);
>     DOM:OMString
domStringName = node.nodeName();
>     QString stringName = domStringName.string();
>     DOM::HTMLLinkElement * htmlLinkEle =
(DOM::HTMLLinkElement*)&(node);
>
>    
if(htmlLinkEle->href().string().contains("#"))
>     {
> 	//if the link refers to itself, get it's onclick
event
>       QString jText =
> htmlLinkEle->getAttribute(DOM:OMString
(QString("onclick"))).string();
>     }
>     exploreDomNode(node, text);
>   }
>   DOM::HTMLCollection  collectionImg =
doc->images();
>
>   for(unsigned long i = 0; i<
collectionImg.length(); i ++)
>   {
>     DOM::Node nodeImg = collection.item( i);
>     if(nodeImg.isNull())
>     {
>
>       continue;
>     }
>     DOM::Node actualNode = nodeImg.firstChild();
>
>    
if(actualNode.nodeName().string()=="IMG")
>     {
>       DOM::Element targetElement(actualNode);
>
>
>
if(targetElement.getAttribute("title").string()==l
inkNameLineEdit->text())
>       {
>         DOM::Element nodeImgEle(nodeImg);
>        
if(nodeImgEle.hasAttribute("onclick"))
>
> 	//This is how you execute the a JS on your page
>          
>
m_part->executeScript(nodeImgEle.getAttribute("oncli
ck").string()); }
>     }
>   }
> }
>
> void JsRunner::exploreDomNode( DOM::Node node)
> {
>   if(node.isNull())
>   {
>     return;
>   }
>   repeat = false;
>   DOM::NodeList childNodeList = node.childNodes();
>   if(childNodeList.length()==0)
>     return;
>   for( unsigned long i = 0 ; i <
childNodeList.length() ; i ++)
>   {
>     QString childString;
>     DOM::Node childNode = childNodeList.item(i);
>     if(childNode.nodeValue().string() ==
linkNameLineEdit->text())
>     {
>       DOM::HTMLLinkElement * htmlLinkEle =
(DOM::HTMLLinkElement*)&(node);
>       QString jText =
> htmlLinkEle->getAttribute(DOM:OMString
(QString("onclick"))).string();
>       if(jText!= QString::null)
>       {
>         m_part->executeScript( jText);
>       }
>       else
>       {
>         QString href =
htmlLinkEle->href().string();
>         m_part->openURL(href);
>       }
>       return;
>     }
>     else
>     {
>       repeat = true;
>     }
>     exploreDomNode(childNode, text);
>   }
> }
>
> >> Visit 
http://mail.kde.org/mailman/listinfo/kde-devel#unsub to
> >> unsubscribe <<



-- 
With best regards, George.
http://www.kchmviewer.net - the first CHM files viewer for Qt/KDE.
http://www.transientmail
.com - a temporary spam-safe e-mail address.
 
>> Visit 
http://mail.kde.org/mailman/listinfo/kde-devel#unsub to
unsubscribe <<

Hi all,

> > I've found the way to emulate click on a node
itself. It is simpler, and
> > generates an actual click without needing to parse
the node content, or
> > without messing with Qt or DOM internals. So far
was tested on images,
> > anchors and buttons - seems to work everywhere.
Here is the code:
> How about sites whose contents can dynamically change
with respect to time.
> It's an honest question, not a debate.

I expect the getRect( DOM::Node ) to return a valid value
when the node is 
changed. It is used internally too, so this should be the
case.

> George i was thinking about Python. Not very strong in
perl
> But this seems to be a lovely project to work on. I'm
game.

That's exactly why I was thinking about Perl  my
knowlegde of Python is 
absolute zero.


-- 
With best regards, George.
http://www.kchmviewer.net - the first CHM files viewer for Qt/KDE.
http://www.transientmail
.com - a temporary spam-safe e-mail address.
 
>> Visit 
http://mail.kde.org/mailman/listinfo/kde-devel#unsub to
unsubscribe <<

Hi George,

On Tuesday 06 Mar 2007 11:57:53 am George Yunaev wrote:
> Hi all,
>
> > > I've found the way to emulate click on a node
itself. It is simpler,
> > > and generates an actual click without needing
to parse the node
> > > content, or without messing with Qt or DOM
internals. So far was tested
> > > on images, anchors and buttons - seems to
work everywhere. Here is the
> > > code:
> >
> > How about sites whose contents can dynamically
change with respect to
> > time. It's an honest question, not a debate.
>
> I expect the getRect( DOM::Node ) to return a valid
value when the node is
> changed. It is used internally too, so this should be
the case.
Ok
> > George i was thinking about Python. Not very
strong in perl
> > But this seems to be a lovely project to work on.
I'm game.
We can then work on the core together. you can work on the
perl bindings and i 
can work on the python bindings. Added advantage being,
we'll be able to 
design it well enough, so that people can write more
bindings for other 
languages.
> That's exactly why I was thinking about Perl  my
knowlegde of Python is
> absolute zero.


 
>> Visit 
http://mail.kde.org/mailman/listinfo/kde-devel#unsub to
unsubscribe <<

Hi George,


On Tuesday 06 Mar 2007 11:40:27 am you wrote:
> Hi Manish and all,
>
> I've found the way to emulate click on a node itself.
It is simpler, and
> generates an actual click without needing to parse the
node content, or
> without messing with Qt or DOM internals. So far was
tested on images,
> anchors and buttons - seems to work everywhere. Here is
the code:
How about sites whose contents can dynamically change with
respect to time. 
It's an honest question, not a debate.

> void KDMainWindow::emulateMouseClick( DOM::Node pNode
)
> {
> 	QRect nr = pNode.getRect();
>
> 	// Make the click in the middle of pic/link/widget
> 	QPoint coords = QPoint( nr.x() + nr.width() / 2,
nr.y() + nr.height() / 2
> );
>
> 	// Convert to viewport
> 	coords = m_browser->view()->contentsToViewport(
coords );
>
> 	// Create the press/release event. Note that postEvent
takes care of
>         // pointers, so no need to delete them.
> 	QEvent * bp = new QMouseEvent (
QEvent::MouseButtonPress, coords,
>                            Qt::LeftButton,
Qt::LeftButton );
> 	QEvent * br = new QMouseEvent (
QEvent::MouseButtonRelease, coords,
>                            Qt::LeftButton,
Qt::LeftButton );
>
> 	// Post events to viewport!!! not to ScrollView!
> 	QApplication::postEvent(
m_browser->view()->viewport(), bp );
> 	QApplication::postEvent(
m_browser->view()->viewport(), br );
> }
>
> Obviously it could be enchanced to emulate double or
right clicks, or even
> dragging (read: testing AJAX apps like Google maps)
coooooooool
> By the way, an attempt to post this event to a widget
created from QObject
> but not from QWidget (for example, KHTMLPart) results
in segfault in
> isActiveWidget(), as Qt does not check that the
recipient is a widget.
>
> 2Manish: if you tell me your KWebTest is integrated (or
going to be) with
> Perl to provide scripting for QA, I'd say that we're
developing the same
> application, and there is little reason to develop two
different, so it
> might be reasonable to combine the forces?
George i was thinking about Python. Not very strong in perl
But this seems to be a lovely project to work on. I'm game.

> 2Others: we need such a web test framework, because
there seems to be no
> other way to automate testing of AJAX apps, which
become more and more
> popular.
Point.
> > On Monday 05 Mar 2007 2:18:26 pm George Yunaev
wrote:
> > Hi George,
> >
> > > Hi all,
> > >
> > > Playing with KHTMLPart and DOM in KDE3.5. The
goal is:
> > > - Load a web page into KHTMLPart (done);
> > > - Obtain and parse DOM tree to find a
DOM::Node with speficic image,
> > > which contains the link (done);
> > > - Activate this link. Here's the problem:
setActiveNode() only sets the
> > > focus on this node. Is there a way to emulate
user behavior like "User
> > > clicked the mouse button on this
DOM::Node"?
> >
> > Yes there is a method, read on..
> >
> > > I don't want to do it by parsing the
appropriate link href and
> > > get()ting it myself - in this case it
actually won't work because the
> > > link connected to JavaScript onclick() event.
Also I'd prefer more
> > > universal solution, which would also work
with generic links, form
> > > buttons etc.
> > >
> > > So far the only way I tnink about is to
obtain node region, translate
> > > it to KHTMLView, and emulate a mouse click by
sending a QMouseEvent.
> > > Not sure it gonna work, and it looks quite
ugly. Are there easier ways?
> >
> > As I understand your problem, you we have three
issues
> >
> > 1) Clicking on simple links (where it href's a
simple url). in that case,
> > a simple openURL will do
> > 2) Clicking on a URL which href's # (itself) but
has an "onclick" event
> > associated with it (in which case you need to run
the script with
> > executeScript())
> > 3) Clicking on an image (the property being the
alt tag here instead of
> > the link name.) Cases (1) and (2) are handled in
the same manner
> >
> > here's some code, which will help you out.
> > (it's shamelessly copy-pasted from my KWebTest)
> >
> > //m_part is the pointer to the khtml part:
> > //linkNameLineEdit is a simple line edit that
contains
> > // EITHER the name of a link OR the
"name" of an image (ALT tag)
> > // This code simulates clicking on a link
(image/text)
> > // WARNING: it is assumed all that the link/images
names are unique.
> > //JsRunner is a class i wrote to simulate all
JavaScript (and later ALL)
> > user events on a web page
> >
> > void JsRunner::executeLinks()
> > {
> >
> >   repeat = true;
> >   if(!m_part)
> >   {
> >
> >     return;
> >   }
> >
> >
> >   DOM::HTMLDocument * doc = new
> > DOM::HTMLDocument(m_part->htmlDocument());
DOM::HTMLCollection 
> > collection = doc->links();
> >
> >
> >   for(unsigned long i = 0; i<
collection.length(); i ++)
> >   {
> >     DOM::Node node = collection.item( i);
> >     DOM:OMString
domStringName = node.nodeName();
> >     QString stringName = domStringName.string();
> >     DOM::HTMLLinkElement * htmlLinkEle =
(DOM::HTMLLinkElement*)&(node);
> >
> >    
if(htmlLinkEle->href().string().contains("#"))
> >     {
> > 	//if the link refers to itself, get it's onclick
event
> >       QString jText =
> > htmlLinkEle->getAttribute(DOM:OMString
(QString("onclick"))).string();
> >     }
> >     exploreDomNode(node, text);
> >   }
> >   DOM::HTMLCollection  collectionImg =
doc->images();
> >
> >   for(unsigned long i = 0; i<
collectionImg.length(); i ++)
> >   {
> >     DOM::Node nodeImg = collection.item( i);
> >     if(nodeImg.isNull())
> >     {
> >
> >       continue;
> >     }
> >     DOM::Node actualNode = nodeImg.firstChild();
> >
> >    
if(actualNode.nodeName().string()=="IMG")
> >     {
> >       DOM::Element targetElement(actualNode);
> >
> >
> >
if(targetElement.getAttribute("title").string()==l
inkNameLineEdit->text()
> >) {
> >         DOM::Element nodeImgEle(nodeImg);
> >        
if(nodeImgEle.hasAttribute("onclick"))
> >
> > 	//This is how you execute the a JS on your page
> >
> >
m_part->executeScript(nodeImgEle.getAttribute("oncli
ck").string()); }
> >     }
> >   }
> > }
> >
> > void JsRunner::exploreDomNode( DOM::Node node)
> > {
> >   if(node.isNull())
> >   {
> >     return;
> >   }
> >   repeat = false;
> >   DOM::NodeList childNodeList =
node.childNodes();
> >   if(childNodeList.length()==0)
> >     return;
> >   for( unsigned long i = 0 ; i <
childNodeList.length() ; i ++)
> >   {
> >     QString childString;
> >     DOM::Node childNode = childNodeList.item(i);
> >     if(childNode.nodeValue().string() ==
linkNameLineEdit->text())
> >     {
> >       DOM::HTMLLinkElement * htmlLinkEle =
> > (DOM::HTMLLinkElement*)&(node); QString jText
=
> > htmlLinkEle->getAttribute(DOM:OMString
(QString("onclick"))).string();
> >       if(jText!= QString::null)
> >       {
> >         m_part->executeScript( jText);
> >       }
> >       else
> >       {
> >         QString href =
htmlLinkEle->href().string();
> >         m_part->openURL(href);
> >       }
> >       return;
> >     }
> >     else
> >     {
> >       repeat = true;
> >     }
> >     exploreDomNode(childNode, text);
> >   }
> > }
> >
> > >> Visit 
http://mail.kde.org/mailman/listinfo/kde-devel#unsub to
> > >> unsubscribe <<


 
>> Visit 
http://mail.kde.org/mailman/listinfo/kde-devel#unsub to
unsubscribe <<