KDevFileManager performance

Hi,

 I'm worried about the performance problems in the
KDevFileManager 
class, due to the KDevProjectItem* way of storing the data.
If these 
classes help later with the model/view stuff, I have no
problem with 
them, but due to their tree-like structure it is hard and
time 
consuming to work with them. For example if you want to get
all the 
files that were added to the project, you would need to go
through this 
structure using a recursive method. To find if a url was
already added 
or not, you would have to do the same. To get information
about the 
item, again, a search through all of the items.
 For this reason I have a suggestion, which is to move
allFiles from 
KDevProject to KDevFileManager. The reasoning is that this
way I could 
implement a file manager which stores the
URL->KDevProjectItem* mapping 
in a QHash or QMap and it will be much faster to retrieve
all the 
above. allFiles could have access directly to this map,
inProject could 
get the QList of allFiles and find there the URL and so on.
It would be 
even more better if allFiles would return a QHash<KUrl, 
KDevProjectFileItem*>.

Comments?

Andras

-- 
Quanta Plus developer - http://quanta.kdewebdev.o
rg
K Desktop Environment - http://www.kde.org
_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel

Here it is a proposal for KDevFileManager.

Andras

-- 
Quanta Plus developer - http://quanta.kdewebdev.o
rg
K Desktop Environment - http://www.kde.org
_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel

On Thursday 17 August 2006 14:57, Andras Mantia wrote:
> Hi,
>
>  I'm worried about the performance problems in the
KDevFileManager
> class, due to the KDevProjectItem* way of storing the
data. If these
> classes help later with the model/view stuff, I have no
problem with
> them, but due to their tree-like structure it is hard
and time
> consuming to work with them. For example if you want to
get all the
> files that were added to the project, you would need to
go through this
> structure using a recursive method. To find if a url
was already added
> or not, you would have to do the same. To get
information about the
> item, again, a search through all of the items.
>  For this reason I have a suggestion, which is to move
allFiles from
> KDevProject to KDevFileManager. The reasoning is that
this way I could
> implement a file manager which stores the
URL->KDevProjectItem* mapping
> in a QHash or QMap and it will be much faster to
retrieve all the
> above. allFiles could have access directly to this map,
inProject could
> get the QList of allFiles and find there the URL and so
on. It would be
> even more better if allFiles would return a
QHash<KUrl,
> KDevProjectFileItem*>.
>
> Comments?
>

Have you done any profiling? Or are you just guessing?
Please come back with 
hard numbers on the performance difference between the
current code and your 
proposal and we can take a look at doing something
different. There's just 
too much other stuff to do ATM.
--
Matt

_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel

Hi!

Can we please put off this discussion.  There are bigger
problems with the 
project API(s) and they'll need to be completely refactored
I'm afraid.  This 
is what currently happens:

KDevProjectController is the controller, but all it really
does at the moment 
is handle loading/unloading the KDevProjectManager_part.  
KDevProjectManager_part is the view, but it calls the
importers to populate 
the data in a 'model'.  This model really isn't a model
at all.  It needs to 
be rewritten to inherit QAbstractItemModel.

Now, this is what _should_ be happening:

KDevProjectController is the controller and it should behave
as such.  The 
controller should be primarily responsible with populating
(via the 
importers), updating and changing the project model.  This
model should 
inherit QAbstractItemModel.  The KDevProjectManager_part
should only be 
responsible for displaying the model.  The only other thing
it should do is 
respond to user requests by forwarding those requests on to 
KDevProjectController which should then update/change the
model in accordance 
with the user's requests.  That is how it should happen.

Note that this isn't only the project API(s).  The way
KDevDocument is handled 
is similarly messed up at the moment.  Right now
KDevDocumentController 
behaves like a true controller, but it stores it's data in
QHash(s).  It 
should directly populate a model that inherits
QAbstractItemModel.  
KDevDocument should be made to inherit KDevItemCollection
and so on...

And then the DocumentView part should only be responsible
for displaying the 
model and forwarding on user requests to
KDevDocumentController.  Right now, 
DocumentView tries to capture all the signals of
KDevDocumentController to 
populate it's own model.  The view is playing intermediary
between the 
controller and the model.  Bad!

So, as you can see all of this needs to be reworked.  I have
plans to do so, 
but there are just so many things to do right now.

Adam

On Thursday 17 August 2006 3:57 pm, Andras Mantia wrote:
> Hi,
>
>  I'm worried about the performance problems in the
KDevFileManager
> class, due to the KDevProjectItem* way of storing the
data. If these
> classes help later with the model/view stuff, I have no
problem with
> them, but due to their tree-like structure it is hard
and time
> consuming to work with them. For example if you want to
get all the
> files that were added to the project, you would need to
go through this
> structure using a recursive method. To find if a url
was already added
> or not, you would have to do the same. To get
information about the
> item, again, a search through all of the items.
>  For this reason I have a suggestion, which is to move
allFiles from
> KDevProject to KDevFileManager. The reasoning is that
this way I could
> implement a file manager which stores the
URL->KDevProjectItem* mapping
> in a QHash or QMap and it will be much faster to
retrieve all the
> above. allFiles could have access directly to this map,
inProject could
> get the QList of allFiles and find there the URL and so
on. It would be
> even more better if allFiles would return a
QHash<KUrl,
> KDevProjectFileItem*>.
>
> Comments?
>
> Andras

_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel

On Friday 18 August 2006 06:54, Adam Treat wrote:
> Hi!
>
> Can we please put off this discussion.  There are
bigger problems with the
> project API(s) and they'll need to be completely
refactored I'm afraid. 
> This is what currently happens:
>
> KDevProjectController is the controller, but all it
really does at the
> moment is handle loading/unloading the
KDevProjectManager_part.
> KDevProjectManager_part is the view, but it calls the
importers to populate
> the data in a 'model'.  This model really isn't a
model at all.  It needs
> to be rewritten to inherit QAbstractItemModel.

So I understand now that this part of KDevelop is not yet
really ready for 
usage by Quanta. Since I am right now at Andras place and we
want to work as 
much as possible in the next days/weeks on Quanta we would
have two options:

1. jump into KDevelop development and try to implement what
is needed for 
Quanta

2. concentrate on Quanta internals that are more or less
independed from 
KDevelop

You could help us making this decision by answering the
following questions:

Do you want us to help in KDevelop? Even if we would need
some guidance to 
fullfill a task?

Which areas in KDevelop are more or less stable already and
could be used by 
us without the risk of many changes in the future?

Is there any estimation when the unstable areas in KDevelop
might become 
stable? 

Please understand that we do not want to put any pressure on
anybody. We just 
try to get the most of our limited time. 

Jens
_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel

On Friday 18 August 2006 02:20, Matt Rogers wrote:
> Have you done any profiling?

Not when I wrote the mail....

> Or are you just guessing?

Yes, but based on reading the code and thinking a little.

> Please come  
> back with hard numbers on the performance difference
between the
> current code and your proposal and we can take a look
at doing
> something different. 

I wrote some code to compare the two behavior. With a test
data of 64 
folders and 289 files loaded into the model, I searched for
all files 
trying to find if they are in the model or not (this is the
same what 
we want, find the KDevProject*Item for a url) using a tree
traversal 
and using a QHash. Using the tree traversal takes between
1-17 ms, 
while using QHash is 0-1 ms, usually 0 here (AMD64 3200+
CPU).
This certainly tells me that the current storage doesn't
scale for many 
files and directories.

I committed the code to KDevProject::inProject so you can
check my 
benchmark.


> There's just too much other stuff to do ATM. 

Jens explained why we are talking about this classes. We
wanted tohave a 
working project implementation for Quanta so we can start to
work on 
other issues.

Andras

-- 
Quanta Plus developer - http://quanta.kdewebdev.o
rg
K Desktop Environment - http://www.kde.org
_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel

On Friday 18 August 2006 12:10 pm, Andras Mantia wrote:
> On Friday 18 August 2006 02:20, Matt Rogers wrote:
> > Have you done any profiling?
>
> Not when I wrote the mail....
>
> > Or are you just guessing?
>
> Yes, but based on reading the code and thinking a
little.
>
> > Please come
> > back with hard numbers on the performance
difference between the
> > current code and your proposal and we can take a
look at doing
> > something different.
>
> I wrote some code to compare the two behavior. With a
test data of 64
> folders and 289 files loaded into the model, I searched
for all files
> trying to find if they are in the model or not (this is
the same what
> we want, find the KDevProject*Item for a url) using a
tree traversal
> and using a QHash. Using the tree traversal takes
between 1-17 ms,
> while using QHash is 0-1 ms, usually 0 here (AMD64
3200+ CPU).
> This certainly tells me that the current storage
doesn't scale for many
> files and directories.
>
> I committed the code to KDevProject::inProject so you
can check my
> benchmark.

Please don't commit benchmarks like that to the actual
code.  Besides, I agree 
that a QHash is probably the way to store this, but it
isn't such a big deal.  
We don't need to debate this right now.

> > There's just too much other stuff to do ATM.
>
> Jens explained why we are talking about this classes.
We wanted tohave a
> working project implementation for Quanta so we can
start to work on
> other issues.

Sure.  Unfortunately, right now the API is subject to a lot
of change.  There 
are no guarantees that the API you are coding against right
now won't change 
drastically even in the short term.

> Andras

_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel

On Friday 18 August 2006 9:31 am, Jens Herden wrote:
> On Friday 18 August 2006 06:54, Adam Treat wrote:
> > Hi!
> >
> > Can we please put off this discussion.  There are
bigger problems with
> > the project API(s) and they'll need to be
completely refactored I'm
> > afraid. This is what currently happens:
> >
> > KDevProjectController is the controller, but all
it really does at the
> > moment is handle loading/unloading the
KDevProjectManager_part.
> > KDevProjectManager_part is the view, but it calls
the importers to
> > populate the data in a 'model'.  This model
really isn't a model at all. 
> > It needs to be rewritten to inherit
QAbstractItemModel.
>
> So I understand now that this part of KDevelop is not
yet really ready for
> usage by Quanta. Since I am right now at Andras place
and we want to work
> as much as possible in the next days/weeks on Quanta we
would have two
> options:
>
> 1. jump into KDevelop development and try to implement
what is needed for
> Quanta
>
> 2. concentrate on Quanta internals that are more or
less independed from
> KDevelop

This is what I think is best.  The API is still going
through some big 
sweeping changes.  Personally, I think you should
concentrate on Quanta 
specifics if you can.

> You could help us making this decision by answering the
following
> questions:
>
> Do you want us to help in KDevelop? Even if we would
need some guidance to
> fullfill a task?

Well, we have some definite plans already for the project
API and for other 
parts.  Unfortunately the plans are mostly in other
people's heads ;)

> Which areas in KDevelop are more or less stable already
and could be used
> by us without the risk of many changes in the future?

The kdevplatform API is still subject to large changes.  All
of it.

> Is there any estimation when the unstable areas in
KDevelop might become
> stable?

Well, when the project stuff is converted to a true
model/view and so on, then 
I would imagine the API's for that would take shape...

> Please understand that we do not want to put any
pressure on anybody. We
> just try to get the most of our limited time.

Sure!  Most of the work right now involves changing our
shared API's.  The 
easiest way to do this is just to disable everything else
and worry about the 
core plugins.  We don't have the CPU cycles to test
everything in Quanta when 
a change is made...  For now, if it is at all possible, can
you guys 
concentrate on Quanta specific stuff?

BTW, I'm gone for the weekend camping.  Any replies will
have to wait until 
Monday at the earliest...

Cheers,

Adam

_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel

> Sure!  Most of the work right now involves changing our
shared API's.  The
> easiest way to do this is just to disable everything
else and worry about
> the core plugins.  We don't have the CPU cycles to
test everything in
> Quanta when a change is made...  For now, if it is at
all possible, can you
> guys concentrate on Quanta specific stuff?

Thanks for the reply. Yes, we can concentrate on Quanta
stuff. So we will do 
so from now on.

> BTW, I'm gone for the weekend camping.  Any replies
will have to wait until
> Monday at the earliest...

Have fun at the weekend camping!

Jens
_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel

On Friday 18 August 2006 16:31, Jens Herden wrote:
> Do you want us to help in KDevelop? Even if we would
need some guidance to
> fullfill a task?
I wish I could ask you to join us and contribute to KDevelop
with your ideas
and code but the reality is quite harsh. Too much is
changing in KDevelop and
as Adam said there're too many things that are only on
developer's heads
and nowhere else. So at this point we can neither guide you
nor provide 
much help. So I agree with Adam that you should better
concentrate on Quanta
internals at this time. But please keep an eye on KDevelop
and don't stop
telling us what you want ;) Your input/questions can be very
important.

_______________________________________________
KDevelop-devel mailing list
KDevelop-develbarney.cs.uni-potsdam.de
http://barney.cs.uni-potsdam.de/mailman/listinfo/
kdevelop-devel