bring speed back into KConfig

Hi, 

the "knotify4 eats cpu" bugreport seems to be
highly popular, so the reason 
for that seems to be the incredible performance loss in
KConfigIni  due to 
the "kconfig refactoring branch" being merged last
autumn by Andreas Pakulat. 

The code now uses highlevel QByteArray operations and even
worse QFile 
operations for operating on the .ini files. My testcase of
kwin.notifyrc, 
this means: 

2000 lines of config

resulting in

16000 ~QByteArray() and 16000 QByteArray() constructions
6000 syscalls (stat() and 2x llseek()) due to
QFile::readLine and 
QFile::atEof()
8000 memcpy operations

in time: 

150ms KConfig4
45ms  via QSettings of Qt 4.4
29ms via KConfig3

I think thats a bit under performing. Anyone interested in
fixing the hot 
spots in that code? I think it is fairly obvious why the
code is extremely 
slow. I wanted to take a look at it myself but I guess I'll
run out of time 
before I get anywhere. 

Thanks,
Dirk
_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

On Tuesday 15 April 2008, Dirk Mueller wrote:
> Hi,
>
> the "knotify4 eats cpu" bugreport seems to be
highly popular, so the reason
> for that seems to be the incredible performance loss in
KConfigIni  due to
> the "kconfig refactoring branch" being merged
last autumn by Andreas
> Pakulat.
>
> The code now uses highlevel QByteArray operations and
even worse QFile
> operations for operating on the .ini files. 

I.e. for parsing the ini-style rc files ?
Uh, this code was _really_ optimized before.

Alex
_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

Dnia wtorek, 15 kwietnia 2008, Dirk Mueller napisał:
> Hi,
>
> the "knotify4 eats cpu" bugreport seems to be
highly popular, so the reason
> for that seems to be the incredible performance loss in
KConfigIni  due to
> the "kconfig refactoring branch" being merged
last autumn by Andreas
> Pakulat.
>
> The code now uses highlevel QByteArray operations 

Lots of reallocations and memory copies instead of simple
moving of pointer.

> and even worse QFile 
> operations for operating on the .ini files. My testcase
of kwin.notifyrc,
> this means:

Attached very simple patch cuts parsing time by ~50% (on hot
cache). It reads 
whole config file in one go, like KDE3 did.
KDE3 also was able to mmap() file instead of reading it, but
readAll() shows 
5% on profile (using google perftools) so there are still
many bigger 
targets.

>
> 2000 lines of config
>
> resulting in
>
> 16000 ~QByteArray() and 16000 QByteArray()
constructions
> 6000 syscalls (stat() and 2x llseek()) due to
QFile::readLine and
> QFile::atEof()
> 8000 memcpy operations
>
> in time:
>
> 150ms KConfig4
> 45ms  via QSettings of Qt 4.4
> 29ms via KConfig3
>
> I think thats a bit under performing. Anyone interested
in fixing the hot
> spots in that code? I think it is fairly obvious why
the code is extremely
> slow. I wanted to take a look at it myself but I guess
I'll run out of time
> before I get anywhere.
>
> Thanks,
> Dirk
> _______________________________________________
> Kde-optimize mailing list
> Kde-optimizekde.org
> ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

Beware!

Hi, long ago I had a deep look at the kde3 code because it
caused a crash.

> > The code now uses highlevel QByteArray operations
and even worse QFile
> > operations for operating on the .ini files.

I also found (at that time) that the code was not perfectly
optimized and it 
was not capable of handling different line ending styles CR
vs LF vs CR LF 
and it did not handle various unicode encodings correctly.

> I.e. for parsing the ini-style rc files ?
> Uh, this code was _really_ optimized before.

Again: the use of mmap() caused exceptions in one network
filesystem (smb at 
that time). Please be very carefull before changing it.

Yours Jürgen



_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

On Wednesday 16 April 2008, Juergen Pfennig wrote:

> I also found (at that time) that the code was not
perfectly optimized and
> it was not capable of handling different line ending
styles CR vs LF vs CR
> LF and it did not handle various unicode encodings
correctly.

there is only one encoding: utf8, so this is lesser of a
concern. the line 
ending style might be a new problem, but it certainly wasn't
an isseu with 
kde3 

> Again: the use of mmap() caused exceptions in one
network filesystem (smb
> at that time). Please be very carefull before changing
it.

I'm also against adding mmap(), it doesn't seem to be a good
idea and has too 
many drawbacks, especially iwth networked filesystems.
However, a general 
parser speedup (in the style that qsettings e.g. did it,
however that code is 
GPL, so we can't use it). 

Greetings,
Dirk

_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

Le mardi 15 avril 2008, Dirk Mueller a écrit :
> Hi,
>
> the "knotify4 eats cpu" bugreport seems to be
highly popular, so the reason
> for that seems to be the incredible performance loss in
KConfigIni  due to
> the "kconfig refactoring branch" being merged
last autumn by Andreas
> Pakulat.

Oh, i though it was a problem in the xine backend, which is
why i did not 
spend much attention to it.
I may have a closer look.

_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

Dnia czwartek, 17 kwietnia 2008, Olivier Goffart napisał:
> Le mardi 15 avril 2008, Dirk Mueller a écrit :
> > Hi,
> >
> > the "knotify4 eats cpu" bugreport seems
to be highly popular, so the
> > reason for that seems to be the incredible
performance loss in KConfigIni
> >  due to the "kconfig refactoring branch"
being merged last autumn by
> > Andreas Pakulat.
>
> Oh, i though it was a problem in the xine backend,
which is why i did not
> spend much attention to it.
> I may have a closer look.

I did some more optimizing to  minimize copying data around.
Instead of using 
QByteArray everywhere I added class BufferFragment with very
similar (bare 
minimum used by parser) API, but operating on allocated
earlier big buffer. 
like left(), trim(), mid(), etc. are only pointer and int
operations.

Results:
 - 500x parsing of kwin.notifyrc takes 1.3s instead of 5.8s
 - KConfig from KDE3 takes 1.4s 
 - kconfig unit test pass

BufferFragment class contains very short functions (most of
them 1-3 lines) 
that could be inlined, so all definitions are in header
file. Is it OK or 
separate .cpp file is necessary?

_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

On Friday 18 April 2008, Jakub Stachowski wrote:
> Dnia czwartek, 17 kwietnia 2008, Olivier Goffart
napisał:
> > Le mardi 15 avril 2008, Dirk Mueller a écrit :
> > > Hi,
> > >
> > > the "knotify4 eats cpu" bugreport
seems to be highly popular, so the
> > > reason for that seems to be the incredible
performance loss in
> > > KConfigIni due to the "kconfig
refactoring branch" being merged last
> > > autumn by Andreas Pakulat.
> >
> > Oh, i though it was a problem in the xine backend,
which is why i did not
> > spend much attention to it.
> > I may have a closer look.
>
> I did some more optimizing to  minimize copying data
around. Instead of
> using QByteArray everywhere I added class
BufferFragment with very similar
> (bare minimum used by parser) API, but operating on
allocated earlier big
> buffer. like left(), trim(), mid(), etc. are only
pointer and int
> operations.
>
> Results:
>  - 500x parsing of kwin.notifyrc takes 1.3s instead of
5.8s
>  - KConfig from KDE3 takes 1.4s
>  - kconfig unit test pass

Very cool 

> BufferFragment class contains very short functions
(most of them 1-3 lines)
> that could be inlined, so all definitions are in header
file. Is it OK or
> separate .cpp file is necessary?

Not sure. Having it inline makes it faster but there can be
(binary ?) 
compatibility issues. I don't know if that matters here,
since it's just an 
internal header.

Alex
_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

On Friday 18 April 2008, Jakub Stachowski wrote:

> I did some more optimizing to  minimize copying data
around. Instead of
> using QByteArray everywhere I added class
BufferFragment with very similar
> (bare minimum used by parser) API, but operating on
allocated earlier big
> buffer. like left(), trim(), mid(), etc. are only
pointer and int
> operations.

Very nice approach, I like it. I haven't read the patch in
detail, but 
browsing over it I think it does the right thing. I have a
couple of small 
additions however: 

a) the readall() might kill the app if its a really big
config file. can you 
add some "chunking" so that it perhaps reads only
128kb or something like 
that at a time, even if its a really big kconfig file?

b) the methods do need clear documentation about the
expected state and 
calling convention, to make the patch easier to review. 

c) please don't add an implicit QByteArray() operator.. add
a toByteArray() 
function and place the conversion explicitely in the code,
so that it is 
clear when the "slow" conversion is happening. 

Again, many thanks for putting time into it. Great work!

Greetings,
Dirk
_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize

On Friday 18 April 2008, Jakub Stachowski wrote:
> Dnia czwartek, 17 kwietnia 2008, Olivier Goffart
napisał:
> > Le mardi 15 avril 2008, Dirk Mueller a écrit :
> > > Hi,
> > >
> > > the "knotify4 eats cpu" bugreport
seems to be highly popular, so the
> > > reason for that seems to be the incredible
performance loss in KConfigIni
> > >  due to the "kconfig refactoring
branch" being merged last autumn by
> > > Andreas Pakulat.
> >
> > Oh, i though it was a problem in the xine backend,
which is why i did not
> > spend much attention to it.
> > I may have a closer look.
> 
> I did some more optimizing to  minimize copying data
around. Instead of using 
> QByteArray everywhere I added class BufferFragment with
very similar (bare 
> minimum used by parser) API, but operating on allocated
earlier big buffer. 
> like left(), trim(), mid(), etc. are only pointer and
int operations.

Interesting. Sounds a bit like the new (since Qt-4.3)
QStringRef, but for 8-bit data instead of unicode data...

-- 
David Faure, faurekde.org, sponsored by Trolltech to work on
KDE,
Konqueror (http://www.konqueror.org
), and KOffice (http://www.koffice.org).
_______________________________________________
Kde-optimize mailing list
Kde-optimizekde.org
ht
tps://mail.kde.org/mailman/listinfo/kde-optimize