Small (un-representative) benchmark on sqlite with blobs

Hi,

David, Kris and myself had a (short) discussion about how to
persist
duchain data. Especially with multi-projects in mind we
might get quite
some data.

As David said he had issues with bdb in KDevelop3.4 we
pondered wether
using a sql db with blobs might be a good idea (with a
really simple
table layout). We can use the Qt db classes here to not have
to fiddle
with the low-level API of the dbs.

So I created a short testcase for reading a 140 Meg file
from disk,
storing it into the db and retrieving it again. All this
done on sqlite,
because mysql and postgresql need more setup to be done
(with sqlite all
one needs to provide is a database name).

I don't have the exact numbers for that anymore, but it
mainly is like
this:
1x140Meg testfile
Fetching from db with cold cache: about 1 minutes
Writing to db: about 40 seconds
fetching from db with cache: 10 seconds
Reading of file from disk with QFile and cold caches: 10
seconds

Now I had another 30 minutes and extended that to a more
suitable test,
using 20 35 meg files of binary data (not sure wether thats
the duchain
size later on, but well...). Storing these 20 files looks
like this:

,----
| andreasmorpheus:~/temp/testsqlite>./testsqlite -create
| true
| Drop table result: QSqlError(1, "Unable to execute
statement", "no such table: f1")
| create table result: QSqlError(-1, "",
"")
| "Reading Start(1xaa):" QDateTime("Mo Jun 25
02:23:57 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:24:01
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:24:01
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:24:16
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(2xaa):" QDateTime("Mo Jun 25
02:24:16 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:24:19
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:24:19
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:24:22
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(3xaa):" QDateTime("Mo Jun 25
02:24:22 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:24:24
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:24:24
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:24:29
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(4xaa):" QDateTime("Mo Jun 25
02:24:29 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:24:31
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:24:31
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:24:34
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(5xaa):" QDateTime("Mo Jun 25
02:24:34 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:24:38
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:24:38
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:24:41
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(1xab):" QDateTime("Mo Jun 25
02:24:41 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:24:44
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:24:44
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:24:49
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(2xab):" QDateTime("Mo Jun 25
02:24:49 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:24:55
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:24:55
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:25:01
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(3xab):" QDateTime("Mo Jun 25
02:25:01 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:25:06
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:25:06
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:25:09
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(4xab):" QDateTime("Mo Jun 25
02:25:09 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:25:14
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:25:14
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:25:18
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(5xab):" QDateTime("Mo Jun 25
02:25:18 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:25:22
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:25:22
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:25:26
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(1xac):" QDateTime("Mo Jun 25
02:25:26 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:25:33
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:25:33
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:25:36
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(2xac):" QDateTime("Mo Jun 25
02:25:36 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:25:39
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:25:39
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:25:42
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(3xac):" QDateTime("Mo Jun 25
02:25:42 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:25:45
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:25:45
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:25:50
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(4xac):" QDateTime("Mo Jun 25
02:25:50 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:25:53
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:25:53
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:25:58
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(5xac):" QDateTime("Mo Jun 25
02:25:58 2007")
| Opening file: true
| read bytes: 36700160
| Reading Done: QDateTime("Mo Jun 25 02:26:01
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:26:01
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:26:05
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(1xad):" QDateTime("Mo Jun 25
02:26:05 2007")
| Opening file: true
| read bytes: 34254415
| Reading Done: QDateTime("Mo Jun 25 02:26:08
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:26:08
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:26:12
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(2xad):" QDateTime("Mo Jun 25
02:26:12 2007")
| Opening file: true
| read bytes: 34254415
| Reading Done: QDateTime("Mo Jun 25 02:26:15
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:26:15
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:26:17
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(3xad):" QDateTime("Mo Jun 25
02:26:17 2007")
| Opening file: true
| read bytes: 34254415
| Reading Done: QDateTime("Mo Jun 25 02:26:20
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:26:20
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:26:23
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(4xad):" QDateTime("Mo Jun 25
02:26:23 2007")
| Opening file: true
| read bytes: 34254415
| Reading Done: QDateTime("Mo Jun 25 02:26:26
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:26:26
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:26:30
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
| "Reading Start(5xad):" QDateTime("Mo Jun 25
02:26:30 2007")
| Opening file: true
| read bytes: 34254415
| Reading Done: QDateTime("Mo Jun 25 02:26:33
2007")
| Executing: "insert into f1(num,data)
values(:num,:blob);"
| Inserting Start: QDateTime("Mo Jun 25 02:26:33
2007")
| true
| Inserting End: QDateTime("Mo Jun 25 02:26:36
2007")
| Inserting result: QSqlError(-1, "",
"")
| lines inserted: 1
`----

Sorry, for the long output. Now as one can see storing such
files is
not taking that much time with hot-caches (the first,
cold-cache case
does take a bit longer).

Now reading the whole data from the database using 1 single
select looks
like this (with somewhat hot caches, I'll do a
cold-cache-case
tomorrow):

,----
| andreasmorpheus:~/temp/testsqlite>./testsqlite
| true
| Fetching Start: QDateTime("Mo Jun 25 02:34:40
2007")
| lines fetched: 0
| Fetching Row number 0 : QDateTime("Mo Jun 25 02:34:43
2007")
| Fetched size: 36700160
| Fetching Row number 0 done: QDateTime("Mo Jun 25
02:34:43 2007")
| Fetching Row number 1 : QDateTime("Mo Jun 25 02:34:45
2007")
| Fetched size: 36700160
| Fetching Row number 1 done: QDateTime("Mo Jun 25
02:34:45 2007")
| Fetching Row number 2 : QDateTime("Mo Jun 25 02:34:50
2007")
| Fetched size: 36700160
| Fetching Row number 2 done: QDateTime("Mo Jun 25
02:34:50 2007")
| Fetching Row number 3 : QDateTime("Mo Jun 25 02:34:52
2007")
| Fetched size: 36700160
| Fetching Row number 3 done: QDateTime("Mo Jun 25
02:34:52 2007")
| Fetching Row number 4 : QDateTime("Mo Jun 25 02:34:55
2007")
| Fetched size: 36700160
| Fetching Row number 4 done: QDateTime("Mo Jun 25
02:34:55 2007")
| Fetching Row number 5 : QDateTime("Mo Jun 25 02:34:57
2007")
| Fetched size: 36700160
| Fetching Row number 5 done: QDateTime("Mo Jun 25
02:34:57 2007")
| Fetching Row number 6 : QDateTime("Mo Jun 25 02:35:00
2007")
| Fetched size: 36700160
| Fetching Row number 6 done: QDateTime("Mo Jun 25
02:35:00 2007")
| Fetching Row number 7 : QDateTime("Mo Jun 25 02:35:02
2007")
| Fetched size: 36700160
| Fetching Row number 7 done: QDateTime("Mo Jun 25
02:35:02 2007")
| Fetching Row number 8 : QDateTime("Mo Jun 25 02:35:03
2007")
| Fetched size: 36700160
| Fetching Row number 8 done: QDateTime("Mo Jun 25
02:35:03 2007")
| Fetching Row number 9 : QDateTime("Mo Jun 25 02:35:05
2007")
| Fetched size: 36700160
| Fetching Row number 9 done: QDateTime("Mo Jun 25
02:35:05 2007")
| Fetching Row number 10 : QDateTime("Mo Jun 25
02:35:17 2007")
| Fetched size: 36700160
| Fetching Row number 10 done: QDateTime("Mo Jun 25
02:35:17 2007")
| Fetching Row number 11 : QDateTime("Mo Jun 25
02:35:46 2007")
| Fetched size: 36700160
| Fetching Row number 11 done: QDateTime("Mo Jun 25
02:35:46 2007")
| Fetching Row number 12 : QDateTime("Mo Jun 25
02:36:36 2007")
| Fetched size: 36700160
| Fetching Row number 12 done: QDateTime("Mo Jun 25
02:36:36 2007")
| Fetching Row number 13 : QDateTime("Mo Jun 25
02:36:54 2007")
| Fetched size: 36700160
| Fetching Row number 13 done: QDateTime("Mo Jun 25
02:36:54 2007")
| Fetching Row number 14 : QDateTime("Mo Jun 25
02:37:18 2007")
| Fetched size: 36700160
| Fetching Row number 14 done: QDateTime("Mo Jun 25
02:37:18 2007")
| Fetching Row number 15 : QDateTime("Mo Jun 25
02:37:26 2007")
| Fetched size: 34254415
| Fetching Row number 15 done: QDateTime("Mo Jun 25
02:37:26 2007")
| Fetching Row number 16 : QDateTime("Mo Jun 25
02:37:36 2007")
| Fetched size: 34254415
| Fetching Row number 16 done: QDateTime("Mo Jun 25
02:37:36 2007")
| Fetching Row number 17 : QDateTime("Mo Jun 25
02:37:47 2007")
| Fetched size: 34254415
| Fetching Row number 17 done: QDateTime("Mo Jun 25
02:37:47 2007")
| Fetching Row number 18 : QDateTime("Mo Jun 25
02:37:55 2007")
| Fetched size: 34254415
| Fetching Row number 18 done: QDateTime("Mo Jun 25
02:37:55 2007")
| Fetching Row number 19 : QDateTime("Mo Jun 25
02:38:12 2007")
| Fetched size: 34254415
| Fetching Row number 19 done: QDateTime("Mo Jun 25
02:38:12 2007")
| Fetching End: QDateTime("Mo Jun 25 02:38:12
2007")
`----

The first 10 entries can be fetched quite good, but then it
gets really
slow. I don't know why this is the case, but I guess the
problem is that
the whole 630 meg database file doesn't fit into my 1G
memory
alltogether. so its delivering the other entries from disk
by doing seek
or some such...

So now everybody can think wether all this is fast eough or
not. Test on
your own disk, the main.cpp and .pro files are attached (it
reads
<num>xa<a-d> and uses foobar as db name).

Andreas

-- 
You will be Told about it Tomorrow.  Go Home and Prepare
Thyself.

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On 25.06.07 02:40:33, Andreas Pakulat wrote:
> Hi,
> 
> David, Kris and myself had a (short) discussion about
how to persist
> duchain data. Especially with multi-projects in mind we
might get quite
> some data.
> 
> As David said he had issues with bdb in KDevelop3.4 we
pondered wether
> using a sql db with blobs might be a good idea (with a
really simple
> table layout). We can use the Qt db classes here to not
have to fiddle
> with the low-level API of the dbs.
> 
> So I created a short testcase for reading a 140 Meg
file from disk,
> storing it into the db and retrieving it again. All
this done on sqlite,
> because mysql and postgresql need more setup to be done
(with sqlite all
> one needs to provide is a database name).

Here are the results from a cold-cache-run. Fetching first:
andreasmorpheus:~/temp/testsqlite>./testsqlite
true
Fetching Start: QDateTime("Mo Jun 25 12:23:05
2007")
lines fetched: 0
Fetching Row number 0 : QDateTime("Mo Jun 25 12:23:08
2007")
Fetched size: 36700160
Fetching Row number 0 done: QDateTime("Mo Jun 25
12:23:08 2007")
Fetching Row number 1 : QDateTime("Mo Jun 25 12:23:10
2007")
Fetched size: 36700160
Fetching Row number 1 done: QDateTime("Mo Jun 25
12:23:10 2007")
Fetching Row number 2 : QDateTime("Mo Jun 25 12:23:18
2007")
Fetched size: 36700160
Fetching Row number 2 done: QDateTime("Mo Jun 25
12:23:18 2007")
Fetching Row number 3 : QDateTime("Mo Jun 25 12:23:20
2007")
Fetched size: 36700160
Fetching Row number 3 done: QDateTime("Mo Jun 25
12:23:20 2007")
Fetching Row number 4 : QDateTime("Mo Jun 25 12:23:23
2007")
Fetched size: 36700160
Fetching Row number 4 done: QDateTime("Mo Jun 25
12:23:23 2007")
Fetching Row number 5 : QDateTime("Mo Jun 25 12:23:25
2007")
Fetched size: 36700160
Fetching Row number 5 done: QDateTime("Mo Jun 25
12:23:25 2007")
Fetching Row number 6 : QDateTime("Mo Jun 25 12:23:30
2007")
Fetched size: 36700160
Fetching Row number 6 done: QDateTime("Mo Jun 25
12:23:30 2007")
Fetching Row number 7 : QDateTime("Mo Jun 25 12:23:32
2007")
Fetched size: 36700160
Fetching Row number 7 done: QDateTime("Mo Jun 25
12:23:32 2007")
Fetching Row number 8 : QDateTime("Mo Jun 25 12:23:34
2007")
Fetched size: 36700160
Fetching Row number 8 done: QDateTime("Mo Jun 25
12:23:34 2007")
Fetching Row number 9 : QDateTime("Mo Jun 25 12:23:36
2007")
Fetched size: 36700160
Fetching Row number 9 done: QDateTime("Mo Jun 25
12:23:36 2007")
Fetching Row number 10 : QDateTime("Mo Jun 25 12:23:41
2007")
Fetched size: 36700160
Fetching Row number 10 done: QDateTime("Mo Jun 25
12:23:41 2007")
Fetching Row number 11 : QDateTime("Mo Jun 25 12:23:47
2007")
Fetched size: 36700160
Fetching Row number 11 done: QDateTime("Mo Jun 25
12:23:47 2007")
Fetching Row number 12 : QDateTime("Mo Jun 25 12:23:53
2007")
Fetched size: 36700160
Fetching Row number 12 done: QDateTime("Mo Jun 25
12:23:53 2007")
Fetching Row number 13 : QDateTime("Mo Jun 25 12:23:58
2007")
Fetched size: 36700160
Fetching Row number 13 done: QDateTime("Mo Jun 25
12:23:58 2007")
Fetching Row number 14 : QDateTime("Mo Jun 25 12:24:04
2007")
Fetched size: 36700160
Fetching Row number 14 done: QDateTime("Mo Jun 25
12:24:04 2007")
Fetching Row number 15 : QDateTime("Mo Jun 25 12:24:12
2007")
Fetched size: 34254415
Fetching Row number 15 done: QDateTime("Mo Jun 25
12:24:12 2007")
Fetching Row number 16 : QDateTime("Mo Jun 25 12:24:17
2007")
Fetched size: 34254415
Fetching Row number 16 done: QDateTime("Mo Jun 25
12:24:17 2007")
Fetching Row number 17 : QDateTime("Mo Jun 25 12:24:23
2007")
Fetched size: 34254415
Fetching Row number 17 done: QDateTime("Mo Jun 25
12:24:23 2007")
Fetching Row number 18 : QDateTime("Mo Jun 25 12:24:28
2007")
Fetched size: 34254415
Fetching Row number 18 done: QDateTime("Mo Jun 25
12:24:28 2007")
Fetching Row number 19 : QDateTime("Mo Jun 25 12:24:32
2007")
Fetched size: 34254415
Fetching Row number 19 done: QDateTime("Mo Jun 25
12:24:32 2007")
Fetching End: QDateTime("Mo Jun 25 12:24:33
2007")

Surprisingly this was way faster, but I guess the reason is
that my
memory was nearly empty as this was done after a cold boot.

Creation of the db looks like this (dropping the table was
taking quite
some time, btw):

andreasmorpheus:~/temp/testsqlite>./testsqlite -create
true
Drop table result: QSqlError(-1, "",
"")
create table result: QSqlError(-1, "",
"")
"Reading Start(1xaa):" QDateTime("Mo Jun 25
12:28:03 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:28:08
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:28:08
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:28:12
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(2xaa):" QDateTime("Mo Jun 25
12:28:12 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:28:14
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:28:14
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:28:18
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(3xaa):" QDateTime("Mo Jun 25
12:28:18 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:28:20
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:28:20
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:28:24
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(4xaa):" QDateTime("Mo Jun 25
12:28:24 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:28:26
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:28:26
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:28:29
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(5xaa):" QDateTime("Mo Jun 25
12:28:29 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:28:33
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:28:33
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:28:37
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(1xab):" QDateTime("Mo Jun 25
12:28:37 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:28:40
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:28:40
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:28:43
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(2xab):" QDateTime("Mo Jun 25
12:28:43 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:28:45
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:28:45
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:28:49
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(3xab):" QDateTime("Mo Jun 25
12:28:49 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:28:51
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:28:51
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:28:55
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(4xab):" QDateTime("Mo Jun 25
12:28:55 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:28:58
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:28:58
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:29:01
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(5xab):" QDateTime("Mo Jun 25
12:29:01 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:29:04
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:29:04
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:29:10
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(1xac):" QDateTime("Mo Jun 25
12:29:10 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:29:16
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:29:16
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:29:23
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(2xac):" QDateTime("Mo Jun 25
12:29:23 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:29:25
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:29:25
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:29:32
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(3xac):" QDateTime("Mo Jun 25
12:29:32 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:29:34
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:29:34
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:29:42
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(4xac):" QDateTime("Mo Jun 25
12:29:42 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:29:45
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:29:45
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:29:55
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(5xac):" QDateTime("Mo Jun 25
12:29:55 2007")
Opening file: true
read bytes: 36700160
Reading Done: QDateTime("Mo Jun 25 12:29:56
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:29:56
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:30:12
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(1xad):" QDateTime("Mo Jun 25
12:30:12 2007")
Opening file: true
read bytes: 34254415
Reading Done: QDateTime("Mo Jun 25 12:30:16
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:30:16
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:30:42
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(2xad):" QDateTime("Mo Jun 25
12:30:42 2007")
Opening file: true
read bytes: 34254415
Reading Done: QDateTime("Mo Jun 25 12:30:45
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:30:45
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:31:02
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(3xad):" QDateTime("Mo Jun 25
12:31:02 2007")
Opening file: true
read bytes: 34254415
Reading Done: QDateTime("Mo Jun 25 12:31:03
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:31:03
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:31:14
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(4xad):" QDateTime("Mo Jun 25
12:31:14 2007")
Opening file: true
read bytes: 34254415
Reading Done: QDateTime("Mo Jun 25 12:31:16
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:31:16
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:31:25
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1
"Reading Start(5xad):" QDateTime("Mo Jun 25
12:31:25 2007")
Opening file: true
read bytes: 34254415
Reading Done: QDateTime("Mo Jun 25 12:31:26
2007")
Executing: "insert into f1(num,data)
values(:num,:blob);"
Inserting Start: QDateTime("Mo Jun 25 12:31:26
2007")
true
Inserting End: QDateTime("Mo Jun 25 12:31:36
2007")
Inserting result: QSqlError(-1, "", "")
lines inserted: 1

Andreas

-- 
Your love life will be happy and harmonious.

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On 25.06.07 10:49:37, Roberto Raggi wrote:
> 
> Hi,
> 
> Il giorno 25/giu/07, alle ore 02:40, Andreas Pakulat ha
scritto:
> 
> > Hi,
> >
> > David, Kris and myself had a (short) discussion
about how to persist
> > duchain data. Especially with multi-projects in
mind we might get  
> > quite
> > some data.
> 
> My problem with SQL is its textual representation. In
general the  
> engines are pretty fasts, but unfortunately you have to
parse and  
> generate SQL statements for every single operation.
Think about the  
> result SQL statement when you ask for all the symbols
in the global  
> namespace  That
takes quite a bit of resources, especially in a  
> real time environment.

Nobody said we're going to store the data in a relational
schema. David
mentioned something like 2 Map-like tables. So the
SQL-statements are
surely no problem. The real data will be stored as Blob,
which is the
whole reason I did this experiment.

> KDevelop is not a compiler, you don't need to  
> store a lot of information in its persistent storage. I
think you can  
> use an approach similar to the per project PCS file I
did for  
> KDevelop 3. The project's PCS file is a dump of the
Code Model,  

a) there's no code-model
b) AFAIK, the duchain is global for all projects, to allow
to lookup
symbols from other projects and save space and time when
projects use
common headers (like a handful of Qt projects). (Kris or
David know this
exactly)

Andreas

-- 
You may get an opportunity for advancement today.  Watch
it!

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

Hi!

Il giorno 25/giu/07, alle ore 12:43, Andreas Pakulat ha
scritto:

> On 25.06.07 10:49:37, Roberto Raggi wrote:
>
> Nobody said we're going to store the data in a
relational schema.  
> David
> mentioned something like 2 Map-like tables. So the
SQL-statements are
> surely no problem. The real data will be stored as
Blob, which is the
> whole reason I did this experiment.

interesting.. so why SQL? you can pretty much use everything
else for  
that.

>
>> KDevelop is not a compiler, you don't need to
>> store a lot of information in its persistent
storage. I think you can
>> use an approach similar to the per project PCS file
I did for
>> KDevelop 3. The project's PCS file is a dump of the
Code Model,
>
> a) there's no code-model
I was talking about kdev3's PCS file 

> b) AFAIK, the duchain is global for all projects, to
allow to lookup
> symbols from other projects and save space and time
when projects use
> common headers (like a handful of Qt projects). (Kris
or David know  
> this
> exactly)
hmm, I'm confused. I think we are talking about different
things   
A DU chain is a Definition-Use chain. That means if you have
a  
definition (e.g. the method setObjectName of QObject) you
can ask for  
the uses of that definition in the current translation unit.
As you  
can see this thing is very dynamic. For example, if at some
point you  
call the method setObjectName() from the function whatver()
of your  
project you will change the DU chain, and that's OK, but as
you can  
see this is just a chain of a definition and a use, and you
need to  
define what is a use and what is a definition (== you need
some sort  
of model). Then you need to know where the definition is
created (==  
scope), and so on.. Maybe the DU chain in KDevelop 4 is a
"super DU  
chain" that is a mixture of scopes, symbol tables,
environments, & co.

ciao robe


_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On 25.06.07 13:20:55, Roberto Raggi wrote:
> Il giorno 25/giu/07, alle ore 12:43, Andreas Pakulat ha
scritto:
> > On 25.06.07 10:49:37, Roberto Raggi wrote:
> >
> > Nobody said we're going to store the data in a
relational schema.  
> > David
> > mentioned something like 2 Map-like tables. So the
SQL-statements are
> > surely no problem. The real data will be stored as
Blob, which is the
> > whole reason I did this experiment.
> 
> interesting.. so why SQL? you can pretty much use
everything else for  
> that.

As I already said: David had a couple of problems with the
BerkleyDB in
KDevelop3. I don't care wether we use SQL or some binary
database,
except that for SQL we already have pretty much everything
in place
(high-level API, sqlite plugin that is almost always built
for QT,

> >> KDevelop is not a compiler, you don't need to
> >> store a lot of information in its persistent
storage. I think you can
> >> use an approach similar to the per project PCS
file I did for
> >> KDevelop 3. The project's PCS file is a dump
of the Code Model,
> >
> > a) there's no code-model
> I was talking about kdev3's PCS file 
> 
> > b) AFAIK, the duchain is global for all projects,
to allow to lookup
> > symbols from other projects and save space and
time when projects use
> > common headers (like a handful of Qt projects).
(Kris or David know  
> > this
> > exactly)
> hmm, I'm confused. I think we are talking about
different things   
> A DU chain is a Definition-Use chain. That means if you
have a  
> definition (e.g. the method setObjectName of QObject)
you can ask for  
> the uses of that definition in the current translation
unit. As you  
> can see this thing is very dynamic. For example, if at
some point you  
> call the method setObjectName() from the function
whatver() of your  
> project you will change the DU chain, and that's OK,
but as you can  
> see this is just a chain of a definition and a use, and
you need to  
> define what is a use and what is a definition (== you
need some sort  
> of model). Then you need to know where the definition
is created (==  
> scope), and so on.. Maybe the DU chain in KDevelop 4 is
a "super DU  
> chain" that is a mixture of scopes, symbol tables,
environments, & co.

I don't know how the  DUChain in KDevelop4 works (what it
builds and
what it doesn't build). I do know however that Hamish
suggested to store
the duchain and the AST together, instead of building a
codemodel and
storing that.

Andreas

-- 
Hope that the day after you die is a nice day.

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On 6/25/07, Andreas Pakulat <apakugmx.de> wrote:
> On 25.06.07 13:20:55, Roberto Raggi wrote:
> > Il giorno 25/giu/07, alle ore 12:43, Andreas
Pakulat ha scritto:
> > > On 25.06.07 10:49:37, Roberto Raggi wrote:
> > >
> > > Nobody said we're going to store the data in
a relational schema.
> > > David
> > > mentioned something like 2 Map-like tables.
So the SQL-statements are
> > > surely no problem. The real data will be
stored as Blob, which is the
> > > whole reason I did this experiment.
> >
> > interesting.. so why SQL? you can pretty much use
everything else for
> > that.
>
> As I already said: David had a couple of problems with
the BerkleyDB in
> KDevelop3. I don't care wether we use SQL or some
binary database,
> except that for SQL we already have pretty much
everything in place
> (high-level API, sqlite plugin that is almost always
built for QT,
>

But.. in KDev3 the database was actually used for queries..
if you are
saying it should all go into a blob, why not simply use a
file on
disk? What does the the database give you at all in this
case?

And while on topic.. all data in one blob?? You're only
interested in
persistence here? It seems to suggest that you will have all
of the
data (duchain, whatever) in memory during normal operation,
but surely
that will use up way too much memory? KDev3 kept (and
persisted) only
the project PCS in memory, and used bdb for lookups against
external
libraries. This to keep the memory usage down. (And with
large
projects, this was a bit too heavy too.)

// jens

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On 25.06.07 15:59:16, Jens Dagerbo wrote:
> On 6/25/07, Andreas Pakulat <apakugmx.de> wrote:
> > On 25.06.07 13:20:55, Roberto Raggi wrote:
> > > Il giorno 25/giu/07, alle ore 12:43, Andreas
Pakulat ha scritto:
> > > > On 25.06.07 10:49:37, Roberto Raggi
wrote:
> > > >
> > > > Nobody said we're going to store the
data in a relational schema.
> > > > David
> > > > mentioned something like 2 Map-like
tables. So the SQL-statements are
> > > > surely no problem. The real data will be
stored as Blob, which is the
> > > > whole reason I did this experiment.
> > >
> > > interesting.. so why SQL? you can pretty much
use everything else for
> > > that.
> >
> > As I already said: David had a couple of problems
with the BerkleyDB in
> > KDevelop3. I don't care wether we use SQL or some
binary database,
> > except that for SQL we already have pretty much
everything in place
> > (high-level API, sqlite plugin that is almost
always built for QT,
> >
> 
> But.. in KDev3 the database was actually used for
queries.. if you are
> saying it should all go into a blob, why not simply use
a file on
> disk? What does the the database give you at all in
this case?

Nothing, except that we have all in one file.

> And while on topic.. all data in one blob?? You're only
interested in
> persistence here? It seems to suggest that you will
have all of the
> data (duchain, whatever) in memory during normal
operation, but surely
> that will use up way too much memory? KDev3 kept (and
persisted) only
> the project PCS in memory, and used bdb for lookups
against external
> libraries. This to keep the memory usage down. (And
with large
> projects, this was a bit too heavy too.)

I'll leave that for David, because I don't know how large
AST+DUChain
stuff might get.

Andreas

-- 
You will be married within a year, and divorced within two.

_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

Hi!

Il giorno 25/giu/07, alle ore 16:33, Andreas Pakulat ha
scritto:
>
>> And while on topic.. all data in one blob?? You're
only interested in
>> persistence here? It seems to suggest that you will
have all of the
>> data (duchain, whatever) in memory during normal
operation, but  
>> surely
>> that will use up way too much memory? KDev3 kept
(and persisted) only
>> the project PCS in memory, and used bdb for lookups
against external
>> libraries. This to keep the memory usage down. (And
with large
>> projects, this was a bit too heavy too.)
>
> I'll leave that for David, because I don't know how
large AST+DUChain
> stuff might get.

Please, don't save the AST. It's not a good idea. The AST
has a lot  
of useless stuff  and I
think you will end up with some GB of data  
for a project of the size of kdelibs.

ciao robe


_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On Monday 25 June 2007 15:59:16 Jens Dagerbo wrote:
> On 6/25/07, Andreas Pakulat <apakugmx.de> wrote:
> But.. in KDev3 the database was actually used for
queries.. if you are
> saying it should all go into a blob, why not simply use
a file on
> disk? What does the the database give you at all in
this case?
>
> And while on topic.. all data in one blob?? You're only
interested in
I was thinking about the file-on-disk thing too. But noone
is talking about 
storing everything in one blob here, of course it would be
many many separate 
blobs, that could be loaded each on demand.

> persistence here? It seems to suggest that you will
have all of the
> data (duchain, whatever) in memory during normal
operation, but surely
> that will use up way too much memory? KDev3 kept (and
persisted) only
> the project PCS in memory, and used bdb for lookups
against external
> libraries. This to keep the memory usage down. (And
with large
> projects, this was a bit too heavy too.)
>
> // jens

The general idea is that every du-chain ever parsed is
stored in a database on 
disk, and can be loaded on-demand when needed. That way,
after some usage, we 
would have a database for most possible versions of all
commonly used 
header-files in the database, and parsing could become
lightning fast even on 
a new project, while still being correct.

The du-chain database would consist of this:
- A mapping from absolute file-names to a list of
environment-matching nodes
- One environment-matching node for each du-chain
- All the du-chains

The environment-matching nodes will be needed to decide
whether one of the 
stored du-chains can be used in a given environment.

We cannot store it all in a flat file, because everything
may change on the 
fly as code changes, including the environment-matching
nodes and the 
du-chains. Any node should also be deletable from the
database any time(for 
example when one a file changes, all other versions of the
file including the 
matching-nodes can be deleted). Also it would be hard to
just load a specific 
needed du-chain in a flat file.

The only option would probably be storing everything in
separate files. But 
that might either become very slow when a lot of small
du-chains need to be 
loaded, or we would need to implement some complex logic to
group du-chains 
that commonly appear together into bigger files, so we could
load them all at 
once.

I think the best solution might be storing it all in a very
simple and 
efficient database, because the management-complexity would
stay within a 
stable database-implementation and we wouldn't need to care
much about it.

greetings, David


_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel

On Tuesday 26 June 2007 10:23:10 Jens Dagerbo wrote:
> I guess this may or may not be sufficient, depending on
what features
> you want to support.
> Example where I don't expect it to work: "Goto
declaration" for a
> referenced function in a correctly included header
would reasonably be
> cheap, but a cool feature would be if you could ask
KDevelop "given
> this here printf() function, what header do I need to
include?". With
> a real query frontend to a database, that could be a
really cheap
> operation, with what you suggest it probably can't.
>
> // jens

A separete symbol-table might be used for this. Once we use
a database, we 
could put the symbol-table into that database for simple
queries.
Currently that's a little too far for me though. 

greetings, David


_______________________________________________
KDevelop-devel mailing list
KDevelop-develkdevelop.org
https://barney.cs.uni-potsdam.de/mailman/listinf
o/kdevelop-devel