db->seq never gets to end

Hopefully this list is okay ...

I am using:

        for (r = db->seq(db, &dbk, &dbd,
R_FIRST); !r;
            r = db->seq(db, &dbk, &dbd, R_NEXT))
{

But sometimes it never ends.

I added a counter. And it would get to hundreds of thousands
and data 
would repeat. I only have less than 1500 keys. I'd also get
dbd.size that 
would be hundreds of thousands of bytes (but should only be
20 bytes).

Is there any way to ask the hash(3) how many elements it
has? Or is the 
way to loop like above and count?

Is there any way to ask what the cursor is? (Is it a
number?)

I think something is causing my cursor to reset. But I can't
tell yet.

It doesn't happen all the time, but when it does it uses up
a ton of 
memory. For now, I "break" if my counter gets too
high.

Any suggestions on how to troubleshoot this would be
appreciated.

Maybe this is sign of a corrupted database. I am just
curious why it 
doesn't happen every time then. I also moved databases out
of the way to 
create new ones almost 20 times now and problem continues.

(This is in latest spamd/grey.c code on NetBSD 3.1 on
multiple systems.)

  Jeremy C. Reed

On Thu, Aug 09, 2007 at 10:57:39PM -0500, Jeremy C. Reed
wrote:
> Hopefully this list is okay ...
> 
> I am using:
> 
>         for (r = db->seq(db, &dbk, &dbd,
R_FIRST); !r;
>             r = db->seq(db, &dbk, &dbd,
R_NEXT)) {
> 
> But sometimes it never ends.
> 
> I added a counter. And it would get to hundreds of
thousands and data 
> would repeat. I only have less than 1500 keys. I'd also
get dbd.size that 
> would be hundreds of thousands of bytes (but should
only be 20 bytes).
> 
> Is there any way to ask the hash(3) how many elements
it has?

Maybe it's just me, but I didn't even realise that you
could
do a sequential scan of a hashed database. dbopen(3) says:

              R_LAST  and  R_PREV  are  available  only  for
 the
              DB_BTREE and DB_RECNO access methods  because 
they
              each  imply  that  the  keys have an inherent
order
              which does not change.

Given that statement, I don't see how R_FIRST and R_NEXT
are
any different, but I'm getting old and confused.

Was this software meant to use a different version of
Berkeley
db?

Regards,
Alistair

On Tue, 14 Aug 2007, Alistair Crooks wrote:

> On Thu, Aug 09, 2007 at 10:57:39PM -0500, Jeremy C.
Reed wrote:
> > Hopefully this list is okay ...
> > 
> > I am using:
> > 
> >         for (r = db->seq(db, &dbk,
&dbd, R_FIRST); !r;
> >             r = db->seq(db, &dbk, &dbd,
R_NEXT)) {
> > 
> > But sometimes it never ends.
> > 
> > I added a counter. And it would get to hundreds of
thousands and data 
> > would repeat. I only have less than 1500 keys. I'd
also get dbd.size that 
> > would be hundreds of thousands of bytes (but
should only be 20 bytes).
> > 
> > Is there any way to ask the hash(3) how many
elements it has?
> 
> Maybe it's just me, but I didn't even realise that you
could
> do a sequential scan of a hashed database. dbopen(3)
says:
> 
>               R_LAST  and  R_PREV  are  available  only
 for  the
>               DB_BTREE and DB_RECNO access methods 
because  they
>               each  imply  that  the  keys have an
inherent order
>               which does not change.
> 
> Given that statement, I don't see how R_FIRST and
R_NEXT are
> any different, but I'm getting old and confused.
> 
> Was this software meant to use a different version of
Berkeley
> db?
> 
> Regards,
> Alistair


I never saw any comment to your first thought.

The "spamd" software was written to use Berkeley
DB as provided in the 
base install of OpenBSD. I assume that it is near same as
ours.

My spamd databases continue to get corrupted and I have many
log entries 
like:

Aug 29 12:26:46 ca spamd[11901]: can't delete 74.220.163.20

mail.pwhosting.com <> <Annelists.reedmedia.net>
from spamd db (No buffer 
space available)

Aug 29 12:25:43 ca spamd[11901]: queueing deletion of
et.paqueta.com.br <> <Kief
ferlists.reR!UF.PUF2uUF^Q

(notice missing end >)

Aug 29 17:04:31 c-0500 spamd[28330]: can't delete
Rhein-Neckar.DE> 
<reedreedmedia.net>5^TUF^DSUF^UhUF^C from spamd db (No
buffer space 
available)


Should a Berkeley DB key and/or data be sanitized to replace
special 
characters before using?

I have been using spamd for over a year. Previously spamd
used BTREE and 
since January been using HASH. Their CVS log says:
"Using DB_BTREE for 
spamd is wrong, order is never required and the rebalancing
really slags 
big databases."

Note I have the problems on two different 3.1 servers (one
i386 and the 
other XEN3_DOMU on i386).

The side effects are that it often fails on "bogus
entry in spamd 
database" so my pf spamd-white table is not updated;
new GREY entries are 
created when WHITE entry for the IP already exists; pf table
is sometimes 
replaced with only a small amount of the WHITE entries; and
sometimes 
spamd loops through database entries thinking it has
hundreds of thousands 
of entries using up all memory (I stopped that by forcing a
max limit to 
amount of entries it can loop through).

  Jeremy C. Reed