6.2-Release and Clamd 0.90 with libpthread.so

Hi,

After upgrading to clamd 0.90 I found that clamd
doesn't work well with libpthread.so.2.

Using libmap.conf entries for libthr.so.2 or libc_r.so.6
worked fine. With both thread libraries clamd works as it
should.

The effect with libpthread is that clamd works very very
slow.
It's only able to scan 1-2 files in 5 seconds instead of
400-500 in the same time with libthr/libc_r.

A ktrace shows the following behaviour:

clamd 0.90 and libpthread.so.2

   1735 clamd    RET   fork 0
   1735 clamd    CALL  kse_release(0xbe7eac20)
   1735 clamd    CALL  kse_wakeup(0x9e4678c)
   1735 clamd    RET   kse_wakeup 0
   1735 clamd    CALL  kse_release(0x9ea3f10)
   1735 clamd    RET   kse_release 0
   1735 clamd    RET   kse_release 0
   1735 clamd    CALL  kse_release(0x805df4c)
   1735 clamd    RET   kse_release 0
   1735 clamd    CALL  kse_release(0x9eabedc)
   1735 clamd    RET   kse_release 0
   1735 clamd    CALL  kse_release(0x805dedc)
   1735 clamd    CALL  kse_wakeup(0x9e4670c)
   1735 clamd    CALL  kse_release(0x9eabedc)
   1735 clamd    RET   kse_release 0
   1735 clamd    RET   kse_wakeup 0
   1735 clamd    CALL  kse_wakeup(0x9e4680c)
   1735 clamd    RET   fork 0
   1735 clamd    CALL  kse_wakeup(0x805740c)
   1735 clamd    RET   kse_wakeup 0
   1735 clamd    RET   kse_release 0
   1735 clamd    CALL  kse_release(0x9ea7f10)
   1735 clamd    RET   kse_release 0
   1735 clamd    CALL  kse_release(0xbef61c10)
   1735 clamd    CALL  kse_release(0x805dedc)
   1735 clamd    CALL  kse_wakeup(0x805740c)
   1735 clamd    RET   kse_wakeup 0
   1735 clamd    RET   kse_release 0
   1735 clamd    CALL  kse_release(0xbec30f50)
   1735 clamd    CALL  kse_wakeup(0x9e4678c)
   1735 clamd    RET   kse_wakeup 0
   1735 clamd    RET   kse_release 0
   1735 clamd    CALL  kse_release(0xbe8fbb90)
   1735 clamd    CALL  kse_wakeup(0x9e4670c)
   1735 clamd    RET   kse_wakeup 0
   1735 clamd    RET   kse_release 0
   1735 clamd    CALL  kse_release(0x9ea7f40)
   1735 clamd    CALL  kse_wakeup(0x9e4680c)
   1735 clamd    RET   kse_release 0
   1735 clamd    RET   kse_release 0
   1735 clamd    RET   kse_wakeup 0
   1735 clamd    CALL  kse_release(0x9ea7ed0)
   1735 clamd    CALL  kse_wakeup(0x805740c)
   1735 clamd    CALL  kse_release(0xbef61bd0)
   1735 clamd    RET   kse_wakeup 0
   1735 clamd    RET   kse_release 0
   1735 clamd    CALL  kse_release(0xbee50bd0)
   1735 clamd    CALL  kse_wakeup(0x9e4678c)
   1735 clamd    RET   kse_wakeup 0
   1735 clamd    RET   kse_release 0

After ~10000 lines which show all like this, clamd actually
opens and scans something, then another ~10000 lines follow
...

I guess it's a bug in libpthread, but it really doesn't
work fine with this lib. Or is it bad programming in clamav
?

Any comments ?

Martin

Martin Blapp, <mbimp.ch> <mbrFreeBSD.org>
------------------------------------------------------------
------
ImproWare AG, UNIXSP & ISP, Zurlindenstrasse 29, 4133
Pratteln, CH
Phone: +41 61 826 93 00 Fax: +41 61 826 93 01
PGP: <finger -l mbrfreebsd.org>
PGP Fingerprint: B434 53FC C87C FE7B 0A18 B84C 8686 EF22
D300 551E
------------------------------------------------------------
------
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Hi,

> Probably because it's fork()ing and trying to call
> non-async-signal-safe functions without doing an
exec()?

The strange thing is that I can't see any any fork() calls
with libc_r or libthr. How's that possible ? Or do you
think
that its stuck at the beginning ? But why does it work then
for some degree ?

Martin
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

On Tue, 20 Feb 2007, Martin Blapp wrote:

>
> Hi,
>
>> Probably because it's fork()ing and trying to call
>> non-async-signal-safe functions without doing an
exec()?
>
> The strange thing is that I can't see any any fork()
calls
> with libc_r or libthr. How's that possible ? Or do you
think
> that its stuck at the beginning ? But why does it work
then
> for some degree ?

I dunno, maybe ktrace was confused and it wasn't a fork()?

-- 
DE
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

On Tue, 20 Feb 2007, Martin Blapp wrote:

>
> Hi,
>
> After upgrading to clamd 0.90 I found that clamd
> doesn't work well with libpthread.so.2.
>
> Using libmap.conf entries for libthr.so.2 or
libc_r.so.6
> worked fine. With both thread libraries clamd works as
it
> should.
>
> The effect with libpthread is that clamd works very
very slow.
> It's only able to scan 1-2 files in 5 seconds instead
of
> 400-500 in the same time with libthr/libc_r.
>
> A ktrace shows the following behaviour:
>
> clamd 0.90 and libpthread.so.2
>
>  1735 clamd    RET   fork 0

Probably because it's fork()ing and trying to call
non-async-signal-safe functions without doing an exec()?

-- 
DE
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Hi Daniel,

Any idea how I can track this down ?

It's rather easy to reproduce ...

fork() is only used in the functions daemonize() and in
virusaction(), which is used to call an external programm
if one is defined (I don't have one).

So fork() should only be called once during going to be
a background daemon.

Martin

Martin Blapp, <mbimp.ch> <mbrFreeBSD.org>
------------------------------------------------------------
------
ImproWare AG, UNIXSP & ISP, Zurlindenstrasse 29, 4133
Pratteln, CH
Phone: +41 61 826 93 00 Fax: +41 61 826 93 01
PGP: <finger -l mbrfreebsd.org>
PGP Fingerprint: B434 53FC C87C FE7B 0A18 B84C 8686 EF22
D300 551E
------------------------------------------------------------
------

On Tue, 20 Feb 2007, Daniel Eischen wrote:

> On Tue, 20 Feb 2007, Martin Blapp wrote:
>
>> 
>> Hi,
>> 
>>> Probably because it's fork()ing and trying to
call
>>> non-async-signal-safe functions without doing
an exec()?
>> 
>> The strange thing is that I can't see any any
fork() calls
>> with libc_r or libthr. How's that possible ? Or do
you think
>> that its stuck at the beginning ? But why does it
work then
>> for some degree ?
>
> I dunno, maybe ktrace was confused and it wasn't a
fork()?
>
> -- 
> DE
> _______________________________________________
> freebsd-threadsfreebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
> To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"
>
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

On Tue, 20 Feb 2007, Martin Blapp wrote:

>
> Hi Daniel,
>
> Any idea how I can track this down ?

Try commenting out the fork() in daemonize().

Also try setting kern.threads.virtual_cpu=1 if you have
more than 1 cpu (this will limit it to 1 KSE).

> It's rather easy to reproduce ...
>
> fork() is only used in the functions daemonize() and
in
> virusaction(), which is used to call an external
programm
> if one is defined (I don't have one).
>
> So fork() should only be called once during going to
be
> a background daemon.

-- 
DE
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Hi,

> Also try setting kern.threads.virtual_cpu=1 if you
have
> more than 1 cpu (this will limit it to 1 KSE).

Our boxes have 2 CPUs and HTT is activated.

Interesting. With kern.threads.virtual_cpu=1 it works also
in the libpthread case, but while there are still some
fork() calls
it works better. With kern.threads.virtual_cpu=2 it's
slower. And
With kern.threads.virtual_cpu=4 its really slow.

If I run clamd in foreground mode everything works as it
should.
More interesting, the CPU load drops significantly
(500-1000%) !

Something is seriosly broken here.

Martin

_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

On Tue, 20 Feb 2007, Martin Blapp wrote:

> Hi,
>
>> Also try setting kern.threads.virtual_cpu=1 if you
have
>> more than 1 cpu (this will limit it to 1 KSE).
>
> Our boxes have 2 CPUs and HTT is activated.
>
> Interesting. With kern.threads.virtual_cpu=1 it works
also
> in the libpthread case, but while there are still some
fork() calls
> it works better. With kern.threads.virtual_cpu=2 it's
slower. And
> With kern.threads.virtual_cpu=4 its really slow.
>
> If I run clamd in foreground mode everything works as
it should.
> More interesting, the CPU load drops significantly
(500-1000%) !
>
> Something is seriosly broken here.

Running in foreground with or without
kern.threads.virtual_cpu
adjusted?

If running in foreground works regardless of
kern.threads.virtual_cpu
(there will still be some inefficiencies here because of
the
scheduler and KSE overhead/contention), then it is probably
due
to the application doing a fork() from a threaded
application
without doing an exec().  This is prohibited by POSIX and
libpthread makes no attempt to allow it.

-- 
DE
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Hi,

> If running in foreground works regardless of
kern.threads.virtual_cpu
> (there will still be some inefficiencies here because
of the
> scheduler and KSE overhead/contention), then it is
probably due
> to the application doing a fork() from a threaded
application
> without doing an exec().  This is prohibited by POSIX
and
> libpthread makes no attempt to allow it.

Yes it works regardless of kern.threads.virtual_cpu.

Yes, fork() is called without exec(). So what's the proper
way ? Start
clamd in foreground mode with

daemon /usr/local/sbin/clamd

And redirect stderr ?

Or can this be solved differently ?

Martin
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Hi,

Useing daemon(3) instead of fork seems to fix the problem.

Thank you very much !

Martin

Martin Blapp, <mbimp.ch> <mbrFreeBSD.org>
------------------------------------------------------------
------
ImproWare AG, UNIXSP & ISP, Zurlindenstrasse 29, 4133
Pratteln, CH
Phone: +41 61 826 93 00 Fax: +41 61 826 93 01
PGP: <finger -l mbrfreebsd.org>
PGP Fingerprint: B434 53FC C87C FE7B 0A18 B84C 8686 EF22
D300 551E
------------------------------------------------------------
------

On Tue, 20 Feb 2007, Daniel Eischen wrote:

> On Tue, 20 Feb 2007, Martin Blapp wrote:
>
>> Hi,
>> 
>>> Also try setting kern.threads.virtual_cpu=1 if
you have
>>> more than 1 cpu (this will limit it to 1 KSE).
>> 
>> Our boxes have 2 CPUs and HTT is activated.
>> 
>> Interesting. With kern.threads.virtual_cpu=1 it
works also
>> in the libpthread case, but while there are still
some fork() calls
>> it works better. With kern.threads.virtual_cpu=2
it's slower. And
>> With kern.threads.virtual_cpu=4 its really slow.
>> 
>> If I run clamd in foreground mode everything works
as it should.
>> More interesting, the CPU load drops significantly
(500-1000%) !
>> 
>> Something is seriosly broken here.
>
> Running in foreground with or without
kern.threads.virtual_cpu
> adjusted?
>
> If running in foreground works regardless of
kern.threads.virtual_cpu
> (there will still be some inefficiencies here because
of the
> scheduler and KSE overhead/contention), then it is
probably due
> to the application doing a fork() from a threaded
application
> without doing an exec().  This is prohibited by POSIX
and
> libpthread makes no attempt to allow it.
>
> -- 
> DE
> _______________________________________________
> freebsd-threadsfreebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
> To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"
>
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Sigh,

>
> Useing daemon(3) instead of fork seems to fix the
problem.

Of course deamon(3) calls fork, and after a short time I
had again the same ktrace output and CPU usage.

Martin
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

> Sigh,
>
>> 
>> Useing daemon(3) instead of fork seems to fix the
problem.
>
> Of course deamon(3) calls fork, and after a short time
I
> had again the same ktrace output and CPU usage.

Using daemon(3) works some times, but from time to time
I see then again the old behaviour, looping kse calls
and some forks().

I wonder why those calls never happen in the libthr
and libc_r cases.

Martin
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

On Wed, 21 Feb 2007, Martin Blapp wrote:

>> Sigh,
>> 
>>> 
>>> Useing daemon(3) instead of fork seems to fix
the problem.
>> 
>> Of course deamon(3) calls fork, and after a short
time I
>> had again the same ktrace output and CPU usage.
>
> Using daemon(3) works some times, but from time to
time
> I see then again the old behaviour, looping kse calls
> and some forks().
>
> I wonder why those calls never happen in the libthr
> and libc_r cases.

libc_r and libthr don't have to kse initialization and
have different locking.

Regardless, I think the code (clamd) is wrong.  There
is no guarantee that fork() from threaded application
will work for libthr and libc_r either.  If clamd needs
to daemonize, it can always exec() itself with an added
argument that means "I've already fork()'d, don't
fork()
again".

-- 
DE
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Hi,

After some busy days I found again time to work on this. I
still have no clue
what could be responsable for the problems I see.

ktrace still shows lot of fork()s, but I can't see where
whey are called.

Running ktrace with libc_r or libthr I can't see a single
call to fork().
Is ktrace lying to me ? Why do I only see fork()s with
libpthread ?

Martin

Martin Blapp, <mbimp.ch> <mbrFreeBSD.org>
------------------------------------------------------------
------
ImproWare AG, UNIXSP & ISP, Zurlindenstrasse 29, 4133
Pratteln, CH
Phone: +41 61 826 93 00 Fax: +41 61 826 93 01
PGP: <finger -l mbrfreebsd.org>
PGP Fingerprint: B434 53FC C87C FE7B 0A18 B84C 8686 EF22
D300 551E
------------------------------------------------------------
------

_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

On Sat, 24 Feb 2007, Martin Blapp wrote:

>
> Hi,
>
> After some busy days I found again time to work on
this. I still have no clue
> what could be responsable for the problems I see.
>
> ktrace still shows lot of fork()s, but I can't see
where whey are called.

ktrace is probably lying to you if the code is really only
fork()ing once.

> Running ktrace with libc_r or libthr I can't see a
single call to fork().
> Is ktrace lying to me ? Why do I only see fork()s with
libpthread ?

The kse upcalls might look like forks to ktrace?

-- 
DE
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Hi,

>> Running ktrace with libc_r or libthr I can't see a
single call to fork().
>> Is ktrace lying to me ? Why do I only see fork()s
with libpthread ?
>
> The kse upcalls might look like forks to ktrace?
>

That could be. Is it easy to fix this for ktrace ?

Anyway, that still doesn't solve the problem for me, since
clamd with libpthreads is extremly CPU hungry and is slow.

I really miss something to profile like strace on solaris
to
see where it spends most CPU cycles. The problem is not
that
it doesn't work, the problem is that it works far too slow.

Do you have any ideas ?

Martin
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

On Sat, 24 Feb 2007, Martin Blapp wrote:

>
> Hi,
>
>>> Running ktrace with libc_r or libthr I can't
see a single call to fork().
>>> Is ktrace lying to me ? Why do I only see
fork()s with libpthread ?
>> 
>> The kse upcalls might look like forks to ktrace?
>> 
>
> That could be. Is it easy to fix this for ktrace ?

I don't know...

> Anyway, that still doesn't solve the problem for me,
since
> clamd with libpthreads is extremly CPU hungry and is
slow.
>
> I really miss something to profile like strace on
solaris to
> see where it spends most CPU cycles. The problem is not
that
> it doesn't work, the problem is that it works far too
slow.
>
> Do you have any ideas ?

Yes, fix clamd so it doesn't fork() without doing an
exec().
POSIX says that behavior is undefined when threaded
applications
fork() and call non-async-signal-safe functions.

-- 
DE
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

I already did that. With libc_r and libthr I don't see
a single fork() here.

> Yes, fix clamd so it doesn't fork() without doing an
exec().
> POSIX says that behavior is undefined when threaded
applications
> fork() and call non-async-signal-safe functions.
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Hi all,

I'm including now clamav-devellists.clamav.net too, since
it's
not clear if this problem is clamd or system library
related.

The problem is that clamd works only well with FreeBSDs
libthr 1:1
kernel threading library. Using libc_r (userland threading
only)
seems to consume already a bit more resources. But clamd
together with
the N:M kernel threads library (libpthreads) is really
slow.

For some strange reason setting the sysctl
kern.threads.virtual_cpu=1 to allow 
only one CPU for clamd works well and takes the CPU load
significantly down.

After reading

http://www.gossamer-threads.com/lists/clamav/users/28795

especially this part:

[...]

>
>We've been running 6.x our scanner boxes for a while
now, but it's only
>been with the more recent security/clamav-devel port
installs that we
>noticed a problem much like this. Most connections to
the daemon (made
>through clamav-milter in our case) timed out, and the
only way to bring
>down the daemon was with a kill -9.
>
>For us, the 20061029 devel snapshot was fine, but the
current one
>(20061217) has problems.
>

[...]

I've now tried to go back to the development snapshot
20061129
and so I've reversed the clamd threading patches up to this
date.
But no luck. First it works well, then it becomes suddenly
slower
and more and more connections time out. Also, only 'kill -9'
kills
clamd. This isn't the case with lc_r and libthr btw ...

Could it be that libpthread from it's kind of nature is a
lot
slower than libthr in that special case ? Maybe clamd
expects system scope 
thread semantics and so libpthreads is slower ... Of course
our current 
workload could then be too much for the poor clamd and a
even more growing 
workload may slow it down due to resource bottlenecks.

--
Martin
_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"

Let me make a summary:

With kern.threads.virtual_cpu=1 clamd works best with
libpthread.
It uses almost no CPU. With kern.threads.virtual_cpu=2 or
kern.threads.virtual_cpu=4 and growing scanner load, the CPU
load
grows exponentially.

The initial fork() seems not to be the problem. It happens
in
the foreground mode (without any fork call) too, but only
if
there is some amount of scanner load.

Setting maxthreads in clamd.conf seems to make the
situation
even worse.

libthr and libc_r work without any problems and ktrace here
doesn't
show any fork() calls at all.

I wonder if something else could get libpthread spinning on
kse activation stuff. I still think something is wrong
here.

Martin

Martin Blapp, <mbimp.ch> <mbrFreeBSD.org>
------------------------------------------------------------
------
ImproWare AG, UNIXSP & ISP, Zurlindenstrasse 29, 4133
Pratteln, CH
Phone: +41 61 826 93 00 Fax: +41 61 826 93 01
PGP: <finger -l mbrfreebsd.org>
PGP Fingerprint: B434 53FC C87C FE7B 0A18 B84C 8686 EF22
D300 551E
------------------------------------------------------------
------

_______________________________________________
freebsd-threadsfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-threa
ds
To unsubscribe, send any mail to
"freebsd-threads-unsubscribefreebsd.org"