prelude_connection_pool_recv() does not wakeup other *python* threads

I use several python threads in a script. One of them (the
"producer")
use prelude, another is a consumer.

When the first thread is "sleeping" in
`prelude_connection_pool_recv()`,
other threads never wake up. My workaround is simple
(example below) but
not very clean nor efficient.


Another workaround should be to modify SWIG code (*1) I
think. But I am
not the only one who use multithread with python. So a
better way should
already exists.

Thanks for ideas.
_anthony

(I'm using libprelude-0.9.14 (with prelude-manager-0.9.8), 
Python 2.4.4c1 and ubuntu 6.10)

{{{# a workaround:

while True:
    # "0" : nowait
    msg = prelude.prelude_connection_pool_recv(__pool, 0)
    if msg is None:
        # sleep from python side
        sleep(0.5)
        continue
    else:
        # we have a prelude_msg, exiting the loop
        break

# after that, I filter msg to get its IDMEF_Alert if it has
one,
# I offer it to the "consumer" thread through a
Queue object
# and I wait for another alert

}}}

*1 :
http://mail.python.org/pipermail/python-li
st/2001-February/071515.html


_______________________________________________
Prelude-user site list
Prelude-userprelude-ids.org
http://www.prelude-ids.org/mailman/listinfo/prelude-user

Le mercredi 29 août 2007 à 11:39 +0000, anthony.loiseau a
écrit :
> I use several python threads in a script. One of them
(the "producer")
> use prelude, another is a consumer.
> 
> When the first thread is "sleeping" in
`prelude_connection_pool_recv()`,
> other threads never wake up. My workaround is simple
(example below) but
> not very clean nor efficient.
> 
> Another workaround should be to modify SWIG code (*1) I
think. But I am
> not the only one who use multithread with python. So a
better way should
> already exists.


Hello Anthony,

Thanks you for the feedback, and the pointer to the SWIG
discussion.
Could you please try the attached patch, and tell me if that
solve your
problem?

You will need to have the latest version of SWIG installed,
and to
compile libprelude using the --with-swig configure switch.

Additionally, this patch add Py_BEGIN_ALLOW_THREADS /
Py_END_ALLOW_THREADS around each call to libprelude C
function from the
SWIG module, so I'm unsure whether it can have unwanted side
effect on
the overall Prelude<->Python interaction.

What would be nice is if you could test it with python heavy
application
(like Prewikka), to see if it cause any regression.

Regards,

-- 
Yoann Vandoorselaere | Responsable R&D / CTO |
PreludeIDS Technologies
Tel: +33 (0)8 70 70 21 58                  Fax: +33(0)4 78
42 21 58
http://www.prelude-ids.com


_______________________________________________
Prelude-user site list
Prelude-userprelude-ids.org
http://www.prelude-ids.org/mailman/listinfo/prelude-user

Thanks for this patch.
I will try it.


Unfortunately, I'm not sure I will be able to appreciate
overall, since
prelude is very new for me (I have no idea of what a
"normal" overall
should be).

I will try and tell you back about the results.
Thanks,
_anthony



On Fri, 2007-08-31 at 15:25 +0200, Yoann Vandoorselaere
wrote:
> Hello Anthony,
> 
> Thanks you for the feedback, and the pointer to the
SWIG discussion.
> Could you please try the attached patch, and tell me if
that solve your
> problem?
> 
> You will need to have the latest version of SWIG
installed, and to
> compile libprelude using the --with-swig configure
switch.
> 
> Additionally, this patch add Py_BEGIN_ALLOW_THREADS /
> Py_END_ALLOW_THREADS around each call to libprelude C
function from the
> SWIG module, so I'm unsure whether it can have unwanted
side effect on
> the overall Prelude<->Python interaction.
> 
> What would be nice is if you could test it with python
heavy application
> (like Prewikka), to see if it cause any regression.
> 
> Regards,
> 

_______________________________________________
Prelude-user site list
Prelude-userprelude-ids.org
http://www.prelude-ids.org/mailman/listinfo/prelude-user

The patch works just fine for me 

python treads are active during pool_recv() and I have not
noticed
overall.

I have sent 10 000 foo alerts from a python program to my
other python
program and it has worked fine with 2 threads running for
the receiver.
(1 min 05 sec to send, 2 min 40 sec to receive with quite
lot debug
messages)

Prewikka interface take 3 min and 20 sec to show 20 000+
alerts



NB1 : My computer is a Intel P4 HT 3GHz (6000 bogomips), 1GB
Ram
      I use prewikka through Firefox2
      (in case this can help you to appreciate an averall)


NB2: prelude functions which appear in my test are :
       - prelude_connection_pool_recv    (thread 1)

       - idmef_path_new                  (thread 2)
       - idmef_path_get
       - idmef_path_destroy
       - idmef_value_get_string

So I have not tested other libprelude's functions at all
against
multi-threading.


Thanks for this patch
_Anthony



On Fri, 2007-08-31 at 15:59 +0200, anthony.loiseau wrote:
> Thanks for this patch.
> I will try it.
> 
> 
> Unfortunately, I'm not sure I will be able to
appreciate overall, since
> prelude is very new for me (I have no idea of what a
"normal" overall
> should be).
> 
> I will try and tell you back about the results.
> Thanks,
> _anthony
> 
> 
> 
> On Fri, 2007-08-31 at 15:25 +0200, Yoann Vandoorselaere
wrote:
> > Hello Anthony,
> > 
> > Thanks you for the feedback, and the pointer to
the SWIG discussion.
> > Could you please try the attached patch, and tell
me if that solve your
> > problem?
> > 
> > You will need to have the latest version of SWIG
installed, and to
> > compile libprelude using the --with-swig configure
switch.
> > 
> > Additionally, this patch add
Py_BEGIN_ALLOW_THREADS /
> > Py_END_ALLOW_THREADS around each call to
libprelude C function from the
> > SWIG module, so I'm unsure whether it can have
unwanted side effect on
> > the overall Prelude<->Python interaction.
> > 
> > What would be nice is if you could test it with
python heavy application
> > (like Prewikka), to see if it cause any
regression.
> > 
> > Regards,
> > 
> 
> _______________________________________________
> Prelude-user site list
> Prelude-userprelude-ids.org
> http://www.prelude-ids.org/mailman/listinfo/prelude-user


_______________________________________________
Prelude-user site list
Prelude-userprelude-ids.org
http://www.prelude-ids.org/mailman/listinfo/prelude-user

Le vendredi 31 août 2007 à 17:47 +0200, anthony.loiseau a
écrit :
> The patch works just fine for me 
> 
> python treads are active during pool_recv() and I have
not noticed
> overall.
> 
> I have sent 10 000 foo alerts from a python program to
my other python
> program and it has worked fine with 2 threads running
for the
> receiver.
> (1 min 05 sec to send, 2 min 40 sec to receive with
quite lot debug
> messages)


Thanks for the feedback, I will apply the patch against both
libprelude
& libpreludedb bindings.

Regards,

-- 
Yoann Vandoorselaere | Responsable R&D / CTO |
PreludeIDS Technologies
Tel: +33 (0)8 70 70 21 58                  Fax: +33(0)4 78
42 21 58
http://www.prelude-ids.com


_______________________________________________
Prelude-user site list
Prelude-userprelude-ids.org
http://www.prelude-ids.org/mailman/listinfo/prelude-user