emerge threaded

I'm trying to setup my superkaramba theme to have the
emerge module
being threaded so the gui is still responsive while it's
checking for
updates. But I'm running into this error

Unhandled exception in thread started by <function main
at 0xb68f356c>
Traceback (most recent call last):
  File "portage_engine.py", line 29, in main
    reload(portage)
  File "/usr/lib/portage/pym/portage.py", line
132, in ?
    signal.signal(signal.SIGCHLD, signal.SIG_DFL)
ValueError: signal only works in main thread

From what I've found so far, I basically won't be able to
do this
because of the signal calls in portage.py. I'm just
wondering if
anybody of anything else I could try or a work around to
allow me to
thread the emerge module?
--
Tom

-- 
gentoo-portage-devgentoo.org mailing list

On Mon, 2006-24-04 at 22:09 -0400, Tom Hosiawa wrote:
> I'm trying to setup my superkaramba theme to have the
emerge module
> being threaded so the gui is still responsive while
it's checking for
> updates. But I'm running into this error
> 
> Unhandled exception in thread started by <function
main at 0xb68f356c>
> Traceback (most recent call last):
>   File "portage_engine.py", line 29, in
main
>     reload(portage)
>   File "/usr/lib/portage/pym/portage.py",
line 132, in ?
>     signal.signal(signal.SIGCHLD, signal.SIG_DFL)
> ValueError: signal only works in main thread
> 
> >From what I've found so far, I basically won't be
able to do this
> because of the signal calls in portage.py. I'm just
wondering if
> anybody of anything else I could try or a work around
to allow me to
> thread the emerge module?
> --
> Tom
> 

>From our experience in porthole portage does not like to
be threaded.
There are a number of environment variables and things that
are not
preserved for portage/emerge to spawn additional processes. 
We also
limit what we do thread for info gathering.

That is why in porthole we use an imported portage for
information
gathering, but for installs/sync/unmerge we spawn a new
process and call
emerge with normal cammand line arguments.


That however could change for the next major version of
portage.  I have
not attempted to try an internal build in the imported
pkgcore, but it
may well work ok.  That is something that is yet to be
tested.  Brian
Harring does not know how pkgcore will handle being
threaded.  It is too
early in it's development.

-- 
Brian <dol-sentelus.net>

-- 
gentoo-portage-devgentoo.org mailing list

On Mon, Apr 24, 2006 at 07:55:17PM -0700, Brian wrote:
> That however could change for the next major version of
portage.  I have
> not attempted to try an internal build in the imported
pkgcore, but it
> may well work ok.  That is something that is yet to be
tested.  Brian
> Harring does not know how pkgcore will handle being
threaded.  It is too
> early in it's development.

Locking has been setup in a few spots (not enabled mostly,
fex vdb 
repo locking)- what's not in place (and matters) is that
internally, 
pkgcore uses pkgcore.util.caching pretty heavily- used to
generate 
unique instances that are reused.

For example, if you get the package instance
sys-apps/portage-2.0.51 
from repo A and shove it in a list (thus holding onto it in
memory), 
if you grab sys-apps/portage-2.0.51 from repo A *again*, you
get the 
same instance back (literally same instance).

So... generation of the instance (or lookup of the old
instance) is 
a critical section- have to ensure that updates to the dict
of what 
pkgs are in memory (whether removal or addition of pkgs) is
atomic so 
you don't get N copies of pkg A.

Fortunately, python's GIL protects deals with the
"pkg a is being 
unalloc'd while it's about to be handed back"- GIL +
ref counting 
prevents that from being an issue.

Aside from that... the only other spots where locking is
required is
1) build directory usage- don't want two instances
executing 
configure/make in the same directory fex, mainly because
ebuild 
execution may mangle things in install phase (which would
break things 
for compile phase).
2) sync locking.  Debatable on that one (best you can manage
is 
invalidating any pkgs from that repo in memory, which can
cause 
issues).
3) cache updates (debatable also).  Downside of races there
is that 
cache regen for a node occurs twice (rather then once),
which needs to 
be weighed against cost of locking overhead...

Usual locking requirements stable has (distdir fex) applies
also.
~harring