Xen2/NetBSD 3.1 DomU running slowly

Every now and then I get reports of a customer's DomU
running slowly and, 
sure enough, when I connect it takes ages to login. However,
it's running 
at 100% idle and the other DomU on the same box is fine. A
reboot of the 
DomU fixes it. When it happened today, I saw the following
on the Dom0 
console (which is running quickly and responsively):

xen_shm_map: ENOMEM
xen_shm_map: ENOMEM1
xvif3.0: can't map packet page

The first two are debugging, I believe, but the third looks
suspicious...

The kernels are 3.1_STABLE from 20th Nov 2006. Have there
been pullups 
that fix this, i.e. should I be updating? I'm not in a
position to upgrade 
to NetBSD 4.0/Xen3 at the moment.

-- 
Stephen

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Every now and then I get reports of a customer's DomU
running  
> slowly and, sure enough, when I connect it takes ages
to login.  
> However, it's running at 100% idle and the other DomU
on the same  
> box is fine. A reboot of the DomU fixes it. When it
happened today,  
> I saw the following on the Dom0 console (which is
running quickly  
> and responsively):
>
> xen_shm_map: ENOMEM
> xen_shm_map: ENOMEM1
> xvif3.0: can't map packet page
>
> The first two are debugging, I believe, but the third
looks  
> suspicious...
>
> The kernels are 3.1_STABLE from 20th Nov 2006. Have
there been  
> pullups that fix this, i.e. should I be updating? I'm
not in a  
> position to upgrade to NetBSD 4.0/Xen3 at the moment.

I've seen the same console messages, repeatedly, in a
Xen2/NetBSD  
4.0beta2 environment (both DOM0 and DOMU) from
Dec'06/Jan'07.

I've not noticed the extreme slowness of the DOMUs though,
but have  
not looked for it either.

Johan


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFGXoyNKJmr+nqSTbYRAs0pAJkBWjTPwevhHZukOJok8NheVIq7gwCd
HzAg
/0UPsqGrYRzA/zjNjTvnG6E=
=eQU5
-----END PGP SIGNATURE-----

On Thu, May 31, 2007 at 09:24:51AM +0100, Stephen Borrill
wrote:
> Every now and then I get reports of a customer's DomU
running slowly and, 
> sure enough, when I connect it takes ages to login.
However, it's running 
> at 100% idle and the other DomU on the same box is
fine. A reboot of the 
> DomU fixes it. When it happened today, I saw the
following on the Dom0 
> console (which is running quickly and responsively):
> 
> xen_shm_map: ENOMEM
> xen_shm_map: ENOMEM1
> xvif3.0: can't map packet page
> 
> The first two are debugging, I believe, but the third
looks suspicious...
> 
> The kernels are 3.1_STABLE from 20th Nov 2006. Have
there been pullups 
> that fix this, i.e. should I be updating? I'm not in a
position to upgrade 
> to NetBSD 4.0/Xen3 at the moment.

You may have an usage that requires more VM space for
inter-domains 
communications than is currenly allocated. You can try
increasing XENSHM_NPAGES
in sys/arch/xen/i386/xen_shm_machdep.c

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Sun, 3 Jun 2007, Manuel Bouyer wrote:
> On Thu, May 31, 2007 at 09:24:51AM +0100, Stephen
Borrill wrote:
>> Every now and then I get reports of a customer's
DomU running slowly and,
>> sure enough, when I connect it takes ages to login.
However, it's running
>> at 100% idle and the other DomU on the same box is
fine. A reboot of the
>> DomU fixes it. When it happened today, I saw the
following on the Dom0
>> console (which is running quickly and
responsively):
>>
>> xen_shm_map: ENOMEM
>> xen_shm_map: ENOMEM1
>> xvif3.0: can't map packet page
>>
>> The first two are debugging, I believe, but the
third looks suspicious...
>>
>> The kernels are 3.1_STABLE from 20th Nov 2006. Have
there been pullups
>> that fix this, i.e. should I be updating? I'm not
in a position to upgrade
>> to NetBSD 4.0/Xen3 at the moment.
>
> You may have an usage that requires more VM space for
inter-domains
> communications than is currenly allocated. You can try
increasing XENSHM_NPAGES
> in sys/arch/xen/i386/xen_shm_machdep.c

On the DOM0 presumably? Would doubling it be a fair
increase?

-- 
Stephen

On Tue, Jul 03, 2007 at 03:15:09PM +0100, Stephen Borrill
wrote:
> On Sun, 3 Jun 2007, Manuel Bouyer wrote:
> >On Thu, May 31, 2007 at 09:24:51AM +0100, Stephen
Borrill wrote:
> >>Every now and then I get reports of a
customer's DomU running slowly and,
> >>sure enough, when I connect it takes ages to
login. However, it's running
> >>at 100% idle and the other DomU on the same box
is fine. A reboot of the
> >>DomU fixes it. When it happened today, I saw
the following on the Dom0
> >>console (which is running quickly and
responsively):
> >>
> >>xen_shm_map: ENOMEM
> >>xen_shm_map: ENOMEM1
> >>xvif3.0: can't map packet page
> >>
> >>The first two are debugging, I believe, but the
third looks suspicious...
> >>
> >>The kernels are 3.1_STABLE from 20th Nov 2006.
Have there been pullups
> >>that fix this, i.e. should I be updating? I'm
not in a position to upgrade
> >>to NetBSD 4.0/Xen3 at the moment.
> >
> >You may have an usage that requires more VM space
for inter-domains
> >communications than is currenly allocated. You can
try increasing 
> >XENSHM_NPAGES
> >in sys/arch/xen/i386/xen_shm_machdep.c
> 
> On the DOM0 presumably?

Yes

> Would doubling it be a fair increase?

Yes, in fact it really depends on how much domUs will do a
full ring of I/Os
at the same time (or how much interfaces in a single domU)


-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Tue, 3 Jul 2007, Manuel Bouyer wrote:
>>> On Thu, May 31, 2007 at 09:24:51AM +0100,
Stephen Borrill wrote:
>>>> Every now and then I get reports of a
customer's DomU running slowly and,
>>>> sure enough, when I connect it takes ages
to login. However, it's running
>>>> at 100% idle and the other DomU on the same
box is fine. A reboot of the
>>>> DomU fixes it. When it happened today, I
saw the following on the Dom0
>>>> console (which is running quickly and
responsively):
>>>>
>>>> xen_shm_map: ENOMEM
>>>> xen_shm_map: ENOMEM1
>>>> xvif3.0: can't map packet page
>>>>
>>>> The first two are debugging, I believe, but
the third looks suspicious...
>>>>
>>>> The kernels are 3.1_STABLE from 20th Nov
2006. Have there been pullups
>>>> that fix this, i.e. should I be updating?
I'm not in a position to upgrade
>>>> to NetBSD 4.0/Xen3 at the moment.
>>>
>>> You may have an usage that requires more VM
space for inter-domains
>>> communications than is currenly allocated. You
can try increasing
>>> XENSHM_NPAGES
>>> in sys/arch/xen/i386/xen_shm_machdep.c
>> Would doubling it be a fair increase?
>
> Yes, in fact it really depends on how much domUs will
do a full ring of I/Os
> at the same time (or how much interfaces in a single
domU)

Single interface in each domU. 2 domUs per box. The domU in
question is 
just running Apache (and some perl cgis) and ipfilter. It
serves about 
50GB of data per month (i.e. not too heavy load). I'd have
thought this 
was pretty standard usage.

-- 
Stephen

On Wed, Aug 29, 2007 at 08:31:59AM +0100, Stephen Borrill
wrote:
> >at the same time (or how much interfaces in a
single domU)
> 
> I've doubled it and also now the domU is the only one
on the box. The 
> performance problems are still happening.

Another thing that can happen (I ran into this recently) is
that when
the ring is full, the transmit stops and won't restart a
soon as there is
some place in the ring, because there's no interrupt
mechanism in the Xen2
interface to allow this (we have to poll for the ring
pointers and notice when
there's some space). Xen3 is much better in this area.

-- 
Manuel Bouyer, LIP6, Universite Paris VI.          
Manuel.Bouyerlip6.fr
     NetBSD: 26 ans d'experience feront toujours la
difference
--