xennetback_ifstart: no mcl_pages crashes XEN2-dom0

Hello!

I am using NetBSD4.0Beta as XEN2-Dom0 (from 10/4/06) with
six 
NetBSD3.0.1 DomUs on an AMD 3700+ server, VIA Chipset with
1GB of RAM 
and Hardware Raid.

 From time to time I get the following error that forces me
to reboot my 
Dom0


May 30 10:07:42 pless /netbsd: xennetback: got only 29 new
mcl pages
May 30 10:07:47 pless /netbsd: xennetback: got only 9 new
mcl pages
May 30 10:07:48 pless /netbsd: xennetback: got only 3 new
mcl pages
May 30 10:07:48 pless /netbsd: xennetback: got only 1 new
mcl pages
May 30 10:07:48 pless /netbsd: xennetback: can't get new mcl
pages (0)
May 30 10:07:48 pless /netbsd: xennetback_ifstart: no
mcl_pages
And the last two lines repeat 1000+ times....

What are those mcl pages?

On the domU machines I let run a lot of RSync cronjobs
copying files 
from another sibling-PC with very similar configuration.
Some of the 
DomUs share directories with nfs. In fact this machine is my

hot-standby-backup-server. The master server does not
complain about 
this problem. (It only complains about ..."/netbsd:
xbdback: domain 6 
sending excessively fragmented I/O" but it's not that
bad I think)


First I thought that it could have to do with a buggy
network driver for 
the onboard Realtec NIC but now there is an Intel NIC built
in and the 
problem still persists.

My system memory is divided as follows:

pless# xm list
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0       63    0  r----    499.4
ap-php4            3      255    0  -b---     26.0    9603
ap-php5            4      127    0  -b---     21.8    9604
dbserver           5      127    0  -b---    101.3    9605
mailserver         7      169    0  -b---     60.5    9607
ns1                2      127    0  -b---     20.4    9602
router             1      127    0  -b---    257.4    9601


pless# xm info
system                 : NetBSD
host                   : pless
release                : 4.0_BETA
version                : NetBSD 4.0_BETA (XEN2_DOM0) #0: Wed
Apr 18 
15:14:53 CEST 2007  rootpless:/xen/usr/src/sys/arch/i386/compile/XEN2_DOM0
machine                : i386
cores                  : 1
hyperthreads_per_core  : 1
cpu_mhz                : 2199
memory                 : 1022
free_memory            : 2



I use an self-built kernel for this NetBSD version with more
SHM pages 
(8192 instead of 2048). I hoped that could solve the problem
but it did not.

options         SHMMAXPGS=8192  # 2048 pages is the default


I read on the list something about XEN-DOM0 needing 32MB for
itself. 
(Date of posting: 4/15/2005 6pm from Manuel Bouyer) Does it
mean that I 
have to increase my 2MB of free memory to 32? That's
actually what I'd 
try next...

Another idea which will be quite time consuming and I would
like to avoid:
Is it worth it to update to a newer Beta-version of NetBSD4
with a newer 
Kernel?


I really hope that there is somebody out there who can help
me 

Thanks in advance!


Yours,
Damian Lubosch

On Wed, May 30, 2007 at 07:39:34PM +0200, Damian Lubosch
wrote:
> May 30 10:07:42 pless /netbsd: xennetback: got only 29
new mcl pages
> May 30 10:07:47 pless /netbsd: xennetback: got only 9
new mcl pages
> May 30 10:07:48 pless /netbsd: xennetback: got only 3
new mcl pages
> May 30 10:07:48 pless /netbsd: xennetback: got only 1
new mcl pages
> May 30 10:07:48 pless /netbsd: xennetback: can't get
new mcl pages (0)
> May 30 10:07:48 pless /netbsd: xennetback_ifstart: no
mcl_pages
> And the last two lines repeat 1000+ times....
> 
> What are those mcl pages?

Pages for mbuf clusters used at network buffers.

Please options(4) (via "man 4 options") and search
for "NMBCLUSTERS".
It will tell you how to increase the number of network
buffers your
kernel is allowed to use.

	Kind regards

-- 
Matthias Scheler                                  http://zhadum.org.uk/

On Wed, May 30, 2007 at 07:39:34PM +0200, Damian Lubosch
wrote:
> Hello!
> 
> I am using NetBSD4.0Beta as XEN2-Dom0 (from 10/4/06)
with six 
> NetBSD3.0.1 DomUs on an AMD 3700+ server, VIA Chipset
with 1GB of RAM 
> and Hardware Raid.
> 
> From time to time I get the following error that forces
me to reboot my 
> Dom0
> 
> 
> May 30 10:07:42 pless /netbsd: xennetback: got only 29
new mcl pages
> May 30 10:07:47 pless /netbsd: xennetback: got only 9
new mcl pages
> May 30 10:07:48 pless /netbsd: xennetback: got only 3
new mcl pages
> May 30 10:07:48 pless /netbsd: xennetback: got only 1
new mcl pages
> May 30 10:07:48 pless /netbsd: xennetback: can't get
new mcl pages (0)
> May 30 10:07:48 pless /netbsd: xennetback_ifstart: no
mcl_pages
> And the last two lines repeat 1000+ times....
> 
> What are those mcl pages?

Memory that dom0 reclaims from the hypervisor to remplace
pages that
we gave to the remote domains when sending packets

> 
> On the domU machines I let run a lot of RSync cronjobs
copying files 
> from another sibling-PC with very similar
configuration. Some of the 
> DomUs share directories with nfs. In fact this machine
is my 
> hot-standby-backup-server. The master server does not
complain about 
> this problem. (It only complains about
..."/netbsd: xbdback: domain 6 
> sending excessively fragmented I/O" but it's not
that bad I think)
> 
> 
> First I thought that it could have to do with a buggy
network driver for 
> the onboard Realtec NIC but now there is an Intel NIC
built in and the 
> problem still persists.
> 
> My system memory is divided as follows:
> 
> pless# xm list
> Name              Id  Mem(MB)  CPU  State  Time(s) 
Console
> Domain-0           0       63    0  r----    499.4
> ap-php4            3      255    0  -b---     26.0   
9603
> ap-php5            4      127    0  -b---     21.8   
9604
> dbserver           5      127    0  -b---    101.3   
9605
> mailserver         7      169    0  -b---     60.5   
9607
> ns1                2      127    0  -b---     20.4   
9602
> router             1      127    0  -b---    257.4   
9601
> 
> 
> pless# xm info
> system                 : NetBSD
> host                   : pless
> release                : 4.0_BETA
> version                : NetBSD 4.0_BETA (XEN2_DOM0)
#0: Wed Apr 18 
> 15:14:53 CEST 2007  rootpless:/xen/usr/src/sys/arch/i386/compile/XEN2_DOM0
> machine                : i386
> cores                  : 1
> hyperthreads_per_core  : 1
> cpu_mhz                : 2199
> memory                 : 1022
> free_memory            : 2
> 
> 
> 
> I use an self-built kernel for this NetBSD version with
more SHM pages 
> (8192 instead of 2048). I hoped that could solve the
problem but it did not.
> 
> options         SHMMAXPGS=8192  # 2048 pages is the
default

No, it's completely different.

> 
> 
> I read on the list something about XEN-DOM0 needing
32MB for itself. 
> (Date of posting: 4/15/2005 6pm from Manuel Bouyer)
Does it mean that I 
> have to increase my 2MB of free memory to 32? That's
actually what I'd 
> try next...

I strongly suspect that there's not enough free ram for the
hypervisor.
I think the 32MB is not counted as part of "xm
info", but it's something
that can't ever be allocated to domains anyway, it's
internal to the
hypervisor. You'll also need some free pages for the
communications between
domains.

> 
> Another idea which will be quite time consuming and I
would like to avoid:
> Is it worth it to update to a newer Beta-version of
NetBSD4 with a newer 
> Kernel?

I don't think it will help.

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

Hello!

Again my dom0 crashed this night.
The dom0 showed the same message as few weeks ago:

Manuel Bouyer wrote:
>> May 30 10:07:48 pless /netbsd: xennetback_ifstart:
no mcl_pages
>> And the last two lines repeat 1000+ times....
>>
>> What are those mcl pages?
>
> Memory that dom0 reclaims from the hypervisor to
remplace pages that
> we gave to the remote domains when sending packets
>

Ok, I think I got it.

>> I read on the list something about XEN-DOM0 needing
32MB for itself.
>> (Date of posting: 4/15/2005 6pm from Manuel Bouyer)
Does it mean that I
>> have to increase my 2MB of free memory to 32?
That's actually what I'd
>> try next...
>
> I strongly suspect that there's not enough free ram for
the hypervisor.

Should the free ram be under dom0? As I understand XEN, the
hypervisor
ist running in dom0, right? The dom0 has 64MB of its own
memory. Since
it isn't doing anything else but providing XEN, I thought it
would be
enough. I could spare some more MB (up to 90) if it is
necessary. Is it?

> I think the 32MB is not counted as part of "xm
info", but it's something
> that can't ever be allocated to domains anyway, it's
internal to the
> hypervisor.

So nobody noticed problems having 2MB free in "xm
info"? The free ram
has no meaning but showing a waste of memory?

> You'll also need some free pages for the communications
between
> domains.

There is a lot of communication between the domUs. I resized
the
NMBCLUSTERS to 2048. Should I increase it even more?

>
>> Another idea which will be quite time consuming and
I would like to
avoid:
>> Is it worth it to update to a newer Beta-version of
NetBSD4 with a newer
>> Kernel?
>
> I don't think it will help.
>

Ok, that will save me some time. Should I maybe upgrade to
XEN 3? Did it
turn out to be more stable? My dom0 is running NetBSD Beta
anyway so a
switch won't cause many changes.

Thanks for any help.

Damian Lubosch

On Fri, Jun 29, 2007 at 09:35:55AM +0200, Damian Lubosch
wrote:
> 
> 
> Hello!
> 
> Again my dom0 crashed this night.
> The dom0 showed the same message as few weeks ago:
> 
> Manuel Bouyer wrote:
> [...]
> 
> >> I read on the list something about XEN-DOM0
needing 32MB for itself.
> >> (Date of posting: 4/15/2005 6pm from Manuel
Bouyer) Does it mean that I
> >> have to increase my 2MB of free memory to 32?
That's actually what I'd
> >> try next...
> >
> > I strongly suspect that there's not enough free
ram for the hypervisor.
> 
> Should the free ram be under dom0?

No, the hypervisor itself (i.e. 'free' in xm info)

> As I understand XEN, the hypervisor
> ist running in dom0, right?

No, the hypervisor is running on its own and control access
to ressources
of all domains, including dom0. More technically, on i386
the hypervisor
runs in ring0, all domains kernels (including dom0) runs in
ring1 and userland
programs in ring3. On plain NetBSD/i386 (and most other unix
variants on i386),
the kernel runs in ring0 and userland in ring3, rings 1 and
2 are unused.

> The dom0 has 64MB of its own memory. Since
> it isn't doing anything else but providing XEN, I
thought it would be
> enough. I could spare some more MB (up to 90) if it is
necessary. Is it?

you can watch with top what's going on, but I suspect 64MB
is enough.

> 
> > I think the 32MB is not counted as part of
"xm info", but it's something
> > that can't ever be allocated to domains anyway,
it's internal to the
> > hypervisor.
> 
> So nobody noticed problems having 2MB free in "xm
info"? The free ram
> has no meaning but showing a waste of memory?

it's really free in the same sense as free in Unix: memory
that can be
allocated to something if needed (it can be creation of new
domains, or
page swapping needed as part of inter-domain
communications)

> 
> There is a lot of communication between the domUs. I
resized the
> NMBCLUSTERS to 2048. Should I increase it even more?

You can watch then usage with netstat -m; but if you don't
get the
"WARNING: mclpool limit reached; increase
NMBCLUSTERS" message,
you're probably allocated enough.

> >
> > I don't think it will help.
> >
> 
> Ok, that will save me some time. Should I maybe upgrade
to XEN 3? Did it
> turn out to be more stable? My dom0 is running NetBSD
Beta anyway so a
> switch won't cause many changes.

It won't change much I guess. I have important prodcution
servers running
very well with Xen2. I also have some running Xen3; all of
them are going
their job 


-- 
Manuel Bouyer, LIP6, Universite Paris VI.          
Manuel.Bouyerlip6.fr
     NetBSD: 26 ans d'experience feront toujours la
difference
--

Manuel Bouyer schrieb:
> On Fri, Jun 29, 2007 at 09:35:55AM +0200, Damian
Lubosch wrote:
>   
>>
>>> I strongly suspect that there's not enough free
ram for the hypervisor.
>>>       
>> Should the free ram be under dom0?
>>     
>
> No, the hypervisor itself (i.e. 'free' in xm info)
>
>   

Ok, I should have read the docs more carefully. Thank you
very much for 
the info. I'll try to run the hypervisor with more memory
now. I freed 
about 50MB of memory for the hypervisor and I hope that
it'll run now 
much more stable

Bye, Damian