Re: input/output errors and out of inodes and file system full

On Tue, Jan 16, 2007 at 01:52:19PM +0000, Matthias Scheler
wrote:
> On Tue, Jan 16, 2007 at 03:26:48AM +0000, reedreedmedia.net wrote:
> > Anyone seen anything like this?
> > 
> > My kernel is netbsd-3.
> 
> This pullup might fix your problem:
> 
> http://mail-index.netbsd.org/source-changes/2007
/01/15/0016.html
> 

I don't think so, this pullup was to fix a possible panic at
boot with more
than one xbd. 
This commit could be the fix:
http://mail-index.netbsd.org/source-changes/2007
/01/01/0033.html

but the author has not submitted a pullup request to
netbsd-3 yet ...

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Tue, Jan 16, 2007 at 01:52:19PM +0000, Matthias Scheler
wrote:
> On Tue, Jan 16, 2007 at 03:26:48AM +0000, reedreedmedia.net wrote:
> > Anyone seen anything like this?
> > 
> > My kernel is netbsd-3.
> 
> This pullup might fix your problem:
> 
> http://mail-index.netbsd.org/source-changes/2007
/01/15/0016.html
> 

I don't think so, this pullup was to fix a possible panic at
boot with more
than one xbd. 
This commit could be the fix:
http://mail-index.netbsd.org/source-changes/2007
/01/01/0033.html

but the author has not submitted a pullup request to
netbsd-3 yet ...

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Tue, 16 Jan 2007, Manuel Bouyer wrote:

> 
> On Tue, Jan 16, 2007 at 01:52:19PM +0000, Matthias
Scheler wrote:
>> On Tue, Jan 16, 2007 at 03:26:48AM +0000, reedreedmedia.net wrote:
>>> Anyone seen anything like this?
>>>
>>> My kernel is netbsd-3.
>>
>> This pullup might fix your problem:
>>
>> http://mail-index.netbsd.org/source-changes/2007
/01/15/0016.html
>>
>
> I don't think so, this pullup was to fix a possible
panic at boot with more
> than one xbd.
> This commit could be the fix:
> http://mail-index.netbsd.org/source-changes/2007
/01/01/0033.html
>
> but the author has not submitted a pullup request to
netbsd-3 yet ...

Hm.  I am the xen provider in question  (we are a small
shop, and I have a 
dayjob, so I haven't gotten around to really looking at it
until now)  but 
using the same kernel, I do get occasional heavy disk load
panics. 
Actually I just triggered one right now (incidentally while
I was 
attempting to manually apply the patch you suggested;  all
my NetBSD boxes 
at the moment are XenUs. )

here's the crash:

uvm_fault(0xc044ec80, 0xcaa8a000, 0, 2) -> 0xe
kernel: page fault trap, code=0
Stopped in pid 8.1 (aiodoned) at       
netbsd:softdep_pageiodone+0x248: 
m
ovl     %edx,0(%eax)
softdep_pageiodone(c09e30fc,3415c,c0745380,c9bab108,c9bab318
) at 
netbsd:softdep_
pageiodone+0x248
uvm_aio_aiodone(c09e30fc,204,c03f755f,0,c047a7e0) at 
netbsd:uvm_aio_aiodone+0x4d
b
uvm_aiodone_daemon(c9bab318,4fc000,c0506000,0,c01001cc) at 
netbsd:uvm_aiodone_da
emon+0xc9
ds          0x11
es          0x11
fs          0x31
gs          0x11
edi         0x800
esi         0x3fff
ebp         0xca504efc
ebx         0
edx         0
ecx         0xcca0bf08
eax         0xcaa8a130
eip         0xc025e0b8  softdep_pageiodone+0x248
cs          0x9
eflags      0x10246
esp         0xca504e94
ss          0x11
netbsd:softdep_pageiodone+0x248:        movl   
%edx,0(%eax)
Stopped in pid 8.1 (aiodoned) at       
netbsd:softdep_pageiodone+0x248: 
m
ovl     %edx,0(%eax)


and the bt
db> bt
softdep_pageiodone(c09e30fc,3415c,c0745380,c9bab108,c9bab318
) at 
netbsd:softdep_
pageiodone+0x248
uvm_aio_aiodone(c09e30fc,204,c03f755f,0,c047a7e0) at 
netbsd:uvm_aio_aiodone+0x4d
b
uvm_aiodone_daemon(c9bab318,4fc000,c0506000,0,c01001cc) at 
netbsd:uvm_aiodone_da
emon+0xc9
db>


(the disk, incidentally, was full, and it does have two xbd
devices, and 
heavy activity on one.)


I've extracted and applied the patch in question; (It
applies cleanly, 
btw, save for the comments)   I'm re-compiling 3.1 with the
patch applied 
presently.

On Tue, 16 Jan 2007, Manuel Bouyer wrote:

> 
> On Tue, Jan 16, 2007 at 01:52:19PM +0000, Matthias
Scheler wrote:
>> On Tue, Jan 16, 2007 at 03:26:48AM +0000, reedreedmedia.net wrote:
>>> Anyone seen anything like this?
>>>
>>> My kernel is netbsd-3.
>>
>> This pullup might fix your problem:
>>
>> http://mail-index.netbsd.org/source-changes/2007
/01/15/0016.html
>>
>
> I don't think so, this pullup was to fix a possible
panic at boot with more
> than one xbd.
> This commit could be the fix:
> http://mail-index.netbsd.org/source-changes/2007
/01/01/0033.html
>
> but the author has not submitted a pullup request to
netbsd-3 yet ...

Hm.  I am the xen provider in question  (we are a small
shop, and I have a 
dayjob, so I haven't gotten around to really looking at it
until now)  but 
using the same kernel, I do get occasional heavy disk load
panics. 
Actually I just triggered one right now (incidentally while
I was 
attempting to manually apply the patch you suggested;  all
my NetBSD boxes 
at the moment are XenUs. )

here's the crash:

uvm_fault(0xc044ec80, 0xcaa8a000, 0, 2) -> 0xe
kernel: page fault trap, code=0
Stopped in pid 8.1 (aiodoned) at       
netbsd:softdep_pageiodone+0x248: 
m
ovl     %edx,0(%eax)
softdep_pageiodone(c09e30fc,3415c,c0745380,c9bab108,c9bab318
) at 
netbsd:softdep_
pageiodone+0x248
uvm_aio_aiodone(c09e30fc,204,c03f755f,0,c047a7e0) at 
netbsd:uvm_aio_aiodone+0x4d
b
uvm_aiodone_daemon(c9bab318,4fc000,c0506000,0,c01001cc) at 
netbsd:uvm_aiodone_da
emon+0xc9
ds          0x11
es          0x11
fs          0x31
gs          0x11
edi         0x800
esi         0x3fff
ebp         0xca504efc
ebx         0
edx         0
ecx         0xcca0bf08
eax         0xcaa8a130
eip         0xc025e0b8  softdep_pageiodone+0x248
cs          0x9
eflags      0x10246
esp         0xca504e94
ss          0x11
netbsd:softdep_pageiodone+0x248:        movl   
%edx,0(%eax)
Stopped in pid 8.1 (aiodoned) at       
netbsd:softdep_pageiodone+0x248: 
m
ovl     %edx,0(%eax)


and the bt
db> bt
softdep_pageiodone(c09e30fc,3415c,c0745380,c9bab108,c9bab318
) at 
netbsd:softdep_
pageiodone+0x248
uvm_aio_aiodone(c09e30fc,204,c03f755f,0,c047a7e0) at 
netbsd:uvm_aio_aiodone+0x4d
b
uvm_aiodone_daemon(c9bab318,4fc000,c0506000,0,c01001cc) at 
netbsd:uvm_aiodone_da
emon+0xc9
db>


(the disk, incidentally, was full, and it does have two xbd
devices, and 
heavy activity on one.)


I've extracted and applied the patch in question; (It
applies cleanly, 
btw, save for the comments)   I'm re-compiling 3.1 with the
patch applied 
presently.

On Sat, 20 Jan 2007, Luke S. Crawford wrote:
> On Tue, 16 Jan 2007, Manuel Bouyer wrote:
>> I don't think so, this pullup was to fix a possible
panic at boot with 
>> more
>> than one xbd.
>> This commit could be the fix:
>> http://mail-index.netbsd.org/source-changes/2007
/01/01/0033.html
>> 
>> but the author has not submitted a pullup request
to netbsd-3 yet ...

Hey, I've been running a 3.1 XenU with this patch for a day
or so now, and 
it looks to have solved my problem;   can it be pulled into
3.1?


thanks!

On Sun, Jan 21, 2007 at 01:52:03PM -0800, Luke S. Crawford
wrote:
> On Sat, 20 Jan 2007, Luke S. Crawford wrote:
> >On Tue, 16 Jan 2007, Manuel Bouyer wrote:
> >>I don't think so, this pullup was to fix a
possible panic at boot with 
> >>more
> >>than one xbd.
> >>This commit could be the fix:
> >>http://mail-index.netbsd.org/source-changes/2007
/01/01/0033.html
> >>
> >>but the author has not submitted a pullup
request to netbsd-3 yet ...
> 
> Hey, I've been running a 3.1 XenU with this patch for a
day or so now, and 
> it looks to have solved my problem;   can it be pulled
into 3.1?

I sent a pullup request yesterday

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Sat, Jan 20, 2007 at 06:05:37PM -0800, Luke S. Crawford
wrote:
> On Tue, 16 Jan 2007, Manuel Bouyer wrote:
> 
> >
> >On Tue, Jan 16, 2007 at 01:52:19PM +0000, Matthias
Scheler wrote:
> >>On Tue, Jan 16, 2007 at 03:26:48AM +0000,
reedreedmedia.net wrote:
> >>>Anyone seen anything like this?
> >>>
> >>>My kernel is netbsd-3.
> >>
> >>This pullup might fix your problem:
> >>
> >>http://mail-index.netbsd.org/source-changes/2007
/01/15/0016.html
> >>
> >
> >I don't think so, this pullup was to fix a possible
panic at boot with more
> >than one xbd.
> >This commit could be the fix:
> >http://mail-index.netbsd.org/source-changes/2007
/01/01/0033.html
> >
> >but the author has not submitted a pullup request
to netbsd-3 yet ...
> 
> Hm.  I am the xen provider in question  (we are a small
shop, and I have a 
> dayjob, so I haven't gotten around to really looking at
it until now)  but 
> using the same kernel, I do get occasional heavy disk
load panics. 
> Actually I just triggered one right now (incidentally
while I was 
> attempting to manually apply the patch you suggested; 
all my NetBSD boxes 
> at the moment are XenUs. )

This looks like corruption in the softdep code (I guess one
of the b_hash
or b_vnbufs lists, or sdpcpool pool). I can't see why this
would happen
only on Xen3 domUs. Also I have seveeral domUs with softdep,
and never
encountered this  ...

-- 
Manuel Bouyer, LIP6, Universite Paris VI.          
Manuel.Bouyerlip6.fr
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Sat, Jan 20, 2007 at 06:05:37PM -0800, Luke S. Crawford
wrote:
> On Tue, 16 Jan 2007, Manuel Bouyer wrote:
> 
> >
> >On Tue, Jan 16, 2007 at 01:52:19PM +0000, Matthias
Scheler wrote:
> >>On Tue, Jan 16, 2007 at 03:26:48AM +0000,
reedreedmedia.net wrote:
> >>>Anyone seen anything like this?
> >>>
> >>>My kernel is netbsd-3.
> >>
> >>This pullup might fix your problem:
> >>
> >>http://mail-index.netbsd.org/source-changes/2007
/01/15/0016.html
> >>
> >
> >I don't think so, this pullup was to fix a possible
panic at boot with more
> >than one xbd.
> >This commit could be the fix:
> >http://mail-index.netbsd.org/source-changes/2007
/01/01/0033.html
> >
> >but the author has not submitted a pullup request
to netbsd-3 yet ...
> 
> Hm.  I am the xen provider in question  (we are a small
shop, and I have a 
> dayjob, so I haven't gotten around to really looking at
it until now)  but 
> using the same kernel, I do get occasional heavy disk
load panics. 
> Actually I just triggered one right now (incidentally
while I was 
> attempting to manually apply the patch you suggested; 
all my NetBSD boxes 
> at the moment are XenUs. )

This looks like corruption in the softdep code (I guess one
of the b_hash
or b_vnbufs lists, or sdpcpool pool). I can't see why this
would happen
only on Xen3 domUs. Also I have seveeral domUs with softdep,
and never
encountered this  ...

-- 
Manuel Bouyer, LIP6, Universite Paris VI.          
Manuel.Bouyerlip6.fr
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Sun, Jan 21, 2007 at 01:52:03PM -0800, Luke S. Crawford
wrote:
> On Sat, 20 Jan 2007, Luke S. Crawford wrote:
> >On Tue, 16 Jan 2007, Manuel Bouyer wrote:
> >>I don't think so, this pullup was to fix a
possible panic at boot with 
> >>more
> >>than one xbd.
> >>This commit could be the fix:
> >>http://mail-index.netbsd.org/source-changes/2007
/01/01/0033.html
> >>
> >>but the author has not submitted a pullup
request to netbsd-3 yet ...
> 
> Hey, I've been running a 3.1 XenU with this patch for a
day or so now, and 
> it looks to have solved my problem;   can it be pulled
into 3.1?

I sent a pullup request yesterday

-- 
Manuel Bouyer <bouyerantioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la
difference
--

On Mon, Jan 22, 2007 at 12:46:44PM +0100, Manuel Bouyer
wrote:
> This looks like corruption in the softdep code (I guess
one of the b_hash
> or b_vnbufs lists, or sdpcpool pool). I can't see why
this would happen
> only on Xen3 domUs. Also I have seveeral domUs with
softdep, and never
> encountered this  ...

Without looking at the code: can't this be a problem with
splbio() not
blocking the interrupts which it should?