fuse hangs in request

Thread: fuse hangs in request_send()

Search this list only Search all lists
fuse hangs in request_send()
Germany	2008-03-03 14:00:15
Hello, with 2.6.22.18 and unionfs-fuse we found a way to make fuse reliably hang in request_send(). Any idea what this could be? (libfuse is 2.6.5) Thanks in advance, Bernd (gdb) l *(request_send+0x277) 0xee7 is in request_send (fs/fuse/dev.c:313). 308 if (req->state == FUSE_REQ_PENDING) { 309 list_del(&req->list); 310 __fuse_put_request(req); 311 } else if (req->state == FUSE_REQ_SENT) { 312 spin_unlock(&fc->lock); 313 wait_event(req->waitq, req->state == FUSE_REQ_FINISHED); 314 spin_lock(&fc->lock); 315 } 316 } 317 [ 853.942702] SysRq : Show Blocked State [ 853.946564] [ 853.946565] free sibling [ 853.955577] task PC stack pid father child younger older [ 853.963369] kate D 0000000100006aae 0 4755 4714 (NOTLB) [ 853.970155] ffff8103ed47fd88 0000000000000086 0000000000000000 ffff81042fbfd670 [ 853.977745] ffff8103ed47fd28 ffffffff80220d93 000000000000000a ffff8103f5b387d0 [ 853.985344] ffff81042fbfd670 000000792272293e 000000000000127b ffff8103f5b38980 [ 853.992720] Call Trace: [ 853.995451] [<ffffffff88199ee7>] :fuse:request_send+0x277/0x2a1 [ 854.001546] [<ffffffff8819d046>] :fuse:fuse_open_common+0x103/0x152 [ 854.008013] [<ffffffff8026eb2b>] __dentry_open+0xd9/0x1aa [ 854.013600] [<ffffffff8026ecb3>] do_filp_open+0x2d/0x3d [ 854.019018] [<ffffffff8026ed07>] do_sys_open+0x44/0xc8 [ 854.024319] [<ffffffff802094de>] system_call+0x7e/0x83 [ 854.029623] [<00002b40865dc992>] [ 854.032992] [ 854.034530] ipmi-sensors D 0000000100007bef 0 4789 3918 (NOTLB) [ 854.041289] ffff8103eb649c50 0000000000000086 0000000000000000 ffff81043ef5abc8 [ 854.048886] ffffffff8024f4d8 ffff8103eb649c5c 0000000000000008 ffff81042909e3c0 [ 854.056476] ffff81042fb715f0 0000007d3cd38b77 0000000000017f85 ffff81042909e570 [ 854.063854] Call Trace: [ 854.066556] [<ffffffff80456606>] __mutex_lock_slowpath+0x5d/0x98 [ 854.072734] [<ffffffff80456482>] mutex_lock+0xa/0xb [ 854.077773] [<ffffffff802769e9>] do_lookup+0x81/0x1ae [ 854.083003] [<ffffffff8027896a>] __link_path_walk+0x87a/0xd0c [ 854.088917] [<ffffffff80278e54>] link_path_walk+0x58/0xe0 [ 854.094483] [<ffffffff802791c1>] do_path_lookup+0x1a0/0x1c4 [ 854.100226] [<ffffffff802799f2>] __user_walk_fd+0x37/0x53 [ 854.105800] [<ffffffff8026f1ef>] sys_faccessat+0x9c/0x153 [ 854.111403] [<ffffffff802094de>] system_call+0x7e/0x83 [ 854.116718] [<00002b4f3d118bb7>] [ 854.120094] ------------------------------------------------------------ ------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ fuse-devel mailing list fuse-devellists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fuse-devel

Re: fuse hangs in request_send()
Germany	2008-03-03 14:14:13
Bernd Schubert wrote: > Hello, > > with 2.6.22.18 and unionfs-fuse we found a way to make fuse reliably hang > in request_send(). Any idea what this could be? (libfuse is 2.6.5) > > > Thanks in advance, > Bernd > > > (gdb) l *(request_send+0x277) > 0xee7 is in request_send (fs/fuse/dev.c:313). > 308 if (req->state == FUSE_REQ_PENDING) { > 309 list_del(&req->list); > 310 __fuse_put_request(req); > 311 } else if (req->state == FUSE_REQ_SENT) { > 312 spin_unlock(&fc->lock); > 313 wait_event(req->waitq, req->state == > FUSE_REQ_FINISHED); > 314 spin_lock(&fc->lock); > 315 } > 316 } > 317 Ah, seems this got inlined, this is in request_wait_answer() ------------------------------------------------------------ ------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ fuse-devel mailing list fuse-devellists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fuse-devel

Re: fuse hangs in request_send()
Germany	2008-03-03 14:39:50
Bernd Schubert wrote: > Hello, > > with 2.6.22.18 and unionfs-fuse we found a way to make fuse reliably hang > in request_send(). Any idea what this could be? (libfuse is 2.6.5) Ah, also forgot this. The last strace output of unionfs-fuse is: [pid 3683] <... futex resumed> ) = 0 [pid 4760] writev(5, [{"20/300", 16}], 1 <unfinished ...> [pid 3683] rt_sigreturn(0x6b6c3c <unfinished ...> [pid 4760] <... writev resumed> ) = 16 [pid 3683] <... rt_sigreturn resumed> ) = -1 EINTR (Interrupted system call) [pid 4760] read(5, <unfinished ...> Somehow this is rather odd, IMHO. According to the trace I sent, the kernel part of fuse tries to open a file with fuse_open_common(). But strace of the userspace part seems to want to read something from /dev/fuse. Nice deadlock, request_send() waits for data from /dev/fuse and libfuse also wants to read from /dev/fuse, but neither wants to write to it. > > > Thanks in advance, > Bernd > > > (gdb) l *(request_send+0x277) > 0xee7 is in request_send (fs/fuse/dev.c:313). > 308 if (req->state == FUSE_REQ_PENDING) { > 309 list_del(&req->list); > 310 __fuse_put_request(req); > 311 } else if (req->state == FUSE_REQ_SENT) { > 312 spin_unlock(&fc->lock); > 313 wait_event(req->waitq, req->state == > FUSE_REQ_FINISHED); > 314 spin_lock(&fc->lock); > 315 } > 316 } > 317 > > > [ 853.942702] SysRq : Show Blocked State > [ 853.946564] > [ 853.946565] free > [ sibling > [ 853.955577] task PC stack pid father child > [ younger older > [ 853.963369] kate D 0000000100006aae 0 4755 4714 (NOTLB) > [ 853.970155] ffff8103ed47fd88 0000000000000086 0000000000000000 > [ ffff81042fbfd670 > [ 853.977745] ffff8103ed47fd28 ffffffff80220d93 000000000000000a > [ ffff8103f5b387d0 > [ 853.985344] ffff81042fbfd670 000000792272293e 000000000000127b > [ ffff8103f5b38980 853.992720] Call Trace: > [ 853.995451] [<ffffffff88199ee7>] :fuse:request_send+0x277/0x2a1 > [ 854.001546] [<ffffffff8819d046>] :fuse:fuse_open_common+0x103/0x152 > [ 854.008013] [<ffffffff8026eb2b>] __dentry_open+0xd9/0x1aa > [ 854.013600] [<ffffffff8026ecb3>] do_filp_open+0x2d/0x3d > [ 854.019018] [<ffffffff8026ed07>] do_sys_open+0x44/0xc8 > [ 854.024319] [<ffffffff802094de>] system_call+0x7e/0x83 > [ 854.029623] [<00002b40865dc992>] > [ 854.032992] > [ 854.034530] ipmi-sensors D 0000000100007bef 0 4789 3918 (NOTLB) > [ 854.041289] ffff8103eb649c50 0000000000000086 0000000000000000 > [ ffff81043ef5abc8 > [ 854.048886] ffffffff8024f4d8 ffff8103eb649c5c 0000000000000008 > [ ffff81042909e3c0 > [ 854.056476] ffff81042fb715f0 0000007d3cd38b77 0000000000017f85 > [ ffff81042909e570 854.063854] Call Trace: > [ 854.066556] [<ffffffff80456606>] __mutex_lock_slowpath+0x5d/0x98 > [ 854.072734] [<ffffffff80456482>] mutex_lock+0xa/0xb > [ 854.077773] [<ffffffff802769e9>] do_lookup+0x81/0x1ae > [ 854.083003] [<ffffffff8027896a>] __link_path_walk+0x87a/0xd0c > [ 854.088917] [<ffffffff80278e54>] link_path_walk+0x58/0xe0 > [ 854.094483] [<ffffffff802791c1>] do_path_lookup+0x1a0/0x1c4 > [ 854.100226] [<ffffffff802799f2>] __user_walk_fd+0x37/0x53 > [ 854.105800] [<ffffffff8026f1ef>] sys_faccessat+0x9c/0x153 > [ 854.111403] [<ffffffff802094de>] system_call+0x7e/0x83 > [ 854.116718] [<00002b4f3d118bb7>] > [ 854.120094] > > > > ------------------------------------------------------------ ------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ ------------------------------------------------------------ ------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ fuse-devel mailing list fuse-devellists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fuse-devel

Re: fuse hangs in request_send()
Hungary	2008-03-04 11:54:47
> Ah, also forgot this. The last strace output of unionfs-fuse is: > > [pid 3683] <... futex resumed> ) = 0 > [pid 4760] writev(5, [{"20/300", 16}], 1 > <unfinished ...> > [pid 3683] rt_sigreturn(0x6b6c3c <unfinished ...> > [pid 4760] <... writev resumed> ) = 16 > [pid 3683] <... rt_sigreturn resumed> ) = -1 EINTR (Interrupted system > call) > [pid 4760] read(5, <unfinished ...> > > Somehow this is rather odd, IMHO. According to the trace I sent, the kernel > part of fuse tries to open a file with fuse_open_common(). > > But strace of the userspace part seems to want to read something > from /dev/fuse. Nice deadlock, request_send() waits for data from /dev/fuse > and libfuse also wants to read from /dev/fuse, but neither wants to write to > it. If there was only one thread, it would be a deadlock, yes. But there are more then one threads (even the short strace snippet shows that). So the interesting question is: what are those other threads doing? Is this reproducible? Can you do a run with '-d' to see which requests have been answered and which haven't? Miklos ------------------------------------------------------------ ------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ fuse-devel mailing list fuse-devellists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fuse-devel

Re: fuse hangs in request_send()
Germany	2008-03-05 04:32:49
Hello Miklos, On Tuesday 04 March 2008 18:54:47 Miklos Szeredi wrote: > > Ah, also forgot this. The last strace output of unionfs-fuse is: > > > > [pid 3683] <... futex resumed> ) = 0 > > [pid 4760] writev(5, [{"20/300", 16}], 1 > > <unfinished ...> > > [pid 3683] rt_sigreturn(0x6b6c3c <unfinished ...> > > [pid 4760] <... writev resumed> ) = 16 > > [pid 3683] <... rt_sigreturn resumed> ) = -1 EINTR (Interrupted system > > call) > > [pid 4760] read(5, <unfinished ...> > > > > Somehow this is rather odd, IMHO. According to the trace I sent, the > > kernel part of fuse tries to open a file with fuse_open_common(). > > > > But strace of the userspace part seems to want to read something > > from /dev/fuse. Nice deadlock, request_send() waits for data from > > /dev/fuse and libfuse also wants to read from /dev/fuse, but neither > > wants to write to it. > > If there was only one thread, it would be a deadlock, yes. But there > are more then one threads (even the short strace snippet shows that). Actually it is even more complex. The strace is almost the same for single threaded unionfs, but we have different unionfs processes for /usr and /etc. In this case the program tried to write to a directory in /usr, which is a link to /etc. I first didn't see this link and therefor did the wrong strace... > > So the interesting question is: what are those other threads doing? > > Is this reproducible? Can you do a run with '-d' to see which > requests have been answered and which haven't? Fortunately it is easily reproducible and I will debug this during the next days. Thanks for your help, Bernd -- Bernd Schubert Q-Leap Networks GmbH ------------------------------------------------------------ ------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ fuse-devel mailing list fuse-devellists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/fuse-devel

[1-5]

about | contact Other archives ( Real Estate discussion Medical topics )