FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process: (swi

Email lists > FreeBSD SMP implementation group

Thread: FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process: (swi

Search this list only Search all lists
FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process: (swi
United States	2007-06-13 21:26:51
Hello, Recently, in developing our driver, we found the core dump did not work on the FBSD OS 6.2. It hung at the beginning of core dump when crash happened. We used the same configuration file to build the kernel on FBSD 6.0 (with SMP options). It worked. It dumped out the whole core file successfully. What is the problem? How to solve this is issue? Or what option should we add to get core dump work if we want to keep SMP option in the kernel configuration? The following are the prints when the dump hung. fatal trap 12 : page fault while in kernel mode cpuid = 1 ; apic id = 00 fault virtual address 0x8 fault code = supervisor read , page not present instruction pointer = 0x20:0xc059ac77 stack pointer = 0x28:0xe7fc2b30 frame pointer = 0x28:0xe7fc2b4c code segment = base 0x0 limit 0xfffff , type 0x1b = DPL0 , pres 1 , def32 1 , gran 1 processor eflags = interrupt enabled , resume , IOPL = 0 current process = 12 (swi1:net) trap number 12 panic : page fault cpuid = 0; uptime 3hours 51 minutes 18 seconds dumping 4030MB in 2 chunks chunk 0 1MB chunk 1 4031 MB (stopped and hung here) Our system: CPU: Dual Core AMD Opteron(tm) Processor 280 (2405.46-MHz 686-class CPU), 4x 1Gb RAM FreeBSD localhost.localdomain 6.2-RELEASE FreeBSD 6.2-RELEASE #1: Wed Jun 13 18:28:22 PDT 2007 rootlocalhost.localdomain:/usr/src/sys/i386/compile/YONGS MPPOLL60 i386 Attached is our kernel build configuration file. Thanks, Yong _______________________________________________ freebsd-smpfreebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-smp To unsubscribe, send any mail to "freebsd-smp-unsubscribefreebsd.org"


Re: FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process:
	2007-06-13 21:43:33
On Wed, Jun 13, 2007 at 07:26:51PM -0700, Yong Rao wrote: > Hello, > > > > Recently, in developing our driver, we found the core dump did not work > on the FBSD OS 6.2. It hung at the beginning of core dump when crash > happened. This can happen when the panic involves the storage subsystem, or if something else is performing I/O (i.e. holding a relevant lock) at the time of panic. Online debugging using ddb helps a lot. Kris _______________________________________________ freebsd-smpfreebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-smp To unsubscribe, send any mail to "freebsd-smp-unsubscribefreebsd.org"

RE: FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process:
United States	2007-06-14 01:21:53
Hi Kris, Thanks for your reply. I have 2 exactly the same equipments. The ONLY difference is the OS. One is running 6.0 FBSD. The other one is 6.2 FBSD. I built the kernels using the same configuration file (with SMP option). My problem is that the OS 6.2 does not dump core when crash happens, while the OS 6.0 does core dump perfectly. Why it is like this? Something wrong with FBSD 6.2 OS when SMP options is enabled? Thanks, Yong -----Original Message----- From: Kris Kennaway [mailto:krisobsecurity.org] Sent: Wednesday, June 13, 2007 7:44 PM To: Yong Rao Cc: freebsd-smpfreebsd.org; Jagjit Choudhary Subject: Re: FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process: (swi1:net) On Wed, Jun 13, 2007 at 07:26:51PM -0700, Yong Rao wrote: > Hello, > > > > Recently, in developing our driver, we found the core dump did not work > on the FBSD OS 6.2. It hung at the beginning of core dump when crash > happened. This can happen when the panic involves the storage subsystem, or if something else is performing I/O (i.e. holding a relevant lock) at the time of panic. Online debugging using ddb helps a lot. Kris _______________________________________________ freebsd-smpfreebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-smp To unsubscribe, send any mail to "freebsd-smp-unsubscribefreebsd.org"

Re: FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process:
	2007-06-14 14:06:03
On Wed, Jun 13, 2007 at 11:21:53PM -0700, Yong Rao wrote: > Hi Kris, > > Thanks for your reply. > > I have 2 exactly the same equipments. The ONLY difference is the OS. One > is running 6.0 FBSD. The other one is 6.2 FBSD. > > I built the kernels using the same configuration file (with SMP option). > My problem is that the OS 6.2 does not dump core when crash happens, > while the OS 6.0 does core dump perfectly. Why it is like this? > Something wrong with FBSD 6.2 OS when SMP options is enabled? Impossible to rule it out of course, but this sounds unlikely. Many of us developers perform dumps on SMP 6.2 systems regularly. There were many changes between 6.0 and 6.2, and perhaps it is as simple as your workload is exercising a slightly different code path or timing. Using DDB you will be able to determine what the other threads are doing at the time of crash and this should help to figure out why dumping is failing. Kris

RE: FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process:
United States	2007-06-14 18:37:43
Hi Kris, We are wondering if there is any connection between the core dump failure and the options DEVICE_POLLING? In our kernel, we have options DEVICE_POLLING, and in our driver (Ethernet) we have the ether_poll_register() function (not registered yet). Do you know if there is any problem with this? (Using polling mode and SMP simultaneously) Thanks, Yong -----Original Message----- From: Kris Kennaway [mailto:krisobsecurity.org] Sent: Thursday, June 14, 2007 12:06 PM To: Yong Rao Cc: Kris Kennaway; freebsd-smpfreebsd.org; Jagjit Choudhary Subject: Re: FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process: (swi1:net) On Wed, Jun 13, 2007 at 11:21:53PM -0700, Yong Rao wrote: > Hi Kris, > > Thanks for your reply. > > I have 2 exactly the same equipments. The ONLY difference is the OS. One > is running 6.0 FBSD. The other one is 6.2 FBSD. > > I built the kernels using the same configuration file (with SMP option). > My problem is that the OS 6.2 does not dump core when crash happens, > while the OS 6.0 does core dump perfectly. Why it is like this? > Something wrong with FBSD 6.2 OS when SMP options is enabled? Impossible to rule it out of course, but this sounds unlikely. Many of us developers perform dumps on SMP 6.2 systems regularly. There were many changes between 6.0 and 6.2, and perhaps it is as simple as your workload is exercising a slightly different code path or timing. Using DDB you will be able to determine what the other threads are doing at the time of crash and this should help to figure out why dumping is failing. Kris _______________________________________________ freebsd-smpfreebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-smp To unsubscribe, send any mail to "freebsd-smp-unsubscribefreebsd.org"

Re: FreeBSD-6.2, SMP, coredump -- fatal trap 12 : page fault while in kernel mode, current process:
	2007-06-14 19:41:20
On Thu, Jun 14, 2007 at 04:37:43PM -0700, Yong Rao wrote: > Hi Kris, > > We are wondering if there is any connection between the core dump > failure and the options DEVICE_POLLING? In our kernel, we have options > DEVICE_POLLING, and in our driver (Ethernet) we have the > ether_poll_register() function (not registered yet). Do you know if > there is any problem with this? (Using polling mode and SMP > simultaneously) I am not sure, sorry. I don't use this option myself. Kris _______________________________________________ freebsd-smpfreebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-smp To unsubscribe, send any mail to "freebsd-smp-unsubscribefreebsd.org"

kernel core dump hangs with the SMP kernel (FreeBSD 6.2)
United States	2007-06-20 18:53:53
Hello, I am able to isolate the problem to kernel configurations which have SMP enabled when used with 2 cpus. With ONE cpu the core dump works ok. I built the kernel with GENERIC, and deliberately crash the kernel (for testing purpose). The core dump works fine. Only added the "options SMP" and crashed the kernel, then prior to any pages being dumped out, it hangs there. Has someone successfully core dumped on a system using SMP kernel with multiple CPUs? I tried on two different boxes (different motherboards, CPUs and hard disks). Both got failed. I tried to enable the DDB, but don't know what to look for when it goes into ddb. Appreciate any pointers. a) The CPU information is CPU: Dual Core AMD Opteron(tm) Processor 280 (2405.47-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x20f12 Stepping = 2 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,AP IC,SEP,MTRR,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,S SE,SSE2,HTT> Features2=0x1<SSE3> AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow > AMD Features2=0x3<LAHF,CMP> Cores per package: 2 b) We also tried on another mother board, which has 2 CPUs. The CPU information is below. CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,AP IC,SEP,MTRR,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HT T,TM,PBE> Features2=0x4400<CNTX-ID,<b14>> real memory = 2147418112 (2047 MB) avail memory = 2096300032 (1999 MB) ACPI APIC Table: <A M I OEMAPIC > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 6 c) The following are the prints when the dump hung. mem dump: start address = 0x4352, len=0x30 Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x4352 fault code = supervisor read, page not present instruction pointer = 0x20:0xc9e9fc92 stack pointer = 0x28:0xebdbdbdc frame pointer = 0x28:0xebdbdbf8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 74231 (pnicdbg) trap number = 12 panic: page fault cpuid = 1 Uptime: 1d18h27m42s Dumping 4030 MB (2 chunks) chunk 0: 1MB (154 pages) ... ok chunk 1: 4031MB (1031776 pages) (stopped and hung here) Thanks, Yong _______________________________________________ freebsd-smpfreebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-smp To unsubscribe, send any mail to "freebsd-smp-unsubscribefreebsd.org"

RE: kernel core dump hangs with the SMP kernel on dual cpu board (FreeBSD 6.2)
United States	2007-07-07 00:01:08
Hello, I am attaching more debugs. After core dump hangs, I am able to get into ddb. But the trace for the thread where core dump is running does not look right. I was able to isolate the dumper function - ad_dump where it seems to be hanging. Hopefully these debugs will give more clues. Please advise on what more to look for. I am new to SMP and still coming up to speed. The following debugs are taken after core dump hangs: a) show pcpu b) bt 100054 -- 100054 is the thread ID, which does the core dump c) alltrace (in attached file "alltrace.txt") d) step and next command in debug a) --- db> show pcpu cpuid = 1 curthread = 0xc9450d80: pid 16 "swi6: Giant taskq" curpcb = 0xe7fe9d90 fpcurthread = none idlethread = 0xc944f600: pid 10 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db> show pcpu 0 cpuid = 0 curthread = 0xc96eb000: pid 894 "disablemeta" curpcb = 0xe9b4fd90 fpcurthread = none idlethread = 0xc944f780: pid 11 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: db> b) --- db> bt 100054 Tracing pid 894 tid 100054 td 0xc96eb000 sched_switch(3225858983,10,0,3382180240,3377804288,...) at sched_switch+331 _end(53,128,606517178,0,0,...) at 3920951848 _end(3377796632,3379230208,3379230476,3377796608,3379229696, ...) at 3377796608 _end(3228796064,3228796064,3228796064,3228796064,3228796064, ...) at 3379230208 null_method(0,0,0,0,0,...) at null_method db> c) --- Please see attached file "alltrace.txt" for details. d) trace --- db> trace Tracing pid 943 tid 100080 td 0xc998d000 kdb_enter(3228283314) at kdb_enter+43 panic(3390017776,3237235458,3956644984,21554,3389785424,...) at panic+295 mtpioctl(3389997312,3237235458,3389811456,3,3382235136,...) at mtpioctl+193 devfs_ioctl_f(3379694976,3237235458,3389811456,3376546688,33 82235136) at devfs_5 ioctl(3382235136,3956645124) at ioctl+874 syscall(59,59,59,3216893976,134516360,...) at syscall+559 Xint0x80_syscall() at Xint0x80_syscall+31 --- syscall (54, FreeBSD ELF32, ioctl), eip = 672367343, esp = 3216893900, ebp - db> s [thread pid 943 tid 100080 ] Stopped at kdb_enter+44: leave db> s [thread pid 943 tid 100080 ] Stopped at kdb_enter+45: ret db> s [thread pid 943 tid 100080 ] Stopped at panic+295: addl $4,%esp db> s [thread pid 943 tid 100080 ] Stopped at panic+298: pushl $561 db> s [thread pid 943 tid 100080 ] Stopped at panic+303: pushl $3228283350 db> s [thread pid 943 tid 100080 ] Stopped at panic+308: pushl $0 db> s [thread pid 943 tid 100080 ] Stopped at panic+310: pushl $3228776128 db> s [thread pid 943 tid 100080 ] Stopped at panic+315: call _mtx_lock_spin_flags db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags: pushl %ebp db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+1: movl %esp,%ebp db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+3: pushl %edi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+4: pushl %esi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+5: pushl %ebx db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+6: movl 8(%ebp),%esi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+9: movl 20(%ebp),%edi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+12: pushl %edi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+13: pushl 16(%ebp) db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+16: movl 12(%ebp),%eax db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+19: orl $9,%eax db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+22: pushl %eax db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+23: pushl %esi db> s [thread pid 943 tid 100080 ] Stopped at _mtx_lock_spin_flags+24: call witness_checkorder db> n After 22 instructions (0 loads, 0 stores), [thread pid 943 tid 100080 ] Stopped at witness_checkorder+1259: ret db> n KDB: stack backtrace: hangs here ... Thanks, Yong Rao Force10 Networks Inc. 408 571 3617 -----Original Message----- From: Yong Rao Sent: Wednesday, June 20, 2007 4:54 PM To: freebsd-smpfreebsd.org Cc: Jagjit Choudhary Subject: kernel core dump hangs with the SMP kernel (FreeBSD 6.2) Hello, I am able to isolate the problem to kernel configurations which have SMP enabled when used with 2 cpus. With ONE cpu the core dump works ok. I built the kernel with GENERIC, and deliberately crash the kernel (for testing purpose). The core dump works fine. Only added the "options SMP" and crashed the kernel, then prior to any pages being dumped out, it hangs there. Has someone successfully core dumped on a system using SMP kernel with multiple CPUs? I tried on two different boxes (different motherboards, CPUs and hard disks). Both got failed. I tried to enable the DDB, but don't know what to look for when it goes into ddb. Appreciate any pointers. a) The CPU information is CPU: Dual Core AMD Opteron(tm) Processor 280 (2405.47-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x20f12 Stepping = 2 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,AP IC,SEP,MTRR,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,S SE,SSE2,HTT> Features2=0x1<SSE3> AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow > AMD Features2=0x3<LAHF,CMP> Cores per package: 2 b) We also tried on another mother board, which has 2 CPUs. The CPU information is below. CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2800.11-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,AP IC,SEP,MTRR,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HT T,TM,PBE> Features2=0x4400<CNTX-ID,<b14>> real memory = 2147418112 (2047 MB) avail memory = 2096300032 (1999 MB) ACPI APIC Table: <A M I OEMAPIC > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 6 c) The following are the prints when the dump hung. mem dump: start address = 0x4352, len=0x30 Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x4352 fault code = supervisor read, page not present instruction pointer = 0x20:0xc9e9fc92 stack pointer = 0x28:0xebdbdbdc frame pointer = 0x28:0xebdbdbf8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 74231 (pnicdbg) trap number = 12 panic: page fault cpuid = 1 Uptime: 1d18h27m42s Dumping 4030 MB (2 chunks) chunk 0: 1MB (154 pages) ... ok chunk 1: 4031MB (1031776 pages) (stopped and hung here) Thanks, Yong _______________________________________________ freebsd-smpfreebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-smp To unsubscribe, send any mail to "freebsd-smp-unsubscribefreebsd.org"

[1-8]

about | contact Other archives ( Real Estate discussion Medical topics )