RE: Unable to change the content of memory usingcrash on a live system

On the other hand, there's nothing to prevent the
ambitious developer from writing their own 
/dev/crash driver that *does* have a write operation
in it, is there?

  -- David Wright, Egenera, Inc. 

> -----Original Message-----
> From: crash-utility-bouncesredhat.com 
> [mailto:crash-utility-bouncesredhat.com] On Behalf Of
Dave Anderson
> Sent: Thursday, March 06, 2008 9:37 AM
> To: Discussion list for crash utility usage,maintenance
and 
> development
> Subject: Re: [Crash-utility] Unable to change the
content of 
> memory usingcrash on a live system
> 
> Dheeraj Sangamkar wrote:
> > I use crash 4.0-3.9 on a live 2.6.9-55 kernel on
i386/i686 as root.
> > 
> > crash> ls -l /dev/crash
> > crw-------  1 root root 10, 61 Mar  5 21:57
/dev/crash
> > crash> ls -l /dev/mem
> > crw-r-----  1 root kmem 1, 1 Mar  5 16:49
/dev/mem
> > crash> q
> > [rootlinux17081 ~]# ls -l /dev/crash /dev/mem
> > ls: /dev/crash: No such file or directory
> > crw-r-----  1 root kmem 1, 1 Mar  5 16:49
/dev/mem
> > [rootlinux17081 ~]# id
> > uid=0(root) gid=0(root) 
> >
groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(whe
el)
> > 
> > So, the /dev/crash file has write permission for
me. The
> > 
> > I am attempting to change the content of some
memory.
> > 
> > crash> struct request_queue 0xf7b933f8
> > struct request_queue {
> >   queue_head = {
> > <SNIP>
> > ...
> > }
> > 
> > crash> struct -o request_queue | grep
in_flight
> >   [476] unsigned int in_flight;
> > crash> eval 0xf7b933f8 + 476
> > hexadecimal: f7b935d4
> >     decimal: 4156110292  (-138857004)
> >       octal: 36756232724
> >      binary: 11110111101110010011010111010100
> > crash> rd f7b935d4
> > f7b935d4:  fffffff1                             
....
> > crash> wr f7b935d4 0
> > wr: cannot write to /dev/crash!
> > 
> > I get the error above even if I change the
ownership of 
> /dev/kmem to 
> > root:root
> > crash> ls -l /dev/mem
> > crw-r-----  1 root root 1, 1 Mar  5 16:49
/dev/mem
> > 
> > Am I doing something wrong? How do I change the
content of 
> memory on a 
> > live system using crash?
> 
> With Red Hat x86 and x86_64 kernels, you can't.
> 
> I feel your pain...
> 
> The crash utility traditionally has had the capability
of writing
> to /dev/mem, which can be a very useful, powerful (and
dangerous)
> tool for kernel debugging.
> 
> But Red Hat deemed the /dev/mem interface as a security
hole,
> and restricted the x86 and x86_64 /dev/mem drivers to
just
> the first 256 pages (1MB) of physical memory, making it
useless
> for the crash utility.  They allowed me to create the
/dev/crash
> driver to replace it -- but it is effectively read-only
because
> the driver has no write file operations handler:
> 
>    static struct file_operations crash_fops = {
>            owner:          THIS_MODULE,
>            llseek:         crash_llseek,
>            read:           crash_read,
>    };
> 
> and so the kernel's vfs_write() returns EINVAL.
> 
> Changing the permission of /dev/mem won't help because
it
> isn't used by the crash utility when /dev/crash
exists.
> 
> Sorry about that,
>    Dave
> 
> 
> --
> Crash-utility mailing list
> Crash-utilityredhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
> 

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

"Wright, David" <dwrightegenera.com> writes:

> On the other hand, there's nothing to prevent the
> ambitious developer from writing their own 
> /dev/crash driver that *does* have a write operation
> in it, is there?

Why bother?  Just unbreak the /dev/mem interface and don't
load the
crash module.

-Jeff

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

Wright, David wrote:
> On the other hand, there's nothing to prevent the
> ambitious developer from writing their own 
> /dev/crash driver that *does* have a write operation
> in it, is there?
> 
>   -- David Wright, Egenera, Inc. 
> 

Nope.  Or removing the restriction from the /dev/mem
driver,
which runs into the per-arch devmem_is_allowed() function.

Here's the "arch/i386/mm/init.c" version:

/*
  * devmem_is_allowed() checks to see if /dev/mem access to
a certain address is
  * valid. The argument is a physical page number.
  *
  * On x86, access has to be given to the first megabyte of
ram because that area
  * contains bios code and data regions used by X and dosemu
and similar apps.
  * Access has to be given to non-kernel-ram areas as well,
these contain the PCI
  * mmio resources as well as potential bios/acpi data
regions.
  */
int devmem_is_allowed(unsigned long pagenr)
{
         if (pagenr <= 256)
                 return 1;
         if (!page_is_ram(pagenr))
                 return 1;
         return 0;
}

Dave

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility

As it turns out, another way to work around the /dev/mem
restriction
is with kprobes.

The devmem_is_allowed() function looks like this, and for
the purposes of using it by the crash utility, we'd like it
to return 1 always:

   int devmem_is_allowed(unsigned long pagenr)
   {
           if (pagenr <= 256)
                   return 1;
           if (!page_is_ram(pagenr))
                   return 1;
           return 0;
   }

I took the sample kretprobes.c file from
Documentation/kprobes.txt
and set a kretprobe in devmem_is_allowed() that forces a
return
value of 1:

   static int ret_handler(struct kretprobe_instance *ri,
struct pt_regs *regs)
   {
           regs->eax = 1;
           return 0;
   }


Here's the "kretprobes.c" module I used:

   #include <linux/kernel.h>
   #include <linux/module.h>
   #include <linux/kprobes.h>

   static const char *probed_func =
"devmem_is_allowed";

   /* Return-probe handler: force return value to be 1. */
   static int ret_handler(struct kretprobe_instance *ri,
struct pt_regs *regs)
   {
   	regs->eax = 1;
   	return 0;
   }

   static struct kretprobe my_kretprobe = {
   	.handler = ret_handler,
   	/* Probe up to 20 instances concurrently. */
   	.maxactive = 20
   };

   static int __init kretprobe_init(void)
   {
   	int ret;
   	my_kretprobe.kp.symbol_name = (char *)probed_func;

   	if ((ret = register_kretprobe(&my_kretprobe)) <
0) {
   		printk("register_kretprobe failed, returned
%dn", ret);
   		return -1;
   	}
   	printk("Planted return probe at %pn",
my_kretprobe.kp.addr);

   	return 0;
   }

   static void __exit kretprobe_exit(void)
   {
   	unregister_kretprobe(&my_kretprobe);
   	printk("kretprobe unregisteredn");
   	/* nmissed > 0 suggests that maxactive was set too
low. */
   	printk("Missed probing %d instances of %sn",
   		my_kretprobe.nmissed, probed_func);
   }

   module_init(kretprobe_init)
   module_exit(kretprobe_exit)
   MODULE_LICENSE("GPL");

And then build it with the supplied Makefile snippet:

obj-m := kretprobes.o
KDIR := /lib/modules/$(shell uname -r)/build
PWD := $(shell pwd)
default:
	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
clean:
	rm -f *.mod.c *.ko *.o

Load the module, and then while it's running, "crash
/dev/mem" will
override its  default usage of "/dev/crash" and
just work.

This was on a RHEL5 kernel, but it should work for RHEL4 as
well:

   $ crash /dev/mem

   crash 4.0-6.1
   Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 
Red Hat, Inc.
   Copyright (C) 2004, 2005, 2006  IBM Corporation
   Copyright (C) 1999-2006  Hewlett-Packard Co
   Copyright (C) 2005, 2006  Fujitsu Limited
   Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
   Copyright (C) 2005  NEC Corporation
   Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
   Copyright (C) 1999, 2000, 2001, 2002  Mission Critical
Linux, Inc.
   This program is free software, covered by the GNU General
Public License,
   and you are welcome to change it and/or distribute copies
of it under
   certain conditions.  Enter "help copying" to
see the conditions.
   This program has absolutely no warranty.  Enter
"help warranty" for details.

   GNU gdb 6.1
   Copyright 2004 Free Software Foundation, Inc.
   GDB is free software, covered by the GNU General Public
License, and you are
   welcome to change it and/or distribute copies of it under
certain conditions.
   Type "show copying" to see the conditions.
   There is absolutely no warranty for GDB.  Type "show
warranty" for details.
   This GDB was configured as
"i686-pc-linux-gnu"...

         KERNEL:
/usr/lib/debug/lib/modules/2.6.18-53.el5/vmlinux
       DUMPFILE: /dev/mem
           CPUS: 2
           DATE: Thu Mar  6 14:43:06 2008
         UPTIME: 23 days, 04:50:13
   LOAD AVERAGE: 0.14, 0.20, 0.20
          TASKS: 175
       NODENAME: crash.boston.redhat.com
        RELEASE: 2.6.18-53.el5
        VERSION: #1 SMP Wed Oct 10 16:34:02 EDT 2007
        MACHINE: i686  (1993 Mhz)
         MEMORY: 511.5 MB
            PID: 15518
        COMMAND: "crash"
           TASK: cb0ffaa0  [THREAD_INFO: d976c000]
            CPU: 0
          STATE: TASK_RUNNING (ACTIVE)

   crash> p panic_on_oops
   panic_on_oops = $2 = 1
   crash> wr panic_on_oops 2
   crash> p panic_on_oops
   panic_on_oops = $3 = 2
   crash>

Dave

--
Crash-utility mailing list
Crash-utilityredhat.com
https://www.redhat.com/mailman/listinfo/crash-utility