List Info

Thread: Malloc behavior
Malloc behavior
user name
 2007-03-10 23:06:04 

  


  

    Hi,

I'm sorry for a so big email, but this might be a bug.

As some of you may know, there is a characteristic of malloc called overcommit, which lets a program "allocate" more virtual memory than is available on the system. I've set overcommit_memory=2 and turned off swap, but I believe the behaviour of malloc is not the expected.



Let me introduce the overcommit, then show the test code, then the results.

//-- Starting introduction of overcommit

According to http://www.die.net/doc/linux/man/man3/malloc.3.html
:

"By default, Linux follows an optimistic memory
allocation strategy. This means that when malloc() returns
non-NULL there is no guarantee that the memory really is available.
This is a really bad bug. In case it turns out that the system is
out of memory, one or more processes will be killed by the infamous
OOM killer. In case Linux is employed under circumstances where it
would be less desirable to suddenly lose some randomly picked
processes, and moreover the kernel version is sufficiently recent,
one can switch off this overcommitting behavior using a command
like

# echo 2 > /proc/sys/vm/overcommit_memory
See also the
kernel Documentation directory, files
vm/overcommit-accounting and sysctl/vm.txt.";


Looking at vm/overcommit-accounting:

";The Linux kernel supports the following overcommit handling modes

0      ; -       Heuristic overcommit handling. Obvious overcommits of

 ;           ;    address space are refused. Used for a typical system. It
     ;           ensures a seriously wild allocation fails while allowing
    ;           ; overcommit to reduce swap usage.  root is allowed to

 ;           ;    allocate slightly more memory in this mode. This is the
   ;           ;  default.

1       -       Always overcommit. Appropriate for some scientific
           ;     applications.

2 ;      -       Don't overcommit. The total address space commit

    ;           ; for the system is not permitted to exceed swap + a
       ;         configurable percentage (default is 50) of physical RAM.
 ;           ;    Depending on the percentage you use, in most situations

       ;         this means a process will not be killed while accessing
  ;           ;   pages but will receive errors on memory allocation as
     ;           appropriate.

The overcommit policy is set via the sysctl `vm.overcommit_memory'.


The overcommit percentage is set via `vm.overcommit_ratio'."

//-- Ending introduction of overcommit

Then I started doing some tests. Below is my the test code:

//---------------------------- Top of test code

#include <stdio.h>
#include <malloc.h>
#include <limits.h>
#include <string.h>
#include <unistd.h>

int
main(void)
{
 &nbsp;  int i;
 &nbsp;  char *p;
   ;  &nbsp;  
4 &nbsp;  char *r = malloc(1048576);

&nbsp; &nbsp; 
 &nbsp;  // Filling some useless space to copy later
&nbsp; &nbsp; for (i = 0; i <= 1048576; i++)
 ; &nbsp; {
 &nbsp;   &nbsp;  *(char*)(r+i) = i; &nbsp;   &nbsp;  
 &nbsp;  }
 &nbsp;  
 &nbsp;  
 &nbsp;  for (i = 1; ; i++)
 ; &nbsp; {
 &nbsp;   &nbsp;  p = malloc(1048576);

&nbsp; &nbsp;  &nbsp;  
 &nbsp;   &nbsp;  if (p == NULL)
&nbsp; &nbsp;  &nbsp;  {
 &nbsp;   &nbsp;   &nbsp;  printf(&quot;malloc %d MB - %pn", i, p);
   ;  &nbsp;   &nbsp;  printf(&quot;Returned nulln";);
 &nbsp;   &nbsp;   &nbsp;  return 0; &nbsp;   &nbsp;   &nbsp;  
 &nbsp;   &nbsp;  }
 &nbsp;   &nbsp;  else

&nbsp; &nbsp;  &nbsp;  {
//   ;  &nbsp;   &nbsp;  printf(&quot;malloc %d MB - %pn", i, p);
   ;  &nbsp;   &nbsp;  if (memcpy(p, r, 1048576) == 0)
 &nbsp;   &nbsp;   &nbsp;  {
 &nbsp;   &nbsp;   &nbsp;   &nbsp;  printf(&quot;Error in memcpyn&quot;);
 ; &nbsp;  &nbsp;   &nbsp;  }
 &nbsp;   &nbsp;   &nbsp;  usleep(5);

 &nbsp;   &nbsp;  }
 &nbsp;   &nbsp;  
 &nbsp;  }
 &nbsp;   &nbsp;   
&nbsp; printf(&quot;Ok, no NULLn";);
  return 0;
}
//---------------------------- End of test code

Conditions:
Kernel: 2.6.9-42.0.10.ELsmp
RAM memory: 2GB

Swap: OFF
overcommit_memory=2 (changed by me)
overcommit_ratio=50 (default)

' free ' ; yields:
&nbsp; &nbsp;   ; &nbsp; &nbsp;           total ; &nbsp; &nbsp;  used   ; &nbsp;  free   ;  shared&nbsp; &nbsp; buffers&nbsp; &nbsp;  cached
Mem: &nbsp; &nbsp; &nbsp; 2065252&nbsp; &nbsp;  602872&nbsp; &nbsp; 1462380&nbsp; &nbsp; &nbsp;   ;  0 &nbsp; &nbsp;  12372 ; &nbsp;  283248

-/+ buffers/cache: &nbsp;   307252&nbsp; &nbsp; 1758000
Swap: &nbsp; &nbsp;   ; &nbsp;       0 &nbsp; &nbsp; &nbsp;   ; 0 &nbsp; &nbsp; &nbsp;   ; 0

There is about 1.4GB free RAM memory.

Then I've executed the test code got:
malloc 178 MB - (nil)
Returned null


According to the kernel document above: 
"The total address space commit for the system is not permitted to exceed swap + a
configurable percentage (default is 50) of physical RAM. Depending on the percentage

you use, in most situations this means a process will not be killed while accessing pages
but will receive errors on memory allocation as appropriate."

Shouldn't I have got something about 1000 MB instead of 178 MB, because 1000 MB is 50% of  physical memory? As long as I have 
1.4GB free, I should be able to allocate those 1000 MB. Are there other (configurable or not) limits for memory allocation? (ulimit is not being used)

Then I repeated the same test, but with swap on. Swap size is 1GB.


Conditions:


Kernel: 2.6.9-42.0.10.ELsmp


RAM memory: 2GB

Swap: 1GB

overcommit_memory=2 (changed by me)

overcommit_ratio=50 (default)



' free ' ; yields:
&nbsp;        &nbsp;  &nbsp; &nbsp; &nbsp; &nbsp;  total ; &nbsp; &nbsp;  used   ; &nbsp;  free   ;  shared&nbsp; &nbsp; buffers&nbsp; &nbsp;  cached
Mem: &nbsp; &nbsp; &nbsp; 2065252&nbsp; &nbsp;  605480&nbsp; &nbsp; 1459772&nbsp; &nbsp; &nbsp;   ;  0 &nbsp; &nbsp;  12628 ; &nbsp;  283512
-/+ buffers/cache: &nbsp;   309340&nbsp; &nbsp; 1755912

Swap:   ; &nbsp; 1048568&nbsp; &nbsp; &nbsp;   ;  0 &nbsp;  1048568

Then I've executde the test code and got:
malloc 1198 MB - (nil)
Returned null

It means this time I've allocated successfuly 1198 MB. Shouldn9;t I have got something about 2GB?



Thanks
Leonardo Pinheiro




  

  
  
   
     
    

  

    Re:  Malloc behavior
  


  

     user name

     2007-03-11 15:58:50
  


  

    
On 11/03/07, Leonardo Vilela Pinheiro <leopinheirogmail.com> wrote:
> Hi,
>
> I'm sorry for a so big email, but this might be a bug.
>
> As some of you may know, there is a characteristic of
malloc called
> overcommit, which lets a program "allocate"
more virtual memory than is
> available on the system. I've set overcommit_memory=2
and turned off swap,
> but I believe the behaviour of malloc is not the
expected.
>
> ...
>
> Conditions:
> Kernel: 2.6.9-42.0.10.ELsmp
> RAM memory: 2GB
> Swap: OFF
> overcommit_memory=2 (changed by me)
> overcommit_ratio=50 (default)
>
> ' free '  yields:
>                      total       used       free    
shared    buffers
> cached
> Mem:       2065252     602872    1462380          0    
 12372     283248
> -/+ buffers/cache:     307252    1758000
> Swap:                0          0          0
>
> There is about 1.4GB free RAM memory.
>
> Then I've executed the test code got:
> malloc 178 MB - (nil)
> Returned null
>
> According to the kernel document above:
> "The total address space commit for the system is
not permitted to exceed
> swap + a
> configurable percentage (default is 50) of physical
RAM. Depending on the
> percentage
> you use, in most situations this means a process will
not be killed while
> accessing pages
> but will receive errors on memory allocation as
appropriate."
>
> Shouldn't I have got something about 1000 MB instead of
178 MB, because 1000
> MB is 50% of  physical memory? As long as I have 1.4GB
free, I should be
> able to allocate those 1000 MB. Are there other
(configurable or not) limits
> for memory allocation? (ulimit is not being used)

I think the memory available with overcommit_memory=2 is for
system as
a whole, not individual processes ...

You need to increase the overcommit_ratio number to get more
memory -
strangely this number can be greater than 100

James Pearson
_______________________________________________
CentOS-devel mailing list
CentOS-develcentos.org
http://lists.centos.org/mailman/listinfo/centos-devel



  


  

    
  

  
  
   
     
    






 [1-2]











   
 





about | contact  Other archives ( Real Estate discussion Medical topics )