Re: improve overcommit docs - Mailing list pgsql-patches

At the time I wrote the original 2.6 was not out even in prerelease,
which is why I was deliberately somewhat vague about it. It is still in
prerelease, and it will in fact work slightly differently from what was
in some 2.4 kernels - there are 2 settings that govern this instead of
1. Here is the 2.6 description straight from
linux-2.6.0-test9/Documentation/vm/overcommit-accounting:

-------------------------------------------
The Linux kernel supports three overcommit handling modes

0       -       Heuristic overcommit handling. Obvious overcommits of
                address space are refused. Used for a typical system. It
                ensures a seriously wild allocation fails while allowing
                overcommit to reduce swap usage.  root is allowed to
                allocate slighly more memory in this mode. This is the
                default.

1       -       No overcommit handling. Appropriate for some scientific
                applications.

2       -       (NEW) strict overcommit. The total address space commit
                for the system is not permitted to exceed swap + a
                configurable percentage (default is 50) of physical RAM.
                Depending on the percentage you use, in most situations
                this means a process will not be killed while accessing
                pages but will receive errors on memory allocation as
                appropriate.

The overcommit policy is set via the sysctl `vm.overcommit_memory'.

The overcommit percentage is set via `vm.overcommit_ratio'.
-----------------------------------------------------



Also note that this is wrong for 2.4:

      run the machine out of memory. If your kernel supports the strict
!     paranoid modes of overcommit handling, you can also relieve this

There are 2 modes: strict (allow commit up to sizeof(swap plus 1/2 RAM) ) and paranoid (allow commit up to sizeof(swap)
).


Wordsmith it however you like

cheers

andrew


Neil Conway wrote:

>This patch makes some improvements to the section of the documentation
>that describes the Linux 2.4 memory overcommit behavior.
>
>I removed the almost content-free assertion that "You will need enough
>swap space to cover your memory needs." If this is intended to
>communicate anything meaningful, can someone rephrase it, please?
>
>This patch also includes a fix for a typo noticed by Robert Treat.
>
>Is this suitable for 7.4 (either the whole patch, or just the typo
>fix)?
>
>-Neil
>
>
>------------------------------------------------------------------------
>
>Index: doc/src/sgml/runtime.sgml
>===================================================================
>RCS file: /var/lib/cvs/pgsql-server/doc/src/sgml/runtime.sgml,v
>retrieving revision 1.218
>diff -c -r1.218 runtime.sgml
>*** doc/src/sgml/runtime.sgml    14 Nov 2003 15:43:22 -0000    1.218
>--- doc/src/sgml/runtime.sgml    16 Nov 2003 02:07:42 -0000
>***************
>*** 1294,1300 ****
>      <para>
>       Unfortunately, there is no well-defined method for determining
>       ideal values for the family of <quote>cost</quote> variables that
>!      below. You are encouraged to experiment and share
>       your findings.
>      </para>
>     </note>
>--- 1294,1300 ----
>      <para>
>       Unfortunately, there is no well-defined method for determining
>       ideal values for the family of <quote>cost</quote> variables that
>!      appear below. You are encouraged to experiment and share
>       your findings.
>      </para>
>     </note>
>***************
>*** 3267,3301 ****
>     <title>Linux Memory Overcommit</title>
>
>     <para>
>!     Linux kernels of version 2.4.* have a poor default memory
>!     overcommit behavior, which can result in the PostgreSQL server
>!     (<filename>postmaster</filename> process) being killed by the
>!     kernel if the memory demands of another process cause the system
>!     to run out of memory.
>     </para>
>
>     <para>
>!     If this happens, you will see a kernel message looking like this
>!     (consult your system documentation and configuration on where to
>!     look for such a message):
>  <programlisting>
>  Out of Memory: Killed process 12345 (postmaster).
>  </programlisting>
>!     And, of course, you will find that your database server has
>!     disappeared.
>     </para>
>
>     <para>
>      To avoid this situation, run <productname>PostgreSQL</productname>
>      on a machine where you can be sure that other processes will not
>      run the machine out of memory. If your kernel supports the strict
>!     and/or paranoid modes of overcommit handling, you can also relieve
>!     this problem by altering the system's default behaviour. This can
>!     be determined by examining the function
>!     <function>vm_enough_memory</> in the file <filename>mm/mmap.c</>
>!     in the kernel source. If this file reveals that the strict and/or
>!     paranoid modes are supported by your kernel, turn one of these
>!     modes on by using
>  <programlisting>
>  sysctl -w vm.overcommit_memory=2
>  </programlisting>
>--- 3267,3302 ----
>     <title>Linux Memory Overcommit</title>
>
>     <para>
>!     In Linux 2.4, the default virtual memory configuration is not
>!     optimal for <productname>PostgreSQL</productname>. Because of the
>!     way that the kernel implements memory overcommit, the kernel may
>!     terminate the <productname>PostgreSQL</productname> server (the
>!     <filename>postmaster</filename> process) if the memory demands of
>!     another process cause the system to run out of virtual memory.
>     </para>
>
>     <para>
>!     If this happens, you will see a kernel message that looks like
>!     this (consult your system documentation and configuration on where
>!     to look for such a message):
>  <programlisting>
>  Out of Memory: Killed process 12345 (postmaster).
>  </programlisting>
>!     This indicates that the <filename>postmaster</filename> process
>!     has been terminated due to memory pressure;
>!     <productname>PostgreSQL</productname> will need to be restarted.
>     </para>
>
>     <para>
>      To avoid this situation, run <productname>PostgreSQL</productname>
>      on a machine where you can be sure that other processes will not
>      run the machine out of memory. If your kernel supports the strict
>!     paranoid modes of overcommit handling, you can also relieve this
>!     problem by altering the system's default behaviour. This can be
>!     determined by examining the function <function>vm_enough_memory</>
>!     in the file <filename>mm/mmap.c</> in the kernel source. If this
>!     file reveals that the strict or paranoid modes are supported by
>!     your kernel, turn one of these modes on by using
>  <programlisting>
>  sysctl -w vm.overcommit_memory=2
>  </programlisting>
>***************
>*** 3304,3315 ****
>  sysctl -w vm.overcommit_memory=3
>  </programlisting>
>      for paranoid mode, or placing an equivalent entry in
>!     <filename>/etc/sysctl.conf</>.
>     </para>
>
>     <caution>
>      <para>
>!      Using these settings in a kernel which does not support these
>       modes will almost certainly increase the danger of the kernel
>       killing the database server, rather than reducing it. If in any
>       doubt, consult a kernel expert or your kernel vendor.
>--- 3305,3320 ----
>  sysctl -w vm.overcommit_memory=3
>  </programlisting>
>      for paranoid mode, or placing an equivalent entry in
>!     <filename>/etc/sysctl.conf</>.  For more information on memory
>!     overcommit handling and the meaning of these kernel settings,
>!     consult the file
>!     <filename>Documentation/vm/overcommit-accounting</filename> in a
>!     recent version of the source code for the Linux kernel.
>     </para>
>
>     <caution>
>      <para>
>!      Using these settings with a kernel which does not support these
>       modes will almost certainly increase the danger of the kernel
>       killing the database server, rather than reducing it. If in any
>       doubt, consult a kernel expert or your kernel vendor.
>***************
>*** 3317,3331 ****
>     </caution>
>
>     <para>
>!     These modes are expected to be supported in all 2.6 and later
>!     kernels. Some vendor 2.4 kernels may also support these modes. It
>!     is, however, known that some vendor documents suggest that they
>!     support them while examination of the kernel source reveals that
>!     they do not.
>!    </para>
>!
>!    <para>
>!     Note, you will need enough swap space to cover all your memory needs.
>     </para>
>    </sect2>
>   </sect1>
>--- 3322,3333 ----
>     </caution>
>
>     <para>
>!     The method described above for changing the kernel's overcommit
>!     behavior should work with Linux kernel 2.6 and later. In addition,
>!     some versions of the 2.4 kernel distributed by some vendors may
>!     also support this functionality. However, it is known that some
>!     vendor documents suggest that they support them while examination
>!     of the kernel source reveals that they do not.
>     </para>
>    </sect2>
>   </sect1>
>
>
>------------------------------------------------------------------------
>
>
>---------------------------(end of broadcast)---------------------------
>TIP 2: you can get off all lists at once with the unregister command
>    (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>
>