Re: Re: Too many open files (was Re: spinlock problems reported earlier) - Mailing list pgsql-hackers

On Mon, 28 Aug 2000, Tom Lane wrote:

> Brook Milligan <brook@biology.nmsu.edu> writes:
> > In any case, if this really follows the POSIX standard, perhaps
> > PostgreSQL code should assume these semantics and work around other
> > cases that don't follow the standard (instead of work around the POSIX
> > cases).
> 
> HP asserts that *they* follow the POSIX standard, and in this case
> I'm more inclined to believe them than the *BSD camp.  A per-process
> limit on open files has existed in most Unices I've heard of; I had
> never heard of a per-userid limit until yesterday.  (And I'm not yet
> convinced that that's actually what *BSD implements; are we sure it's
> not just a typo in the man page?)
> 
> 64 or so for _SC_OPEN_MAX is not really what I'm worried about anyway.
> IIRC, we've heard reports that some platforms return values in the
> thousands, ie, essentially telling each process it can have the whole
> kernel FD table, and it's that behavior that I'm speculating is causing
> Marc's problem.
> 
> Marc, could you check what is returned by sysconf(_SC_OPEN_MAX) on your
> box?  And/or check to see how many files each backend is actually
> holding open?

> ./t
4136


> sysctl kern.maxfiles
kern.maxfiles: 32768


> cat t.c
#include <stdio.h>
#include <unistd.h>

main()
{ printf("%ld\n", sysconf(_SC_OPEN_MAX));
}

okay, slightly difficult since they come and go, but using the database
that is used for the search engine, with just a psql session:

pgsql# lsof -p 85333
COMMAND    PID  USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
postgres 85333 pgsql  cwd   VDIR  13,131088      3072   7936 /pgsql/data2/udmsearch
postgres 85333 pgsql  rtd   VDIR  13,131072       512      2 /
postgres 85333 pgsql  txt   VREG  13,131084   4651486 103175 /pgsql/bin/postgres
postgres 85333 pgsql  txt   VREG  13,131076     77648 212924 /usr/libexec/ld-elf.so.1
postgres 85333 pgsql  txt   VREG  13,131076     11860  56504 /usr/lib/libdescrypt.so.2
postgres 85333 pgsql  txt   VREG  13,131076    120736  56525 /usr/lib/libm.so.2
postgres 85333 pgsql  txt   VREG  13,131076     34336  56677 /usr/lib/libutil.so.3
postgres 85333 pgsql  txt   VREG  13,131076    154128  57068 /usr/lib/libreadline.so.4
postgres 85333 pgsql  txt   VREG  13,131076    270100  56532 /usr/lib/libncurses.so.5
postgres 85333 pgsql  txt   VREG  13,131076    570064  56679 /usr/lib/libc.so.4
postgres 85333 pgsql    0r  VCHR        2,2       0t0   7967 /dev/null
postgres 85333 pgsql    1w  VREG  13,131084       995 762037 /pgsql/logs/postmaster.5432.61308
postgres 85333 pgsql    2w  VREG  13,131084 316488878 762038 /pgsql/logs/5432.61308
postgres 85333 pgsql    3r  VREG  13,131088      1752   8011 /pgsql/data2/udmsearch/pg_internal.init
postgres 85333 pgsql    4u  VREG  13,131084  22757376  15922 /pgsql/data/pg_log
postgres 85333 pgsql    5u  unix 0xd46a3300       0t0        ->0xd469a540
postgres 85333 pgsql    6u  VREG  13,131084      8192  15874 /pgsql/data/pg_variable
postgres 85333 pgsql    7u  VREG  13,131088     16384   7982 /pgsql/data2/udmsearch/pg_class
postgres 85333 pgsql    8u  VREG  13,131088     32768   7980 /pgsql/data2/udmsearch/pg_class_relname_index
postgres 85333 pgsql    9u  VREG  13,131088     81920   7985 /pgsql/data2/udmsearch/pg_attribute
postgres 85333 pgsql   10u  VREG  13,131088     65536   7983 /pgsql/data2/udmsearch/pg_attribute_relid_attnum_index
postgres 85333 pgsql   11u  VREG  13,131088      8192   7945 /pgsql/data2/udmsearch/pg_trigger
postgres 85333 pgsql   12u  VREG  13,131088      8192   7993 /pgsql/data2/udmsearch/pg_am
postgres 85333 pgsql   13u  VREG  13,131088     16384   7977 /pgsql/data2/udmsearch/pg_index
postgres 85333 pgsql   14u  VREG  13,131088      8192   7988 /pgsql/data2/udmsearch/pg_amproc
postgres 85333 pgsql   15u  VREG  13,131088     16384   7991 /pgsql/data2/udmsearch/pg_amop
postgres 85333 pgsql   16u  VREG  13,131088     73728   7961 /pgsql/data2/udmsearch/pg_operator
postgres 85333 pgsql   17u  VREG  13,131088     16384   7976 /pgsql/data2/udmsearch/pg_index_indexrelid_index
postgres 85333 pgsql   18u  VREG  13,131088     32768   7960 /pgsql/data2/udmsearch/pg_operator_oid_index
postgres 85333 pgsql   19u  VREG  13,131088     16384   7976 /pgsql/data2/udmsearch/pg_index_indexrelid_index
postgres 85333 pgsql   20u  VREG  13,131088     16384   7942 /pgsql/data2/udmsearch/pg_trigger_tgrelid_index
postgres 85333 pgsql   21u  VREG  13,131084      8192  15921 /pgsql/data/pg_shadow
postgres 85333 pgsql   22u  VREG  13,131084      8192  15918 /pgsql/data/pg_database
postgres 85333 pgsql   23u  VREG  13,131088      8192   7952 /pgsql/data2/udmsearch/pg_rewrite
postgres 85333 pgsql   24u  VREG  13,131088     16384   7941 /pgsql/data2/udmsearch/pg_type
postgres 85333 pgsql   25u  VREG  13,131088     16384   7940 /pgsql/data2/udmsearch/pg_type_oid_index
postgres 85333 pgsql   26u  VREG  13,131088         0   7938 /pgsql/data2/udmsearch/pg_user
postgres 85333 pgsql   27u  VREG  13,131088    188416   7984 /pgsql/data2/udmsearch/pg_attribute_relid_attnam_index
postgres 85333 pgsql   28u  VREG  13,131088     65536   7959 /pgsql/data2/udmsearch/pg_operator_oprname_l_r_k_index
postgres 85333 pgsql   29u  VREG  13,131088     16384   7981 /pgsql/data2/udmsearch/pg_class_oid_index
postgres 85333 pgsql   30u  VREG  13,131088     40960   7948 /pgsql/data2/udmsearch/pg_statistic
postgres 85333 pgsql   31u  VREG  13,131088     32768   7947 /pgsql/data2/udmsearch/pg_statistic_relid_att_index
postgres 85333 pgsql   32u  VREG  13,131088    212992   7958 /pgsql/data2/udmsearch/pg_proc
postgres 85333 pgsql   33u  VREG  13,131088     49152   7957 /pgsql/data2/udmsearch/pg_proc_oid_index


when running a vacuum on the database, the only changes appear to be
adding (and removing when done) those tables that are currently being
vacuumed ... so, it appears, ~48 or so files opened ...