Re: fallocate / posix_fallocate for new WAL file creation (etc...) - Mailing list pgsql-hackers

On 2013-11.06 17:28, Jon Nelson wrote:
> There hasn't been much activity here recently.  I'm curious, then, if
> there are questions that I can answer.
> It may be useful to summarize some things here:
>
> - the purpose of the patch is to use posix_fallocate when creating new
> WAL files, because it's (usually) much quicker
> - using posix_fallocate is *one* system call versus 2048 calls to write(2)
> - additionally, using posix_fallocate /guarantees/ that the filesystem
> has space for the WAL file (by spec)
> - reportedly (difficult to test or prove), using posix_fallocate *may*
> reduce file fragmentation
> - the (limited) testing I've done bears this out: the more new WAL
> file creation there is, the more the improvement. Once the number of
> WAL files reaches a constant point, there does not appear to be either
> a positive or a negative performance impact. This is as expected.
> - a test program (C) was also written and used which creates,
> allocates, and then writes to files as fast as possible. This test
> program also shows the expected performance benefits.
> - the performance benefits range from a few percent up to about 15 percent

tried the test program of the patch at least a bit.

Retrieved it from:
http://www.postgresql.org/message-id/attachment/29088/test_fallocate.c

on an oldish linux box (Kernel 2.6.32, x86_64) following    $ gcc -o test_fallocate test_fallocate.c
a short    $ ./test_fallocate foo 1 1
yields:
without posix_fallocate: 1 open/close iterations, 1 rewrite in 26.1993s
with posix_fallocate: 1 open/close iterations, 1 rewrite in 13.3299s

on another box (Kernel 3.2.0, x86_64) same procedure yields:
without posix_fallocate: 1 open/close iterations, 1 rewrite in 19.1972s
with posix_fallocate: 1 open/close iterations, 1 rewrite in 9.9280s

Note, when trying gcc -O2 test_fallocate.c fails to compile with:

In file included from /usr/include/fcntl.h:252:0,                 from test_fallocate.c:3:
In function ‘open’,    inlined from ‘main’ at test_fallocate.c:68:16:
/usr/include/x86_64-linux-gnu/bits/fcntl2.h:51:24: error: call to 
‘__open_missing_mode’ declared with attribute error: open with O_CREAT 
in second argument needs 3 arguments


> Concerns:
> - some were concerned that the spec makes no claims about
> posix_fallocate being able to guarantee that the space allocated has
> zeroes in it. This was discussed here and on the Linux Kernel mailing
> list, wherein the expected behavior is that it does provide zeroes
> - most systems don't allocate a great many new WAL files, so the
> performance benefit is small
> - <your concern here>
>
> Benefits:
> - new WAL file allocate is much quicker, more efficient (fewer system calls)
> - the patch is (reportedly - I'm not a good judge here!) quite small

HTH,
Stefan.