Re: [pgsql-hackers-win32] SRA Win32 sync() code - Mailing list pgsql-patches

On Monday 17 November 2003 11:16, Bruce Momjian wrote:
> Tom Lane wrote:
> > > Do we know that having the background writer fsync a file that was
> > > written by a backend cause all the data to fsync?  I think I could
> > > write a program to test this by timing each of these tests:
> >
> > That might prove something about the particular platform you tested it
> > on; but it would not speak to the real problem, which is what we can
> > assume is true on every platform...
>
> The attached program does test if fsync can be used on a file descriptor
> after the file is closed and then reopened.  I see:
>
>     write                  0.000613
>     write & fsync          0.001727
>     write, close & fsync   0.001633

ArchLinux, maxtor IDE HDD, write cache enabled.

[shridhar@daithan tmp]$ gcc -o test_fsync test_fsync.c
[shridhar@daithan tmp]$ ./test_fsync
write                  0.002403
write & fsync          0.009423
write, close & fsync   0.006457
[shridhar@daithan tmp]$ uname -a
Linux daithan 2.4.21 #1 SMP Tue Jul 8 19:41:52 PDT 2003 i686 unknown

> Anyway, if we find all our platforms can pass this test, we might be
> able to allow backends to do their own writes and just record the file
> name somewhere for the checkpointer to fsync.  It also shows write/fsync
> was 3x slower than simple write.
>
> Does anyone have a platform where the last duration is significantly
> different from the middle timing?

Does 30% difference above count as significant?

Assuming fsync on a file descriptor flushes dirty buffers of that file, from
all processes, would following be sufficient?

1. Open WAL with O_SYNC|O_DIRECT (Later whereever possible) And issue fsync on
WAL files whenever required.

2. Use regular writes for data files and fsync them in background.

May be if background process is the only one that issues any fsync on data
files, that could maximize overall system throughput.

Say, all backends write to a datafile and signal the background writer, that
they are blocked on this write to complete. BGWriter could chunk all such
requests and flush them/fsync when there is enough disk activity. Hopefully
none of them would be stalled for too long. That way slowest part of the
system i.e the disk will be kept full of load.

Besides since WAL writes are synchornous, backgrounds can safely push a write
and move to further business, most of the times. I guess BGWriter has to
fsync the data files anyways to recycle a WAL segment.

In idle conditions, this mechanism should not be a problem.

Just a thought. Does this take care of sync?

> I am keeping this discussion on patches because of the C program
> attachment.

I dropped win32 list. I am not subscribed to it. Just getting thread out of
it.

I will write a short program which writes to a file in different processes and
attempts to fsync them from only one. Let's see what that turns out.

 Shridhar