PG 7.1.2 Crash: cannot read xlog dir - Mailing list pgsql-admin

My situation: PG 7.1.2, Redhat 7.2, running in a chroot jail on a "VDS"
server at my new ISP. I can't recompile anything, can't upgrade PG
(basically, I'm stuck with 7.1.2).

This issue was previously noted in a thread in late 2002. The actual thread
that Tom Lane suggests it might be a permissions issue is missing from the
archive, but I found it in Google's cache ( for two Webcrawler docs:
http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=%22cannot+read+xlog+d
ir%22+&btnG=Google+Search ). As to why they aren't on
archives.postgresql.org ... ya got me.

I changed permissions to the most permissive setting I know (0777), plus I
own the directory, I own the files, and I own the postmaster process, so the
only thing I can think is that 'readdir' is badly linked or has some freaky
kernel interaction. I have Python, perl and PHP on the system, and they all
use 'opendir' and 'readdir' and 'closedir' just fine on the pg_xlog
directory.

My problem: I've deduced that the 'readdir' call is broken in my PG. I
examined the source code for 7.1 very very thoroughly (
http://developer.postgresql.org/cvsweb.cgi/pgsql-server/src/backend/access/t
ransam/xlog.c?rev=1.65.2.1&content-type=text/x-cvsweb-markup&only_with_tag=R
EL7_1_STABLE see MoveOfflineLogs). What I've found is that 'opendir' seems
to open the directory fine (does not return a NULL value), but when
'readdir' tries to grab a filename something bombs with a file system error
'No such file or directory' and it returns a NULL and 'errno' gets set. The
strange thing is that it gets in there ONCE and does ONE file
(0000000000000000) and then it won't do anymore, ever again, until I stop
the server and run initdb again.

At this point I know that there's nothing wrong with the XLOG directory or
the files in it, because PG has been writing transactions fine for 7-8 hours
up to this point. It can only be a bad 'readdir' call.

My question: Is there some runtime setting I can use to prevent
MoveOfflineLogs() from ever being called? I would MUCH rather have a couple
of old XLOGs lying around than a fatal crash. Maybe by CHECKPOINTing every
hour or something ... I've tried playing with a bunch of different WAL
settings and ... I can't stop MoveOfflineLogs from being called.

Please keep in mind my hands are tied, and I can't recompile and I can't
upgrade. Even if I could upgrade, I imagine that 'readdir' would still be
broken, and I'd still have this issue.

If anybody can think of a workaround I'd really appreciate it. I've been
racking my brain on this for a week.

Thanks

-Keith


==================

Here's the log.

/usr/local/pgsql/bin/postmaster: reaping dead processes...
/usr/local/pgsql/bin/postmaster: CleanupProc: pid 24626 exited with status 0
XLogFlush: rqst 0/12259528; wrt 0/0; flsh 0/0
XLogFlush: rqst 0/17078212; wrt 0/17078248; flsh 0/17078248
XLogFlush: rqst 0/17078152; wrt 0/17078248; flsh 0/17078248
XLogFlush: rqst 0/0; wrt 0/17078248; flsh 0/17078248
INSERT @ 0/17078248: prev 0/17078212; xprev 0/0; xid 0: XLOG - checkpoint:
redo 0/17078248; undo 0/0; sui 28; xid 3495; oid 36603; online
XLogFlush: rqst 0/17078312; wrt 0/17078248; flsh 0/17078248
DEBUG:  MoveOfflineLogs: remove 0000000000000000
FATAL 2:  MoveOfflineLogs: cannot read xlog dir: No such file or directory
DEBUG:  proc_exit(2)
DEBUG:  shmem_exit(2)
DEBUG:  exit(2)
/usr/local/pgsql/bin/postmaster: reaping dead processes...
/usr/local/pgsql/bin/postmaster: CleanupProc: pid 24736 exited with status
512
Server process (pid 24736) exited with status 512 at Sat May 31 09:57:57
2003
Terminating any active server processes...
Server processes were terminated at Sat May 31 09:57:57 2003
Reinitializing shared memory and semaphores
invoking IpcMemoryCreate(size=1236992)
DEBUG:  database system was interrupted at 2003-05-31 09:57:57 EDT
DEBUG:  CheckPoint record at (0, 17078248)
DEBUG:  Redo record at (0, 17078248); Undo record at (0, 0); Shutdown FALSE
DEBUG:  NextTransactionId: 3495; NextOid: 36603
DEBUG:  database system was not properly shut down; automatic recovery in
progress...
DEBUG:  ReadRecord: record with zero len at (0, 17078312)
DEBUG:  redo is not required
INSERT @ 0/17078312: prev 0/17078248; xprev 0/0; xid 0: XLOG - checkpoint:
redo 0/17078312; undo 0/0; sui 28; xid 3495; oid 36603; shutdown
XLogFlush: rqst 0/17078376; wrt 0/17078312; flsh 0/17078312
FATAL 2:  MoveOfflineLogs: cannot read xlog dir: No such file or directory
DEBUG:  proc_exit(2)
DEBUG:  shmem_exit(2)
DEBUG:  exit(2)

=========================

Here's the code from 7.1.

static void
MoveOfflineLogs(uint32 log, uint32 seg)
{
        DIR                   *xldir;
        struct dirent *xlde;
        char                lastoff[32];
        char                path[MAXPGPATH];

        Assert(XLOG_archive_dir[0] == 0);        /* ! implemented yet */

        xldir = opendir(XLogDir);
        if (xldir == NULL)
                elog(STOP, "MoveOfflineLogs: cannot open xlog dir: %m");

        sprintf(lastoff, "%08X%08X", log, seg);

        errno = 0;
        while ((xlde = readdir(xldir)) != NULL)
        {
                if (strlen(xlde->d_name) == 16 &&
                        strspn(xlde->d_name, "0123456789ABCDEF") == 16 &&
                        strcmp(xlde->d_name, lastoff) <= 0)
                {
                        elog(LOG, "MoveOfflineLogs: %s %s",
(XLOG_archive_dir[0]) ?
                                 "archive" : "remove", xlde->d_name);
                        sprintf(path, "%s%c%s", XLogDir, SEP_CHAR,
xlde->d_name);
                        if (XLOG_archive_dir[0] == 0)
                                unlink(path);
                }
                errno = 0;
        }
        if (errno)
                elog(STOP, "MoveOfflineLogs: cannot read xlog dir: %m");
        closedir(xldir);
}