Thread: [BUGS] BUG #14736: Crash on postgresql server by autovacuum worker process

The following bug has been logged on the website:

Bug reference:      14736
Logged by:          JOE
Email address:      jothiprasath21@gmail.com
PostgreSQL version: 9.3.14
Operating system:   CentOS release 6.7 (Final)
Description:

I got the following stack trace after the crash and i'm not sure what were
the queries executing at the point of time. 
This got failed in XLogFileInit () and it is running fine on a restart.

Core was generated by `postgres: autovacuum worker process   icon                        '.
Program terminated with signal 6, Aborted.
#0  0x00007fdda6972625 in raise (sig=<value optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install
postgresql93-server-9.3.14-1PGDG.rhel6.x86_64
(gdb) bt full
#0  0x00007fdda6972625 in raise (sig=<value optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64       resultvar = 0       pid = <value optimized out>       selftid = <value
optimizedout>
 
#1  0x00007fdda6973e05 in abort () at abort.c:92       save_stage = 2       act = {__sigaction_handler = {sa_handler =
0x7fff6fe91f80,
sa_sigaction = 0x7fff6fe91f80}, sa_mask = {__val = {12037408, 0, 7663537,
140735070937200, 8898152,             140589959244632, 140589986960592, 140735070937312, 4294967295,
140735070937312, 1, 11638160, 0, 0, 0, 0}}, sa_flags = -1474629440,
sa_restorer = 0x1}       sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x0000000000731859 in errfinish ()
No symbol table info available.
#3  0x00000000004b7e93 in XLogFileInit ()
No symbol table info available.
#4  0x00000000004b813d in ?? ()
No symbol table info available.
#5  0x00000000004b9987 in XLogFlush ()
No symbol table info available.
#6  0x00000000004aa3e8 in ?? ()
No symbol table info available.
#7  0x00000000004aa66d in ?? ()
No symbol table info available.
#8  0x00000000004ab975 in CommitTransactionCommand ()
No symbol table info available.
#9  0x000000000057da0a in vacuum ()
No symbol table info available.
#10 0x0000000000610175 in ?? ()
No symbol table info available.
#11 0x0000000000610616 in ?? ()
No symbol table info available.
#12 0x00000000006106e6 in StartAutoVacWorker ()
No symbol table info available.
#13 0x000000000061af52 in ?? ()
No symbol table info available.
#14 <signal handler called>
No symbol table info available.
#15 0x00007fdda6a21393 in __select_nocancel () at
../sysdeps/unix/syscall-template.S:82
No locals.
#16 0x000000000061c380 in PostmasterMain ()
No symbol table info available.
#17 0x00000000005b8428 in main ()

Thanks



--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

jothiprasath21@gmail.com writes:
> I got the following stack trace after the crash and i'm not sure what were
> the queries executing at the point of time. 

The postmaster log would probably have told you what happened ... did you
save it?

Just looking at the stack trace, I'm thinking it probably was a file
write error (maybe just out-of-disk-space?) that got promoted to PANIC
because we were inside a critical section for commit.  If so, it's
operating as designed.
        regards, tom lane


-- 
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

I don't think this could be because of out-of-disk-space.

$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 14498
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  3.6G  3.9G  48% /
tmpfs           1.8G     0  1.8G   0% /dev/shm
/dev/xvdb        20G  7.5G   12G  41% /mywork
/dev/xvdc       4.0G  1.7G  2.2G  44% /mnt

The postgres logs just show the below. (Crash happened on 03, July and no
specific logs)
LOG:  database system was shut down at 2017-06-10 00:03:12 GMT
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

Thanks, JOE



--
View this message in context:
http://www.postgresql-archive.org/BUG-14736-Crash-on-postgresql-server-by-autovacuum-worker-process-tp5970386p5970544.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.


-- 
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

Hello Tom

Could you please let me know what are the other possibilities that could
have caused this crash?
As i mentioned above, there is no supporting log and disk space is available
very much.
I would like to conclude on this with possibilities of this crash and any
measures to avoid this in future.

Thanks



--
View this message in context:
http://www.postgresql-archive.org/BUG-14736-Crash-on-postgresql-server-by-autovacuum-worker-process-tp5970386p5971103.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.


-- 
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

jothiprasath216 <jothiprasath21@gmail.com> writes:
> Could you please let me know what are the other possibilities that could
> have caused this crash?

There was, absolutely positively, a log message emitted by that
elog/ereport call before it called abort().  If you didn't find it
then you're looking in the wrong place, or you have a broken
logging configuration.

Without that message nor any other information, it's impossible to
speculate further.
        regards, tom lane


-- 
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

The log configuration in postgresql.conf file is as below,

datestyle = 'iso, mdy'
default_text_search_config = 'pg_catalog.english'
lc_messages = 'en_US.UTF-8'
lc_monetary = 'en_US.UTF-8'
lc_numeric = 'en_US.UTF-8'
lc_time = 'en_US.UTF-8'
listen_addresses = '*'
log_directory = 'pg_log'
log_filename = 'postgresql-%a.log'
log_rotation_age = '1d'
log_rotation_size = 0
log_truncate_on_rotation = on
logging_collector = on
max_connections = 100
port = 5432
shared_buffers = '32MB'

With this configuration, i'm left with only one log file to search for the
error log, in which i could not find any error specific error logs.
I have already attached the final logs which are present in the
corresponding log file. 
That is, no logs after "LOG:  autovacuum launcher started"

Apart from this, i could not find any other postresql log file in the
machine.
Is there any other way to get the error logs for this crash?



--
View this message in context:
http://www.postgresql-archive.org/BUG-14736-Crash-on-postgresql-server-by-autovacuum-worker-process-tp5970386p5971124.html
Sent from the PostgreSQL - bugs mailing list archive at Nabble.com.


-- 
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs

On 13 July 2017 at 11:04, jothiprasath216 <jothiprasath21@gmail.com> wrote:

> With this configuration, i'm left with only one log file to search for the
> error log, in which i could not find any error specific error logs.
> I have already attached the final logs which are present in the
> corresponding log file.
> That is, no logs after "LOG:  autovacuum launcher started"

I suppose we already know there was definitely some kind of I/O error
when writing the transaction log it's not a huge stretch to imagine
the same error may have prevented the log from being written. Possibly
the disk was full briefly and then the condition eased. Or possibly a
hardware fault of some kind. Filesystem errors can cause the
filesystem to be remounted ro which someone perhaps "fixed" or
rebooted the system subsequently?

One thing I was going to mention was to check "df -i" as well which
people often don't think of.

If this is a reoccurring problem you could configure the logs to be
sent remotely to a different system.

-- 
greg


-- 
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs