Thread: stuck spinlock detected at dynahash.c
today I found in logs: WARNING: pgstat wait timeout WARNING: pgstat wait timeout PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981 PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981 ... LOG: server process (PID 8078) was terminated by signal 6: Aborted LOG: terminating any other active server processes WARNING: terminating connection because of crash of another server process DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited a .... LOG: archiver process (PID 6277) exited with exit code 1 FATAL: the database system is in recovery mode .... LOG: all server processes terminated; reinitializing ..... LOG: database system was interrupted; last known up at 2010-10-06 15:03:32 MSD LOG: database system was not properly shut down; automatic recovery in progress LOG: redo starts at B7/3FF16C28 LOG: record with zero length at B7/4B810F58 LOG: redo done at B7/4B810F28 LOG: last completed transaction was at log time 2010-10-06 15:06:44.254586+04 LOG: autovacuum launcher started LOG: database system is ready to accept connections what was that? I google only http://archives.postgresql.org/pgsql-general/2009-11/msg00871.php postgresql-8.4.4-1.fc13.x86_64 -- All bugs reserved
Excerpts from Timon's message of mié oct 06 07:35:44 -0400 2010: > today I found in logs: > > WARNING: pgstat wait timeout > WARNING: pgstat wait timeout > PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981 > PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981 What platform is this? Are you running on some sort of virtualized system? -- Álvaro Herrera <alvherre@commandprompt.com> The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support
no, simple dedicated server with Linux Fedora 13 x86_64 kernel-2.6.33.6-147.2.4.fc13.x86_64 2010/10/6 Alvaro Herrera <alvherre@commandprompt.com>: > Excerpts from Timon's message of mié oct 06 07:35:44 -0400 2010: >> today I found in logs: >> >> WARNING: pgstat wait timeout >> WARNING: pgstat wait timeout >> PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981 >> PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981 > > What platform is this? Are you running on some sort of virtualized > system? > > > -- > Álvaro Herrera <alvherre@commandprompt.com> > The PostgreSQL Company - Command Prompt, Inc. > PostgreSQL Replication, Consulting, Custom Development, 24x7 support > -- All bugs reserved
Timon <timosha@gmail.com> writes:
> 2010/10/6 Alvaro Herrera <alvherre@commandprompt.com>:
>>> WARNING: �pgstat wait timeout
>>> WARNING: �pgstat wait timeout
>>> PANIC: �stuck spinlock (0x7fd877761af8) detected at dynahash.c:981
>>> PANIC: �stuck spinlock (0x7fd877761af8) detected at dynahash.c:981
>>
>> What platform is this? �Are you running on some sort of virtualized
>> system?
> no, simple dedicated server with Linux Fedora 13 x86_64
> kernel-2.6.33.6-147.2.4.fc13.x86_64
There's not a lot we can do with only this much information. I'd
suggest running the postmaster with core dumps enabled (ulimit -c
unlimited) so that if it happens again, the PANICs will result in
core dumps that you can get stack traces from.
regards, tom lane
---------- Forwarded message ---------- From: Timon <timosha@gmail.com> Date: 2010/10/8 Subject: Re: [GENERAL] stuck spinlock detected at dynahash.c To: Tom Lane <tgl@sss.pgh.pa.us> I have this (thanks abrtd): ------------------------------- Crash ID: 26:a6235a787c4e9c6c5ab4f6eadd2168371b4e0fe3 Last crash: Wed 06 Oct 2010 03:13:42 PM MSD Analyzer: CCpp Component: postgresql Package: postgresql-server-8.4.4-1.fc13 Command: 'postgres: autovacuum worker process ' '' '' '' '' '' Executable: /usr/bin/postgres System: Fedora release 13 (Goddard), kernel 2.6.33.6-147.2.4.fc13.x86_64 Rating: 0 Coredump file: /var/spool/abrt/ccpp-1286363576-2966/coredump Reason: Process /usr/bin/postgres was killed by signal 6 (SIGABRT) Hostname: xxxxxxxxx Backtrace: BFD: Warning: /var/spool/abrt/ccpp-1286363576-2966/coredump is truncated: expected core file size >= 8587022336, found: 65536. warning: core file may not match specified executable file. [New Thread 2966] Cannot access memory at address 0x33d3820108 Cannot access memory at address 0x33d3820108 Cannot access memory at address 0x33d3820108 Cannot access memory at address 0x33d3820108 Cannot access memory at address 0x33d3820108 Failed to read a valid object file image from memory. Core was generated by `postgres: autovacuum worker process '. Program terminated with signal 6, Aborted. #0 0x00000033d3a329a5 in ?? () Thread 1 (Thread 2966): #0 0x00000033d3a329a5 in ?? () No symbol table info available. Cannot access memory at address 0x7fffaebbae78 Cannot access memory at address 0x33d3820108 From To Syms Read Shared Object Library 0x000000306fe00af0 0x000000306fe18904 Yes /lib64/ld-linux-x86-64.so.2 No symbol "__abort_msg" in current context. No symbol "__glib_assert_msg" in current context. rax 0x0 0 rbx 0xac0840 11274304 rcx 0xffffffffffffffff -1 rdx 0x6 6 rsi 0xb96 2966 rdi 0xb96 2966 rbp 0x16 0x16 rsp 0x7fffaebbae78 0x7fffaebbae78 r8 0x7fd87ee427c0 140567818545088 r9 0x616e796420746120 7020682340065370400 r10 0x8 8 r11 0x206 518 r12 0x0 0 r13 0x282d870 42129520 r14 0x282d870 42129520 r15 0x7da50b 8234251 rip 0x33d3a329a5 0x33d3a329a5 eflags 0x206 [ PF IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 No function contains program counter for selected frame. ------------------------------- maybe this can help a little 2010/10/7 Tom Lane <tgl@sss.pgh.pa.us>: > Timon <timosha@gmail.com> writes: >> 2010/10/6 Alvaro Herrera <alvherre@commandprompt.com>: >>>> WARNING: pgstat wait timeout >>>> WARNING: pgstat wait timeout >>>> PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981 >>>> PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981 >>> >>> What platform is this? Are you running on some sort of virtualized >>> system? > >> no, simple dedicated server with Linux Fedora 13 x86_64 >> kernel-2.6.33.6-147.2.4.fc13.x86_64 > > There's not a lot we can do with only this much information. I'd > suggest running the postmaster with core dumps enabled (ulimit -c > unlimited) so that if it happens again, the PANICs will result in > core dumps that you can get stack traces from. > > regards, tom lane > -- All bugs reserved -- All bugs reserved