Thread: Postgresql Crasching

Postgresql Crasching

From

Márcio Antônio Sepp

Date:

25 January 2019, 18:21:43

Hi all,


We are experiencing the follow error in last few weeks:

Jan 25 16:03:51 FreeBSD11 kernel: pid 78639 (postgres), uid 770: exited on
signal 11 (core dumped)
Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-1] 2019-01-25 16:03:51.135 -02
[73403] WARNING:  terminating connection because of crash of another server
process
Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-2] 2019-01-25 16:03:51.135 -02
[73403] DETAIL:  The postmaster has commanded this server process to roll
back the current transacti
on and exit, because another server process exited abnormally and possibly
corrupted shared memory.
Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-3] 2019-01-25 16:03:51.135 -02
[73403] HINT:  In a moment you should be able to reconnect to the database
and repeat your command.


This is our test machine and some days ago we try to increase memory and
vcpu, but the problem persist.


Some information:
root@FreeBSD11:~ # uname -a
FreeBSD FreeBSD11.2 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27
08:16:24 UTC 2018
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64


root@FreeBSD11:~ # swapinfo -h
Device          1K-blocks     Used    Avail Capacity
/dev/da0s1b       4194268       0B     4.0G     0%


root@FreeBSD11:~ # df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    141G     13G    117G    10%    /
devfs          1.0K    1.0K      0B   100%    /dev


# select version();
PostgreSQL 11.0 on amd64-portbld-freebsd11.2, compiled by FreeBSD clang
version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0), 64-bit


Any suggestion?

Tks a lot


--
Att
Márcio A. Sepp

Re: Postgresql Crasching

From

Adrian Klaver

Date:

25 January 2019, 18:24:21

On 1/25/19 10:21 AM, Márcio Antônio Sepp wrote:
> 
> Hi all,
> 
> 
> We are experiencing the follow error in last few weeks:
> 
> Jan 25 16:03:51 FreeBSD11 kernel: pid 78639 (postgres), uid 770: exited on
> signal 11 (core dumped)
> Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-1] 2019-01-25 16:03:51.135 -02
> [73403] WARNING:  terminating connection because of crash of another server
> process
> Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-2] 2019-01-25 16:03:51.135 -02
> [73403] DETAIL:  The postmaster has commanded this server process to roll
> back the current transacti
> on and exit, because another server process exited abnormally and possibly
> corrupted shared memory.
> Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-3] 2019-01-25 16:03:51.135 -02
> [73403] HINT:  In a moment you should be able to reconnect to the database
> and repeat your command.

What is the logs just before the above?

> 
> 
> This is our test machine and some days ago we try to increase memory and
> vcpu, but the problem persist.
> 
> 
> Some information:
> root@FreeBSD11:~ # uname -a
> FreeBSD FreeBSD11.2 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27
> 08:16:24 UTC 2018
> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
> 
> 
> root@FreeBSD11:~ # swapinfo -h
> Device          1K-blocks     Used    Avail Capacity
> /dev/da0s1b       4194268       0B     4.0G     0%
> 
> 
> root@FreeBSD11:~ # df -h
> Filesystem     Size    Used   Avail Capacity  Mounted on
> /dev/da0s1a    141G     13G    117G    10%    /
> devfs          1.0K    1.0K      0B   100%    /dev
> 
> 
> # select version();
> PostgreSQL 11.0 on amd64-portbld-freebsd11.2, compiled by FreeBSD clang
> version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0), 64-bit
> 
> 
> Any suggestion?
> 
> Tks a lot
> 
> 
> --
> Att
> Márcio A. Sepp
> 
> 
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com

Re: Postgresql Crasching

From

Tom Lane

Date:

25 January 2019, 18:28:17

=?iso-8859-1?Q?M=E1rcio_Ant=F4nio_Sepp?= <marcio@zyontecnologia.com.br> writes:
> We are experiencing the follow error in last few weeks:
> Jan 25 16:03:51 FreeBSD11 kernel: pid 78639 (postgres), uid 770: exited on
> signal 11 (core dumped)

> # select version();
> PostgreSQL 11.0 on amd64-portbld-freebsd11.2, compiled by FreeBSD clang
> version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0), 64-bit

First thing to try is updating to 11.1, just to see if problem is already
fixed.  If not, find out what query is crashing (see postmaster log),
and if you can get a stack trace out of one of the core files, that'd
be pretty helpful too.

https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend

We might need to ask for a self-contained test case, if the stack
trace doesn't make it clear what's failing.

            regards, tom lane

RES: Postgresql Crasching

From

Márcio Antônio Sepp

Date:

25 January 2019, 18:50:24

>
> On 1/25/19 10:21 AM, Márcio Antônio Sepp wrote:
> >
> > Hi all,
> >
> >
> > We are experiencing the follow error in last few weeks:
> >
> > Jan 25 16:03:51 FreeBSD11 kernel: pid 78639 (postgres), uid 770: exited
> on
> > signal 11 (core dumped)
> > Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-1] 2019-01-25 16:03:51.135
> -02
> > [73403] WARNING:  terminating connection because of crash of another
> server
> > process
> > Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-2] 2019-01-25 16:03:51.135
> -02
> > [73403] DETAIL:  The postmaster has commanded this server process to roll
> > back the current transacti
> > on and exit, because another server process exited abnormally and
> possibly
> > corrupted shared memory.
> > Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-3] 2019-01-25 16:03:51.135
> -02
> > [73403] HINT:  In a moment you should be able to reconnect to the
> database
> > and repeat your command.
>
> What is the logs just before the above?


Jan 25 16:01:55 FreeBSD11 postgres[71605]: [6-2] 2019-01-25 16:01:55.733 -02 [71605] DETAIL:  Key (tdoemp, codemp,
codfil,origem, sercai, numdoc, tdopes, codpes, datpar)=(1, 59 
84305, 1, 55, 1, 160, 1, 5984305000107, 201901) is not present in table "e026pag_p2019".
Jan 25 16:01:55 FreeBSD11 postgres[71605]: [6-3] 2019-01-25 16:01:55.733 -02 [71605] STATEMENT:  COMMIT
Jan 25 16:02:35 FreeBSD11 postgres[73403]: [6-1] 2019-01-25 16:02:35.184 -02 [73403] ERROR:  insert or update on table
"e026par_p2019"violates foreign key constraint "e026par_p 
2019_tdoemp_fkey"
Jan 25 16:02:35 FreeBSD11 postgres[73403]: [6-2] 2019-01-25 16:02:35.184 -02 [73403] DETAIL:  Key (tdoemp, codemp,
codfil,origem, sercai, numdoc, tdopes, codpes, datpar)=(1, 59 
84305, 1, 55, 1, 160, 1, 5984305000107, 201901) is not present in table "e026pag_p2019".
Jan 25 16:02:35 FreeBSD11 postgres[73403]: [6-3] 2019-01-25 16:02:35.184 -02 [73403] STATEMENT:  COMMIT
Jan 25 16:03:51 FreeBSD11 kernel: pid 78639 (postgres), uid 770: exited on signal 11 (core dumped)


> >
> > This is our test machine and some days ago we try to increase memory and
> > vcpu, but the problem persist.
> >
> >
> > Some information:
> > root@FreeBSD11:~ # uname -a
> > FreeBSD FreeBSD11.2 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #0: Thu Sep
> 27
> > 08:16:24 UTC 2018
> > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
> >
> >
> > root@FreeBSD11:~ # swapinfo -h
> > Device          1K-blocks     Used    Avail Capacity
> > /dev/da0s1b       4194268       0B     4.0G     0%
> >
> >
> > root@FreeBSD11:~ # df -h
> > Filesystem     Size    Used   Avail Capacity  Mounted on
> > /dev/da0s1a    141G     13G    117G    10%    /
> > devfs          1.0K    1.0K      0B   100%    /dev
> >
> >
> > # select version();
> > PostgreSQL 11.0 on amd64-portbld-freebsd11.2, compiled by FreeBSD clang
> > version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0), 64-
> bit
> >

Re: RES: Postgresql Crasching

From

Adrian Klaver

Date:

25 January 2019, 20:28:06

On 1/25/19 10:50 AM, Márcio Antônio Sepp wrote:
>>
>> On 1/25/19 10:21 AM, Márcio Antônio Sepp wrote:
>>>
>>> Hi all,
>>>
>>>
>>> We are experiencing the follow error in last few weeks:
>>>
>>> Jan 25 16:03:51 FreeBSD11 kernel: pid 78639 (postgres), uid 770: exited
>> on
>>> signal 11 (core dumped)
>>> Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-1] 2019-01-25 16:03:51.135
>> -02
>>> [73403] WARNING:  terminating connection because of crash of another
>> server
>>> process
>>> Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-2] 2019-01-25 16:03:51.135
>> -02
>>> [73403] DETAIL:  The postmaster has commanded this server process to roll
>>> back the current transacti
>>> on and exit, because another server process exited abnormally and
>> possibly
>>> corrupted shared memory.
>>> Jan 25 16:03:51 FreeBSD11 postgres[73403]: [7-3] 2019-01-25 16:03:51.135
>> -02
>>> [73403] HINT:  In a moment you should be able to reconnect to the
>> database
>>> and repeat your command.
>>
>> What is the logs just before the above?
> 
> 
> Jan 25 16:01:55 FreeBSD11 postgres[71605]: [6-2] 2019-01-25 16:01:55.733 -02 [71605] DETAIL:  Key (tdoemp, codemp,
codfil,origem, sercai, numdoc, tdopes, codpes, datpar)=(1, 59
 
> 84305, 1, 55, 1, 160, 1, 5984305000107, 201901) is not present in table "e026pag_p2019".
> Jan 25 16:01:55 FreeBSD11 postgres[71605]: [6-3] 2019-01-25 16:01:55.733 -02 [71605] STATEMENT:  COMMIT
> Jan 25 16:02:35 FreeBSD11 postgres[73403]: [6-1] 2019-01-25 16:02:35.184 -02 [73403] ERROR:  insert or update on
table"e026par_p2019" violates foreign key constraint "e026par_p
 
> 2019_tdoemp_fkey"
> Jan 25 16:02:35 FreeBSD11 postgres[73403]: [6-2] 2019-01-25 16:02:35.184 -02 [73403] DETAIL:  Key (tdoemp, codemp,
codfil,origem, sercai, numdoc, tdopes, codpes, datpar)=(1, 59
 
> 84305, 1, 55, 1, 160, 1, 5984305000107, 201901) is not present in table "e026pag_p2019".
> Jan 25 16:02:35 FreeBSD11 postgres[73403]: [6-3] 2019-01-25 16:02:35.184 -02 [73403] STATEMENT:  COMMIT
> Jan 25 16:03:51 FreeBSD11 kernel: pid 78639 (postgres), uid 770: exited on signal 11 (core dumped)
> 

Hmm, a 1:16 between the query failing and the crash. Hard to say if 
there is a cause and effect. I would follow Tom Lane's advice and see if 
you can get a stack trace.


-- 
Adrian Klaver
adrian.klaver@aklaver.com