Thread: Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Pendekar Dikala Senja

Date:

15 January 2020, 04:02:38

Here are the logs and our configuration.

Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.

As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu

LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch

Distributor ID: Ubuntu

Description: Ubuntu 16.04.6 LTS

Release: 16.04

Codename: xenial

Can you help us to find out the solution and configuration?

Thanks for your help

From: Michael Paquier
Sent: Tuesday, January 14, 2020 6:58 PM
To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> I got error when replicate with native logical method from PostgreSQL 11.6
> to PostgreSQL 12.1, I will connecting table partition to table partition
> from Version 11.6 to Version 12.1, but after all tables already replicated
> around more than 2 hours, server can't access and recovery mode. please help
> to advice, below log.

Could you provide a backtrace? Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:
https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

Attachment

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Michael Paquier

Date:

15 January 2020, 04:42:57

On Wed, Jan 15, 2020 at 04:02:38AM +0000, Pendekar Dikala Senja wrote:
> Can you help us to find out the solution and configuration?

A more detailed backtrace would be nice.  Here are some guidelines:
https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend
--
Michael

Attachment

signature.asc

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Mahendra Singh Thalor

Date:

15 January 2020, 10:02:57

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
> Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL 11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already replicated
> > around more than 2 hours, server can't access and recovery mode. please help
> > to advice, below log.
>
> Could you provide a backtrace? Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Pendekar Dikala Senja

Date:

15 January 2020, 10:08:42

Hi,

right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend

From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Pendekar Dikala Senja

Date:

16 January 2020, 03:20:26

following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.

sudo gdb -p 28164

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word".

Attaching to process 28164

ptrace: No such process.

(gdb) set pagination off

(gdb) set logging file debuglog.txt

(gdb) set logging on

Copying output to debuglog.txt.

(gdb) cont

The program is not being run.

(gdb) detach

The program is not being run.

(gdb) b errfinish

No symbol table is loaded. Use the "file" command.

Make breakpoint pending on future shared library load? (y or [n]) n

(gdb)

thanks

From: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,

right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend

Attachment

log.txt

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Mahendra Singh Thalor

Date:

16 January 2020, 03:30:03

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <pendekar.senja@outlook.com> wrote:

following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.

sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.

Yes, every time pid will change. You have to give '-c" option while stating the server to get call stack.

You have to give "gdb postgres data/core.pid".

This command should be fired from your bin(binray) folder.

Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)

thanks

From: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
> Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL 11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already replicated
> > around more than 2 hours, server can't access and recovery mode. please help
> > to advice, below log.
>
> Could you provide a backtrace? Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Pendekar Dikala Senja

Date:

16 January 2020, 03:44:34

I didn't see where the path pg_ctl

$./pg_ctl -D /data/postgres/data -l logfile -c start

-su: ./pg_ctl: No such file or directory

$ pg_ctl -D /data/postgres/data -l logfile -c start

pg_ctl: command not found

on datadir I did not see core.PID, let me know where the dir.

$ ls -lrth

total 144K

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_twophase

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_tblspc

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_snapshots

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_serial

drwx------ 4 postgres postgres 4.0K Jan 2 16:34 pg_multixact

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_dynshmem

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_commit_ts

-rw------- 1 postgres postgres 3 Jan 2 16:34 PG_VERSION

-rw------- 1 postgres postgres 88 Jan 2 16:34 postgresql.auto.conf

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_xact

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_subtrans

drwx------ 6 postgres postgres 4.0K Jan 2 16:44 base

drwx------ 5 postgres postgres 4.0K Jan 3 18:29 pg_replslot

-rw------- 1 postgres postgres 127 Jan 15 12:38 postmaster.opts

drwx------ 2 postgres postgres 4.0K Jan 16 01:00 pg_log

-rw------- 1 postgres postgres 47 Jan 16 01:00 current_logfiles

drwx------ 3 postgres postgres 52K Jan 16 11:00 pg_wal

drwx------ 4 postgres postgres 4.0K Jan 16 11:40 pg_logical

-rw------- 1 postgres postgres 93 Jan 16 11:40 postmaster.pid

drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat_tmp

drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_notify

drwx------ 2 postgres postgres 4.0K Jan 16 11:40 global

drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat

$ cd postmaster.pid

-su: cd: postmaster.pid: Not a directory

$ more postmaster.pid

15803

/data/postgres/data

1579063112

5432

/var/run/postgresql

5432001 169312256

ready

From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 7:30 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <pendekar.senja@outlook.com> wrote:

following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.

sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.

Yes, every time pid will change. You have to give '-c" option while stating the server to get call stack.

You have to give "gdb postgres data/core.pid".

This command should be fired from your bin(binray) folder.

Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)

thanks

From: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
> Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL 11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already replicated
> > around more than 2 hours, server can't access and recovery mode. please help
> > to advice, below log.
>
> Could you provide a backtrace? Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Pendekar Dikala Senja

Date:

16 January 2020, 03:47:55

please find below :

sudo gdb -p 15803

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word".

Attaching to process 15803

Reading symbols from /usr/lib/postgresql/12/bin/postgres...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/.build-id/b1/7c21299099640a6d863e423d99265824e7bb16.debug...done.

done.

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.23.so...done.

done.

Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.23.so...done.

done.

Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.23.so...done.

done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libicui18n.so.55...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libicuuc.so.55...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libsystemd.so.0...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.23.so...done.

done.

Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.23.so...done.

done.

Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libresolv-2.23.so...done.

done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi.so.3...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls.so.30...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libicudata.so.55...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libselinux.so.1...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.20...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libheimntlm.so.0...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.26...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libasn1.so.8...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libhcrypto.so.4...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libroken.so.18...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libidn.so.11...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.6...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.6...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.4...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libpcre.so.3...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libwind.so.0...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libheimbase.so.1...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libhx509.so.5...(no debugging symbols found)...done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libsqlite3.so.0...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.23.so...done.

done.

Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging symbols found)...done.

Reading symbols from /usr/lib/postgresql/12/lib/pg_stat_statements.so...(no debugging symbols found)...done.

Reading symbols from /usr/lib/postgresql/12/lib/pg_partman_bgw.so...(no debugging symbols found)...done.

Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.23.so...done.

done.

0x00007f5ce4bf85b3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:84

84 ../sysdeps/unix/syscall-template.S: No such file or directory.

From: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Sent: Wednesday, January 15, 2020 7:44 PM
To: Mahendra Singh Thalor <mahi6run@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

I didn't see where the path pg_ctl

$./pg_ctl -D /data/postgres/data -l logfile -c start

-su: ./pg_ctl: No such file or directory

$ pg_ctl -D /data/postgres/data -l logfile -c start

pg_ctl: command not found

on datadir I did not see core.PID, let me know where the dir.

$ ls -lrth

total 144K

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_twophase

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_tblspc

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_snapshots

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_serial

drwx------ 4 postgres postgres 4.0K Jan 2 16:34 pg_multixact

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_dynshmem

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_commit_ts

-rw------- 1 postgres postgres 3 Jan 2 16:34 PG_VERSION

-rw------- 1 postgres postgres 88 Jan 2 16:34 postgresql.auto.conf

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_xact

drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_subtrans

drwx------ 6 postgres postgres 4.0K Jan 2 16:44 base

drwx------ 5 postgres postgres 4.0K Jan 3 18:29 pg_replslot

-rw------- 1 postgres postgres 127 Jan 15 12:38 postmaster.opts

drwx------ 2 postgres postgres 4.0K Jan 16 01:00 pg_log

-rw------- 1 postgres postgres 47 Jan 16 01:00 current_logfiles

drwx------ 3 postgres postgres 52K Jan 16 11:00 pg_wal

drwx------ 4 postgres postgres 4.0K Jan 16 11:40 pg_logical

-rw------- 1 postgres postgres 93 Jan 16 11:40 postmaster.pid

drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat_tmp

drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_notify

drwx------ 2 postgres postgres 4.0K Jan 16 11:40 global

drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat

$ cd postmaster.pid

-su: cd: postmaster.pid: Not a directory

$ more postmaster.pid

15803

/data/postgres/data

1579063112

5432

/var/run/postgresql

5432001 169312256

ready

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <pendekar.senja@outlook.com> wrote:

following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.

sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.

Yes, every time pid will change. You have to give '-c" option while stating the server to get call stack.

You have to give "gdb postgres data/core.pid".

This command should be fired from your bin(binray) folder.

Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)

thanks

From: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
> Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL 11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already replicated
> > around more than 2 hours, server can't access and recovery mode. please help
> > to advice, below log.
>
> Could you provide a backtrace? Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Christophe Pettus

Date:

16 January 2020, 18:46:55

We've been encountering the same bug; here's a stack trace from the crashed process:

(gdb) bt
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:427
#1  0x00005582bda1fb57 in memcpy (__len=425318492, __src=<optimized out>, __dest=<optimized out>) at
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#2  tts_virtual_materialize (slot=0x5582bebca418) at ./build/../src/backend/executor/execTuples.c:235
#3  0x00005582bda21179 in ExecFetchSlotHeapTuple (slot=0x5582bebca418, materialize=<optimized out>,
shouldFree=0x7ffeccfccc37)at ./build/../src/backend/executor/execTuples.c:1624 
#4  0x00005582bd8b4e9c in heapam_tuple_update (relation=0x7f0bf1a92430, otid=0x5582bebca6a8, slot=0x5582bebca418,
cid=943,snapshot=<optimized out>, crosscheck=0x0, wait=true, 
    tmfd=0x7ffeccfcccc0, lockmode=0x7ffeccfcccbc, update_indexes=0x7ffeccfccd37) at
./build/../src/backend/access/heap/heapam_handler.c:325
#5  0x00005582bd8e4306 in table_tuple_update (update_indexes=0x7ffeccfccd37, lockmode=0x7ffeccfcccbc,
tmfd=0x7ffeccfcccc0,wait=true, crosscheck=0x0, snapshot=0x0, 
    cid=<optimized out>, slot=0x5582bebca418, otid=0x5582bebca6a8, rel=0x7f0bf1a92430) at
./build/../src/include/access/tableam.h:1261
#6  simple_table_tuple_update (rel=0x7f0bf1a92430, otid=otid@entry=0x5582bebca6a8, slot=slot@entry=0x5582bebca418,
snapshot=0x0,update_indexes=update_indexes@entry=0x7ffeccfccd37) 
    at ./build/../src/backend/access/table/tableam.c:325
#7  0x00005582bda1d462 in ExecSimpleRelationUpdate (estate=estate@entry=0x5582bebd1970,
epqstate=epqstate@entry=0x7ffeccfcce10,searchslot=searchslot@entry=0x5582bebca678, 
    slot=slot@entry=0x5582bebca418) at ./build/../src/backend/executor/execReplication.c:493
#8  0x00005582bdb1a188 in apply_handle_update (s=s@entry=0x7ffeccfd4420) at
./build/../src/backend/replication/logical/worker.c:750
#9  0x00005582bdb1affe in apply_dispatch (s=0x7ffeccfd4420) at ./build/../src/backend/replication/logical/worker.c:968
#10 LogicalRepApplyLoop (last_received=8839008698464) at ./build/../src/backend/replication/logical/worker.c:1176
#11 ApplyWorkerMain (main_arg=<optimized out>) at ./build/../src/backend/replication/logical/worker.c:1734
#12 0x00005582bdae972e in StartBackgroundWorker () at ./build/../src/backend/postmaster/bgworker.c:834
#13 0x00005582bdaf579c in do_start_bgworker (rw=0x5582beafcca0) at ./build/../src/backend/postmaster/postmaster.c:5770
#14 maybe_start_bgworkers () at ./build/../src/backend/postmaster/postmaster.c:5996
#15 0x00005582bdaf6235 in sigusr1_handler (postgres_signal_arg=<optimized out>) at
./build/../src/backend/postmaster/postmaster.c:5167
#16 <signal handler called>
#17 0x00007f1013bfaff7 in __GI___select (nfds=nfds@entry=8, readfds=readfds@entry=0x7ffeccfd5470,
writefds=writefds@entry=0x0,exceptfds=exceptfds@entry=0x0, 
    timeout=timeout@entry=0x7ffeccfd53d0) at ../sysdeps/unix/sysv/linux/select.c:41
#18 0x00005582bdaf67db in ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1668
#19 0x00005582bdaf8513 in PostmasterMain (argc=5, argv=0x5582beaaa090) at
./build/../src/backend/postmaster/postmaster.c:1377
#20 0x00005582bd86ecf6 in main (argc=5, argv=0x5582beaaa090) at ./build/../src/backend/main/main.c:228
(gdb)
--
-- Christophe Pettus
   xof@thebuild.com

Re: BUG #16205: background worker "logical replication worker" (PID25218) was terminated by signal 11: Segmentation

From

Michael Paquier

Date:

16 January 2020, 23:45:14

On Thu, Jan 16, 2020 at 10:46:55AM -0800, Christophe Pettus wrote:
> We've been encountering the same bug; here's a stack trace from the
> crashed process:
>
> [stack trace]

I cannot say if Pendekar's bug is the same issue, but yours looks very
familiar to that:
https://www.postgresql.org/message-id/16129-a0c0f48e71741e5f@postgresql.org

And this has already been fixed, in time for 12.2.
--
Michael

Attachment

signature.asc