Segfault in pg_stat_activity - Mailing list pgsql-bugs

Version info:

Postgres 9.5alpha1 (postgresql-9.5_9.5~alpha1-1.pgdg80+1_amd64.deb)

Distribution: Debian 8.0 (Jessie),
Architecture: AMD64
Libc: 2.19-18

Since I've installed 9.5alpha1 a week ago, I've seen several segfaults
in the log file:

% grep 'server process .* was terminated by signal' *.log
postgresql-2015-07-21_072520.log:2015-07-21 07:26:02 GMT [1600]: [4-1] user=
=3D,db=3D,pid=3D1600 LOG:  server process (PID 25320) was terminated by sig=
nal 11: Segmentation fault
postgresql-2015-07-23_082800.log:2015-07-23 11:00:56 GMT [1600]: [9-1] user=
=3D,db=3D,pid=3D1600 LOG:  server process (PID 28864) was terminated by sig=
nal 11: Segmentation fault
postgresql-2015-07-23_082800.log:2015-07-23 12:10:58 GMT [1600]: [14-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 6530) was terminated by sig=
nal 11: Segmentation fault
postgresql-2015-07-23_082800.log:2015-07-23 13:05:47 GMT [1600]: [19-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 14376) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-23_134706.log:2015-07-23 13:55:42 GMT [1600]: [24-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 21482) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-24_000000.log:2015-07-24 08:21:10 GMT [1600]: [29-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 18045) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-24_083758.log:2015-07-24 09:05:41 GMT [1600]: [34-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 24665) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-24_093924.log:2015-07-24 09:50:59 GMT [1600]: [39-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 31328) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-24_093924.log:2015-07-24 11:10:55 GMT [1600]: [44-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 10480) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-24_093924.log:2015-07-24 11:50:55 GMT [1600]: [49-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 16160) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-24_093924.log:2015-07-24 11:55:48 GMT [1600]: [54-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 16947) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-24_093924.log:2015-07-24 13:26:02 GMT [1600]: [59-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 30542) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-27_000000.log:2015-07-27 07:55:59 GMT [1600]: [64-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 22295) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-27_115156.log:2015-07-27 15:10:20 GMT [1600]: [69-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 21239) was terminated by si=
gnal 11: Segmentation fault

% grep -A2  'server process .* was terminated by signal' *.log
postgresql-2015-07-21_072520.log:2015-07-21 07:26:02 GMT [1600]: [4-1] user=
=3D,db=3D,pid=3D1600 LOG:  server process (PID 25320) was terminated by sig=
nal 11: Segmentation fault
postgresql-2015-07-21_072520.log-2015-07-21 07:26:02 GMT [1600]: [5-1] user=
=3D,db=3D,pid=3D1600 DETAIL:  Failed process was running: SELECT DISTINCT u=
sename,usename FROM pg_stat_activity ORDER BY 1
postgresql-2015-07-21_072520.log-2015-07-21 07:26:02 GMT [1600]: [6-1] user=
=3D,db=3D,pid=3D1600 LOG:  terminating any other active server processes
--
postgresql-2015-07-23_082800.log:2015-07-23 11:00:56 GMT [1600]: [9-1] user=
=3D,db=3D,pid=3D1600 LOG:  server process (PID 28864) was terminated by sig=
nal 11: Segmentation fault
postgresql-2015-07-23_082800.log-2015-07-23 11:00:56 GMT [1600]: [10-1] use=
r=3D,db=3D,pid=3D1600 DETAIL:  Failed process was running: SELECT DISTINCT =
usename,usename FROM pg_stat_activity ORDER BY 1
postgresql-2015-07-23_082800.log-2015-07-23 11:00:56 GMT [1600]: [11-1] use=
r=3D,db=3D,pid=3D1600 LOG:  terminating any other active server processes
--
postgresql-2015-07-23_082800.log:2015-07-23 12:10:58 GMT [1600]: [14-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 6530) was terminated by sig=
nal 11: Segmentation fault
postgresql-2015-07-23_082800.log-2015-07-23 12:10:58 GMT [1600]: [15-1] use=
r=3D,db=3D,pid=3D1600 DETAIL:  Failed process was running: SELECT DISTINCT =
usename,usename FROM pg_stat_activity ORDER BY 1
postgresql-2015-07-23_082800.log-2015-07-23 12:10:58 GMT [1600]: [16-1] use=
r=3D,db=3D,pid=3D1600 LOG:  terminating any other active server processes
[...]
--
postgresql-2015-07-27_000000.log:2015-07-27 07:55:59 GMT [1600]: [64-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 22295) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-27_000000.log-2015-07-27 07:55:59 GMT [1600]: [65-1] use=
r=3D,db=3D,pid=3D1600 DETAIL:  Failed process was running: SELECT DISTINCT =
usename,usename FROM pg_stat_activity ORDER BY 1
postgresql-2015-07-27_000000.log-2015-07-27 07:55:59 GMT [1600]: [66-1] use=
r=3D,db=3D,pid=3D1600 LOG:  terminating any other active server processes
--
postgresql-2015-07-27_115156.log:2015-07-27 15:10:20 GMT [1600]: [69-1] use=
r=3D,db=3D,pid=3D1600 LOG:  server process (PID 21239) was terminated by si=
gnal 11: Segmentation fault
postgresql-2015-07-27_115156.log-2015-07-27 15:10:20 GMT [1600]: [70-1] use=
r=3D,db=3D,pid=3D1600 DETAIL:  Failed process was running: SELECT DISTINCT =
usename,usename FROM pg_stat_activity ORDER BY 1
postgresql-2015-07-27_115156.log-2015-07-27 15:10:20 GMT [1600]: [71-1] use=
r=3D,db=3D,pid=3D1600 LOG:  terminating any other active server processes

All the crashed processes were running

"SELECT DISTINCT usename,usename FROM pg_stat_activity ORDER BY 1"

but running that query does not normally produce a crash (as a rough
guess, the chance is about 1%).

Here is the backtrace from the core file produced by the latest crash:


Core was generated by `postgres: postgres template1 [local] SELECT         =
                          '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fb94d314289 in memcpy (__len=3D200, __src=3D0x7fb948719ad8, __de=
st=3D0x7fb94f4c1bb0) at /usr/include/x86_64-linux-gnu/bits/string3.h:51
51        return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__de=
st));
(gdb) bt
#0  0x00007fb94d314289 in memcpy (__len=3D200, __src=3D0x7fb948719ad8, __de=
st=3D0x7fb94f4c1bb0) at /usr/include/x86_64-linux-gnu/bits/string3.h:51
#1  pgstat_read_current_status () at /tmp/buildd/postgresql-9.5-9.5~alpha1/=
build/../src/backend/postmaster/pgstat.c:3000
#2  0x00007fb94d318db1 in pgstat_read_current_status () at /tmp/buildd/post=
gresql-9.5-9.5~alpha1/build/../src/backend/postmaster/pgstat.c:2434
#3  pgstat_fetch_stat_numbackends () at /tmp/buildd/postgresql-9.5-9.5~alph=
a1/build/../src/backend/postmaster/pgstat.c:2435
#4  0x00007fb94d403051 in pg_stat_get_activity (fcinfo=3D0x7ffe57b2e480) at=
 /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/utils/adt/pgsta=
tfuncs.c:534
#5  0x00007fb94d278362 in ExecMakeTableFunctionResult (funcexpr=3D0x7fb94f3=
7e890, econtext=3D0x7fb94f37bb10, argContext=3D<optimized out>, expectedDes=
c=3D0x7fb94f380758, randomAccess=3D0 '\000')
    at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/=
execQual.c:2188
#6  0x00007fb94d28d512 in FunctionNext (node=3Dnode@entry=3D0x7fb94f37ba00)=
 at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/nod=
eFunctionscan.c:94
#7  0x00007fb94d27a539 in ExecScanFetch (recheckMtd=3D0x7fb94d28d260 <Funct=
ionRecheck>, accessMtd=3D0x7fb94d28d270 <FunctionNext>, node=3D0x7fb94f37ba=
00) at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/=
execScan.c:82
#8  ExecScan (node=3Dnode@entry=3D0x7fb94f37ba00, accessMtd=3DaccessMtd@ent=
ry=3D0x7fb94d28d270 <FunctionNext>, recheckMtd=3DrecheckMtd@entry=3D0x7fb94=
d28d260 <FunctionRecheck>)
    at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/=
execScan.c:132
#9  0x00007fb94d28d568 in ExecFunctionScan (node=3Dnode@entry=3D0x7fb94f37b=
a00) at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor=
/nodeFunctionscan.c:268
#10 0x00007fb94d272d58 in ExecProcNode (node=3Dnode@entry=3D0x7fb94f37ba00)=
 at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/exe=
cProcnode.c:442
#11 0x00007fb94d285513 in ExecHashJoin (node=3Dnode@entry=3D0x7fb94f37ae60)=
 at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/nod=
eHashjoin.c:154
#12 0x00007fb94d272cd8 in ExecProcNode (node=3Dnode@entry=3D0x7fb94f37ae60)=
 at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/exe=
cProcnode.c:477
#13 0x00007fb94d28cdc2 in ExecNestLoop (node=3Dnode@entry=3D0x7fb94f37ab40)=
 at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/nod=
eNestloop.c:123
#14 0x00007fb94d272cf8 in ExecProcNode (node=3Dnode@entry=3D0x7fb94f37ab40)=
 at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/exe=
cProcnode.c:469
#15 0x00007fb94d28f071 in ExecSort (node=3Dnode@entry=3D0x7fb94f37a960) at =
/tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/nodeSor=
t.c:103
#16 0x00007fb94d272cb8 in ExecProcNode (node=3Dnode@entry=3D0x7fb94f37a960)=
 at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/exe=
cProcnode.c:488
#17 0x00007fb94d28f325 in ExecUnique (node=3Dnode@entry=3D0x7fb94f37a6e0) a=
t /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/nodeU=
nique.c:69
#18 0x00007fb94d272c78 in ExecProcNode (node=3Dnode@entry=3D0x7fb94f37a6e0)=
 at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/exe=
cProcnode.c:504
#19 0x00007fb94d26fc8e in ExecutePlan (dest=3D0x7fb94f36ccc8, direction=3D<=
optimized out>, numberTuples=3D0, sendTuples=3D<optimized out>, operation=
=3DCMD_SELECT, planstate=3D0x7fb94f37a6e0, estate=3D0x7fb94f37a550)
    at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/executor/=
execMain.c:1549
#20 standard_ExecutorRun (queryDesc=3D0x7fb94f351a70, direction=3D<optimize=
d out>, count=3D0) at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/ba=
ckend/executor/execMain.c:337
#21 0x00007fb94d37a30f in PortalRunSelect (portal=3Dportal@entry=3D0x7fb94f=
378540, forward=3Dforward@entry=3D1 '\001', count=3D0, count@entry=3D922337=
2036854775807, dest=3Ddest@entry=3D0x7fb94f36ccc8)
    at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/tcop/pque=
ry.c:946
#22 0x00007fb94d37b980 in PortalRun (portal=3Dportal@entry=3D0x7fb94f378540=
, count=3Dcount@entry=3D9223372036854775807, isTopLevel=3DisTopLevel@entry=
=3D1 '\001', dest=3Ddest@entry=3D0x7fb94f36ccc8, altdest=3Daltdest@entry=3D=
0x7fb94f36ccc8,=20
    completionTag=3DcompletionTag@entry=3D0x7ffe57b2f0c0 "") at /tmp/buildd=
/postgresql-9.5-9.5~alpha1/build/../src/backend/tcop/pquery.c:790
#23 0x00007fb94d378feb in exec_simple_query (query_string=3D0x7fb94f3147c0 =
"SELECT DISTINCT usename,usename FROM pg_stat_activity ORDER BY 1") at /tmp=
/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/tcop/postgres.c:1104
#24 PostgresMain (argc=3D<optimized out>, argv=3Dargv@entry=3D0x7fb94f2bfd7=
0, dbname=3D0x7fb94f2bfd50 "template1", username=3D<optimized out>) at /tmp=
/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/tcop/postgres.c:4025
#25 0x00007fb94d116ac7 in BackendRun (port=3D0x7fb94f2c00d0) at /tmp/buildd=
/postgresql-9.5-9.5~alpha1/build/../src/backend/postmaster/postmaster.c:4159
#26 BackendStartup (port=3D0x7fb94f2c00d0) at /tmp/buildd/postgresql-9.5-9.=
5~alpha1/build/../src/backend/postmaster/postmaster.c:3835
#27 ServerLoop () at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/bac=
kend/postmaster/postmaster.c:1609
#28 0x00007fb94d31e9b1 in PostmasterMain (argc=3D5, argv=3D<optimized out>)=
 at /tmp/buildd/postgresql-9.5-9.5~alpha1/build/../src/backend/postmaster/p=
ostmaster.c:1254
#29 0x00007fb94d117c72 in main (argc=3D5, argv=3D0x7fb94f27d1e0) at /tmp/bu=
ildd/postgresql-9.5-9.5~alpha1/build/../src/backend/main/main.c:221

    hp

--=20
   _  | Peter J. Holzer    | I want to forget all about both belts and
|_|_) |                    | suspenders; instead, I want to buy pants=20
| |   | hjp@hjp.at         | that actually fit.
__/   | http://www.hjp.at/ |   -- http://noncombatant.org/