Thread: BUG #18273: Incorrect memory access occurs when gist__int index is building on data containing 2^31-1

The following bug has been logged on the website:

Bug reference:      18273
Logged by:          Alexander Lakhin
Email address:      exclusion@gmail.com
PostgreSQL version: 16.1
Operating system:   Ubuntu 22.04
Description:

The following script:
CREATE EXTENSION intarray;
CREATE TABLE t(a int[]);
\copy t from 'contrib/intarray/data/test__int.data'
INSERT INTO t VALUES(array[1,2,101,102,201,202,2147483647]);
INSERT INTO t VALUES(array[1,2,101,102,201,202,2147483647]);

CREATE INDEX gist_idx ON t USING gist (a gist__int_ops);

triggers a server crash for me with the stack trace:
Core was generated by `postgres: law regression [local] CREATE INDEX
                        '.
Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x00007f7363c27be3 in g_int_decompress (fcinfo=0x7ffcd5de1960) at
_int_gist.c:347
347                                     *dr++ = j;
(gdb) bt
#0  0x00007f7363c27be3 in g_int_decompress (fcinfo=0x7ffcd5de1960) at
_int_gist.c:347
#1  0x0000556ef169f9ff in FunctionCall1Coll (flinfo=0x556ef259f460,
collation=100, arg1=140723896588928) at fmgr.c:1110
#2  0x0000556ef0f573c0 in gistdentryinit (giststate=0x556ef259e238, nkey=0,
e=0x7ffcd5de1a80, k=140133595572752, r=0x7f7363c5c108, pg=0x7f73652f6000 "",
o=12, l=false, isNull=false) at gistutil.c:562
#3  0x0000556ef0f5706c in gistchoose (r=0x7f7363c5c108, p=0x7f73652f6000 "",
it=0x556ef25a31b8, giststate=0x556ef259e238) at gistutil.c:456
#4  0x0000556ef0f462ff in gistdoinsert (r=0x7f7363c5c108,
itup=0x556ef25a31b8, freespace=819, giststate=0x556ef259e238,
heapRel=0x7f7363c5b008, is_build=true) at gist.c:749
#5  0x0000556ef0f4a236 in gistBuildCallback (index=0x7f7363c5c108,
tid=0x556ef256296c, values=0x7ffcd5de21a0, isnull=0x7ffcd5de2180,
tupleIsAlive=true, state=0x7ffcd5de2670) at gistbuild.c:929
#6  0x0000556ef0f81333 in heapam_index_build_range_scan
(heapRelation=0x7f7363c5b008, indexRelation=0x7f7363c5c108,
indexInfo=0x556ef2564768, allow_sync=true, anyvisible=false, progress=true,
start_blockno=0, numblocks=4294967295, 
    callback=0x556ef0f4a126 <gistBuildCallback>,
callback_state=0x7ffcd5de2670, scan=0x556ef2562910) at
heapam_handler.c:1703
#7  0x0000556ef0f48b58 in table_index_build_scan (table_rel=0x7f7363c5b008,
index_rel=0x7f7363c5c108, index_info=0x556ef2564768, allow_sync=true,
progress=true, callback=0x556ef0f4a126 <gistBuildCallback>, 
    callback_state=0x7ffcd5de2670, scan=0x0) at
../../../../src/include/access/tableam.h:1781
#8  0x0000556ef0f48fc7 in gistbuild (heap=0x7f7363c5b008,
index=0x7f7363c5c108, indexInfo=0x556ef2564768) at gistbuild.c:317
#9  0x0000556ef104bccc in index_build (heapRelation=0x7f7363c5b008,
indexRelation=0x7f7363c5c108, indexInfo=0x556ef2564768, isreindex=false,
parallel=true) at index.c:3042
#10 0x0000556ef10487ff in index_create (heapRelation=0x7f7363c5b008,
indexRelationName=0x556ef2474f50 "gist_idx", indexRelationId=16512,
parentIndexRelid=0, parentConstraintId=0, relFileNumber=0,
indexInfo=0x556ef2564768, 
    indexColNames=0x556ef24a0c68, accessMethodObjectId=783, tableSpaceId=0,
collationObjectId=0x556ef24a0cb8, classObjectId=0x556ef24a0cd0,
coloptions=0x556ef24a0ce8, reloptions=0, flags=0, constr_flags=0, 
    allow_system_table_mods=false, is_internal=false,
constraintId=0x7ffcd5de2b08) at index.c:1265
#11 0x0000556ef115795f in DefineIndex (relationId=16507,
stmt=0x556ef2475138, indexRelationId=0, parentIndexId=0,
parentConstraintId=0, total_parts=-1, is_alter_table=false,
check_rights=true, check_not_in_use=true, skip_build=false, 
    quiet=false) at indexcmds.c:1166
#12 0x0000556ef14b541b in ProcessUtilitySlow (pstate=0x556ef2564658,
pstmt=0x556ef2475248, queryString=0x556ef2474568 "CREATE INDEX gist_idx ON t
USING gist (a gist__int_ops);", context=PROCESS_UTILITY_TOPLEVEL,
params=0x0, 
    queryEnv=0x0, dest=0x556ef2475508, qc=0x7ffcd5de3540) at
utility.c:1553
#13 0x0000556ef14b428b in standard_ProcessUtility (pstmt=0x556ef2475248,
queryString=0x556ef2474568 "CREATE INDEX gist_idx ON t USING gist (a
gist__int_ops);", readOnlyTree=false, context=PROCESS_UTILITY_TOPLEVEL,
params=0x0, 
    queryEnv=0x0, dest=0x556ef2475508, qc=0x7ffcd5de3540) at
utility.c:1078
#14 0x0000556ef14b3167 in ProcessUtility (pstmt=0x556ef2475248,
queryString=0x556ef2474568 "CREATE INDEX gist_idx ON t USING gist (a
gist__int_ops);", readOnlyTree=false, context=PROCESS_UTILITY_TOPLEVEL,
params=0x0, queryEnv=0x0, 
    dest=0x556ef2475508, qc=0x7ffcd5de3540) at utility.c:530
#15 0x0000556ef14b1a39 in PortalRunUtility (portal=0x556ef24eea88,
pstmt=0x556ef2475248, isTopLevel=true, setHoldSnapshot=false,
dest=0x556ef2475508, qc=0x7ffcd5de3540) at pquery.c:1158
#16 0x0000556ef14b1cb0 in PortalRunMulti (portal=0x556ef24eea88,
isTopLevel=true, setHoldSnapshot=false, dest=0x556ef2475508,
altdest=0x556ef2475508, qc=0x7ffcd5de3540) at pquery.c:1315
#17 0x0000556ef14b10fa in PortalRun (portal=0x556ef24eea88,
count=9223372036854775807, isTopLevel=true, run_once=true,
dest=0x556ef2475508, altdest=0x556ef2475508, qc=0x7ffcd5de3540) at
pquery.c:791
#18 0x0000556ef14a9e09 in exec_simple_query (query_string=0x556ef2474568
"CREATE INDEX gist_idx ON t USING gist (a gist__int_ops);") at
postgres.c:1274
#19 0x0000556ef14aee77 in PostgresMain (dbname=0x556ef24ac9e8 "regression",
username=0x556ef2470818 "law") at postgres.c:4637
#20 0x0000556ef13cfca5 in BackendRun (port=0x556ef249c770) at
postmaster.c:4464
#21 0x0000556ef13cf531 in BackendStartup (port=0x556ef249c770) at
postmaster.c:4192
#22 0x0000556ef13cb876 in ServerLoop () at postmaster.c:1782
#23 0x0000556ef13cb120 in PostmasterMain (argc=3, argv=0x556ef246e750) at
postmaster.c:1466
#24 0x0000556ef127f55f in main (argc=3, argv=0x556ef246e750) at main.c:198

As far as I can see, the loop for j shown below:
Datum
g_int_decompress(PG_FUNCTION_ARGS)
{
...
    for (i = 0; i < lenin; i += 2)
        for (j = din[i]; j <= din[i + 1]; j++)
            if ((!i) || *(dr - 1) != j)
                *dr++ = j;
overflows when dln[i] == dln[i+1] == 2147483647.

I saw also other outcomes with slightly modified data, such as:
WARNING:  problem in alloc set GiST temporary context: corrupt header in
block 0x557d109dbe90
WARNING:  problem in alloc set GiST temporary context: bad chunk size for
chunk 0x557d109dbeb8 in block 0x557d109dbe90
WARNING:  problem in alloc set GiST temporary context: bad block offset for
chunk 0x557d109dbeb8 in block 0x557d109dbe90
ERROR:  pfree called with invalid pointer 0x557d109df728 (header
0x80000e9a80000e99)
WARNING:  problem in alloc set GiST temporary context: req size > alloc size
for chunk 0x557d109dbeb8 in block 0x557d109dbe90

Reproduced on REL_12_STABLE .. master.

PG Bug reporting form <noreply@postgresql.org> writes:
> As far as I can see, the loop for j shown below:
> Datum
> g_int_decompress(PG_FUNCTION_ARGS)
> {
> ...
>     for (i = 0; i < lenin; i += 2)
>         for (j = din[i]; j <= din[i + 1]; j++)
>             if ((!i) || *(dr - 1) != j)
>                 *dr++ = j;
> overflows when dln[i] == dln[i+1] == 2147483647.

Yeah, you are right.  Probably the simplest fix is to widen j to
int64.  I see that somebody has already int64-ified g_int_compress
and internal_size (else we'd not get this far), but they missed this.

Will fix, thanks for report!

            regards, tom lane