Thread: Query planning on partitioned table causes postgres 13.4 to consume all memory
Query planning on partitioned table causes postgres 13.4 to consume all memory
From
Duncan Sands
Date:
Postgresql version: 13.4
O/S version: Ubuntu 21.04
How to reproduce:
(1) Create "data" schema and its tables by executing the commands from the
attached file d.sql. Each of the two created tables has less than 20 rows.
(2) Execute or EXPLAIN this query
DELETE FROM data.files
WHERE path=ANY(
SELECT path
FROM (
SELECT meta, path,
first_value(path)
OVER (PARTITION BY meta ORDER BY priority) AS first
FROM data.files
) f,
data.metadata m
WHERE f.path <> f.first
AND f.meta = m.id
AND m.syd = 667
);
(3) Observe that the EXPLAIN/query never completes, and memory usage goes up
and up.
For example:
duncan=> \i /tmp/d.sql
...
duncan=> DELETE FROM data.files
WHERE path=ANY(
SELECT path
FROM (
SELECT meta, path,
first_value(path)
OVER (PARTITION BY meta ORDER BY priority) AS first
FROM data.files
) f,
data.metadata m
WHERE f.path <> f.first
AND f.meta = m.id
AND m.syd = 667
);
... wailing and gnashing of teeth ...
Attachment
Re: Query planning on partitioned table causes postgres 13.4 to consume all memory
From
Tom Lane
Date:
Duncan Sands <duncan.sands@deepbluecap.com> writes:
> [ planning DELETE on a thousand-partition table takes forever ]
FWIW, this situation has been very much improved for v14 [1].
In older branches, the best advice I can give you is "don't use
so many partitions". Especially not with hash partitioning,
where the query WHERE clause generally won't translate to any
useful pruning of the partitions.
(Personally, I think that hash partitioning is an evil that
we shouldn't have implemented at all. Or at least there
should be stronger warnings about it in the manual than there
are now.)
regards, tom lane
[1] https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=86dc90056
Re: Query planning on partitioned table causes postgres 13.4 to consume all memory
From
Duncan Sands
Date:
Hi Tom, On 19/09/2021 18:03, Tom Lane wrote: > Duncan Sands <duncan.sands@deepbluecap.com> writes: >> [ planning DELETE on a thousand-partition table takes forever ] > > FWIW, this situation has been very much improved for v14 [1]. thanks, part (2) of that commit indeed looks like it should solve it. Best wishes, Duncan. > In older branches, the best advice I can give you is "don't use > so many partitions". Especially not with hash partitioning, > where the query WHERE clause generally won't translate to any > useful pruning of the partitions. > > (Personally, I think that hash partitioning is an evil that > we shouldn't have implemented at all. Or at least there > should be stronger warnings about it in the manual than there > are now.) > > regards, tom lane > > [1] https://git.postgresql.org/gitweb/?p=postgresql.git&a=commitdiff&h=86dc90056 >