On 11/7/2025 10:16, Amit Langote wrote:
> Hi,
>
> I’ve been experimenting with an optimization that reduces executor
> overhead by avoiding unnecessary attribute deformation. Specifically,
> if the executor knows which attributes are actually needed by a plan
> node’s targetlist and qual, it can skip deforming unused columns
> entirely.
Sounds promising. However, I'm not sure we're on the same page. Do you
mean by the proposal an optimisation of slot_deform_heap_tuple() by
providing it with a bitmapset of requested attributes? In this case,
tuple header requires one additional flag to indicate a not-null, but
unfilled column, to detect potential issues.
>
> In a proof-of-concept patch, I initially computed the needed
> attributes during ExecInitSeqScan by walking the plan’s qual and
> targetlist to support deforming only what’s needed when evaluating
> expressions in ExecSeqScan() or the variant thereof (I started with
> SeqScan to keep the initial patch minimal). However, adding more work
> to ExecInit* adds to executor startup cost, which we should generally
> try to reduce. It also makes it harder to apply the optimization
> uniformly across plan types.
I'm not sure if a lot of work will be added. However, cached generic
plan execution should avoid any unnecessary overhead.
>
> I’d now like to propose computing the needed attributes at planning
> time instead. This can be done at the bottom of create_plan_recurse,
> after the plan node has been constructed. A small helper like
> record_needed_attrs(plan) can walk the node’s targetlist and qual
> using pull_varattnos() and store the result in a new Bitmapset
> *attr_used field in the Plan struct. System attributes returned by
> pull_varattnos() can be filtered out during this step, since they're
> either not relevant to deformation or not performance sensitive.
Why do you choose the Plan node? It seems it is relevant to only Scan
nodes. Does it mean extension of the CustomScan API?
> With both patches in place, heap tuple deforming can skip over unused
> attributes entirely. For example, on a 30-column table where the first
> 15 columns are fixed-width, the query:
>
> select sum(a_1) from foo where a_10 = $1;
>
> which references only two fixed-width columns, ran nearly 2x faster
> with the optimization in place (with heap pages prewarmed into
> shared_buffers).
It may be profitable. However, I often encounter cases where a table has
20-40 columns, with arbitrarily mixed fixed and variable-width columns.
And fetching columns by index on a 30-something column is painful. And
in this area, Postgres may gain more profit by adding cost on the column
number in the order_qual_clauses() - in [1] I attempted to explain how
and why it should work.
[1]
https://open.substack.com/pub/danolivo/p/on-expressions-reordering-in-postgres
--
regards, Andrei Lepikhov
