The attached patch adds an optional callback to support special optimization if ForeignScan/CustomScan are located under the Limit node in plan-tree.
Our sort node wisely switches the behavior when we can preliminary know exact number of rows to be produced, because all the Sort node has to return is the top-k rows when it is located under the Limit node. It is much lightweight workloads than sorting of entire input rows when nrows is not small.
In my case, this information is very useful because GPU can complete its sorting operations mostly on L1-grade memory if we can preliminary know the top-k value is enough small and fits to size of the fast memory.
Probably, it is also valuable for Fujita-san's case because this information allows to attach "LIMIT k" clause on the remote query of postgres_fdw. It will reduce amount of the network traffic and remote CPU consumption once we got support of sort pushdown.
One thing we need to pay attention is cost estimation on the planner stage. In the existing code, only create_ordered_paths() and create_merge_append_path() considers the limit clause for cost estimation of sorting. They use the 'limit_tuples' of PlannerInfo; we can reference the structure when extension adds ForeignPath/CustomPath, so I think we don't need a special enhancement on the planner stage.
I believe this hook is gets called at execution time. So to push LIMIT clause like you said above we should use "limit_tuples" at the time of planning and then use this hook to optimize at runtime, right?
Apart from that, attached patch applies cleanly on latest sources and found no issues with make or with regressions.
However this patch is an infrastructure for any possible optimization when foreign/customscan is under LIMIT.
So look good to me.
I quickly tried adding a hook support in postgres_fdw, and it gets called correctly when we have foreignscan with LIMIT (limit being evaluated on local server).
So code wise no issue. Also add this hook details in documentation.
Thanks
Thanks, -- NEC Business Creation Division / PG-Strom Project KaiGai Kohei <kaigai@ak.jp.nec.com>
