Re: Conflict detection for update_deleted in logical replication - Mailing list pgsql-hackers

On Fri, Dec 20, 2024 at 12:41 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
>
> In the test scenarios already shared on -hackers [1], where pgbench was run only on the publisher node in a pub-sub
setup,no performance degradation was observed on either node. 
>
>
>
> In contrast, when pgbench was run only on the subscriber side with detect_update_deleted=on [2], the TPS performance
wasreduced due to dead tuple accumulation. This performance drop depended on the wal_receiver_status_interval—larger
intervalsresulted in more dead tuple accumulation on the subscriber node. However, after the improvement in patch
v16-0002,which dynamically tunes the status request, the default TPS reduction was limited to only 1%. 
>
>
>
> We performed more benchmarks with the v16-patches where pgbench was run on both the publisher and subscriber,
focusingon TPS performance. To summarize the key observations: 
>
>  - No performance impact on the publisher as dead tuple accumulation does not occur on the publisher.
>
>  - The performance is reduced on the subscriber side (TPS reduction (~50%) [3] ) due to dead tuple retention for the
conflictdetection when detect_update_deleted=on. 
>
>  - Performance reduction happens only on the subscriber side, as workload on the publisher is pretty high and the
applyworkers must wait for the amount of transactions with earlier timestamps to be applied and flushed before
advancingthe non-removable XID to remove dead tuples. 
>
>  - To validate this further, we modified the patch to check only each transaction's commit_time and advance the
non-removableXID if the commit_time is greater than candidate_xid_time. The benchmark results[4] remained consistent,
showingsimilar performance reduction. This confirms that the performance impact on the subscriber side is a reasonable
behaviorif we want to detect the update_deleted conflict reliably. 
>
>
>
> We have also tested similar scenarios in physical streaming replication, to see the effect of enabling the
hot_standby_feedbackand recovery_min_apply_delay. The benchmark results[5] showed performance reduction in these cases
aswell, though impact was less compared to the update_deleted scenario because the physical walreceiver does not need
towait for specified WAL to be applied before sending the hot standby feedback message. However, as the
recovery_min_apply_delayincreased, a similar TPS reduction (~50%) was observed, aligning with the behavior seen in the
update_deletedcase. 
>

The first impression after seeing such a performance dip will be not
to use such a setting but as the primary reason is that one
purposefully wants to retain dead tuples both in physical replication
and pub-sub environment, it is an expected outcome. Now, it is
possible that in real world people may not use exactly the setup we
have used to check the worst-case performance. For example, for a
pub-sub setup, one could imagine that writes happen on two nodes N1,
and N2 (both will be publisher nodes) and then all the changes from
both nodes will be assembled in the third node N3 (a subscriber node).
Or, the subscriber node, may not be set up for aggressive writes, Or,
one would be okay not to detect update_delete conflicts with complete
accuracy.

>
>
> Based on the above, I think the performance reduction observed with the update_deleted patch is expected and
necessarybecause the patch's main goal is to retain dead tuples for reliable conflict detection. Reducing this
retentionperiod would compromise the accuracy of update_deleted detection. 
>

The point related to dead tuple accumulation (or database bloat) with
this setting should be documented similarly to what we document for
hot_standby_feedback. See hot_standby_feedback description in docs
[1].

[1] - https://www.postgresql.org/docs/devel/runtime-config-replication.html#RUNTIME-CONFIG-REPLICATION-STANDBY

--
With Regards,
Amit Kapila.