Currently we log conflicts to the server's log file and updates, this
approach has limitations, 1) Difficult to query and analyze, parsing
plain text log files for conflict details is inefficient. 2) Lack of
structured data, key conflict attributes (table, operation, old/new
data, LSN, etc.) are not readily available in a structured, queryable
format. 3) Difficult for external monitoring tools or custom
resolution scripts to consume conflict data directly.
This proposal aims to address these limitations by introducing a
conflict log history table, providing a structured, and queryable
record of all logical replication conflicts. This should be a
configurable option whether to log into the conflict log history
table, server logs or both.
This proposal has two main design questions:
===================================
1. How do we store conflicting tuples from different tables?
Using a JSON column to store the row data seems like the most flexible
solution, as it can accommodate different table schemas.
2. Should this be a system table or a user table?
a) System Table: Storing this in a system catalog is simple, but
catalogs aren't designed for ever-growing data. While pg_large_object
is an exception, this is not what we generally do IMHO.
b) User Table: This offers more flexibility. We could allow a user to
specify the table name during CREATE SUBSCRIPTION. Then we choose to
either create the table internally or let the user create the table
with a predefined schema.
A potential drawback is that a user might drop or alter the table.
However, we could mitigate this risk by simply logging a WARNING if
the table is configured but an insertion fails.
I am currently working on a POC patch for the same, but will post that
once we have some thoughts on design choices.
Schema for the conflict log history table may look like this, although
there is a room for discussion on this.
Note: I think these fields are self explanatory so I haven't
explained them here.
conflict_log_table (
logid SERIAL PRIMARY KEY,
subid OID,
schema_id OID,
table_id OID,
conflict_type TEXT NOT NULL,
operation_type TEXT NOT NULL,
replication_origin TEXT,
remote_commit_ts TIMESTAMPTZ,
local_commit_ts TIMESTAMPTZ,
ri_key JSON,
remote_tuple JSON,
local_tuple JSON,
);
Credit: Thanks to Amit Kapila for discussing this offlist and
providing some valuable suggestions.
--
Regards,
Dilip Kumar
Google
