Re: CDC/ETL system on top of logical replication with pgoutput, custom client - Mailing list pgsql-hackers

On Mon, Aug 7, 2023 at 1:46 PM José Neves <rafaneves3@msn.com> wrote:
>
> Humm, that's... challenging. I faced some issues after "the fix" because I had a couple of transactions with 25k
updates,and I had to split it to be able to push to our event messaging system, as our max message size is 10MB.
Relyingon commit time would mean that all transaction operations will have the same timestamp. If something goes wrong
whilemy worker is pushing that transaction data chunks, I will duplicate some data in the next run, so... this wouldn't
allowme to deal with data duplication. 
> Is there any other way that you see to deal with it?
>
> Right now I only see an option, which is to store all processed LSNs on the other side of the ETL. I'm trying to
avoidthat overhead. 
>

Sorry, I don't understand your system enough to give you suggestions
but if you have any questions related to how logical replication work
then I might be able to help.

--
With Regards,
Amit Kapila.