Re: post-freeze damage control - Mailing list pgsql-hackers

On 4/12/24 22:27, Tomas Vondra wrote:
> 
> 
> On 4/12/24 08:42, David Steele wrote:
>> On 4/11/24 20:26, Tomas Vondra wrote:
>>> On 4/11/24 03:52, David Steele wrote:
>>>> On 4/11/24 10:23, Tom Kincaid wrote:
>>>>>
>>>>> The extensive Beta process we have can be used to build confidence we
>>>>> need in a feature that has extensive review and currently has no known
>>>>> issues or outstanding objections.
>>>>
>>>> I did have objections, here [1] and here [2]. I think the complexity,
>>>> space requirements, and likely performance issues involved in restores
>>>> are going to be a real problem for users. Some of these can be addressed
>>>> in future releases, but I can't escape the feeling that what we are
>>>> releasing here is half-baked.
>>>>
>>> I do not think it's half-baked. I certainly agree there are limitations,
>>> and there's all kinds of bells and whistles we could add, but I think
>>> the fundamental infrastructure is corrent and a meaningful step forward.
>>> Would I wish it to handle .tar for example? Sure I would. But I think
>>> it's something we can add in the future - if we require all of this to
>>> happen in a single release, it'll never happen.
>>
>> I'm not sure that I really buy this argument, anyway. It is not uncommon
>> for significant features to spend years in development before they are
>> committed. This feature went from first introduction to commit in just
>> over six months. Obviously Robert had been working on it for a while,
>> but for a feature this large six months is a sprint.
>>
> 
> Sure, but it's also not uncommon for significant features to be
> developed incrementally, over multiple releases, introducing the basic
> infrastructure first, and then expanding the capabilities later. I'd
> cite logical decoding/replication and parallel query as examples of this
> approach.
> 
> It's possible there's some fundamental flaw in the WAL summarization?
> Sure, I can't rule that out, although I find it unlikely. Could there be
> bugs? Sure, that's possible, but that applies to all code.
> 
> But it seems to me all the comments are about the client side, not about
> the infrastructure. Which is fair, I certainly agree it'd be nice to
> handle more use cases with less effort, but I still think the patch is a
> meaningful step forward.

Yes, my comments are all about the client code. I like the 
implementation of the WAL summarizer a lot. I don't think there is a 
fundamental flaw in the design, either, but I wouldn't be surprised if 
there are bugs. That's life in software development biz.

Even for the summarizer, though, I do worry about the complexity of 
maintaining it over time. It seems like it would be very easy to 
introduce a bug and have it go unnoticed until it causes problems in the 
field. A lot of testing was done outside of the test suite for this 
feature and I'm not sure if we can rely on that focus with every release.

For me an incremental approach would be to introduce the WAL summarizer 
first. There are already plenty of projects that do page-level 
incremental (WAL-G, pg_probackup, pgBackRest) and could help shake out 
the bugs. Then introduce the client tools later when they are more 
robust. Or, release the client tools now but mark them as experimental 
or something so people know that changes are coming and they don't get 
blindsided by that in the next release. Or, at the very least, make the 
caveats very clear so users can make an informed choice.

Regards,
-David