Re: File based Incremental backup v8 - Mailing list pgsql-hackers

On Sat, Mar 7, 2015 at 5:45 PM, Gabriele Bartolini
<gabriele.bartolini@2ndquadrant.it> wrote:
>> By the way, unless I'm missing something, this patch only seems to
>> include the code to construct an incremental backup, but no tools
>> whatsoever to do anything useful with it once you've got it.
>
> As stated previously, Marco is writing a tool called pg_restorebackup (the
> prototype in Python has been already posted) to be included in the core. I
> am in Australia now and not in the office so I cannot confirm it, but I am
> pretty sure he had already written it and was about to send it to the list.

Gabriele, the deadline for the last CommitFest was three weeks ago.
Having a patch that you are "about to send to the list" is not good
enough at this point.

>>   I think that's 100% unacceptable.
>
> I agree, that's why pg_restorebackup written in C is part of this patch.
> See: https://wiki.postgresql.org/wiki/Incremental_backup

No, it *isn't* part of this patch.  You may have a plan to add it to
this patch, but that's not the same thing.

> The goal has always been to provide "file-based incremental backup". I can
> assure that this has always been our compass and the direction to follow.

Regardless of community feedback?  OK.  Let's see how that works out for you.

> I repeat that, using pg_restorebackup, this patch will transparently let
> users benefit from incremental backup even when it will be moved to an
> internal block-level logic. Users will continue to execute pg_basebackup and
> pg_restorebackup, ignoring that with - for example 9.5 - it is file-based
> (saving between 50-70% of space and time) of block level - for example 9.6.

I understand that.  But I also understand that in other cases it's
going to be slower than a full backup.  This problem has been pointed
out several times, and you're just refusing to admit that it's a real
issue.  A user with a bunch of tables where only the rows near the end
of the file get updated is going to repeatedly read those files until
it hits the first modified block and then rewind and reread the whole
file.  I pointed this problem out back in early October and suggested
some ways of fixing it; Heikki followed up with his own suggestions
for modifying my idea.  Instead of implementing any of that, or even
discussing it, you're still plugging away on a design that no
committer has endorsed and that several committers obviously have
concerns about.

It's also pretty clear that nobody likes the backup profile, at least
in the form it exists today.  But it's still here, many patch versions
later.

I think there's absolutely no point in spending more time on this for
9.5.  At least 4 committers have looked at it and none of them are
convinced by the current design; feedback from almost half a year ago
hasn't been incorporated; obviously-needed parts of the patch
(pg_restorebackup) are missing weeks after the last CF deadline.
Let's mark this Rejected in the CF app and move on.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company