Thread: PostgreSQL logical incremental backup for version 9.6 and above

Hi,

 

I think you can user pgBackRest

 

_________________________________

Cordialement, Pascal CROZET

DBA - Qualis Consulting

• www.qualis-consulting.com • 04 78 22 74 90
• Le Bois des Côtes 1 – Bâtiment A
• 300 Route Nationale 6 – 69760 LIMONEST
_________________________________

 

De : girish R G peetle
Envoyé le :mercredi 6 novembre 2019 18:03
À : pgsql-admin
Objet :PostgreSQL logical incremental backup for version 9.6 and above

 

Hi, 

I've a requirement to backup incremental data from PostgreSQL database server which are hosted as 'database as a service' like 'Amazon RDS' , Azure Database etc. 

Full dump can be taken using pg_dump. But is there is a way to capture incremental changes to the database ?  I read about logical replication slots which can be created for each database to capture incremental changes. 

But how do we know from where we should backup the SQL statements (as we need need skip those statements already backed up by pg_dump). 

I thinking of following steps. 

Step 1: Create logical replication slot

Step 2: Run pg_dump ( for full load ). 

Step 3: Capture the changes since last pg_dump using replication slot - How to know this information ?

Thanks in advance for any advise. 

Thanks

Girish

Hi,
For integrity you can use the -k option in initdb command while you initialize the new database cluster

Sent from my iPhone

> On Nov 7, 2019, at 11:18, Asomba Djala <asombadjala8@gmail.com> wrote:
>
> 
> Hello,
>
> I have a question, SQL Server has a tool DBCC checkdb and I believe Oracle it's VerifyDB. Does Postgres have something similar and if not, does anyone have steps or queries to do a sanity check or database integrity check with PgAdmin4?
>
> In advance,
>
> Thank you for your help.
>
> Aso.

Hi,
For integrity you can use the -k option in initdb command while you initialize the new database cluster

Sent from my iPhone

> On Nov 7, 2019, at 11:18, Asomba Djala <asombadjala8@gmail.com> wrote:
>
> 
> Hello,
>
> I have a question, SQL Server has a tool DBCC checkdb and I believe Oracle it's VerifyDB. Does Postgres have something similar and if not, does anyone have steps or queries to do a sanity check or database integrity check with PgAdmin4?
>
> In advance,
>
> Thank you for your help.
>
> Aso.

Ok all,

I’ve to talk to my collegues working with AWS to provide a best answer.

_________________________________

Cordialement, Pascal CROZET

DBA - Qualis Consulting

• www.qualis-consulting.com • 04 78 22 74 90
• Le Bois des Côtes 1 – Bâtiment A
• 300 Route Nationale 6 – 69760 LIMONEST
_________________________________

 

De : MichaelDBA
Envoyé le :mercredi 6 novembre 2019 20:32
À : girish R G peetle
Cc : PASCAL CROZET; pgsql-admin
Objet :Re: RE : PostgreSQL logical incremental backup for version 9.6 and above

 

Hi,
You can use pgbackrest to store WAL on AWS S3, but you need a binary backup from which to apply WAL to do PITR. pg_dump is not a binary backup.  AWS RDS does not allow access to the underlying file system.  So you cannot use pgbackrest to do backup or restore, or use the WAL you saved, let's say on S3, to do PITR.

Regards,
Michael Vitale

girish R G peetle wrote on 11/6/2019 2:26 PM:

Hi Pascal, 

Thanks for the suggestion. Looks like pgBackRest relies on WAL archive logs. But for Amazon RDS or Azure Database WAL logs are not accessible. We can only do pg_dump and logical replication. 

So pgBackRest won't work for my case. 

Thanks

Girish

On Wed, Nov 6, 2019 at 2:05 PM PASCAL CROZET <pascal.crozet@qualis-consulting.com> wrote:

Hi,

 

I think you can user pgBackRest

 

_________________________________

Cordialement, Pascal CROZET

DBA - Qualis Consulting

• www.qualis-consulting.com • 04 78 22 74 90
• Le Bois des Côtes 1 – Bâtiment A
• 300 Route Nationale 6 – 69760 LIMONEST
_________________________________

 

De : girish R G peetle
Envoyé le :mercredi 6 novembre 2019 18:03
À : pgsql-admin
Objet :PostgreSQL logical incremental backup for version 9.6 and above

 

Hi, 

I've a requirement to backup incremental data from PostgreSQL database server which are hosted as 'database as a service' like 'Amazon RDS' , Azure Database etc. 

Full dump can be taken using pg_dump. But is there is a way to capture incremental changes to the database ?  I read about logical replication slots which can be created for each database to capture incremental changes. 

But how do we know from where we should backup the SQL statements (as we need need skip those statements already backed up by pg_dump). 

I thinking of following steps. 

Step 1: Create logical replication slot

Step 2: Run pg_dump ( for full load ). 

Step 3: Capture the changes since last pg_dump using replication slot - How to know this information ?

Thanks in advance for any advise. 

Thanks

Girish

Hello!

pg_probackup has dedicated command for PostgreSQL instance verification,
which rely on amcheck, page checksum comparison and page header sanity
checks.
It can be used on running databases and can be used even if you do not
use pg_probackup for backup/restore purposes.

https://github.com/postgrespro/pg_probackup/blob/master/Documentation.md#checkdb
https://github.com/postgrespro/pg_probackup/blob/master/Documentation.md#verifying-a-cluster

On 11/7/19 10:47 AM, Asomba Djala wrote:
> Hello,
>
> I have a question, SQL Server has a tool DBCC checkdb and I believe
> Oracle it's VerifyDB. Does Postgres have something similar and if not,
> does anyone have steps or queries to do a sanity check or database
> integrity check with PgAdmin4?
>
> In advance,
>
> Thank you for your help.
>
> Aso.

--
Grigory Smolkin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company