Thread: Standby problem after restore_command Implementation

Standby problem after restore_command Implementation

From

"Edson F. Lidorio"

Date:

04 May 2015, 00:57:17

Hello,

I'm having trouble on Standby after the implementation of the
restore_command.
I performed all the settings and it worked normally and after restart
the slave server, started to generate errors in the log of the slave:

I'm using Debian 8 with PostgreSQL 9.4.1 on x86_64-unknown-linux-gnu,
compiled by gcc-4.9. real (Debian 4.9.2-10) 4.9.2, 64-bit


Slave error log:
5/3/2015 16:46:01 BRT [10210-1] @ Replicator [unknown] error: WAL
segment requested 00000001000000000000002C has been removed
5/3/2015 16:46:05 BRT [10211-1] @ Replicator [unknown] error: WAL
segment requested 00000001000000000000002C has been removed
5/3/2015 16:46:10 BRT [10214-1] @ Replicator [unknown] error: WAL
segment requested 00000001000000000000002C has been removed
5/3/2015 16:46:15 BRT [10216-1] @ Replicator [unknown] error: WAL
segment requested 00000001000000000000002C has been removed


Master error log
5/3/2015 19:13:35 BRT [3339-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed
5/3/2015 19:13:40 BRT [3341-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed
5/3/2015 19:13:44 BRT [3343-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed

Settings files are as follows:

master

postgresql.conf

listen_addresses = '*'
wal_level = hot_standby
archive_mode = on
archive_command = 'cp "%p" /mnt/server/archivedir/"%f"'
max_wal_senders = 2
wal_keep_segments = 20


pg_hba.conf
host     replication     replicador      192.168.0.112/32 trust

secondary

postgresql.conf

listen_addresses = '*'
hot_standby = on


pg_hba.conf
host     all             all             0.0.0.0/0               md5

recover.conf em (/var/lib/postgresql/9.4/main)

standby_mode=on
primary_conninfo='host=192.168.0.100 user=replicador application_name=
jessie-stby'
trigger_file='/tmp/pgtrigger'
restore_command = 'cp /mnt/server/archivedir/%f %p'
archive_cleanup_command = 'pg_archivecleanup /mnt/server/archivedir %r'

Re: Standby problem after restore_command Implementation

From

Adrian Klaver

Date:

05 May 2015, 13:46:06

On 05/03/2015 05:57 PM, Edson F. Lidorio wrote:
> Hello,
>
> I'm having trouble on Standby after the implementation of the
> restore_command.
> I performed all the settings and it worked normally and after restart
> the slave server, started to generate errors in the log of the slave:

So is that implying that you had the standby running without the
restore_command?

>
> I'm using Debian 8 with PostgreSQL 9.4.1 on x86_64-unknown-linux-gnu,
> compiled by gcc-4.9. real (Debian 4.9.2-10) 4.9.2, 64-bit
>
>
> Slave error log:
> 5/3/2015 16:46:01 BRT [10210-1] @ Replicator [unknown] error: WAL
> segment requested 00000001000000000000002C has been removed
> 5/3/2015 16:46:05 BRT [10211-1] @ Replicator [unknown] error: WAL
> segment requested 00000001000000000000002C has been removed
> 5/3/2015 16:46:10 BRT [10214-1] @ Replicator [unknown] error: WAL
> segment requested 00000001000000000000002C has been removed
> 5/3/2015 16:46:15 BRT [10216-1] @ Replicator [unknown] error: WAL
> segment requested 00000001000000000000002C has been removed
>
>
> Master error log
> 5/3/2015 19:13:35 BRT [3339-1] @ Replicator [unknown] error: WAL segment
> requested 00000001000000000000002C has been removed
> 5/3/2015 19:13:40 BRT [3341-1] @ Replicator [unknown] error: WAL segment
> requested 00000001000000000000002C has been removed
> 5/3/2015 19:13:44 BRT [3343-1] @ Replicator [unknown] error: WAL segment
> requested 00000001000000000000002C has been removed
>
> Settings files are as follows:
>
> master
>
> postgresql.conf
>
> listen_addresses = '*'
> wal_level = hot_standby
> archive_mode = on
> archive_command = 'cp "%p" /mnt/server/archivedir/"%f"'
> max_wal_senders = 2
> wal_keep_segments = 20
>
>
> pg_hba.conf
> host     replication     replicador      192.168.0.112/32 trust
>
> secondary
>
> postgresql.conf
>
> listen_addresses = '*'
> hot_standby = on
>
>
> pg_hba.conf
> host     all             all             0.0.0.0/0               md5
>
> recover.conf em (/var/lib/postgresql/9.4/main)
>
> standby_mode=on
> primary_conninfo='host=192.168.0.100 user=replicador application_name=
> jessie-stby'
> trigger_file='/tmp/pgtrigger'
> restore_command = 'cp /mnt/server/archivedir/%f %p'
> archive_cleanup_command = 'pg_archivecleanup /mnt/server/archivedir %r'
>
>
>


--
Adrian Klaver
adrian.klaver@aklaver.com

Re: Standby problem after restore_command Implementation

From

Adrian Klaver

Date:

05 May 2015, 14:09:48

On 05/05/2015 07:05 AM, Edson F. Lidorio wrote:

CCing list.

> Em 2015-05-05 10:45, Adrian Klaver escreveu:
>
>> On 05/03/2015 05:57 PM, Edson F. Lidorio wrote:
>>> Hello, I'm having trouble on Standby after the implementation of the
>>> restore_command. I performed all the settings and it worked normally
>>> and after restart the slave server, started to generate errors in the
>>> log of the slave:
>> So is that implying that you had the standby running without the restore_command?
>>> I'm using Debian 8 with PostgreSQL 9.4.1 on x86_64-unknown-linux-gnu,
>>> compiled by gcc-4.9. real (Debian 4.9.2-10) 4.9.2, 64-bit Slave error
>>> log: 5/3/2015 16:46:01 BRT [10210-1] @ Replicator [unknown] error:
>>> WAL segment requested 00000001000000000000002C has been removed
>>> 5/3/2015 16:46:05 BRT [10211-1] @ Replicator [unknown] error: WAL
>>> segment requested 00000001000000000000002C has been removed 5/3/2015
>>> 16:46:10 BRT [10214-1] @ Replicator [unknown] error: WAL segment
>>> requested 00000001000000000000002C has been removed 5/3/2015 16:46:15
>>> BRT [10216-1] @ Replicator [unknown] error: WAL segment requested
>>> 00000001000000000000002C has been removed Master error log 5/3/2015
>>> 19:13:35 BRT [3339-1] @ Replicator [unknown] error: WAL segment
>>> requested 00000001000000000000002C has been removed 5/3/2015 19:13:40
>>> BRT [3341-1] @ Replicator [unknown] error: WAL segment requested
>>> 00000001000000000000002C has been removed 5/3/2015 19:13:44 BRT
>>> [3343-1] @ Replicator [unknown] error: WAL segment requested
>>> 00000001000000000000002C has been removed Settings files are as
>>> follows: master postgresql.conf listen_addresses = '*' wal_level =
>>> hot_standby archive_mode = on archive_command = 'cp "%p"
>>> /mnt/server/archivedir/"%f"' max_wal_senders = 2 wal_keep_segments =
>>> 20 pg_hba.conf host replication replicador 192.168.0.112/32 trust
>>> secondary postgresql.conf listen_addresses = '*' hot_standby = on
>>> pg_hba.conf host all all 0.0.0.0/0 md5 recover.conf em
>>> (/var/lib/postgresql/9.4/main) standby_mode=on
>>> primary_conninfo='host=192.168.0.100 user=replicador
>>> application_name= jessie-stby' trigger_file='/tmp/pgtrigger'
>>> restore_command = 'cp /mnt/server/archivedir/%f %p'
>>> archive_cleanup_command = 'pg_archivecleanup /mnt/server/archivedir %r'
>> --
>> Adrian Klaver
>> adrian.klaver@aklaver.com  <mailto:adrian.klaver@aklaver.com>
>>
> Yes,
> It was working.

So what steps did you take to go from streaming only to streaming and
archiving?

I suspect there was a gap in the stop/restart that allowed a WAL file to
get recycled before the archiving started.

>
>


--
Adrian Klaver
adrian.klaver@aklaver.com

Re: Standby problem after restore_command Implementation

From

Melvin Davidson

Date:

05 May 2015, 14:22:36

It's possible you have wal_keep_segments set too low. What happens is that the master will keep the wals ( in your case 20) after processing them, before sending them off to the great black hole in the network (deleting) and making them unavailable to the standby. Try increasing wal_keep_segments = 100.

On Tue, May 5, 2015 at 10:09 AM, Adrian Klaver <adrian.klaver@aklaver.com> wrote:

On 05/05/2015 07:05 AM, Edson F. Lidorio wrote:

CCing list.

Em 2015-05-05 10:45, Adrian Klaver escreveu:

On 05/03/2015 05:57 PM, Edson F. Lidorio wrote:
Hello, I'm having trouble on Standby after the implementation of the
restore_command. I performed all the settings and it worked normally
and after restart the slave server, started to generate errors in the
log of the slave:
So is that implying that you had the standby running without the restore_command?
I'm using Debian 8 with PostgreSQL 9.4.1 on x86_64-unknown-linux-gnu,
compiled by gcc-4.9. real (Debian 4.9.2-10) 4.9.2, 64-bit Slave error
log: 5/3/2015 16:46:01 BRT [10210-1] @ Replicator [unknown] error:
WAL segment requested 00000001000000000000002C has been removed
5/3/2015 16:46:05 BRT [10211-1] @ Replicator [unknown] error: WAL
segment requested 00000001000000000000002C has been removed 5/3/2015
16:46:10 BRT [10214-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed 5/3/2015 16:46:15
BRT [10216-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed Master error log 5/3/2015
19:13:35 BRT [3339-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed 5/3/2015 19:13:40
BRT [3341-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed 5/3/2015 19:13:44 BRT
[3343-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed Settings files are as
follows: master postgresql.conf listen_addresses = '*' wal_level =
hot_standby archive_mode = on archive_command = 'cp "%p"
/mnt/server/archivedir/"%f"' max_wal_senders = 2 wal_keep_segments =
20 pg_hba.conf host replication replicador 192.168.0.112/32 trust
secondary postgresql.conf listen_addresses = '*' hot_standby = on
pg_hba.conf host all all 0.0.0.0/0 md5 recover.conf em
(/var/lib/postgresql/9.4/main) standby_mode=on
primary_conninfo='host=192.168.0.100 user=replicador
application_name= jessie-stby' trigger_file='/tmp/pgtrigger'
restore_command = 'cp /mnt/server/archivedir/%f %p'
archive_cleanup_command = 'pg_archivecleanup /mnt/server/archivedir %r'
--
Adrian Klaver
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>

Yes,
It was working.

So what steps did you take to go from streaming only to streaming and archiving?

I suspect there was a gap in the stop/restart that allowed a WAL file to get recycled before the archiving started.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Melvin Davidson
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Re: Standby problem after restore_command Implementation

From

Adrian Klaver

Date:

05 May 2015, 14:59:30

On 05/05/2015 07:23 AM, Edson F. Lidorio wrote:

Please keep this on the list, for two reasons:

1) More eyes looking is better.

2) I will be off line shortly.

> Em 2015-05-05 11:09, Adrian Klaver escreveu:
>
>> On 05/05/2015 07:05 AM, Edson F. Lidorio wrote:
>>
>> CCing list.
>>> Em 2015-05-05 10:45, Adrian Klaver escreveu:
>>>> On 05/03/2015 05:57 PM, Edson F. Lidorio wrote:
>>>>> Hello, I'm having trouble on Standby after the implementation of
>>>>> the restore_command. I performed all the settings and it worked
>>>>> normally and after restart the slave server, started to generate
>>>>> errors in the log of the slave:
>>>> So is that implying that you had the standby running without the
>>>> restore_command?
>>>>> I'm using Debian 8 with PostgreSQL 9.4.1 on
>>>>> x86_64-unknown-linux-gnu, compiled by gcc-4.9. real (Debian
>>>>> 4.9.2-10) 4.9.2, 64-bit Slave error log: 5/3/2015 16:46:01 BRT
>>>>> [10210-1] @ Replicator [unknown] error: WAL segment requested
>>>>> 00000001000000000000002C has been removed 5/3/2015 16:46:05 BRT
>>>>> [10211-1] @ Replicator [unknown] error: WAL segment requested
>>>>> 00000001000000000000002C has been removed 5/3/2015 16:46:10 BRT
>>>>> [10214-1] @ Replicator [unknown] error: WAL segment requested
>>>>> 00000001000000000000002C has been removed 5/3/2015 16:46:15 BRT
>>>>> [10216-1] @ Replicator [unknown] error: WAL segment requested
>>>>> 00000001000000000000002C has been removed Master error log 5/3/2015
>>>>> 19:13:35 BRT [3339-1] @ Replicator [unknown] error: WAL segment
>>>>> requested 00000001000000000000002C has been removed 5/3/2015
>>>>> 19:13:40 BRT [3341-1] @ Replicator [unknown] error: WAL segment
>>>>> requested 00000001000000000000002C has been removed 5/3/2015
>>>>> 19:13:44 BRT [3343-1] @ Replicator [unknown] error: WAL segment
>>>>> requested 00000001000000000000002C has been removed Settings files
>>>>> are as follows: master postgresql.conf listen_addresses = '*'
>>>>> wal_level = hot_standby archive_mode = on archive_command = 'cp
>>>>> "%p" /mnt/server/archivedir/"%f"' max_wal_senders = 2
>>>>> wal_keep_segments = 20 pg_hba.conf host replication replicador
>>>>> 192.168.0.112/32 trust secondary postgresql.conf listen_addresses =
>>>>> '*' hot_standby = on pg_hba.conf host all all 0.0.0.0/0 md5
>>>>> recover.conf em (/var/lib/postgresql/9.4/main) standby_mode=on
>>>>> primary_conninfo='host=192.168.0.100 user=replicador
>>>>> application_name= jessie-stby' trigger_file='/tmp/pgtrigger'
>>>>> restore_command = 'cp /mnt/server/archivedir/%f %p'
>>>>> archive_cleanup_command = 'pg_archivecleanup /mnt/server/archivedir %r'
>>>> -- Adrian Klaver adrian.klaver@aklaver.com
>>>> <mailto:adrian.klaver@aklaver.com> <mailto:adrian.klaver@aklaver.com
>>>> <mailto:adrian.klaver@aklaver.com>>
>>> Yes, It was working.
>> So what steps did you take to go from streaming only to streaming and archiving?
>>
>> I suspect there was a gap in the stop/restart that allowed a WAL file to get recycled before the archiving started.
>>
>> --
>> Adrian Klaver
>> adrian.klaver@aklaver.com  <mailto:adrian.klaver@aklaver.com>
>>
> A question thatI have!
> astheslave servercan seethis folderin themaster?
> /mnt/server/archivedir/

That would depend on your setup.

Can you log in to the standby and manually copy files from that directory?

>
> Sorrymy doubtsI'mbasicbeginner!
>
>


--
Adrian Klaver
adrian.klaver@aklaver.com

Re: Standby problem after restore_command Implementation

From

"Edson F. Lidorio"

Date:

05 May 2015, 15:36:14

On 05-05-2015 11:22, Melvin Davidson wrote:

It's possible you have wal_keep_segments set too low. What happens is that the master will keep the wals ( in your case 20) after processing them, before sending them off to the great black hole in the network (deleting) and making them unavailable to the standby. Try increasing wal_keep_segments = 100.

On Tue, May 5, 2015 at 10:09 AM, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 05/05/2015 07:05 AM, Edson F. Lidorio wrote:

CCing list.

Em 2015-05-05 10:45, Adrian Klaver escreveu:

On 05/03/2015 05:57 PM, Edson F. Lidorio wrote:
Hello, I'm having trouble on Standby after the implementation of the
restore_command. I performed all the settings and it worked normally
and after restart the slave server, started to generate errors in the
log of the slave:
So is that implying that you had the standby running without the restore_command?
I'm using Debian 8 with PostgreSQL 9.4.1 on x86_64-unknown-linux-gnu,
compiled by gcc-4.9. real (Debian 4.9.2-10) 4.9.2, 64-bit Slave error
log: 5/3/2015 16:46:01 BRT [10210-1] @ Replicator [unknown] error:
WAL segment requested 00000001000000000000002C has been removed
5/3/2015 16:46:05 BRT [10211-1] @ Replicator [unknown] error: WAL
segment requested 00000001000000000000002C has been removed 5/3/2015
16:46:10 BRT [10214-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed 5/3/2015 16:46:15
BRT [10216-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed Master error log 5/3/2015
19:13:35 BRT [3339-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed 5/3/2015 19:13:40
BRT [3341-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed 5/3/2015 19:13:44 BRT
[3343-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed Settings files are as
follows: master postgresql.conf listen_addresses = '*' wal_level =
hot_standby archive_mode = on archive_command = 'cp "%p"
/mnt/server/archivedir/"%f"' max_wal_senders = 2 wal_keep_segments =
20 pg_hba.conf host replication replicador 192.168.0.112/32 trust
secondary postgresql.conf listen_addresses = '*' hot_standby = on
pg_hba.conf host all all 0.0.0.0/0 md5 recover.conf em
(/var/lib/postgresql/9.4/main) standby_mode=on
primary_conninfo='host=192.168.0.100 user=replicador
application_name= jessie-stby' trigger_file='/tmp/pgtrigger'
restore_command = 'cp /mnt/server/archivedir/%f %p'
archive_cleanup_command = 'pg_archivecleanup /mnt/server/archivedir %r'
--
Adrian Klaver
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>

Yes,
It was working.

So what steps did you take to go from streaming only to streaming and archiving?

I suspect there was a gap in the stop/restart that allowed a WAL file to get recycled before the archiving started.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Melvin Davidson
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Increased the wal_keep_segments = 100 and keeps popping up the same message:

A question that I have!

as the slave server can see this folder in the master?

/mnt/server/archivedir/

Sorry my doubts I'm basic beginner!

Re: Standby problem after restore_command Implementation

From

Melvin Davidson

Date:

05 May 2015, 16:05:30

Did you change the setting on the master?

Did you remember to reload the config on the master after changing?

Did you rebuild the Standby server after making the change and reloading?

IE:
psql -U postgres

SELECT pg_reload_conf();

You must do ALL of the above to have any chance of making things work.

On Tue, May 5, 2015 at 11:35 AM, Edson F. Lidorio <edson@openmailbox.org> wrote:

On 05-05-2015 11:22, Melvin Davidson wrote:
It's possible you have wal_keep_segments set too low. What happens is that the master will keep the wals ( in your case 20) after processing them, before sending them off to the great black hole in the network (deleting) and making them unavailable to the standby. Try increasing wal_keep_segments = 100.

On Tue, May 5, 2015 at 10:09 AM, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 05/05/2015 07:05 AM, Edson F. Lidorio wrote:

CCing list.

Em 2015-05-05 10:45, Adrian Klaver escreveu:

On 05/03/2015 05:57 PM, Edson F. Lidorio wrote:
Hello, I'm having trouble on Standby after the implementation of the
restore_command. I performed all the settings and it worked normally
and after restart the slave server, started to generate errors in the
log of the slave:
So is that implying that you had the standby running without the restore_command?
I'm using Debian 8 with PostgreSQL 9.4.1 on x86_64-unknown-linux-gnu,
compiled by gcc-4.9. real (Debian 4.9.2-10) 4.9.2, 64-bit Slave error
log: 5/3/2015 16:46:01 BRT [10210-1] @ Replicator [unknown] error:
WAL segment requested 00000001000000000000002C has been removed
5/3/2015 16:46:05 BRT [10211-1] @ Replicator [unknown] error: WAL
segment requested 00000001000000000000002C has been removed 5/3/2015
16:46:10 BRT [10214-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed 5/3/2015 16:46:15
BRT [10216-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed Master error log 5/3/2015
19:13:35 BRT [3339-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed 5/3/2015 19:13:40
BRT [3341-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed 5/3/2015 19:13:44 BRT
[3343-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed Settings files are as
follows: master postgresql.conf listen_addresses = '*' wal_level =
hot_standby archive_mode = on archive_command = 'cp "%p"
/mnt/server/archivedir/"%f"' max_wal_senders = 2 wal_keep_segments =
20 pg_hba.conf host replication replicador 192.168.0.112/32 trust
secondary postgresql.conf listen_addresses = '*' hot_standby = on
pg_hba.conf host all all 0.0.0.0/0 md5 recover.conf em
(/var/lib/postgresql/9.4/main) standby_mode=on
primary_conninfo='host=192.168.0.100 user=replicador
application_name= jessie-stby' trigger_file='/tmp/pgtrigger'
restore_command = 'cp /mnt/server/archivedir/%f %p'
archive_cleanup_command = 'pg_archivecleanup /mnt/server/archivedir %r'
--
Adrian Klaver
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>

Yes,
It was working.

So what steps did you take to go from streaming only to streaming and archiving?

I suspect there was a gap in the stop/restart that allowed a WAL file to get recycled before the archiving started.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Melvin Davidson
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.
Increased the wal_keep_segments = 100 and keeps popping up the same message:

A question that I have!
as the slave server can see this folder in the master?
/mnt/server/archivedir/

Sorry my doubts I'm basic beginner!

Melvin Davidson
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Re: Standby problem after restore_command Implementation

From

"Edson F. Lidorio"

Date:

05 May 2015, 17:00:07

Em 2015-05-05 13:05, Melvin Davidson escreveu:

Did you change the setting on the master?
Did you remember to reload the config on the master after changing?
Did you rebuild the Standby server after making the change and reloading?
IE:
psql -U postgres
SELECT pg_reload_conf();

You must do ALL of the above to have any chance of making things work.

On Tue, May 5, 2015 at 11:35 AM, Edson F. Lidorio <edson@openmailbox.org> wrote:
On 05-05-2015 11:22, Melvin Davidson wrote:
It's possible you have wal_keep_segments set too low. What happens is that the master will keep the wals ( in your case 20) after processing them, before sending them off to the great black hole in the network (deleting) and making them unavailable to the standby. Try increasing wal_keep_segments = 100.

On Tue, May 5, 2015 at 10:09 AM, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 05/05/2015 07:05 AM, Edson F. Lidorio wrote:

CCing list.

Em 2015-05-05 10:45, Adrian Klaver escreveu:

On 05/03/2015 05:57 PM, Edson F. Lidorio wrote:
Hello, I'm having trouble on Standby after the implementation of the
restore_command. I performed all the settings and it worked normally
and after restart the slave server, started to generate errors in the
log of the slave:
So is that implying that you had the standby running without the restore_command?
I'm using Debian 8 with PostgreSQL 9.4.1 on x86_64-unknown-linux-gnu,
compiled by gcc-4.9. real (Debian 4.9.2-10) 4.9.2, 64-bit Slave error
log: 5/3/2015 16:46:01 BRT [10210-1] @ Replicator [unknown] error:
WAL segment requested 00000001000000000000002C has been removed
5/3/2015 16:46:05 BRT [10211-1] @ Replicator [unknown] error: WAL
segment requested 00000001000000000000002C has been removed 5/3/2015
16:46:10 BRT [10214-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed 5/3/2015 16:46:15
BRT [10216-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed Master error log 5/3/2015
19:13:35 BRT [3339-1] @ Replicator [unknown] error: WAL segment
requested 00000001000000000000002C has been removed 5/3/2015 19:13:40
BRT [3341-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed 5/3/2015 19:13:44 BRT
[3343-1] @ Replicator [unknown] error: WAL segment requested
00000001000000000000002C has been removed Settings files are as
follows: master postgresql.conf listen_addresses = '*' wal_level =
hot_standby archive_mode = on archive_command = 'cp "%p"
/mnt/server/archivedir/"%f"' max_wal_senders = 2 wal_keep_segments =
20 pg_hba.conf host replication replicador 192.168.0.112/32 trust
secondary postgresql.conf listen_addresses = '*' hot_standby = on
pg_hba.conf host all all 0.0.0.0/0 md5 recover.conf em
(/var/lib/postgresql/9.4/main) standby_mode=on
primary_conninfo='host=192.168.0.100 user=replicador
application_name= jessie-stby' trigger_file='/tmp/pgtrigger'
restore_command = 'cp /mnt/server/archivedir/%f %p'
archive_cleanup_command = 'pg_archivecleanup /mnt/server/archivedir %r'
--
Adrian Klaver
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>

Yes,
It was working.

So what steps did you take to go from streaming only to streaming and archiving?

I suspect there was a gap in the stop/restart that allowed a WAL file to get recycled before the archiving started.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

--
Melvin Davidson
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.
Increased the wal_keep_segments = 100 and keeps popping up the same message:

A question that I have!
as the slave server can see this folder in the master?
/mnt/server/archivedir/

Sorry my doubts I'm basic beginner!

--
Melvin Davidson
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

After you restart the server and it didnt work I'd done reload.

I'm going to do another standby charge today, and see what happens.

Could you explain that doubt my previous message.

I have a slight feeling that the problem may also be in the access to the files folder of the wal.

Re: Standby problem after restore_command Implementation

From

Adrian Klaver

Date:

05 May 2015, 17:31:39

On 05/05/2015 08:35 AM, Edson F. Lidorio wrote:
> On 05-05-2015 11:22, Melvin Davidson wrote:
>> It's possible you have wal_keep_segments set too low. What happens is
>> that the master will keep the wals ( in your case 20) after processing
>> them, before sending them off to the great black hole in the network
>> (deleting) and making them unavailable to the standby. Try increasing
>> wal_keep_segments = 100.
>>

> Increased thewal_keep_segments=100andkeeps popping upthe same message:

The message did not come through. Also increasing wal_keep_segments only
works for the future, it will not deal with a WAL file that has already
been recycled before the setting was changed.

>
> A question thatI have!
> astheslave servercan seethis folderin themaster?
> /mnt/server/archivedir/
>

So have you tried logging into the standby and then accessing the
directory on the master manually. In other words copy a file from the
master to standby using cp from the standby command line.

> Sorrymy doubtsI'mbasicbeginner!

--
Adrian Klaver
adrian.klaver@aklaver.com

Re: Standby problem after restore_command Implementation

From

"Edson F. Lidorio"

Date:

05 May 2015, 18:05:43

Em 2015-05-05 14:31, Adrian Klaver escreveu:

On 05/05/2015 08:35 AM, Edson F. Lidorio wrote:
On 05-05-2015 11:22, Melvin Davidson wrote:
It's possible you have wal_keep_segments set too low. What happens is that the master will keep the wals ( in your case 20) after processing them, before sending them off to the great black hole in the network (deleting) and making them unavailable to the standby. Try increasing wal_keep_segments = 100.
Increased thewal_keep_segments=100andkeeps popping upthe same message:
The message did not come through. Also increasing wal_keep_segments only works for the future, it will not deal with a WAL file that has already been recycled before the setting was changed.
A question thatI have! astheslave servercan seethis folderin themaster? /mnt/server/archivedir/
So have you tried logging into the standby and then accessing the directory on the master manually. In other words copy a file from the master to standby using cp from the standby command line.
Sorrymy doubtsI'mbasicbeginner!
-- 
Adrian Klaver
adrian.klaver@aklaver.com

OK, I'm going to do this test too.

-- 
Edson F. Lidorio

Atenção: Este e-mail pode conter anexos no formato ODF (Open Document
Format)/ABNT (extensões odt, ods, odp, odb, odg). Antes de pedir os anexos
em outro formato, você pode instalar gratuita e livremente o LibreOffice (
http://pt-br.libreoffice.org/).

Re: Standby problem after restore_command Implementation

From

Melvin Davidson

Date:

05 May 2015, 18:21:33

Have you allowed access to user "replication" in the master pg_hba.conf and done a
SELECT pg_reload_conf()?

Can you ping the master from the standby?

Did you RTFM?
http://www.postgresql.org/docs/9.2/interactive/warm-standby.html

25.2.3 Preparing the Master for Standby Servers
25.2.4 Setting Up a Standby Server

On Tue, May 5, 2015 at 2:05 PM, Edson F. Lidorio <edson@openmailbox.org> wrote:

Em 2015-05-05 14:31, Adrian Klaver escreveu:
On 05/05/2015 08:35 AM, Edson F. Lidorio wrote:
On 05-05-2015 11:22, Melvin Davidson wrote:
It's possible you have wal_keep_segments set too low. What happens is that the master will keep the wals ( in your case 20) after processing them, before sending them off to the great black hole in the network (deleting) and making them unavailable to the standby. Try increasing wal_keep_segments = 100.
Increased thewal_keep_segments=100andkeeps popping upthe same message:
The message did not come through. Also increasing wal_keep_segments only works for the future, it will not deal with a WAL file that has already been recycled before the setting was changed.
A question thatI have! astheslave servercan seethis folderin themaster? /mnt/server/archivedir/
So have you tried logging into the standby and then accessing the directory on the master manually. In other words copy a file from the master to standby using cp from the standby command line.
Sorrymy doubtsI'mbasicbeginner!
-- 
Adrian Klaver
adrian.klaver@aklaver.com
OK, I'm going to do this test too.

-- Edson F. Lidorio Atenção: Este e-mail pode conter anexos no formato ODF (Open Document Format)/ABNT (extensões odt, ods, odp, odb, odg). Antes de pedir os anexos em outro formato, você pode instalar gratuita e livremente o LibreOffice ( http://pt-br.libreoffice.org/).

Melvin Davidson
I reserve the right to fantasize. Whether or not you
wish to share my fantasy is entirely up to you.

Re: Standby problem after restore_command Implementation

From

Adrian Klaver

Date:

05 May 2015, 22:09:55

On 05/05/2015 11:21 AM, Melvin Davidson wrote:
> Have you allowed access to user "replication" in the master pg_hba.conf
> and done a
> SELECT pg_reload_conf()?
>
> Can you ping the master from the standby?
>
> Did you RTFM?
> http://www.postgresql.org/docs/9.2/interactive/warm-standby.html
>
> 25.2.3 Preparing the Master for Standby Servers
> 25.2.4 Setting Up a Standby Server


See the OPs original post:

http://www.postgresql.org/message-id/5546C3DC.5070808@openmailbox.org

The OP had streaming only replication working, so the basics are there.
The wheels fell off when WAL archiving was introduced. Best guess is
this is a timing issue, the relevant WALs got recycled while archiving
was being set up and the standby was down.


>
>
>
> On Tue, May 5, 2015 at 2:05 PM, Edson F. Lidorio <edson@openmailbox.org
> <mailto:edson@openmailbox.org>> wrote:
>
>     __
>
>     Em 2015-05-05 14:31, Adrian Klaver escreveu:
>
>>     On 05/05/2015 08:35 AM, Edson F. Lidorio wrote:
>>>     On 05-05-2015 11:22, Melvin Davidson wrote:
>>>>     It's possible you have wal_keep_segments set too low. What
>>>>     happens is that the master will keep the wals ( in your case 20)
>>>>     after processing them, before sending them off to the great
>>>>     black hole in the network (deleting) and making them unavailable
>>>>     to the standby. Try increasing wal_keep_segments = 100.
>>>     Increased thewal_keep_segments=100andkeeps popping upthe same
>>>     message:
>>     The message did not come through. Also increasing wal_keep_segments only works for the future, it will not deal
witha WAL file that has already been recycled before the setting was changed. 
>>>     A question thatI have! astheslave servercan seethis folderin
>>>     themaster? /mnt/server/archivedir/
>>     So have you tried logging into the standby and then accessing the directory on the master manually. In other
wordscopy a file from the master to standby using cp from the standby command line. 
>>>     Sorrymy doubtsI'mbasicbeginner!
>>     --
>>     Adrian Klaver
>>     adrian.klaver@aklaver.com  <mailto:adrian.klaver@aklaver.com>
>>
>     OK,I'm going to dothis testtoo.
>
>     --
>     Edson F. Lidorio
>
>     Atenção: Este e-mail pode conter anexos no formato ODF (Open Document
>     Format)/ABNT (extensões odt, ods, odp, odb, odg). Antes de pedir os anexos
>     em outro formato, você pode instalar gratuita e livremente o LibreOffice (
>     http://pt-br.libreoffice.org/).
>
>
>
>
> --
> *Melvin Davidson*
> I reserve the right to fantasize.  Whether or not you
> wish to share my fantasy is entirely up to you.


--
Adrian Klaver
adrian.klaver@aklaver.com

Re: Standby problem after restore_command Implementation

From

Venkata Balaji N

Date:

05 May 2015, 22:49:27

On Tue, May 5, 2015 at 7:57 AM, Edson F. Lidorio <edson@openmailbox.org> wrote:

On 04-05-2015 00:46, Venkata Balaji N wrote:
You do not see the above WAL file in the archive directory ? "/mnt/server/archivedir" is shared between master and slave databases ? The file may have been removed from the archive location.
Strange !!
Yes, this specified folder file "/ mnt / server / ArchiveDir".
At this shared folder between master and slave.

I suspect that archive cleanup job is cleaning up the archives. "archive_cleanup_command" removes any archives not needed for recovery.

Since the the archive destination is shared across master and slave. Archives are removed and Standby is not getting the required archives.

To avoid this, before you do any tests, i suggest remove "archive_cleanup_command" parameter and see if you are still facing the same problem.

Regards,

Venkata Balaji N

Fujitsu Australia