Thread: pg_backrest restore & streaming replication
Hello,
I have a 2 host setup: master & replica.
Both hosts share a NAS directory mounted at /backup, which is where pg_backrest backups and WAL archives are saved.
We had a recent outage where WAL logs and pg_backrest could not save to /backup, and as a result the replica got really behind. The situation has since been resolved in terms of getting WAL logs and backups correctly being archived into /backup.
I just did a restore on my replica using the most recent backup instead of having it catch up via WAL logs.
This is the command I used: /usr/lib/backrest/bin/pg_backrest.pl --stanza=prod-9.4 restore --log-level-console=info --db-path=/var/lib/postgresql/9.4/replica --thread-max=4 --delta
After starting the replica back up, I noticed that it was no longer doing streaming replication. Also I noticed the logs said something about not finding WAL logs that it expected to find (not sure if these two things are related).
Here is the bottom part of the replica's logs on startup:
2016-03-17 14:14:40.601 T00 INFO: getting WAL segment 00000001000002020000001D
2016-03-17 14:14:40.858 CDT [2635-107]: LOG: restored log file "00000001000002020000001D" from archive
2016-03-17 14:14:41.214 T00 INFO: getting WAL segment 00000001000002020000001E
2016-03-17 14:14:41.216 T00 INFO: 00000001000002020000001E was not found in the archive repository
2016-03-17 14:14:41.235 CDT [2635-108]: LOG: redo done at 202/1DFFFE78
2016-03-17 14:14:41.235 CDT [2635-109]: LOG: last completed transaction was at log time 2016-03-17 14:13:18.23722-05
2016-03-17 14:14:41.527 T00 INFO: getting WAL segment 00000001000002020000001D
2016-03-17 14:14:41.755 CDT [2635-110]: LOG: restored log file "00000001000002020000001D" from archive
2016-03-17 14:14:41.964 T00 INFO: getting WAL segment 00000002.history
2016-03-17 14:14:41.967 T00 INFO: 00000002.history was not found in the archive repository
2016-03-17 14:14:41.992 CDT [2635-111]: LOG: selected new timeline ID: 2
2016-03-17 14:14:42.197 T00 INFO: getting WAL segment 00000001.history
2016-03-17 14:14:42.199 T00 INFO: 00000001.history was not found in the archive repository
2016-03-17 14:14:42.262 CDT [2635-112]: LOG: archive recovery complete
2016-03-17 14:14:42.263 CDT [2639-1]: LOG: checkpoint starting: end-of-recovery immediate wait
2016-03-17 14:15:16.906 CDT [2639-2]: LOG: checkpoint complete: wrote 166544 buffers (31.8%); 0 transaction log file(s) added, 0 removed, 0 recycled; write=1.744 s, sync=32.890 s, total=34.643 s; sync files=551, longest=3.991 s, average=0.059 s
Thank you,
-Aldo
Aldo, * Aldo Sarmiento (aldo@bigpurpledot.com) wrote: > This is the command I used: /usr/lib/backrest/bin/pg_backrest.pl > --stanza=prod-9.4 restore --log-level-console=info > --db-path=/var/lib/postgresql/9.4/replica --thread-max=4 --delta > After starting the replica back up, I noticed that it was no longer doing > streaming replication. Also I noticed the logs said something about not > finding WAL logs that it expected to find (not sure if these two things are > related). The pgbackrest restore command which you used doesn't appear to have included any of the settings necessary to set up a streaming replica. I'm guessing they also weren't included in the pg_backrest.conf file. The only parameters that pgbackrest sets in the recovery.conf by default are those necessary to acquire the necessary XLOGs during restore (the recovery_command option). To add other options to the recovery.conf file (such as those required to start the system as a replica), you can use the --recovery-option to pgbackrest, or add a '[stanza:restore:recovery-option]' section to the pg_backrest.conf file being used. The pgbackrest user guide includes an example of setting up the pg_backrest.conf file with the necessary options: http://www.pgbackrest.org/user-guide.html The "not found" messages are completely normal and expected, PG will ask for XLOGs which don't exist and pgbackrest is just logging that it was asked for an XLOG which didn't exist. Thanks! Stephen