Re: Sync Rep: First Thoughts on Code - Mailing list pgsql-hackers

Hi,

On Thu, Dec 11, 2008 at 7:09 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
>> > Recent changes I have requested in the architecture are:
>> > * making archiving optional on primary, so we don't need to send WAL
>> > data *twice*.
>>
>> Agreed. I'm not so much worried about the bandwidth, but it's a lot of
>> extra work from administration point of view. It's very hard to get it
>> right, so that you eliminate windows like the above.
>>
>> As the patch stands, if you turn off archiving in the primary, and the
>> standby ever disconnects, even for only a few seconds, the standby will
>> miss any WAL generated until it reconnects, and without archiving
>> there's no way for the standby to get hold of the missed WAL.
>
> I described earlier that archiving would turn back on again if the
> replication ever failed (with correct synchronisation).
>
> All I've asked for is the ability to turn on and turn back on archiving,
> yes, with synchronisation so its safe.
>
> Personally, I think people will laugh if we tell them we decided to ship
> all the data twice and couldn't see another way. That's the kind of
> thing people give presentations at PGcon about...


OK, I will add such archiving feature. My new design of archiving is as follows.

Primary
----------
I extend archive_mode as follows and make the user be able to choose the
archiving strategy on the primary.

- always The primary always archives the WAL. This is compatible with current (<=8.3) archive_mode = on.

- none The primary always doesn't archive the WAL. This is compatible with current archive_mode = off.

- standalone The primary doesn't archive the WAL only during replication. If replication is not in progress, the
primaryarchives the WAL. That is, the primary switches the modes whenever replication starts / ends.
 
 [FLS->SLS] When replication starts, the primary disable archiving *after* the switched WAL file is archived. WAL
streamingdoesn't need to wait for disablement of archiving, so the processing on the primary isn't blocked by starting
ofreplication. But, both WAL streaming and archiving would be in progress for a while (until the switched WAL file is
archived)after
 
replication starts.
 [SLS->FLS] When replication starts, the primary restarts archiving immediately. This also doesn't block the processing
onthe primary. But, this might cause loss of some files from an archive if archiving is slow on the standby. The
primaryshould look for the last archived file (by the standby) from an archive and restart archiving from the
subsequentfile? Of course, the primary cannot archive it if it's already removed on the primary.
 

Standby
-----------
I would add new option for achiving during recovery into recovery.conf
(recovery_archive_mode). Though this option is similar to archive_mode,
merging them would confuse the user more, I think. Or, I should merge?
And, do you want to configure the archive command only for recovery?
If so, I would add new option to specify the archive command during
recovery (recovery_archive_command).

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center