Thread: behave of --create-slot option
Hi
I am writing a article about PostgreSQL 11 features. Now I am looking on new option --create-slot option of pg_basebackup command.
I don't understand to use case for this option, because It fails when requested slot already exists. I cannot to imagine use case for this. If I write some scripts, then I prefer the behave like "create if doesn't exist or do nothing".
Any repeated running of script with this option should to fail. Is it required? Why?
Regards
Pavel
2018-05-28 16:41 GMT-03:00 Pavel Stehule <pavel.stehule@gmail.com>: > I am writing a article about PostgreSQL 11 features. Now I am looking on new > option --create-slot option of pg_basebackup command. > > I don't understand to use case for this option, because It fails when > requested slot already exists. I cannot to imagine use case for this. If I > write some scripts, then I prefer the behave like "create if doesn't exist > or do nothing". > In prior versions, you should create a slot (via SQL function or streaming replication protocol) *before* run pg_basebackup. In 11, use --create-slot option and you don't need an extra step (of course, you should drop that slot after the end of backup -- if that is not a new standby. It also does not make sense to use _persistent_ replication slots for backup because you are probably archiving WAL). IMHO the use case is new standbys that will use replication slots. > Any repeated running of script with this option should to fail. Is it > required? Why? > As I said, you should have an extra step to drop that slot (even before 11). There is no "create replication slots if not exists". If you are repeatedly running a script, why don't you let pg_basebackup use temporary replication slots? -- Euler Taveira Timbira - http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento
On 29 May 2018 at 03:41, Pavel Stehule <pavel.stehule@gmail.com> wrote:
HiI am writing a article about PostgreSQL 11 features. Now I am looking on new option --create-slot option of pg_basebackup command.I don't understand to use case for this option, because It fails when requested slot already exists. I cannot to imagine use case for this. If I write some scripts, then I prefer the behave like "create if doesn't exist or do nothing".Any repeated running of script with this option should to fail. Is it required? Why?
The slot is intended to then be used by a replica that was created by pg_basebackup. I think it writes the slot name into recovery.conf; if it doesn't, it should.
So you use a unique slot name for each replica.
2018-05-29 3:28 GMT+02:00 Craig Ringer <craig@2ndquadrant.com>:
On 29 May 2018 at 03:41, Pavel Stehule <pavel.stehule@gmail.com> wrote:HiI am writing a article about PostgreSQL 11 features. Now I am looking on new option --create-slot option of pg_basebackup command.I don't understand to use case for this option, because It fails when requested slot already exists. I cannot to imagine use case for this. If I write some scripts, then I prefer the behave like "create if doesn't exist or do nothing".Any repeated running of script with this option should to fail. Is it required? Why?The slot is intended to then be used by a replica that was created by pg_basebackup. I think it writes the slot name into recovery.conf; if it doesn't, it should.So you use a unique slot name for each replica.
I understand so slot should be unique. But same result (unique rep slot) can be done, if it does nothing when slot exists already. This behave is not idempotent.
Maybe I am search problem, where it is not. Just, when I see some "create object" option, I expect any way, how I can enforce "--if-exists", because it was necessary in major cases.
Regards
Pavel
--
On 29 May 2018 at 11:51, Pavel Stehule <pavel.stehule@gmail.com> wrote:
I understand so slot should be unique. But same result (unique rep slot) can be done, if it does nothing when slot exists already. This behave is not idempotent.Maybe I am search problem, where it is not. Just, when I see some "create object" option, I expect any way, how I can enforce "--if-exists", because it was necessary in major cases.
If the slot already exists, don't you expect it to be in use for an existing replica?
I guess what you seem to want is to be able to delete the replica then re-clone it and use the same slot?
Generally I'd expect you to delete the slot when you remove the replica. Really what this says to me is that we should have a way to delete the upstream slot when we promote or decommission a physical replica.
2018-05-29 6:11 GMT+02:00 Craig Ringer <craig@2ndquadrant.com>:
On 29 May 2018 at 11:51, Pavel Stehule <pavel.stehule@gmail.com> wrote:I understand so slot should be unique. But same result (unique rep slot) can be done, if it does nothing when slot exists already. This behave is not idempotent.Maybe I am search problem, where it is not. Just, when I see some "create object" option, I expect any way, how I can enforce "--if-exists", because it was necessary in major cases.If the slot already exists, don't you expect it to be in use for an existing replica?
the slot name is unique, so I don't expect it - when I use some name from script
I guess what you seem to want is to be able to delete the replica then re-clone it and use the same slot?
maybe. Now, it looks "asymmetric"
Generally I'd expect you to delete the slot when you remove the replica. Really what this says to me is that we should have a way to delete the upstream slot when we promote or decommission a physical replica.
When I think about this option and designed behave, I am inclined to think so it can be hard to use, and better create slot outside, and outside drop slot too.
I hope so I understand to motivation for this option - but I see issues:
1. pg_basebackup can fail from some other reasons, but there is not special status for this issue.
2. when I use this option in any script, then I should to remove possibly exists slot before, to minimize risk of fail of this command.
I understand, current behave can be wanted like protection against unwanted running some deployment script. But can be problematic too, when pg_basebackup or or some other fails from unexpected reasons.
Regards
Pavel
--
2018-05-29 1:31 GMT-03:00 Pavel Stehule <pavel.stehule@gmail.com>:
> I hope so I understand to motivation for this option - but I see issues:
>
> 1. pg_basebackup can fail from some other reasons, but there is not special
> status for this issue.
> 2. when I use this option in any script, then I should to remove possibly
> exists slot before, to minimize risk of fail of this command.
>
If pg_basebackup failed for some other reason *after* the replication
slot was created (say, permission problem) then we should try to
cleanup the old slot. That should be the behavior if we want to try to
be idempotent ("try" because if we have a network problem it won't be
possible to remove the replication slot).
--
Euler Taveira Timbira -
http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento
2018-05-29 16:53 GMT+02:00 Euler Taveira <euler@timbira.com.br>:
2018-05-29 1:31 GMT-03:00 Pavel Stehule <pavel.stehule@gmail.com>:
> I hope so I understand to motivation for this option - but I see issues:
>
> 1. pg_basebackup can fail from some other reasons, but there is not special
> status for this issue.
> 2. when I use this option in any script, then I should to remove possibly
> exists slot before, to minimize risk of fail of this command.
>
If pg_basebackup failed for some other reason *after* the replication
slot was created (say, permission problem) then we should try to
cleanup the old slot. That should be the behavior if we want to try to
be idempotent ("try" because if we have a network problem it won't be
possible to remove the replication slot).
It has sense for me
Pavel
--
Euler Taveira Timbira -
http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento
On Tue, May 29, 2018 at 09:21:00PM +0200, Pavel Stehule wrote:
> 2018-05-29 16:53 GMT+02:00 Euler Taveira <euler@timbira.com.br>:
>> If pg_basebackup failed for some other reason *after* the replication
>> slot was created (say, permission problem) then we should try to
>> cleanup the old slot. That should be the behavior if we want to try to
>> be idempotent ("try" because if we have a network problem it won't be
>> possible to remove the replication slot).
>
> It has sense for me
Hm. There could be an argument for improving the user experience here
so as some cleanup is at least attempted except if --no-clean is defined
by the caller when --create-slot is used. Do we want an open item for
this issue?
--
Michael
Attachment
On Wed, May 30, 2018 at 2:00 PM, Michael Paquier <michael@paquier.xyz> wrote: > Hm. There could be an argument for improving the user experience here > so as some cleanup is at least attempted except if --no-clean is defined > by the caller when --create-slot is used. Do we want an open item for > this issue? Sounds like new development to me. This isn't a bug. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Thu, May 31, 2018 at 10:59:04PM -0400, Robert Haas wrote: > On Wed, May 30, 2018 at 2:00 PM, Michael Paquier <michael@paquier.xyz> wrote: >> Hm. There could be an argument for improving the user experience here >> so as some cleanup is at least attempted except if --no-clean is defined >> by the caller when --create-slot is used. Do we want an open item for >> this issue? > > Sounds like new development to me. This isn't a bug. Still, it seems to me that the user experience is a bit horrible with this new interface of pg_basebackup. If --create-slot is used, then a slot is created before starting a backup. If the slot already exists, then pg_basebackup complains and exits. In order to drop the slot with a only user who has replication access rights (because nobody is really going to have a user who has SQL access so as the slot is dropped), then the only simple way is to use pg_receivewal --drop-slot, making the whole flow inconsistent. pg_basebackup is usually in server-side packages, and pg_receivewal is on the client side. The server packages requiring the client packages, then we are sure that pg_basebackup will drag in pg_receivewal. Still, shouldn't there be a --drop-slot option in pg_basebackup? In this case, if --drop-slot is used, then the slot is dropped and pg_basebackup exits immediately. -- Michael
Attachment
2018-06-01 9:00 GMT-03:00 Michael Paquier <michael@paquier.xyz>: > On Thu, May 31, 2018 at 10:59:04PM -0400, Robert Haas wrote: >> On Wed, May 30, 2018 at 2:00 PM, Michael Paquier <michael@paquier.xyz> wrote: >>> Hm. There could be an argument for improving the user experience here >>> so as some cleanup is at least attempted except if --no-clean is defined >>> by the caller when --create-slot is used. Do we want an open item for >>> this issue? >> >> Sounds like new development to me. This isn't a bug. > > Still, it seems to me that the user experience is a bit horrible with > this new interface of pg_basebackup. If --create-slot is used, then a > slot is created before starting a backup. If the slot already exists, > then pg_basebackup complains and exits. In order to drop the slot with > a only user who has replication access rights (because nobody is really > going to have a user who has SQL access so as the slot is dropped), then > the only simple way is to use pg_receivewal --drop-slot, making the > whole flow inconsistent. pg_basebackup is usually in server-side > packages, and pg_receivewal is on the client side. The server packages > requiring the client packages, then we are sure that pg_basebackup will > drag in pg_receivewal. > Debian and derivatives put pg_basebackup in the client package (indeed, it a client program). A possible fix is to drop the slot if there is an error. However, if the problem is connectivity, we can't drop it. In this case, we can print a user-friendly error saying that the user should drop that slot before try again). If we follow this idea, then I consider it to be a bug fix. > Still, shouldn't there be a --drop-slot option in pg_basebackup? In > this case, if --drop-slot is used, then the slot is dropped and > pg_basebackup exits immediately. > I don't like it. You should send an extra command to recover from an error. It also does not follow the idempotent behavior. I know that you are trying to mimic pg_receivewal options but I prefer another option that removes the slot if it exists (say, --drop-slot-if-exists). In this case, I consider this option as new development. -- Euler Taveira Timbira - http://www.timbira.com.br/ PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento
2018-06-01 18:36 GMT+02:00 Euler Taveira <euler@timbira.com.br>:
2018-06-01 9:00 GMT-03:00 Michael Paquier <michael@paquier.xyz>:
> On Thu, May 31, 2018 at 10:59:04PM -0400, Robert Haas wrote:
>> On Wed, May 30, 2018 at 2:00 PM, Michael Paquier <michael@paquier.xyz> wrote:
>>> Hm. There could be an argument for improving the user experience here
>>> so as some cleanup is at least attempted except if --no-clean is defined
>>> by the caller when --create-slot is used. Do we want an open item for
>>> this issue?
>>
>> Sounds like new development to me. This isn't a bug.
>
> Still, it seems to me that the user experience is a bit horrible with
> this new interface of pg_basebackup. If --create-slot is used, then a
> slot is created before starting a backup. If the slot already exists,
> then pg_basebackup complains and exits. In order to drop the slot with
> a only user who has replication access rights (because nobody is really
> going to have a user who has SQL access so as the slot is dropped), then
> the only simple way is to use pg_receivewal --drop-slot, making the
> whole flow inconsistent. pg_basebackup is usually in server-side
> packages, and pg_receivewal is on the client side. The server packages
> requiring the client packages, then we are sure that pg_basebackup will
> drag in pg_receivewal.
>
Debian and derivatives put pg_basebackup in the client package
(indeed, it a client program). A possible fix is to drop the slot if
there is an error. However, if the problem is connectivity, we can't
drop it. In this case, we can print a user-friendly error saying that
the user should drop that slot before try again). If we follow this
idea, then I consider it to be a bug fix.
> Still, shouldn't there be a --drop-slot option in pg_basebackup? In
> this case, if --drop-slot is used, then the slot is dropped and
> pg_basebackup exits immediately.
>
I don't like it. You should send an extra command to recover from an
error. It also does not follow the idempotent behavior. I know that
you are trying to mimic pg_receivewal options but I prefer another
option that removes the slot if it exists (say,
--drop-slot-if-exists). In this case, I consider this option as new
development.
+1
Pavel
--
Euler Taveira Timbira -
http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento