Thread: Too high rate of progress information from pg_basebackup
Hi,
it looks that pg_basebackup, when using --progress information, outputs
so much information that it puts cpu pressure on terminal application
I'm using - whether it's simple xterm/rxvt, "screen" or "tmux" terminal
multiplexers.
My specific usecase is setting multiple slave servers for multiple
clusters in AWS, so the data transmission between servers is pretty
fast, but the jump host, that I run all the pg_basebackup (via ssh)
from, is not really beefy, and with 4 concurrent pg_basebackups, I've
seen tmux use up to 80% of CPU of the jump host.
I filled bug report for tmux too, but since the problem can be repeated
in xterm, rxvt, and screen, I think that perhaps it should be handled in
some way in pg_basebackup too?
For completeness sake:
we're using pg_basebackup from 9.3.11 on ubuntu trusty.
In my quick sanity-check test, it looks that progress information in our
case is outputted with the speed of ~ 320KB/s. Which seems excessive.
Best regards,
depesz
--
The best thing about modern society is how easy it is to avoid contact with it.
http://depesz.com/
On Wed, May 25, 2016 at 12:37 PM, hubert depesz lubaczewski < depesz@depesz.com> wrote: > Hi, > it looks that pg_basebackup, when using --progress information, outputs > so much information that it puts cpu pressure on terminal application > I'm using - whether it's simple xterm/rxvt, "screen" or "tmux" terminal > multiplexers. > > My specific usecase is setting multiple slave servers for multiple > clusters in AWS, so the data transmission between servers is pretty > fast, but the jump host, that I run all the pg_basebackup (via ssh) > from, is not really beefy, and with 4 concurrent pg_basebackups, I've > seen tmux use up to 80% of CPU of the jump host. > > I filled bug report for tmux too, but since the problem can be repeated > in xterm, rxvt, and screen, I think that perhaps it should be handled in > some way in pg_basebackup too? > > For completeness sake: > we're using pg_basebackup from 9.3.11 on ubuntu trusty. > > In my quick sanity-check test, it looks that progress information in our > case is outputted with the speed of ~ 320KB/s. Which seems excessive. > Hi! This was fixed in pg_basebackup in 9.4, see commit 8198a321c9ea2072c5acde438f4d19638e04f6f2. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
On Wed, May 25, 2016 at 12:52:55PM +0200, Magnus Hagander wrote:
> This was fixed in pg_basebackup in 9.4, see commit
> 8198a321c9ea2072c5acde438f4d19638e04f6f2.
Hmm .. - we can't upgrade to 9.4. Any chance to have this backported to
9.3? It doesn't look like new functionality, more like bugfix, to me.
Best regards,
depesz
--
The best thing about modern society is how easy it is to avoid contact with it.
http://depesz.com/
On Wed, May 25, 2016 at 12:54 PM, hubert depesz lubaczewski < depesz@depesz.com> wrote: > On Wed, May 25, 2016 at 12:52:55PM +0200, Magnus Hagander wrote: > > This was fixed in pg_basebackup in 9.4, see commit > > 8198a321c9ea2072c5acde438f4d19638e04f6f2. > > Hmm .. - we can't upgrade to 9.4. Any chance to have this backported to > 9.3? It doesn't look like new functionality, more like bugfix, to me. > > IIRC that was discussed at the time and we decided not to. The patch is fairly simple though, it wouldn't surprise me if it applies cleanly to 9.3 if you want to do a one-off build. Though I guess that means we could also reconsider it for backpatching now that we haven't seen side-effects. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
On Wed, May 25, 2016 at 12:56:18PM +0200, Magnus Hagander wrote:
> IIRC that was discussed at the time and we decided not to.
Fair enough.
> The patch is fairly simple though, it wouldn't surprise me if it applies
> cleanly to 9.3 if you want to do a one-off build. Though I guess that means
> we could also reconsider it for backpatching now that we haven't seen
> side-effects.
Yeah, that's a no-go for us. I guess that's one more reason to upgrade.
Thanks Magnus,
Best regards,
depesz
--
The best thing about modern society is how easy it is to avoid contact with it.
http://depesz.com/
On Wed, May 25, 2016 at 3:58 AM, hubert depesz lubaczewski <depesz@depesz.com> wrote: > On Wed, May 25, 2016 at 12:56:18PM +0200, Magnus Hagander wrote: >> IIRC that was discussed at the time and we decided not to. > > Fair enough. FWIW, there is one conflict at the top of pg_basebackup.c with a set of variables declared: last_progress_report should be declared as non-static for consistency. So that's rather minimal. -- Michael