Thread: Replication: GZIP compression in WAL sender/receiver processes communication?

On Tue, Dec 10, 2013 at 8:13 AM, Dmitry Koterov <dmitry@koterov.ru> wrote:
> Hello.
>
> Is there a way to compress the traffic between master and slave during the
> replication?.. The streaming gzip would be quite efficient for that.
>
> (WAL archiving is not too good for this purpose because of high lag. I just
> need to minimize the cross-datacenter traffic keeping the replication lag
> low.)
Not directly AFAIK, but this would be a good argument for the
implementation of hooks in walsender and walreceiver that could allow
to use such customization of the stream sent and received.

Note that there is also a patch pending for 9.4 that would make
possible the compression of full page writes, reducing globally the
amount of WAL produced by server in exchange of some CPU work to
compress and decompress the data.
--
Michael

On 12/9/2013 3:13 PM, Dmitry Koterov wrote:
> Is there a way to compress the traffic between master and slave during
> the replication?.. The streaming gzip would be quite efficient for that.
>
> (WAL archiving is not too good for this purpose because of high lag. I
> just need to minimize the cross-datacenter traffic keeping the
> replication lag low.)

run it through a ssh tunnel that has compression turned on ?



--
john r pierce                                      37N 122W
somewhere on the middle of the left coast

Hello,

Yes, gzip compression can be used for compressing WAL traffic during
streaming replication  Following tools can be used in this regard.
SSL compression-SSL support is built in PostgreSQL. You need to ensure
you have OpenSSL library support in your PostgreSQL installation.
Also, you can compress WAL traffic by setting up SSH tunneling between
master and standby and turn on compression while setting up SSH
tunnel. Following link can be followed for the same.
http://www.postgresql.org/docs/9.3/static/ssh-tunnels.html
Ofcourse, these are the solutions outside PostgreSQL.


On Tue, Dec 10, 2013 at 4:43 AM, Dmitry Koterov <dmitry@koterov.ru> wrote:
>
> Hello.
>
> Is there a way to compress the traffic between master and slave during the replication?.. The streaming gzip would be
quiteefficient for that. 
>
> (WAL archiving is not too good for this purpose because of high lag. I just need to minimize the cross-datacenter
traffickeeping the replication lag low.) 

On Mon, Dec 9, 2013 at 3:13 PM, Dmitry Koterov <dmitry@koterov.ru> wrote:
> Is there a way to compress the traffic between master and slave during the
> replication?.. The streaming gzip would be quite efficient for that.

Take a look at the ssh_tunnel.sh [1] tool. This is a wrapper around
SSH tunnel with compression, a watchdog and lock management. Very
useful for cross data center streaming.

[1] https://github.com/grayhemp/pgcookbook/blob/master/ssh_tunnel_with_compression_setup.md

--
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA

http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (901) 903-0499, +7 (988) 888-1979
gray.ru@gmail.com