Re: [REVIEW] Re: Compression of full-page-writes - Mailing list pgsql-hackers

On Fri, Dec 26, 2014 at 4:16 PM, Michael Paquier <michael.paquier@gmail.com> wrote:
> On Fri, Dec 26, 2014 at 3:24 PM, Fujii Masao <masao.fujii@gmail.com> wrote:
>> pglz_compress() and pglz_decompress() still use PGLZ_Header, so the frontend
>> which uses those functions needs to handle PGLZ_Header. But it basically should
>> be handled via the varlena macros. That is, the frontend still seems to need to
>> understand the varlena datatype. I think we should avoid that. Thought?
> Hm, yes it may be wiser to remove it and make the data passed to pglz
> for varlena 8 bytes shorter..

OK, here is the result of this work, made of 3 patches.

The first two patches move pglz stuff to src/common and make it a frontend utility entirely independent on varlena and its related metadata.
- Patch 1 is a simple move of pglz to src/common, with PGLZ_Header still present. There is nothing amazing here, and that's the broken version that has been reverted in 966115c.
- The real stuff comes with patch 2, that implements the removal of PGLZ_Header, changing the APIs of compression and decompression to pglz to not have anymore toast metadata, this metadata being now localized in tuptoaster.c. Note that this patch protects the on-disk format (tested with pg_upgrade from 9.4 to a patched HEAD server). Here is how the APIs of compression and decompression look like with this patch, simply performing operations from a source to a destination:
extern int32 pglz_compress(const char *source, int32 slen, char *dest,
                          const PGLZ_Strategy *strategy);
extern int32 pglz_decompress(const char *source, char *dest,
                          int32 compressed_size, int32 raw_size);
The return value of those functions is the number of bytes written in the destination buffer, and 0 if operation failed. This is aimed to make backend as well more pluggable. The reason why patch 2 exists (it could be merged with patch 1), is to facilitate the review and the changes made to pglz to make it an entirely independent facility.

Patch 3 is the FPW compression, changed to fit with those changes. Note that as PGLZ_Header contains the raw size of the compressed data, and that it does not exist, it is necessary to store the raw length of the block image directly in the block image header with 2 additional bytes. Those 2 bytes are used only if wal_compression is set to true thanks to a boolean flag, so if wal_compression is disabled, the WAL record length is exactly the same as HEAD, and there is no penalty in the default case. Similarly to previous patches, the block image is compressed without its hole.

Patches, as well as the test script and the results are attached.
Regards,
--
Michael