Re: Introduce pg_receivewal gzip compression tests - Mailing list pgsql-hackers

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Monday, July 12th, 2021 at 08:42, Michael Paquier <michael@paquier.xyz> wrote:

> On Fri, Jul 09, 2021 at 11:26:58AM +0000, Georgios wrote:
>
> > As suggested on a different thread [1], pg_receivewal can increase it's test
> >
> > coverage. There exists a non trivial amount of code that handles gzip
> >
> > compression. The current patch introduces tests that cover creation of gzip
> >
> > compressed WAL files and the handling of gzip partial segments. Finally the
> >
> > integrity of the compressed files is verified.
>
> -         # Verify compressed file's integrity
>
>
> -         my $gzip_is_valid = system_log('gzip', '--test', $gzip_wals[0]);
>
>
> -         is($gzip_is_valid, 0, "program gzip verified file's integrity");
>
>
>
> libz and gzip are usually split across different packages, hence there
>
> is no guarantee that this command is always available (same comment as
>
> for LZ4 from a couple of days ago).


Of course. Though while going for it, I did find in Makefile.global.in:

  TAR = @TAR@
  XGETTEXT = @XGETTEXT@

  GZIP    = gzip
  BZIP2   = bzip2

  DOWNLOAD = wget -O $@ --no-use-server-timestamps

Which is also used by GNUmakefile.in

  distcheck: dist
      rm -rf $(dummy)
      mkdir $(dummy)
      $(GZIP) -d -c $(distdir).tar.gz | $(TAR) xf -
      install_prefix=`cd $(dummy) && pwd`; \


This to my understanding means that gzip is expected to exist.
If this is correct, then simply checking for the headers should
suffice, since that is the only dependency for the files to be
created.

If this is wrong, then I will add the discovery code as in the
other patch.

>
> -                 [
>
>
> -                         'pg_receivewal', '-D',     $stream_dir, '--verbose',
>
>
> -                         '--endpos',      $nextlsn, '-Z', '5'
>
>
> -                 ],
>
>
>
> I would keep the compression level to a minimum here, to limit CPU
>
> usage but still compress something faster.
>
> -         # Verify compressed file's integrity
>
>
> -         my $gzip_is_valid = system_log('gzip', '--test', $gzip_wals[0]);
>
>
> -         is($gzip_is_valid, 0, "program gzip verified file's integrity");
>
>
>
> Shouldn't this be coded as a loop going through @gzip_wals?

I would hope that there is only one gz file created. There is a line
further up that tests exactly that.

+   is (scalar(@gzip_wals), 1, "one gzip compressed WAL was created");


Then there should also be a partial gz file which is tested further ahead.

Cheers,
//Georgios

> -----------------------------------------------------------
>
> Michael