Thread: [HACKERS] Running make check-world in buildfarm (was Re: [COMMITTERS] pgsql:Use SASLprep to normalize passwords for SCRAM authentication.)

On 04/08/2017 04:23 AM, Tom Lane wrote:
> Heikki Linnakangas <heikki.linnakangas@iki.fi> writes:
>> Use SASLprep to normalize passwords for SCRAM authentication.
>
> The test script that this adds appears to fail unless the environment
> selects a UTF8-based locale.  On my RHEL6 machine, I see:
>
> LANG=C make check            fail
> LANG=en_US.iso88591 make check        fail
> LANG=en_US.utf8 make check        ok

Fixed, thanks!

> I'm surprised that more of the buildfarm hasn't fallen over.

Hmm. It looks like none of the buildfarm members are running the 
authentication tests. Nor recovery tests, nor subscription tests. We're 
missing a trick here, at least some of the buildfarm members really 
ought to run "make check-world", we're missing a lot of coverage otherwise.

- Heikki

On Sat, Apr 8, 2017 at 7:33 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> Hmm. It looks like none of the buildfarm members are running the
> authentication tests. Nor recovery tests, nor subscription tests. We're
> missing a trick here, at least some of the buildfarm members really ought to
> run "make check-world", we're missing a lot of coverage otherwise.

I recall that Andrew has been favoring as much as possible one folder
path per test series in the buildfarm client (perhaps to keep the
tests separated and have the logs easier to analyze?). I would not
mind much if this is replaced by a pure make check-world, which is
what most of the serious hackers do, or at least a make check running
from src/test to save us a lot of maintenance pain.
-- 
Michael

On 2017-04-08 23:01:06 +0900, Michael Paquier wrote:
> On Sat, Apr 8, 2017 at 7:33 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> > Hmm. It looks like none of the buildfarm members are running the
> > authentication tests. Nor recovery tests, nor subscription tests. We're
> > missing a trick here, at least some of the buildfarm members really ought to
> > run "make check-world", we're missing a lot of coverage otherwise.
> 
> I recall that Andrew has been favoring as much as possible one folder
> path per test series in the buildfarm client (perhaps to keep the
> tests separated and have the logs easier to analyze?). I would not
> mind much if this is replaced by a pure make check-world, which is
> what most of the serious hackers do, or at least a make check running
> from src/test to save us a lot of maintenance pain.

I think it's partially knowing which target failed, and which
regression.diffs to display.  If we were able to revamp check-world so
it outputs a list of targets the regression machinery were able to run
individually, it'd probably help?

- Andres

On 04/08/2017 10:11 AM, Andrew Dunstan wrote:
>
>
> -------- Forwarded Message --------
> Subject:     Re: [HACKERS] Running make check-world in buildfarm (was Re:
> [COMMITTERS] pgsql: Use SASLprep to normalize passwords for SCRAM
> authentication.)
> Date:     Sat, 8 Apr 2017 07:05:54 -0700
> From:     Andres Freund <andres@anarazel.de>
> To:     Michael Paquier <michael.paquier@gmail.com>, Andrew Dunstan
> <andrew@dunslane.net>
> CC:     Heikki Linnakangas <hlinnaka@iki.fi>, Tom Lane
> <tgl@sss.pgh.pa.us>, pgsql-hackers <pgsql-hackers@postgresql.org>
>
>
>
> On 2017-04-08 23:01:06 +0900, Michael Paquier wrote:
> > On Sat, Apr 8, 2017 at 7:33 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
> > > Hmm. It looks like none of the buildfarm members are running the
> > > authentication tests. Nor recovery tests, nor subscription tests. We're
> > > missing a trick here, at least some of the buildfarm members really ought to
> > > run "make check-world", we're missing a lot of coverage otherwise.
> > 
> > I recall that Andrew has been favoring as much as possible one folder
> > path per test series in the buildfarm client (perhaps to keep the
> > tests separated and have the logs easier to analyze?). I would not
> > mind much if this is replaced by a pure make check-world, which is
> > what most of the serious hackers do, or at least a make check running
> > from src/test to save us a lot of maintenance pain.
>
> I think it's partially knowing which target failed, and which
> regression.diffs to display.  If we were able to revamp check-world so
> it outputs a list of targets the regression machinery were able to run
> individually, it'd probably help?
>



Yes, I don't want just to run check-world.

I am aware of a few test sets that need to be added, and I'm planning on
doing that this weekend, in fact. Specifically: recovery, subscription,
authentication and SSL. Peter Eisentraut raised this with me about a
week ago.

Instead of just adding targets to check-world, perhaps we need to look
at what we can do so that the buildfarm client can discover what checks
it might run and run them, just as we specify test schedules for pg_regress.

cheers

andrew

-- 

Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> I think it's partially knowing which target failed, and which
>> regression.diffs to display.  If we were able to revamp check-world so
>> it outputs a list of targets the regression machinery were able to run
>> individually, it'd probably help?

> Yes, I don't want just to run check-world.

Yup.  The situation with the TAP tests (bin-check step) is already a
usability fail: when there's a failure, your first problem is to root
through megabytes of poorly-differentiated logs just to figure out
what actually failed.  Treating all of check-world as a single buildfarm
step would be a disaster.

> Instead of just adding targets to check-world, perhaps we need to look
> at what we can do so that the buildfarm client can discover what checks
> it might run and run them, just as we specify test schedules for pg_regress.

+1.  In the meantime, is there any chance of breaking down bin-check into
a separate step per src/bin/ subdirectory?
        regards, tom lane

On 04/08/2017 05:26 PM, Andrew Dunstan wrote:
> Yes, I don't want just to run check-world.
>
> I am aware of a few test sets that need to be added, and I'm planning on
> doing that this weekend, in fact. Specifically: recovery, subscription,
> authentication and SSL. Peter Eisentraut raised this with me about a
> week ago.

Thanks, that'd be great!

Beware that src/test/ssl is not safe to run on a multi-user system, 
because it allows connections from localhost with well-known user 
certificates. I've been running it on chipmunk for a while, with the 
attached module script.

- Heikki


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 04/08/2017 12:11 PM, Tom Lane wrote:
> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>>> I think it's partially knowing which target failed, and which
>>> regression.diffs to display.  If we were able to revamp check-world so
>>> it outputs a list of targets the regression machinery were able to run
>>> individually, it'd probably help?
>> Yes, I don't want just to run check-world.
> Yup.  The situation with the TAP tests (bin-check step) is already a
> usability fail: when there's a failure, your first problem is to root
> through megabytes of poorly-differentiated logs just to figure out
> what actually failed.  Treating all of check-world as a single buildfarm
> step would be a disaster.
>
>> Instead of just adding targets to check-world, perhaps we need to look
>> at what we can do so that the buildfarm client can discover what checks
>> it might run and run them, just as we specify test schedules for pg_regress.
> +1.  In the meantime, is there any chance of breaking down bin-check into
> a separate step per src/bin/ subdirectory?
>
>             

Possibly. I will look when I go to do the missing checks, later today or
tomorrow.

cheers

andrew




-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 04/08/2017 02:49 PM, Andrew Dunstan wrote:
>
> On 04/08/2017 12:11 PM, Tom Lane wrote:
>> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>>>> I think it's partially knowing which target failed, and which
>>>> regression.diffs to display.  If we were able to revamp check-world so
>>>> it outputs a list of targets the regression machinery were able to run
>>>> individually, it'd probably help?
>>> Yes, I don't want just to run check-world.
>> Yup.  The situation with the TAP tests (bin-check step) is already a
>> usability fail: when there's a failure, your first problem is to root
>> through megabytes of poorly-differentiated logs just to figure out
>> what actually failed.  Treating all of check-world as a single buildfarm
>> step would be a disaster.
>>
>>> Instead of just adding targets to check-world, perhaps we need to look
>>> at what we can do so that the buildfarm client can discover what checks
>>> it might run and run them, just as we specify test schedules for pg_regress.
>> +1.  In the meantime, is there any chance of breaking down bin-check into
>> a separate step per src/bin/ subdirectory?
>>
>>             
> Possibly. I will look when I go to do the missing checks, later today or
> tomorrow.
>


OK, crake is running this code now.  See
<https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2017-04-09%2001%3A58%3A15?

I've left off the SSL tests for now. We should look into how we can do
that more safely. Meanwhile Heikki is running the tests.

Note that some of these tests are quite expensive in terms of time,
particularly recover, subscription and pg_rewind.

This who want to play along can get the bleeding edge code from git,
either by cloning or grabbing a zip of the latest code. I have one or
two things I want to do before wrapping up another client release.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services