Re: Test code is worth the space - Mailing list pgsql-hackers

On Mon, Aug 17, 2015 at 02:04:40PM -0500, Jim Nasby wrote:
> On 8/16/15 8:48 AM, Greg Stark wrote:
> >On Sun, Aug 16, 2015 at 7:33 AM, Noah Misch <noah@leadboat.com> wrote:
> >>When I've just spent awhile implementing a behavior change, the test diffs are
> >>a comforting sight.  They confirm that the test suite exercises the topic I
> >>just changed.  Furthermore, most tests today do not qualify under this
> >>stringent metric you suggest.  The nature of pg_regress opposes it.
> >
> >It's not a comforting sight for me. It means I need to change the test
> >results and then of course the tests will pass. So they didn't really
> >test anything and I don't really know whether the changes broke
> >anything. Any test I had to adjust for my change was effectively
> >worthless.
> >
> >This is precisely my point about pg_regress and why it's the reason
> >there's pressure not to have extensive tests. It's useful to have some
> >end-to-end black box tests like this but it's self-limiting. You can't
> >test many details without tying the tests to specific behaviour.
> >
> >I have worked with insanely extensive testing suites that didn't have
> >this problem. But they were unit tests for code that was structured
> >around testability. When the API is designed to be testable and you
> >have good infrastructure for mocking up the environment and comparing
> >function results in a much narrower way than just looking at text
> >output you can test the correctness of each function without tying the
> >test to the specific behaviour.
> >
> >*That* gives a real comfort. When you change a function to behave
> >entirely differently and can still run all the tests and see that all
> >the code that depends on it still operate correctly then you know you
> >haven't broken anything. In that case it actually *speeds up*
> >development rather than slowing it down. It lets newbie developers
> >hack on code where they don't necessarily know all the downstream
> >dependent code and not be paralyzed by fear that they'll break
> >something.
> 
> As someone who oversaw a database test suite with almost 500 test files
> (each testing multiple cases), I completely agree. Our early framework was
> actually similar to pg_regress and that worked "OK" up to about 200 test
> files, but it got very unwieldy very quickly. We switched to pgTap and it
> was far easier to work with.

My own position is based on having maintained a pg_regress suite an order of
magnitude larger than that.  I don't know why that outcome was so different.

> I suspect any effort to significantly improve Postgres test coverage is
> doomed until there's an alternative to pg_regress.

There is the src/test/perl/TestLib.pm harness.