Thread: [HACKERS] Cost of src/test/recovery and .../subscription tests
So I updated longfin to the new release of the buildfarm client,
and was quite dismayed by the fact that its cycle time went
from 16 minutes to 24. Some of that might be random effects like
the state of the kernel disk caches, but a large chunk of it
--- over 5 minutes --- evidently is from src/test/recovery/,
which the buildfarm script didn't run before and now does.
I am going to say flat out that that's unacceptably long for
a test script that will be run dozens of times a day by the
buildfarm. There isn't any other test script that takes more
than circa 90 seconds on that machine, and I don't think this
one should either.
I'm a bit dubious about whether src/test/subscription is being
reasonably thrifty of test time either. Do we really need twice
the runtime of the entire core regression suite to test that?
There was already a good argument not to enable the TAP tests
on slower buildfarm animals, and if nothing is done about these
additions, I'm afraid that such testing will be put permanently
out of reach. I don't even want to think about how long it
might take a CLOBBER_CACHE_ALWAYS animal to run these tests.
regards, tom lane
On 04/19/2017 01:28 PM, Tom Lane wrote: > So I updated longfin to the new release of the buildfarm client, > and was quite dismayed by the fact that its cycle time went > from 16 minutes to 24. Some of that might be random effects like > the state of the kernel disk caches, but a large chunk of it > --- over 5 minutes --- evidently is from src/test/recovery/, > which the buildfarm script didn't run before and now does. > > I am going to say flat out that that's unacceptably long for > a test script that will be run dozens of times a day by the > buildfarm. There isn't any other test script that takes more > than circa 90 seconds on that machine, and I don't think this > one should either. > > I'm a bit dubious about whether src/test/subscription is being > reasonably thrifty of test time either. Do we really need twice > the runtime of the entire core regression suite to test that? > > There was already a good argument not to enable the TAP tests > on slower buildfarm animals, and if nothing is done about these > additions, I'm afraid that such testing will be put permanently > out of reach. I don't even want to think about how long it > might take a CLOBBER_CACHE_ALWAYS animal to run these tests. You can disable the extra tests by adding this to the buildfarm command line: --skip-steps=misc-check cheers andrew > > regards, tom lane > > -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Wed, Apr 19, 2017 at 1:28 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > So I updated longfin to the new release of the buildfarm client, > and was quite dismayed by the fact that its cycle time went > from 16 minutes to 24. Some of that might be random effects like > the state of the kernel disk caches, but a large chunk of it > --- over 5 minutes --- evidently is from src/test/recovery/, > which the buildfarm script didn't run before and now does. > > I am going to say flat out that that's unacceptably long for > a test script that will be run dozens of times a day by the > buildfarm. There isn't any other test script that takes more > than circa 90 seconds on that machine, and I don't think this > one should either. I think that's bunk. If there are tests that are part of those test suites that are taking a long time to run and not providing meaningful coverage, then that's something that can be improved. However, I reject the argument that a test running for a long time is in itself bad. I'd rather have tests that run for a long time (and thus get run less often) than have no tests. Mind you, I'm not entirely sanguine about the large increase in time that it takes to run make check-world on my local machine vs. a few years ago. It's annoying, and optimizing it would be good. I think the TAP test framework encourages people to add possibly-lengthy tests pretty freely, because you can stick the test inside of a loop and run it in many slightly different configurations, so people do, maybe without giving enough thought to the value of those tests. But outlawing long-running tests is not the right solution to that problem. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Apr 19, 2017 at 1:28 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I am going to say flat out that that's unacceptably long for
>> a test script that will be run dozens of times a day by the
>> buildfarm. There isn't any other test script that takes more
>> than circa 90 seconds on that machine, and I don't think this
>> one should either.
> I think that's bunk. If there are tests that are part of those test
> suites that are taking a long time to run and not providing meaningful
> coverage, then that's something that can be improved. However, I
> reject the argument that a test running for a long time is in itself
> bad. I'd rather have tests that run for a long time (and thus get run
> less often) than have no tests.
That's fine, the problem is that these are now in the buildfarm schedule,
meaning they are no longer in the "get run less often" group.
I've been poking at this further, and it seems like a lot of the problem
is not so much excessive coverage as badly implemented "wait" logic
causing the test to run a lot longer than it needs to. I already fixed
one such issue in 7834d20b5, and there seem to be more, both in the test
scaffolding and the code-under-test. Don't know how reasonable it is
to try to fix all this stuff at this point in the development cycle, but
I'm continuing to look into it.
regards, tom lane