Re: parallel.c is not marked as test covered - Mailing list pgsql-hackers

I wrote:
> Depending on what the percentage actually is, maybe we could treat
> this like the "random" test, and allow a failure to be disregarded
> overall?  But that doesn't seem very nice either, in view of our
> increasing reliance on automated testing.  If "random" were failing
> 90% of the time on some buildfarm critters, that would probably
> indicate a real problem, but we'd likely not realize it for a long time.

BTW, this is getting a bit off-topic for this thread, but: I got
curious about whether any such effect might actually exist, and decided
to grep the buildfarm logs to see how many times a failure of the
"random" test had been ignored.  The answer is that there are 35 such
occurrences in otherwise-successful buildfarm runs, out of something
like 650000 successful runs in the buildfarm database.  This is in the
noise, really.   There are more occurrences than that of cases where
"random                   ... failed (ignored)" appeared in a failed
run, which more than likely means that some other regression test script
caused a crash with collateral damage to this test.

This seems like pretty good evidence that we should remove the "ignored"
marking for the random test, and maybe remove that functionality from
pg_regress altogether.  We could probably adjust the test to decrease
its risk-of-failure by another factor of ten or so, if anyone feels like
0.005% failure probability is too high.

Raw data attached for amusement's sake.

            regards, tom lane

    sysname    |      snapshot       |      stage      |                                          logline
                           

---------------+---------------------+-----------------+-------------------------------------------------------------------------------------------
 orangutan     | 2015-01-02 08:00:09 | OK              |      random                   ... failed (ignored)
 dingo         | 2015-01-22 17:29:03 | Check           |      random                   ... failed (ignored)
 fulmar        | 2015-01-26 03:03:17 | OK              | test random                   ... failed (ignored)
 nightjar      | 2015-02-03 22:27:13 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 dromedary     | 2015-03-09 18:57:28 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 olinguito     | 2015-03-09 18:59:08 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 rover_firefly | 2015-03-09 19:04:00 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 lapwing       | 2015-03-09 19:15:01 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 prairiedog    | 2015-03-09 19:20:35 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 dingo         | 2015-03-09 19:29:00 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 binturong     | 2015-03-09 19:42:28 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 shearwater    | 2015-03-09 19:58:15 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 magpie        | 2015-04-25 20:59:48 | OK              | test random                   ... failed (ignored)
 anole         | 2015-04-28 21:25:27 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 binturong     | 2015-05-07 16:55:15 | pg_upgradeCheck | test random                   ... failed (ignored) (test
processexited with exit code 2) 
 dingo         | 2015-05-07 16:58:01 | pg_upgradeCheck | test random                   ... failed (ignored) (test
processexited with exit code 2) 
 binturong     | 2015-05-08 18:56:02 | pg_upgradeCheck | test random                   ... failed (ignored) (test
processexited with exit code 2) 
 reindeer      | 2015-05-15 06:00:01 | OK              |      random                   ... failed (ignored)
 anole         | 2015-05-28 17:58:54 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 anole         | 2015-06-01 00:30:15 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 crake         | 2015-06-01 18:53:02 | OK              | test random                   ... failed (ignored)
 sittella      | 2015-06-01 22:50:36 | OK              | test random                   ... failed (ignored)
 anole         | 2015-06-01 22:58:24 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 blesbok       | 2015-06-27 18:49:43 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 anole         | 2015-06-28 21:35:02 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 binturong     | 2015-07-03 23:45:49 | pg_upgradeCheck | test random                   ... failed (ignored) (test
processexited with exit code 2) 
 chipmunk      | 2015-07-13 09:29:10 | OK              | test random                   ... failed (ignored)
 kouprey       | 2015-07-19 04:43:10 | OK              |      random                   ... failed (ignored)
 gharial       | 2015-07-19 18:54:12 | OK              |      random                   ... failed (ignored)
 frogmouth     | 2015-07-21 01:11:35 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 frogmouth     | 2015-07-21 13:15:31 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 leech         | 2015-07-28 12:13:11 | pg_upgradeCheck | test random                   ... failed (ignored) (test
processexited with exit code 2) 
 frogmouth     | 2015-07-30 01:13:06 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-05 16:44:12 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-05 23:49:11 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-06 02:22:15 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-06 03:04:08 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-06 12:30:33 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-06 18:30:58 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-06 21:22:12 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-07 02:58:39 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-07 14:41:35 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 mandrill      | 2015-08-07 20:54:58 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 capybara      | 2015-08-13 16:05:05 | OK              | test random                   ... failed (ignored)
 cockatiel     | 2015-08-26 05:36:47 | OK              | test random                   ... failed (ignored)
 mandrill      | 2015-09-07 20:47:19 | OK              | test random                   ... failed (ignored)
 treepie       | 2015-09-11 08:41:36 | OK              |      random                   ... failed (ignored)
 guaibasaurus  | 2015-09-27 02:23:01 | OK              |      random                   ... failed (ignored)
 axolotl       | 2015-10-13 18:01:18 | OK              | test random                   ... failed (ignored)
 okapi         | 2015-10-19 21:15:01 | OK              |      random                   ... failed (ignored)
 rover_firefly | 2015-11-06 06:04:00 | OK              | test random                   ... failed (ignored)
 spurfowl      | 2015-11-07 21:21:03 | OK              |      random                   ... failed (ignored)
 minisauripus  | 2015-11-26 03:42:28 | OK              | test random                   ... failed (ignored)
 jacana        | 2015-12-28 01:05:17 | OK              | test random                   ... failed (ignored)
 spurfowl      | 2016-01-07 15:27:01 | OK              |      random                   ... failed (ignored)
 tern          | 2016-01-14 00:41:10 | OK              |      random                   ... failed (ignored)
 guaibasaurus  | 2016-01-25 03:43:01 | OK              |      random                   ... failed (ignored)
 longfin       | 2016-02-05 13:19:05 | OK              | test random                   ... failed (ignored)
 gharial       | 2016-02-11 03:15:43 | OK              | test random                   ... failed (ignored)
 quokka        | 2016-02-26 15:00:06 | OK              | test random                   ... failed (ignored)
 fulmar        | 2016-03-03 12:01:03 | OK              | test random                   ... failed (ignored)
 jaguarundi    | 2016-03-09 04:52:00 | OK              | test random                   ... failed (ignored)
 nightjar      | 2016-04-05 23:30:57 | OK              | test random                   ... failed (ignored)
 skink         | 2016-04-07 16:00:01 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 skink         | 2016-04-07 16:20:01 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 skink         | 2016-04-07 16:40:01 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 gull          | 2016-04-15 02:20:05 | OK              |      random                   ... failed (ignored)
 piculet       | 2016-04-15 02:30:01 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 pademelon     | 2016-04-16 02:47:16 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 pademelon     | 2016-04-16 03:30:47 | Check           |      random                   ... failed (ignored) (test
processexited with exit code 2) 
 hornet        | 2016-05-06 15:52:14 | OK              | test random                   ... failed (ignored)
 mastodon      | 2016-05-08 04:00:01 | OK              | test random                   ... failed (ignored)
 guaibasaurus  | 2016-06-01 02:58:01 | OK              | test random                   ... failed (ignored)
 handfish      | 2016-06-02 21:55:47 | InstallCheck-C  | test random                   ... failed (ignored)
 handfish      | 2016-06-12 02:01:32 | OK              | test random                   ... failed (ignored)
 spoonbill     | 2016-06-14 23:00:05 | OK              | test random                   ... failed (ignored)
(76 rows)