Re: BUG #19340: Wrong result from CORR() function - Mailing list pgsql-bugs

From Tom Lane
Subject Re: BUG #19340: Wrong result from CORR() function
Date
Msg-id 931689.1764807066@sss.pgh.pa.us
Whole thread Raw
In response to Re: BUG #19340: Wrong result from CORR() function  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-bugs
I wrote:
> Poking at this, I soon found a test case where even with the separate
> sqrt() calls we'd produce a result slightly outside [-1, 1] (running
> this test over more values of x is sufficient).  So now I think we
> should do both the separate sqrt and the clamp.

Per CI results, on some platforms the roundoff error is different from
what I observe, producing a value just less than 1 rather than just
more.  That doesn't invalidate needing the clamp, but it does mean
that we can't use that test case just like that.  I'm inclined to
remove the change of extra_float_digits, but keep the test case.

            regards, tom lane



pgsql-bugs by date:

Previous
From: Daniel Gustafsson
Date:
Subject: Re: BUG #19095: Test if function exit() is used fail when linked static
Next
From: Michael Paquier
Date:
Subject: Re: BUG #19095: Test if function exit() is used fail when linked static