Re: BUG #19340: Wrong result from CORR() function - Mailing list pgsql-bugs

From Tom Lane
Subject Re: BUG #19340: Wrong result from CORR() function
Date
Msg-id 2107637.1765001376@sss.pgh.pa.us
Whole thread Raw
In response to Re: BUG #19340: Wrong result from CORR() function  (Tom Lane <tgl@sss.pgh.pa.us>)
Responses Re: BUG #19340: Wrong result from CORR() function
List pgsql-bugs
I wrote:
> Dean Rasheed <dean.a.rasheed@gmail.com> writes:
>> Looking at float8_regr_accum(), I think it would be preferable to
>> arrange for it to leave Sxx, Syy, and Sxy zero until distinct X and Y
>> values are seen. I.e., something like this:

> That seems like a good idea.  I was initially worried that the extra
> isnan() checks would slow down aggregation noticeably in the normal
> case where we soon discover that the inputs aren't all equal.

BTW, re-reading the patch, I now think we should drop the initial

    if (isnan(commonX) || isnan(commonY))

test, instead bulling ahead with computing tmpX/tmpY/scale, and
only skip the updates of Sxx/Syy/Sxy when we have constant inputs.
Using that initial test is optimizing for constant inputs at the
expense of non-constant inputs, which seems like the wrong way
to bet.

            regards, tom lane



pgsql-bugs by date:

Previous
From: Tom Lane
Date:
Subject: Re: BUG #19341: REPLACE() fails to match final character when using nondeterministic ICU collation
Next
From: Dean Rasheed
Date:
Subject: Re: BUG #19340: Wrong result from CORR() function