On Tue, Apr 22, 2008 at 4:25 PM, Martijn van Oosterhout <
kleptog@svana.org> wrote:
On Tue, Apr 22, 2008 at 10:31:53AM +0200, Zdenek Kotala wrote:
> When you are able detect ordering difference you are able also check if it
> is important for the test or not without any extra effort. Only what we
> need is put some flag to test that order is not important.
Not true. Sorting the file is going jumble all the results together.
Since we perform many tests in one file, you're not going to be able to
seperate them.
> Regression test MUST BE bulletproof. If you get a error you must know that
> it is really error (in postgresql or regtest) and must be fixed. When you
> start to ignore some errors because it can happen sometimes you fall in the
> trap soon.
I think people are misunderstanding. You posted a bunch of diffs with
that comment that they *appeared* to only be ordering differences. How
good did you check? If an 8 become a 9 chances are you'd miss it.
Having a second test checking the sorted results would at least
preclude the chance that there really is something wrong.
It was a guide, not a way of getting out of tests.
In the past, I had faced and tried to work on this exact problem... here's what I had in mind:
in the .expected file, we would demarcate the section of lines we expect to come in any order, by using two special markers. Then, when comparing the actual output with expected output, we would take the demarcated group of lines, and the corresponding lines from actual output, and compare them after sorting.
For eg.
foo.expected:
select * from tenk where col1 <= 3 limit 3;
col1 | col2 | col3
-------------------------
?unsorted_result_start
1 | 10 | 100
2 | 20 | 200
3 | 30 | 300
?unsorted_result_end
foo.out:
select * from tenk where col1 <= 3 limit 3;
col1 | col2 | col3
-------------------------
3 | 30 | 300
2 | 20 | 200
1 | 10 | 100
So, the diff program should discard the lines beginning with '?' (meta character), and then sort and match exactly the same number of lines.
There's another option of putting these '?' lines in a separate file with corresponding begin/end line numbers of the unsorted group, and using this as a parameter to the diffing program.
Of course, this needs a change in the (standard) diff that we use from pg_regress!
Best regards,
--
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com
EnterpriseDB
http://www.enterprisedb.comMail sent from my BlackLaptop device
