Re: Reviewing freeze map code - Mailing list pgsql-hackers

On 05/06/2016 02:29 PM, Andres Freund wrote:
> Hi,
>
>
> On 2016-05-06 14:17:13 -0700, Joshua D. Drake wrote:
>> How do I test?
>>
>> Is there a script I can run?
>
> Unfortunately there's few interesting things to test with pre-made
> scripts. There's no relevant OS dependency here, so each already
> existing test doesn't really lead to significantly increased coverage
> being run by other people.  Generally, when testing for correctness
> issues, it's often of limited benefit to run tests written by the author
> of reviewer - such scripts will usually just test things either has
> thought of.  The dangerous areas are the ones neither author or reviewer
> has considered.

I can't argue with that.

>
>
>> Are there specific things I can do to try and break it?
>
> Upgrade clusters using pg_upgrade and make sure things like index only
> scans still work and yield correct data.  Set up workloads that involve
> freezing, and check that less WAL (and not more!) is generated with 9.6
> than with 9.5.  Make sure queries still work.
>
>
>> What are we looking for exactly?
>
> Data corruption, efficiency problems.
>

I am really not trying to be difficult here but Data Corruption is an 
easy one... what is the metric we accept as an efficiency problem?

>
>> A lot of -hackers seem to forget that although we have 100 -hackers, we have
>> 10000 "consultant/practitioners". Could I read the code and with a weekend
>> of WTF and -hackers questions figure out what is going on, yes but a lot of
>> people couldn't and I don't have the time.
>
> I think tests without reading the code are quite sensible and
> important. And it perfectly makes sense to ask for information about
> what to test.  But fundamentally testing is a lot of work, as is writing
> and reviewing code; unless you're really really good at destructive
> testing, you won't find much in a 15 minute break.
>

Yes, this is true but with a proper testing framework, I don't need a 15 
minute break. I need 1 hour to configure, the rest just "happens" and 
reports back.

I have cycles to test, I have team members to help test (as do *lots* of 
other people) but sometimes we just get lost in how to help.

>
>> You want me (or people like me) to test more? Give us an easy way to
>> do it.
>
> Useful additional testing and easy just don't go well together. By the
> time I have made it easy I've done the testing that's needed.

I don't know that I can agree with this. A proper harness allows you to 
execute: go.sh and boom... 2, 4, even 8 hours later you get a report. I 
will not argue that it isn't easy to implement but I know it can be done.

>> Otherwise, we do what we can, which is try and interface on the things that
>> will directly and immediately affect us (like keywords and syntax).
>
> The amount of bikeshedding on -hackers steals energy and time for
> actually working on stuff, including testing. So I have little sympathy
> for the amount of bike shedding done.

Insuring a reasonable and thought out interface for users is not bike 
shedding, it is at least as important and possibly more important than 
any feature we add.

Sincerely,

JD

>
> Greetings,
>
> Andres Freund
>


-- 
Command Prompt, Inc.                  http://the.postgres.company/                        +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.