Re: Page Checksums - Mailing list pgsql-hackers

On 12/19/2011 02:44 PM, Kevin Grittner wrote:
> I was thinking that we would warn when such was found, set hint bits
> as needed, and rewrite with the new CRC.  In the unlikely event that
> it was a torn hint-bit-only page update, it would be a warning about
> something which is a benign side-effect of the OS or hardware crash.
> The argument was that it could happen months later, and people
> might not remember the crash.  My response to that is: don't let it
> wait that long.  By forcing a vacuum of all possibly-affected tables
> (or all tables if the there's no way to rule any of them out), you
> keep it within recent memory.

Cleanup that requires a potentially unbounded in size VACUUM to sort out 
doesn't sound like a great path to wander down.  Ultimately any CRC 
implementation is going to want a "scrubbing" feature like those found 
in RAID arrays eventually, one that wanders through all database pages 
looking for literal bitrot.  And pushing in priority requests for things 
to check to the top of its queue may end up being a useful feature 
there.  But if you need all that infrastructure just to get the feature 
launched, that's a bit hard to stomach.

Also, as someone who follows Murphy's Law as my chosen religion, I would 
expect this situation could be exactly how flaky hardware would first 
manifest itself:  server crash and a bad CRC on the last thing written 
out.  And in that case, the last thing you want to do is assume things 
are fine, then kick off a VACUUM that might overwrite more good data 
with bad.  The sort of bizarre, "that should never happen" cases are the 
ones I'd most like to see more protection against, rather than excusing 
them and going on anyway.

> There's also *always" a possibility that CRC error is a false
> positive -- if only the bytes in the CRC were damaged.  We're
> talking quantitative changes here, not qualitative.

The main way I expect to validate this sort of thing is with an as yet 
unwritten function to grab information about a data block from a standby 
server for this purpose, something like this:

Master:  Computed CRC A, Stored CRC B; error raised because A!=B
Standby:  Computed CRC C, Stored CRC D

If C==D && A==C, the corruption is probably overwritten bits of the CRC B.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us