Thread: Cache lookup failed for type 70385664

For a certain record in our database I'm getting cache lookup failures
(ERROR:  cache lookup failed for type 70385664). And only for one of
the 2 array columns in that record.

The table definition is:

\d inhoudingen
                        Table "public.inhoudingen"
        Column        |            Type             | Modifiers
----------------------+-----------------------------+-----------
 verkoop_id           | bigint                      | not null
 regel                | smallint                    | not null
 bedrag               | numeric(12,2)               | not null
 berekend_op          | timestamp(0) with time zone |
 berekend_bedrag      | numeric(12,2)               |
 comp_bedrag          | numeric(12,4)[]             |
 comp_naam            | character varying(25)[]     |
 omschrijving         | character varying(100)      |
Indexes:
    "geen_dubbele_idx" PRIMARY KEY, btree (verkoop_id, regel)
Foreign-key constraints:
    "inhoudingen_fk" FOREIGN KEY (verkoop_id) REFERENCES verkoop(id)
ON UPDATE CASCADE ON DELETE CASCADE
    "inhoudingen_fk1" FOREIGN KEY (regel) REFERENCES regels(id) MATCH
FULL ON UPDATE CASCADE ON DELETE RESTRICT


And the column comp_naam is giving this error for one specific record.

The database was put in shutdown mode today without a reason (as in,
we don't know why it did that, yet) and since then it's giving this
error.

I'm now wondering what I can do to fix this. Do I need to delete that
record, or is there a way to recover it?

Regards,
Wessel van Norel

Oh, probably found the cause of the sudden restarts:

Jun 23 09:39:13 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[25239] core dumped: /var/core/core_global.postgres
Jun 23 09:45:56 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[26455] core dumped: /var/core/core_global.postgres
Jun 23 09:56:22 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[29292] core dumped: /var/core/core_global.postgres
Jun 23 10:18:00 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[3838] core dumped: /var/core/core_global.postgres
Jun 23 10:27:23 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[5653] core dumped: /var/core/core_global.postgres
Jun 23 10:38:06 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[7929] core dumped: /var/core/core_global.postgres
Jun 23 10:49:28 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[10192] core dumped: /var/core/core_global.postgres
Jun 23 10:58:59 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[12072] core dumped: /var/core/core_global.postgres
Jun 23 11:05:57 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[13262] core dumped: /var/core/core_global.postgres
Jun 23 11:18:04 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[16759] core dumped: /var/core/core_global.postgres
Jun 23 11:30:25 ph1phys01 genunix: [ID 603404 kern.notice] NOTICE: core_log: postgres[19267] core dumped: /var/core/core_global.postgres

It's on a solaris 10 system.

pstack of the last coredump gives me:

core '/var/core/core_global.postgres' of 19267: /usr/bin/postgres -D /postgresql/server/db_ph
 080833ed slot_deform_tuple (8507718, 8560640, 8500da8, 0, 0, 8560e48) + 1a9
 08500000 ???????? ()

Not sure if that's of any help though... We are running postgresql 8.2.3 on solaris 10.

Regards,
Wessel van Norel

This looks like data corruption on that record. Or possibly on
multiple records.

I would:

a) update to the latest bug-fix release of 8.2 asap. I don't see any
fixed bugs which would cause this specific type of error but there are
a lot of them and I could have missed it.


http://www.postgresql.org/docs/8.2/static/release-8-2-13.html
http://www.postgresql.org/docs/8.2/static/release-8-2-12.html
http://www.postgresql.org/docs/8.2/static/release-8-2-11.html
http://www.postgresql.org/docs/8.2/static/release-8-2-10.html
http://www.postgresql.org/docs/8.2/static/release-8-2-9.html
http://www.postgresql.org/docs/8.2/static/release-8-2-8.html
http://www.postgresql.org/docs/8.2/static/release-8-2-7.html
http://www.postgresql.org/docs/8.2/static/release-8-2-6.html
http://www.postgresql.org/docs/8.2/static/release-8-2-5.html
http://www.postgresql.org/docs/8.2/static/release-8-2-4.html

I would also run a good memory checker on this machine, like memtest86
or something like that. Also, check SMART diagnostics for your drives
or the raid status.

But you'll still have to deal with filtering out the corrupted records
from your data. You should be able to delete records even if
individual data within them is corrupted. If you have corruption in
other places you could have a trickier time though.

If you get the "ctid" column from the bad records you could use dd to
extract the block they're on (the first number in the ctid in 8kB
blocks) and post that. It's possible the nature of the corruption
would be clear -- such as a single bit error which makes bad memory a
prime suspect or random bits of garbage from another type of file
which makes the filesystem a suspect. But it's not going to really
help you fix the problem much.

--
greg
http://mit.edu/~gsstark/resume.pdf