Thread: BUG #5185: postgres error "invalid page header in block xx of relation "pg_proc_proname_args_nsp_index"
BUG #5185: postgres error "invalid page header in block xx of relation "pg_proc_proname_args_nsp_index"
From
"Siwat Siwarborvorn"
Date:
The following bug has been logged online:
Bug reference: 5185
Logged by: Siwat Siwarborvorn
Email address: pg-support@xsidekick.com
PostgreSQL version: 8.3.1
Operating system: Aix 5.3 64bit
Description: postgres error "invalid page header in block xx of
relation "pg_proc_proname_args_nsp_index"
Details:
Machine & OS Information
------------------
- AIX 5.3 64 bit
/home/admin >uname -a
AIX AYUD3 3 5 00CE5C1F4C00
/home/admin >oslevel -s
5300-09-03-0918
- Machine is IBM P570 with 16 CPU, Memory 16GB
- postgresql 8.3.1 compile with options
export CC="gcc -maix64"
export LDFLAGS='-Wl,-bbigtoc'
export OBJECT_MODE=64
export PATH=/usr/local/bin:$PATH
./configure --prefix=/usr/local/pgsql-8.3.1 --without-readline
--without-zlib \
--enable-thread-safety
make
make install
- We create database cluster on filesystem which is on EMC SAN.
- database cluster is on direcotry /pg
/home/admin >df /pg
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/lvpgpro 524288000 134222744 75% 14265 1% /pg
- Currently we have about 700 connections run concurrently.
Scenario
--------
See "invalid page header" error in postgres log.
Error Example
-------
somchai:2009-11-11 16:35:29 THAIST:ERROR: invalid page header in block 49
of relation "pg_proc_proname_args_nsp_index"
somchai:2009-11-11 16:35:43 THAIST:ERROR: invalid page header in block 49
of relation "pg_proc_proname_args_nsp_index"
Bug Information
---------------
1) It happend on 2nd Nov, 3rd Nov and 11th Nov.
2) The error messages are similar. They show error on the same index,
pg_proc_proname_args_nsp_index.
3) On each day the messages show on the same block +1. But the block
numbers
are difference on each day they happened
For example,
On 2nd Nov, the error messages show on block # 13,14.
On 11th Nov, The error massages show on block # 48,49.
4) Only queries use the error block(s) on the index will failed.
5) We usually shutdown the machine on 6.00pm and boot on 6.30am
On the 1st and 3rd times, we see the error after 02.00pm. We quite sure
that
the index was ok and user could use it before we see the error. However on
2nd
time it happend right away when a user used the index.
6) Every time we fixe by reindex pg_proc_proname_args_nsp_index.
7) On 7th Nov, We ran pg_dump on the database and restored to a new
database cluster on a new file system to make sure that this time we won't
use the same disk area.
8) No error on AIX and EMC SAN
Postgresql config file ( Skip lines begin with #)
----------------------
listen_addresses = 'localhost, 172.1.1.7'
# (change requires restart)
port = 5432 # (change requires restart)
max_connections = 1000 # (change requires restart)
shared_buffers = 3GB # min 128kB or max_connections*16kB
max_fsm_pages = 500000 # min max_fsm_relations*16, 6 bytes each
# (change requires restart)
max_fsm_relations = 2000 # min 100, ~70 bytes each
# (change requires restart)
checkpoint_segments = 9 # in logfile segments, min 1, 16MB each
random_page_cost = 3.0 # same scale as above
logging_collector = on # Enable capturing of stderr and csvlog
log_directory = 'pg_log' # Directory where log files are
written
# Can be absolute or relative to
PGDATA
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # Log file name pattern.
# happen after that much log
# output. 0 to disable.
log_rotation_size = 0
log_line_prefix = '%u:%t:' # Special values:
# %u = user name
# e.g. '<%u%%%d> '
log_statement = 'mod' # none, ddl, mod, all
track_activities = on
track_counts = on
autovacuum = on # enable autovacuum subprocess?
# '' uses only default tablespace
datestyle = 'iso, dmy'
client_encoding = win874 # actually, defaults to database
# encoding
lc_messages = 'en_US' # locale for system error message
# strings
lc_monetary = 'en_US' # locale for monetary formatting
lc_numeric = 'en_US' # locale for number formatting
lc_time = 'en_US' # locale for time
formatting
default_text_search_config = 'pg_catalog.english'
default_with_oids = on
Re: BUG #5185: postgres error "invalid page header in block xx of relation "pg_proc_proname_args_nsp_index"
From
Heikki Linnakangas
Date:
Siwat Siwarborvorn wrote: > - postgresql 8.3.1 compile with options Although there isn't anything that might explain this issue, you should upgrade to the latest 8.3 minor release. > 6) Every time we fixe by reindex pg_proc_proname_args_nsp_index. Did you keep a copy of the data files? What do the corrupt pages look like? Are they empty, perhaps, or filled with garbage, or random bit flips or what? That index shouldn't be very big, so you could just copy the corresponding file from the data directory, gzip it and send it by email, so that others can take a look. Might want to check it a hex editor first to make sure there isn't anything sensitive, though. Or you can send it off-list if you prefer. Use "SELECT relfilenode FROM pg_class WHERE relname='pg_proc_proname_args_nsp_index'" to find out the filename in the data directory. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Re: BUG #5185: postgres error "invalid page header in block xx of relation "pg_proc_proname_args_nsp_index"
From
Heikki Linnakangas
Date:
Siwat Siwarborvorn wrote: > From the information from log file below, seem the index file has error > in block 13. Yeah. That block is garbled with some completely unrelated data, see attachment. It looks like an operating system or disk subsystem bug to me. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com