Thread: BUG #17938: could not open shared memory segment "/PostgreSQL.615216676": No such file or directory

The following bug has been logged on the website:

Bug reference:      17938
Logged by:          Onkobu Tanaake
Email address:      onkobu@onkobutanaake.de
PostgreSQL version: 15.3
Operating system:   Gentoo
Description:

After upgrade from 14 to 15 from time to time this error occurs. A snapshot
of a responsive database looks like this:

$> ls -la /dev/shm/PostgreSQL.*
-rw------- 1 postgres postgres 1048576 May 22 19:18
/dev/shm/PostgreSQL.2644390168
-rw------- 1 postgres postgres   26176 May 22 19:14
/dev/shm/PostgreSQL.3338605124

$> lsof | grep PostgreSQL
postgres  16944                   postgres  mem       REG               0,49
    26176   
       2 /dev/shm/PostgreSQL.3338605124
postgres  16945                   postgres  mem       REG               0,49
    26176   
       2 /dev/shm/PostgreSQL.3338605124
postgres  16946                   postgres  mem       REG               0,49
    26176   
       2 /dev/shm/PostgreSQL.3338605124
postgres  16948                   postgres  mem       REG               0,49
    26176   
       2 /dev/shm/PostgreSQL.3338605124
postgres  16949                   postgres  mem       REG               0,49
  1048576   
       3 /dev/shm/PostgreSQL.2644390168
postgres  16949                   postgres  mem       REG               0,49
    26176   
       2 /dev/shm/PostgreSQL.3338605124
postgres  16950                   postgres  mem       REG               0,49
  1048576   
       3 /dev/shm/PostgreSQL.2644390168
postgres  16950                   postgres  mem       REG               0,49
    26176   
       2 /dev/shm/PostgreSQL.3338605124
postgres  16958                   postgres  mem       REG               0,49
  1048576   
       3 /dev/shm/PostgreSQL.2644390168
postgres  16958                   postgres  mem       REG               0,49
    26176   
       2 /dev/shm/PostgreSQL.3338605124

$> df -h
Filesystem             Size  Used Avail Use% Mounted on
[…]
shm                     32G  1.1M   32G   1% /dev/shm
[…]

Config (postgresql.conf) is as follows:

max_wal_size = 2GB
min_wal_size = 1GB
effective_cache_size = 8GB
log_timezone = 'Europe/Berlin'
datestyle = 'iso, mdy'
timezone = 'Europe/Berlin'
default_text_search_config = 'pg_catalog.english'
plperl.on_init = 'use utf8; use re; package utf8; require
"utf8_heavy.pl";'

I don't use shared_memory_type and also not dynamic_shared_memory_type so
they're default (mmap for the first, posix for the second). Got the settings
from https://pgtune.leopard.in.ua/ for a conservative use on a 32GByte 8
core/ 16 threads machine.

Building goes along the lines of an ebuild
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-db/postgresql/postgresql-15.3.ebuild#n117.
I enabled icu, kerberos, nls, pam, ssl, threads, uuid, xml and zlib.

The server does not run any systemd but is pure Init-V.  So it is very
unlikely that a systemd process causes this.Also no container in use. Init
process directly runs postgres server. Kernel is 5.15.80 and was the same
for version 14. I also don't log in with the postgres-user which is also
impossible (disabled in /etc/shadow). WIthin 7 days this happens very
likely/ at least once. How can I track this down further?

On Mon, May 22, 2023 at 06:10:43PM +0000, PG Bug reporting form wrote:
> The server does not run any systemd but is pure Init-V.  So it is very
> unlikely that a systemd process causes this.Also no container in use. Init
> process directly runs postgres server. Kernel is 5.15.80 and was the same
> for version 14. I also don't log in with the postgres-user which is also
> impossible (disabled in /etc/shadow). WIthin 7 days this happens very
> likely/ at least once. How can I track this down further?>

Are you sure that your system does not run systemd and enable
RemoveIPC?  Even if the PostgreSQL service is not running with it, you
would be able to see see issue as long as the parameter is enabled:
https://www.postgresql.org/message-id/BACA8550-4F3D-41D0-A0E1-5BF307BCD648@elevated-dev.com
--
Michael

At Tue, 23 May 2023 07:43:11 +0900, Michael Paquier <michael@paquier.xyz> wrote in 
> On Mon, May 22, 2023 at 06:10:43PM +0000, PG Bug reporting form wrote:
> > The server does not run any systemd but is pure Init-V.  So it is very
> > unlikely that a systemd process causes this.Also no container in use. Init
> > process directly runs postgres server. Kernel is 5.15.80 and was the same
> > for version 14. I also don't log in with the postgres-user which is also
> > impossible (disabled in /etc/shadow). WIthin 7 days this happens very
> > likely/ at least once. How can I track this down further?> 
> 
> Are you sure that your system does not run systemd and enable
> RemoveIPC?  Even if the PostgreSQL service is not running with it, you
> would be able to see see issue as long as the parameter is enabled:
> https://www.postgresql.org/message-id/BACA8550-4F3D-41D0-A0E1-5BF307BCD648@elevated-dev.com

The OP says:
> I don't use shared_memory_type and also not dynamic_shared_memory_type so
> they're default (mmap for the first, posix for the second). Got the settings

It looks like the OP might be misunderstanding
dynamic_shared_memory_type. Outside of Windows, it first prefers
posix, then sysv and finally mmap. Thus it is quite usual to use posix
for DSM segments. Actually posix is defaultly used for
dynamic_shared_memory_type on my Rocky 9 box.

I can't pinpoint why the "No such file or directory" is comnig up,
though.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

At Wed, 24 May 2023 09:57:35 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in 
> dynamic_shared_memory_type. Outside of Windows, it first prefers
> posix, then sysv and finally mmap. Thus it is quite usual to use posix
> for DSM segments. Actually posix is defaultly used for

It is written in the documentation.

https://www.postgresql.org/docs/15/runtime-config-resource.html#GUC-DYNAMIC-SHARED-MEMORY-TYPE

> The use of the mmap option, which is not the default on any
> platform, is generally discouraged because the operating system may
> write modified pages back to disk repeatedly, increasing system I/O
> load; however, it may be useful for debugging, when the pg_dynshmem
> directory is stored on a RAM disk, or when other shared memory
> facilities are not available.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

I finally found the source: a broken test in a script that initiates a 
chroot environment. For some reason the author decided to write this:

test -L /dev/shm && rm /dev/shm && mkdir /dev/shm

Instead of re-creating a symlink based /dev/shm as true mountpoint it 
always did so.

Since the issue came up from time to time without any clear relation I 
scanned scripts on the machine that require root permissions – not so 
may to dig through.

This bogus check with immediate action was replaced with a bail out of 
the script in case this would be true. No bug in PostgreSQL.


Am 24.05.23 um 03:09 schrieb Kyotaro Horiguchi:
> At Wed, 24 May 2023 09:57:35 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in
>> dynamic_shared_memory_type. Outside of Windows, it first prefers
>> posix, then sysv and finally mmap. Thus it is quite usual to use posix
>> for DSM segments. Actually posix is defaultly used for
> 
> It is written in the documentation.
> 
> https://www.postgresql.org/docs/15/runtime-config-resource.html#GUC-DYNAMIC-SHARED-MEMORY-TYPE
> 
>> The use of the mmap option, which is not the default on any
>> platform, is generally discouraged because the operating system may
>> write modified pages back to disk repeatedly, increasing system I/O
>> load; however, it may be useful for debugging, when the pg_dynshmem
>> directory is stored on a RAM disk, or when other shared memory
>> facilities are not available.
> 
> regards.
> 

-- 
------
Web: http://onkobutanaake.de - mit RSS-Feed
Verschlüsselte/ signierte Elektropost (PGP/GPG) an diese Adresse bevorzugt,
öffentlicher Schlüssel http://onkobutanaake.de/onkobutanaake.asc
-
Encrypted/ signed electro-mail (PGP/GPG) to this address preferred,
public key http://onkobutanaake.de/onkobutanaake.asc

Fingerprint 6261 2530 89DA 93E3 2D9F  A0C2 B660 481D 9F77 A942

On Mon, May 29, 2023 at 04:56:09PM +0200, Onkobu wrote:
> This bogus check with immediate action was replaced with a bail out of the
> script in case this would be true. No bug in PostgreSQL.

Cool.  Thanks for the feedback!
--
Michael