Thread: Swap on postgres master server
Hello,
we are running Postgres 10.5 with master slave replication.
These are our custom params
archive_command = 'pgbackrest --stanza=cmdprod archive-push %p' # command to use to archive a logfile segment
archive_mode = on # enables archiving; off, on, or always
checkpoint_completion_target = 0.7 # checkpoint target duration, 0.0 - 1.0
default_text_search_config = 'pg_catalog.english' #
datestyle = 'iso, mdy' #
effective_cache_size = 120GB #
hot_standby = on # "on" allows queries during recovery
lc_messages = 'en_US.UTF-8' # locale for system error message
lc_monetary = 'en_US.UTF-8' # locale for monetary formatting
lc_numeric = 'en_US.UTF-8' # locale for number formatting
lc_time = 'en_US.UTF-8' # locale for time formatting
listen_addresses = '*' # defaults to 'localhost', '*' = all
log_autovacuum_min_duration = 1000ms # -1 disables, 0 logs all actions and
log_checkpoints = on #
log_line_prefix = '%t [%r] [%p]: [%l-1] db=%d,user=%u ' #
log_lock_waits = on # log lock waits >= deadlock_timeout
log_min_duration_statement = 1000ms # -1 is disabled, 0 logs all statements
log_statement = 'ddl' # none, ddl, mod, all
log_temp_files = 1024kB # log temporary files equal or larger
maintenance_work_mem = 2GB #
max_connections = 220 #
max_parallel_workers_per_gather = 8 # taken from max_worker_processes
max_wal_size = 2GB #
min_wal_size = 1GB #
pg_stat_statements.max = 10000 #
pg_stat_statements.track = all #
port = 5432 # port number which Postgres listen
shared_buffers = 10GB #
shared_preload_libraries = 'pg_stat_statements' # (change requires restart)
synchronous_standby_names = '1 ( "usnyh2" )' # comment out during upgrade
track_activity_query_size = 16384 # (change requires restart)
track_io_timing = on #
wal_buffers = 16MB #
wal_keep_segments = 100 #
wal_level = replica # minimal, replica, or logical
work_mem = 600MB #
This server is on Centos 7 and the strange thing is that we see a lot of swap usage :
[root@usnyh-cmd1 ~]# free -m
total used free shared buff/cache available
Mem: 257652 7555 5559 12804 244536 236036
Swap: 16383 7326 9057
7GB used.
But can't see it from any of the commands like top etc.
I am sure it is postgres because it is the only service running on that machine.
Is there anything we can do?
On the sync slave, the usage is just 400MB.
Any trick?
Thanks a lot,
Nicola
Are you sure that swap is used actively? Maybe it had just been used during backup or something.
Look after SwapIn/SwapOut (si/so) it should be '0'
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 12 89344 46608 586384 0 0 12 8 30 86 0 0 99 0 0
If you want to see the amount of ram used by each program with childs run this:
ps -A --sort -rss -o comm,pmem | awk '
NR == 1 { print; next }
{ a[$1] += $2 }
END {
for (i in a) {
printf "%-15s\t%s\n", i, a[i];
}
}
'
On Tue, Oct 16, 2018 at 11:04 AM Nicola Contu <nicola.contu@gmail.com> wrote:
Hello,we are running Postgres 10.5 with master slave replication.These are our custom paramsarchive_command = 'pgbackrest --stanza=cmdprod archive-push %p' # command to use to archive a logfile segmentarchive_mode = on # enables archiving; off, on, or alwayscheckpoint_completion_target = 0.7 # checkpoint target duration, 0.0 - 1.0default_text_search_config = 'pg_catalog.english' #datestyle = 'iso, mdy' #effective_cache_size = 120GB #hot_standby = on # "on" allows queries during recoverylc_messages = 'en_US.UTF-8' # locale for system error messagelc_monetary = 'en_US.UTF-8' # locale for monetary formattinglc_numeric = 'en_US.UTF-8' # locale for number formattinglc_time = 'en_US.UTF-8' # locale for time formattinglisten_addresses = '*' # defaults to 'localhost', '*' = alllog_autovacuum_min_duration = 1000ms # -1 disables, 0 logs all actions andlog_checkpoints = on #log_line_prefix = '%t [%r] [%p]: [%l-1] db=%d,user=%u ' #log_lock_waits = on # log lock waits >= deadlock_timeoutlog_min_duration_statement = 1000ms # -1 is disabled, 0 logs all statementslog_statement = 'ddl' # none, ddl, mod, alllog_temp_files = 1024kB # log temporary files equal or largermaintenance_work_mem = 2GB #max_connections = 220 #max_parallel_workers_per_gather = 8 # taken from max_worker_processesmax_wal_size = 2GB #min_wal_size = 1GB #pg_stat_statements.max = 10000 #pg_stat_statements.track = all #port = 5432 # port number which Postgres listenshared_buffers = 10GB #shared_preload_libraries = 'pg_stat_statements' # (change requires restart)synchronous_standby_names = '1 ( "usnyh2" )' # comment out during upgradetrack_activity_query_size = 16384 # (change requires restart)track_io_timing = on #wal_buffers = 16MB #wal_keep_segments = 100 #wal_level = replica # minimal, replica, or logicalwork_mem = 600MB #This server is on Centos 7 and the strange thing is that we see a lot of swap usage :[root@usnyh-cmd1 ~]# free -mtotal used free shared buff/cache availableMem: 257652 7555 5559 12804 244536 236036Swap: 16383 7326 90577GB used.But can't see it from any of the commands like top etc.I am sure it is postgres because it is the only service running on that machine.Is there anything we can do?On the sync slave, the usage is just 400MB.Any trick?Thanks a lot,Nicola
Nicola Contu wrote: > we are running Postgres 10.5 with master slave replication. > > This server is on Centos 7 and the strange thing is that we see a lot of swap usage : > > [root@usnyh-cmd1 ~]# free -m > total used free shared buff/cache available > Mem: 257652 7555 5559 12804 244536 236036 > Swap: 16383 7326 9057 > > 7GB used. > > But can't see it from any of the commands like top etc. > I am sure it is postgres because it is the only service running on that machine. > > Is there anything we can do? > On the sync slave, the usage is just 400MB. I recommend that you set "vm.swappiness = 0" using "sysctl" to discourage the operating system from swapping as much as possible. Yours, Laurenz Albe -- Cybertec | https://www.cybertec-postgresql.com
No it is not probably used, because I can't find it in any way as I said.
I run your commands :
[root@usnyh-cmd1 ~]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 2 7505332 14545468 13692 241436784 0 0 47 115 0 0 11 2 84 2 0
3 2 7505332 14490408 13692 241436928 0 0 0 248 6153 2013 3 1 93 3 0
1 2 7505332 14474672 13692 241436800 0 0 0 1124 4905 1454 3 1 93 3 0
4 2 7505332 14381156 13692 241436832 0 0 0 96 5322 1782 2 1 94 3 0
^C
[root@usnyh-cmd1 ~]# ps -A --sort -rss -o comm,pmem | awk '
> NR == 1 { print; next }
> { a[$1] += $2 }
> END {
> for (i in a) {
> printf "%-15s\t%s\n", i, a[i];
> }
> }
> '
COMMAND %MEM
kworker/42:2 0
kworker/60:6H 0
kworker/60:2H 0
kdmwork-253:2 0
ksoftirqd/60 0
postmaster 15.2
kworker/9:0H 0
So I'm just asking why it is still there with free -m and if there is any way to free that up if it is not used.
Thanks
Il giorno mar 16 ott 2018 alle ore 11:18 Hans Schou <hans.schou@gmail.com> ha scritto:
Are you sure that swap is used actively? Maybe it had just been used during backup or something.Look after SwapIn/SwapOut (si/so) it should be '0'$ vmstat 1procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----r b swpd free buff cache si so bi bo in cs us sy id wa st1 0 12 89344 46608 586384 0 0 12 8 30 86 0 0 99 0 0If you want to see the amount of ram used by each program with childs run this:ps -A --sort -rss -o comm,pmem | awk 'NR == 1 { print; next }{ a[$1] += $2 }END {for (i in a) {printf "%-15s\t%s\n", i, a[i];}}'On Tue, Oct 16, 2018 at 11:04 AM Nicola Contu <nicola.contu@gmail.com> wrote:Hello,we are running Postgres 10.5 with master slave replication.These are our custom paramsarchive_command = 'pgbackrest --stanza=cmdprod archive-push %p' # command to use to archive a logfile segmentarchive_mode = on # enables archiving; off, on, or alwayscheckpoint_completion_target = 0.7 # checkpoint target duration, 0.0 - 1.0default_text_search_config = 'pg_catalog.english' #datestyle = 'iso, mdy' #effective_cache_size = 120GB #hot_standby = on # "on" allows queries during recoverylc_messages = 'en_US.UTF-8' # locale for system error messagelc_monetary = 'en_US.UTF-8' # locale for monetary formattinglc_numeric = 'en_US.UTF-8' # locale for number formattinglc_time = 'en_US.UTF-8' # locale for time formattinglisten_addresses = '*' # defaults to 'localhost', '*' = alllog_autovacuum_min_duration = 1000ms # -1 disables, 0 logs all actions andlog_checkpoints = on #log_line_prefix = '%t [%r] [%p]: [%l-1] db=%d,user=%u ' #log_lock_waits = on # log lock waits >= deadlock_timeoutlog_min_duration_statement = 1000ms # -1 is disabled, 0 logs all statementslog_statement = 'ddl' # none, ddl, mod, alllog_temp_files = 1024kB # log temporary files equal or largermaintenance_work_mem = 2GB #max_connections = 220 #max_parallel_workers_per_gather = 8 # taken from max_worker_processesmax_wal_size = 2GB #min_wal_size = 1GB #pg_stat_statements.max = 10000 #pg_stat_statements.track = all #port = 5432 # port number which Postgres listenshared_buffers = 10GB #shared_preload_libraries = 'pg_stat_statements' # (change requires restart)synchronous_standby_names = '1 ( "usnyh2" )' # comment out during upgradetrack_activity_query_size = 16384 # (change requires restart)track_io_timing = on #wal_buffers = 16MB #wal_keep_segments = 100 #wal_level = replica # minimal, replica, or logicalwork_mem = 600MB #This server is on Centos 7 and the strange thing is that we see a lot of swap usage :[root@usnyh-cmd1 ~]# free -mtotal used free shared buff/cache availableMem: 257652 7555 5559 12804 244536 236036Swap: 16383 7326 90577GB used.But can't see it from any of the commands like top etc.I am sure it is postgres because it is the only service running on that machine.Is there anything we can do?On the sync slave, the usage is just 400MB.Any trick?Thanks a lot,Nicola
I guess you can run swapoff (followed by swapon). That will free up
whatever is currently swapped. Beware if the system is actively
swapping then swapoff can take some time. But it seems not in your
case.
On Tue, 16 Oct 2018 at 10:48, Nicola Contu <nicola.contu@gmail.com> wrote:
>
> No it is not probably used, because I can't find it in any way as I said.
>
> I run your commands :
>
> [root@usnyh-cmd1 ~]# vmstat 1
> procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
> r b swpd free buff cache si so bi bo in cs us sy id wa st
> 3 2 7505332 14545468 13692 241436784 0 0 47 115 0 0 11 2 84 2 0
> 3 2 7505332 14490408 13692 241436928 0 0 0 248 6153 2013 3 1 93 3 0
> 1 2 7505332 14474672 13692 241436800 0 0 0 1124 4905 1454 3 1 93 3 0
> 4 2 7505332 14381156 13692 241436832 0 0 0 96 5322 1782 2 1 94 3 0
> ^C
> [root@usnyh-cmd1 ~]# ps -A --sort -rss -o comm,pmem | awk '
> > NR == 1 { print; next }
> > { a[$1] += $2 }
> > END {
> > for (i in a) {
> > printf "%-15s\t%s\n", i, a[i];
> > }
> > }
> > '
> COMMAND %MEM
> kworker/42:2 0
> kworker/60:6H 0
> kworker/60:2H 0
> kdmwork-253:2 0
> ksoftirqd/60 0
> postmaster 15.2
> kworker/9:0H 0
>
> So I'm just asking why it is still there with free -m and if there is any way to free that up if it is not used.
>
> Thanks
>
> Il giorno mar 16 ott 2018 alle ore 11:18 Hans Schou <hans.schou@gmail.com> ha scritto:
>>
>> Are you sure that swap is used actively? Maybe it had just been used during backup or something.
>>
>> Look after SwapIn/SwapOut (si/so) it should be '0'
>> $ vmstat 1
>> procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
>> r b swpd free buff cache si so bi bo in cs us sy id wa st
>> 1 0 12 89344 46608 586384 0 0 12 8 30 86 0 0 99 0 0
>>
>> If you want to see the amount of ram used by each program with childs run this:
>> ps -A --sort -rss -o comm,pmem | awk '
>> NR == 1 { print; next }
>> { a[$1] += $2 }
>> END {
>> for (i in a) {
>> printf "%-15s\t%s\n", i, a[i];
>> }
>> }
>> '
>>
>>
>> On Tue, Oct 16, 2018 at 11:04 AM Nicola Contu <nicola.contu@gmail.com> wrote:
>>>
>>> Hello,
>>> we are running Postgres 10.5 with master slave replication.
>>>
>>> These are our custom params
>>>
>>> archive_command = 'pgbackrest --stanza=cmdprod archive-push %p' # command to use to archive a logfile segment
>>> archive_mode = on # enables archiving; off, on, or always
>>> checkpoint_completion_target = 0.7 # checkpoint target duration, 0.0 - 1.0
>>> default_text_search_config = 'pg_catalog.english' #
>>> datestyle = 'iso, mdy' #
>>> effective_cache_size = 120GB #
>>> hot_standby = on # "on" allows queries during recovery
>>> lc_messages = 'en_US.UTF-8' # locale for system error message
>>> lc_monetary = 'en_US.UTF-8' # locale for monetary formatting
>>> lc_numeric = 'en_US.UTF-8' # locale for number formatting
>>> lc_time = 'en_US.UTF-8' # locale for time formatting
>>> listen_addresses = '*' # defaults to 'localhost', '*' = all
>>> log_autovacuum_min_duration = 1000ms # -1 disables, 0 logs all actions and
>>> log_checkpoints = on #
>>> log_line_prefix = '%t [%r] [%p]: [%l-1] db=%d,user=%u ' #
>>> log_lock_waits = on # log lock waits >= deadlock_timeout
>>> log_min_duration_statement = 1000ms # -1 is disabled, 0 logs all statements
>>> log_statement = 'ddl' # none, ddl, mod, all
>>> log_temp_files = 1024kB # log temporary files equal or larger
>>> maintenance_work_mem = 2GB #
>>> max_connections = 220 #
>>> max_parallel_workers_per_gather = 8 # taken from max_worker_processes
>>> max_wal_size = 2GB #
>>> min_wal_size = 1GB #
>>> pg_stat_statements.max = 10000 #
>>> pg_stat_statements.track = all #
>>> port = 5432 # port number which Postgres listen
>>> shared_buffers = 10GB #
>>> shared_preload_libraries = 'pg_stat_statements' # (change requires restart)
>>> synchronous_standby_names = '1 ( "usnyh2" )' # comment out during upgrade
>>> track_activity_query_size = 16384 # (change requires restart)
>>> track_io_timing = on #
>>> wal_buffers = 16MB #
>>> wal_keep_segments = 100 #
>>> wal_level = replica # minimal, replica, or logical
>>> work_mem = 600MB #
>>>
>>> This server is on Centos 7 and the strange thing is that we see a lot of swap usage :
>>>
>>> [root@usnyh-cmd1 ~]# free -m
>>> total used free shared buff/cache available
>>> Mem: 257652 7555 5559 12804 244536 236036
>>> Swap: 16383 7326 9057
>>>
>>> 7GB used.
>>>
>>> But can't see it from any of the commands like top etc.
>>> I am sure it is postgres because it is the only service running on that machine.
>>>
>>> Is there anything we can do?
>>> On the sync slave, the usage is just 400MB.
>>>
>>> Any trick?
>>>
>>> Thanks a lot,
>>> Nicola
Ok thanks I will try that.
But do you know if there is any way to avoid this?
vm.swapiness? or anything on the postgres conf?
Il giorno mar 16 ott 2018 alle ore 15:17 Bob Jolliffe <bobjolliffe@gmail.com> ha scritto:
I guess you can run swapoff (followed by swapon). That will free up
whatever is currently swapped. Beware if the system is actively
swapping then swapoff can take some time. But it seems not in your
case.
On Tue, 16 Oct 2018 at 10:48, Nicola Contu <nicola.contu@gmail.com> wrote:
>
> No it is not probably used, because I can't find it in any way as I said.
>
> I run your commands :
>
> [root@usnyh-cmd1 ~]# vmstat 1
> procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
> r b swpd free buff cache si so bi bo in cs us sy id wa st
> 3 2 7505332 14545468 13692 241436784 0 0 47 115 0 0 11 2 84 2 0
> 3 2 7505332 14490408 13692 241436928 0 0 0 248 6153 2013 3 1 93 3 0
> 1 2 7505332 14474672 13692 241436800 0 0 0 1124 4905 1454 3 1 93 3 0
> 4 2 7505332 14381156 13692 241436832 0 0 0 96 5322 1782 2 1 94 3 0
> ^C
> [root@usnyh-cmd1 ~]# ps -A --sort -rss -o comm,pmem | awk '
> > NR == 1 { print; next }
> > { a[$1] += $2 }
> > END {
> > for (i in a) {
> > printf "%-15s\t%s\n", i, a[i];
> > }
> > }
> > '
> COMMAND %MEM
> kworker/42:2 0
> kworker/60:6H 0
> kworker/60:2H 0
> kdmwork-253:2 0
> ksoftirqd/60 0
> postmaster 15.2
> kworker/9:0H 0
>
> So I'm just asking why it is still there with free -m and if there is any way to free that up if it is not used.
>
> Thanks
>
> Il giorno mar 16 ott 2018 alle ore 11:18 Hans Schou <hans.schou@gmail.com> ha scritto:
>>
>> Are you sure that swap is used actively? Maybe it had just been used during backup or something.
>>
>> Look after SwapIn/SwapOut (si/so) it should be '0'
>> $ vmstat 1
>> procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
>> r b swpd free buff cache si so bi bo in cs us sy id wa st
>> 1 0 12 89344 46608 586384 0 0 12 8 30 86 0 0 99 0 0
>>
>> If you want to see the amount of ram used by each program with childs run this:
>> ps -A --sort -rss -o comm,pmem | awk '
>> NR == 1 { print; next }
>> { a[$1] += $2 }
>> END {
>> for (i in a) {
>> printf "%-15s\t%s\n", i, a[i];
>> }
>> }
>> '
>>
>>
>> On Tue, Oct 16, 2018 at 11:04 AM Nicola Contu <nicola.contu@gmail.com> wrote:
>>>
>>> Hello,
>>> we are running Postgres 10.5 with master slave replication.
>>>
>>> These are our custom params
>>>
>>> archive_command = 'pgbackrest --stanza=cmdprod archive-push %p' # command to use to archive a logfile segment
>>> archive_mode = on # enables archiving; off, on, or always
>>> checkpoint_completion_target = 0.7 # checkpoint target duration, 0.0 - 1.0
>>> default_text_search_config = 'pg_catalog.english' #
>>> datestyle = 'iso, mdy' #
>>> effective_cache_size = 120GB #
>>> hot_standby = on # "on" allows queries during recovery
>>> lc_messages = 'en_US.UTF-8' # locale for system error message
>>> lc_monetary = 'en_US.UTF-8' # locale for monetary formatting
>>> lc_numeric = 'en_US.UTF-8' # locale for number formatting
>>> lc_time = 'en_US.UTF-8' # locale for time formatting
>>> listen_addresses = '*' # defaults to 'localhost', '*' = all
>>> log_autovacuum_min_duration = 1000ms # -1 disables, 0 logs all actions and
>>> log_checkpoints = on #
>>> log_line_prefix = '%t [%r] [%p]: [%l-1] db=%d,user=%u ' #
>>> log_lock_waits = on # log lock waits >= deadlock_timeout
>>> log_min_duration_statement = 1000ms # -1 is disabled, 0 logs all statements
>>> log_statement = 'ddl' # none, ddl, mod, all
>>> log_temp_files = 1024kB # log temporary files equal or larger
>>> maintenance_work_mem = 2GB #
>>> max_connections = 220 #
>>> max_parallel_workers_per_gather = 8 # taken from max_worker_processes
>>> max_wal_size = 2GB #
>>> min_wal_size = 1GB #
>>> pg_stat_statements.max = 10000 #
>>> pg_stat_statements.track = all #
>>> port = 5432 # port number which Postgres listen
>>> shared_buffers = 10GB #
>>> shared_preload_libraries = 'pg_stat_statements' # (change requires restart)
>>> synchronous_standby_names = '1 ( "usnyh2" )' # comment out during upgrade
>>> track_activity_query_size = 16384 # (change requires restart)
>>> track_io_timing = on #
>>> wal_buffers = 16MB #
>>> wal_keep_segments = 100 #
>>> wal_level = replica # minimal, replica, or logical
>>> work_mem = 600MB #
>>>
>>> This server is on Centos 7 and the strange thing is that we see a lot of swap usage :
>>>
>>> [root@usnyh-cmd1 ~]# free -m
>>> total used free shared buff/cache available
>>> Mem: 257652 7555 5559 12804 244536 236036
>>> Swap: 16383 7326 9057
>>>
>>> 7GB used.
>>>
>>> But can't see it from any of the commands like top etc.
>>> I am sure it is postgres because it is the only service running on that machine.
>>>
>>> Is there anything we can do?
>>> On the sync slave, the usage is just 400MB.
>>>
>>> Any trick?
>>>
>>> Thanks a lot,
>>> Nicola