CPU load spikes when CentOS tries to reclaim 'cached' memory

For the past few days, we've been seeing unexpected extremely high CPU spikes
in our system. We observed the following: the 'free' memory would go down to
lower than 300 MB; at that point, 'cached' slowly starts to go down, and
then CPU starts to go way up.

It's almost as if the OS was not releasing 'cached' memory fast enough for
Postgres. Is that analysis correct? Is there a way to fix this?

Here's the session:

 04:58:37 load average: 2.37, free: 532, cached: 22852
 04:58:57 load average: 1.91, free: 451, cached: 22859
 04:59:17 load average: 1.82, free: 469, cached: 22866
 04:59:57 load average: 1.57, free: 387, cached: 22884
 05:00:17 load average: 3.03, free: 574, cached: 22632
 05:00:37 load average: 5.51, free: 268, cached: 22267
 05:00:58 load average: 21.44, free: 237, cached: 21704
 05:01:18 load average: 42.98, free: 243, cached: 21061
 05:01:38 load average: 63.38, free: 266, cached: 20410
 05:01:58 load average: 78.69, free: 315, cached: 20135
 05:02:19 load average: 89.82, free: 214, cached: 20034
 05:02:39 load average: 99.06, free: 253, cached: 19873
 05:02:59 load average: 105.60, free: 390, cached: 19497
 05:03:20 load average: 110.10, free: 394; cached: 19290

Here are the pertinent machine and OS and Postgres details:
RAM: 32 gigs
CPU: 24 cores;  Intel(R) Xeon(R) CPU X7460  @ 2.66GHz
RAID 10
vm.swappiness=0
 PostgreSQL 9.1.11 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.4.7
20120313 (Red Hat 4.4.7-3), 64-bit
 Linux ps2db 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux
postgres=# SELECT name, current_setting(name), source
postgres-#   FROM pg_settings
postgres-#   WHERE source NOT IN ('default', 'override');
             name             |        current_setting        |
source
------------------------------+-------------------------------+----------------------
 application_name             | psql                          | client
 archive_command              | /bin/true                     |
configuration file
 archive_mode                 | on                            |
configuration file
 autovacuum_analyze_threshold | 50                            |
configuration file
 autovacuum_freeze_max_age    | 800000000                     |
configuration file
 autovacuum_naptime           | 5min                          |
configuration file
 autovacuum_vacuum_threshold  | 50                            |
configuration file
 bytea_output                 | escape                        |
configuration file
 checkpoint_completion_target | 0.7                           |
configuration file
 checkpoint_segments          | 128                           |
configuration file
 checkpoint_timeout           | 15min                         |
configuration file
 checkpoint_warning           | 30s                           |
configuration file
 client_encoding              | UTF8                          | client
 constraint_exclusion         | partition                     |
configuration file
 cpu_index_tuple_cost         | 0.005                         |
configuration file
 cpu_operator_cost            | 0.0025                        |
configuration file
 cpu_tuple_cost               | 0.01                          |
configuration file
 custom_variable_classes      | pg_stat_statements            |
configuration file
 DateStyle                    | ISO, MDY                      |
configuration file
 default_statistics_target    | 100                           |
configuration file
 default_text_search_config   | pg_catalog.english            |
configuration file
 effective_cache_size         | 16GB                          |
configuration file
 effective_io_concurrency     | 1                             |
configuration file
 enable_material              | off                           |
configuration file
 escape_string_warning        | on                            |
configuration file
 hot_standby                  | on                            |
configuration file
 lc_messages                  | C                             |
configuration file
 lc_monetary                  | en_US.UTF-8                   |
configuration file
 lc_numeric                   | en_US.UTF-8                   |
configuration file
 lc_time                      | en_US.UTF-8                   |
configuration file
 listen_addresses             | *                             |
configuration file
 log_autovacuum_min_duration  | 0                             |
configuration file
 log_checkpoints              | on                            |
configuration file
 log_connections              | on                            |
configuration file
 log_destination              | csvlog                        |
configuration file
 log_directory                | pg_log                        |
configuration file
 log_disconnections           | on                            |
configuration file
 log_filename                 | postgresql.log.ps2db.%H       |
configuration file
 log_line_prefix              | %t [%d] [%u] [%p]: [%l-1] %h  |
configuration file
 log_lock_waits               | on                            |
configuration file
 log_min_duration_statement   | 0                             |
configuration file
 log_rotation_age             | 1h                            |
configuration file
 log_temp_files               | 0                             |
configuration file
 log_timezone                 | Canada/Pacific                | environment
variable
 log_truncate_on_rotation     | on                            |
configuration file
 logging_collector            | on                            |
configuration file
 maintenance_work_mem         | 1GB                           |
configuration file
 max_connections              | 500                           |
configuration file
 max_locks_per_transaction    | 512                           |
configuration file
 max_stack_depth              | 2MB                           | environment
variable
 max_standby_streaming_delay  | 90min                         |
configuration file
 max_wal_senders              | 6                             |
configuration file
 pg_stat_statements.max       | 10000                         |
configuration file
 pg_stat_statements.track     | all                           |
configuration file
 port                         | 5432                          |
configuration file
 random_page_cost             | 4                             |
configuration file
 shared_buffers               | 6GB                           |
configuration file
 shared_preload_libraries     | pg_stat_statements            |
configuration file
 standard_conforming_strings  | off                           |
configuration file
 stats_temp_directory         | /ram_postgres_stats           |
configuration file
 temp_buffers                 | 16MB                          |
configuration file
 TimeZone                     | Canada/Pacific                | environment
variable
 wal_keep_segments            | 64                            |
configuration file
 wal_level                    | hot_standby                   |
configuration file
 work_mem                     | 8MB                           |
configuration file
(65 rows)





--
View this message in context:
http://postgresql.1045698.n5.nabble.com/CPU-load-spikes-when-CentOS-tries-to-reclaim-cached-memory-tp5806122.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.